Forem: JD Costa

From Rails to Elixir: Safe Rewrites with NOOP Deployments

JD Costa — Wed, 16 Jan 2019 10:22:11 +0000

This is the second of a series of articles about my journey going from Ruby on Rails to Elixir/Phoenix.
\Part-1 From Rails to Elixir: Know Your App
\Part-2 From Rails to Elixir: Safe Rewrites with NOOP Deployments

Recap

I'm porting a Price Tracker App originally built with Ruby on Rails to Elixir/Phoenix. My main motivation is to reduce infrastructure costs as I'm currently paying 24 USD/month for a virtual machine on Google Cloud, the smallest possible to support the whole system. Please read Part-1 for a comprehensive write-up on why I decided to do this migration.

Refactor Anxiety

Being able to improve your existing software for the better is always a good thing and often times a rare privilege. Improvements may come in different forms: you can add missing tests for a less tested feature, modernize and tighten up security by upgrading dependencies, refactor a chunk of code for all the good reasons and sometimes even port an entire piece to a more suitable technology that will enhance flexibility/reliability/performance.

It's not all roses, though. Replacing something that is currently working and serving its purpose often comes with the great responsibility of having to deliver a surrogate that works at least as well as the legacy stuff. There's also the moment of the switch over, which require careful planning, partial shutdowns and in some cases it even demands for a full stop and restart of the entire system. Never a good thing.

These great concerns of 1) not wanting to disrupt the running system during the transition phase and 2) wanting the new replacement to deliver what is expected; are two quite dramatic sources of developer anxiety.

Stress Less with the 6 R's

During my previous experience maintaining a steadily growing product, I learned a bunch about continuously evolving a 5 years old codebase without affecting the ongoing business. As a small team, we couldn't afford unexpected failures in production provoked by reckless deployments that would steal everyone's attention and effectively halt development.

For that reason, we were always very thoughtful about how refactors or any change in general would go into production.

The informal techniques we used varied depending on the type and scale of the migration coming ahead. They would usually fall into one of the 6 Application Migration Strategies - The 6 R's formulated by Amazon AWS in the context of the Cloud:

1\ Rehosting
2\ Replatforming
3\ Repurchasing
4\ Refactoring (re-architecting)
5\ Retire
6\ Retain

The line that separates them is fuzzy but I find the list very useful to clearly define the boundaries of the work I have in hands. It gives me clarity of thought and helps me plan the initiative. Let's say we are moving our infrastructure away from Amazon AWS and into Google Cloud - that's rehosting and therefore we should take every precaution inherent to moving one system from one place to the other. It makes it sound simple, and that is reassuring.

I would also recommend employing one, and only one, migration strategy at a time. If you need to rehost from AWS into Google Cloud and also replatform from Ruby on Rails into Elixir/Phoenix, then do one after the other and don't try to do it all at the same time. It might be tempting to do both at once but if you really think about it, that's effectively deploying a predominantly unknown piece of software in an equally unknown environment. Even with proper testing an staging servers in place, results can be unpredictable.

In my experience, migration success is inversely proportional to the underlying complexity. If you can, make sure that you keep your migrations as simple and unitary as possible.

NOOP Deployments

Perhaps my favorite technique of them all is what I call (and maybe someone before me?) the NOOP Deployment.

NOOP (No Operation) is a programming concept used to describe an action that literally does nothing. Its origin goes back to the Assembly language where it acts as an instruction that has no effect in the system. Wikipedia knows better.

So what exactly is a NOOP Deployment? It's a deployment technique where the new code is placed where it should live in production, but in an inactive state. I'm going to stretch this idea even further by deploying a completely blank Elixir app to the production infrastructure, with no custom code whatsoever, and let it live in there in an inert state while implementation progresses.

To illustrate this idea, the following picture shows exactly how I feel when I have to spend weeks or even months working on an application from top to bottom and the time finally comes to ship it to production:

It doesn't seem right, does it? You don't know exactly if the new terrain will be capable of supporting the structure, or if the old materials will play well in the new environment. Now imagine that there are other houses next to it, you certainly don't want to bring them down along if a catastrophe happens while you're unloading your traveling house. It's not going to do any good to your neighborhood credit.

That's exactly how migrations can go wrong. Deploy a defective application into a production infrastructure close to components of a running system and all hell breaks loose. Think databases going bonkers with connection bursts, external services being bombarded with requests or any other unexpected behavior.

For the sake of my own sanity, I prefer to do it more like this:

Find the terrain, lay the foundations and build upon. A lot less can go wrong when development is mindful of infrastructure and good things will arise when implementation is done in sync with deployment.

Martin Fowler also builds on this idea by introducing the StranglerApplication. It's a concept inspired by the strangler vines in Australia, which "seed in the upper branches of a fig tree and gradually work their way down the tree until they root in the soil". This is a great metaphor for subsystems that are created on the edge of old systems and grow gradually, slowly taking over.

I'm going to follow this approach to port my Ruby on Rails app to Elixir/Phoenix. The plan is to generate a blank Elixir application and place it next to the running Rails app. They will share the same virtual machine as the main goal of the project is to keep a slimmed down version of the existing server.

The Current State of Elixir Deployments

I've heard José Valim in a podcast (around minute 10:00) talking about the recent future of the language and saying that the one thing he still wants to resolve is deployment. He believes that deployment is so important that it should be a concern of the language itself.

Pursuing this idea, he joined forces with Paul Schoenfelder from Dockyard, creator of Distillery, in order to unify ideas from the community, test them in Distillery and eventually integrate a polished version into the Elixir core. Distillery is a rewrite of Erlang's release handling for Elixir and seems to be, from the perspective of a true outsider, one of the most popular solutions for deploying Elixir apps.

This comprehensive article written by Paul explains how they are planning to test their ideas in Distillery 2.0 and moving them into Elixir core once ready. It is very clear that configuration is their main concern, they want to get that right before it reaches the core of the language.

I saw other alternatives available, but to be honest they didn't catch my attention as much as Distillery, even more so after learning that its most recent version is basically a precursor for what's going to be available in the standard lib.

Production Release with Distillery 2.0

At this point, I'm expecting something similar to Capistrano, the go-to deployment thing for Ruby on Rails people. Add it as a dependency, generate a few files, customize it as needed and run a command to put it en route to the destination.

Distillery seemed very complete at first glance. It has lovely docs that guide you through the installation process as well as a few guides to follow for different use cases. The one about umbrella projects came in handy.

So, drop it like it's hot:

defp deps do
  [
    {:distillery, "~> 2.0"}
  ]
end

Install and generate default configuration files:

$ mix release.init

This gave me a rel/config.exs file with a few pre-defined values that can be customized as desidered. The most important lines might be:

environment :prod do
  # ...
end

release :shopxir do
  # ...
end

In my case, it generated a release with the same name as the umbrella project (shopxir) and included the only child application so far - jobs. This child application is a blank/empty Elixir app. There's nothing in there yet.

It also defines two environments, dev and prod. The latter is configured to include the Erlang runtime in the production release by default and doesn't include the source code.

Finally, run the command to build the release:

$ MIX_ENV=prod mix release

Which will fill _build/prod/ with a bunch of files and folders. This command has a nice output showing a few possible further actions one can do with the newly compiled release such as starting in the background, starting in the foreground and even a way to connect interactively with the running app (remote_console).

Just to be sure that my release is well packed, I'm going to add a simple static module to the jobs child app that I can call after the deployment in order to verify if it went as expected.

defmodule Ping do
  def ping do
    IO.puts "Pong."
  end
end

Rebuild and reconnect with remote_console:

$ MIX_ENV=prod mix release
$ _build/prod/rel/shopxir/bin/shopxir start # in the background
$ _build/prod/rel/shopxir/bin/shopxir remote_console
...
iex(shopxir@127.0.0.1)1> Ping.ping
Pong.
:ok

We're rolling 👍

I'm not going to discuss finer details about how releases can be customized because I'm not yet very familiar with them. As implementation goes and new features are moved from Rails into Elixir, I'll certainly need to dive deeper into this setup, but I'm making the decision to do it incrementally and as needed. For now, all I want is a live Elixir app placed in the server.

Deployment

Now, one thing that Distillery doesn't do is the actual heavy lifting of transporting the new version into the production infrastructure, the termination of the running app and initialization of the new version. They offer a few guides based on stuff like Docker and systemd, which is ok-ish, but I was looking for a more off-the-shelf tool. This was when I discovered that I'm a spoiled Ruby kiddo 👶

Docker is cool, I use it a lot locally to run external services such as PostgreSQL and Redis during development. Together with docker-compose, it makes it easy to organize external dependencies per project. For production though, it just feels difficult: configuration management, build servers, makefiles, orchestration, registries, discovery, Dockerfile, swarm, Kubernetes, etcd... I'm tired already.

If Docker in production is your thing, please take it easy on me. I probably just need to learn it properly.

On the other hand, Capistrano and its plugins did everything for me in the Ruby world. This thing takes care of downloading the latest release from a git repository, keeps a collection of releases in separate folders, compiles assets, runs migrations, stops/(re)starts the application and background workers, notifies monitoring services and much much more in a very simple way with almost zero configuration. It's also highly extensible.

I wanted something similar in the Elixir world but I didn't find anything so complete and straightforward.

Luckily, there's a very similar tool called Ansistrano based on Ansible (the provisioning framework) that is heavily inspired in Capistrano as the name says. It fits my needs because I'm already using Ansible for server provisioning.

Ansible has been a constant in all my projects since a few years ago. I use it primarily for infrastructure provisioning but it can be used for deployments too. It's a very versatile tool, people use it to do many different things related to manipulation of remote or even local machines.

In summary, Ansistrano will do the following:

1\ Build a new release locally using Distillery
2\ Create a folder in the remote server and copy the new release into it
3\ Symlink the new folder so that it is interpreted as the current release
4\ Create/update a systemd service that will execute the application
5\ Flip the switch to the new release by restarting the corresponding systemd service

Common deployment tasks such as running database migrations and asset compilation are expected to be handled using Distillery hooks, that's why they are not present in the previous list.

systemd is running the app in foreground mode and sends a SIGTERM by default for termination. This is enough for now because there are no running tasks such as HTTP requests being handled by Phoenix or other long-running processes in the background.

The whole thing (including template files) has a little over 50 lines of Ansible declarations. Check them out in this Gist.

Summary & Next Up

System rewrites are hard and a great source of developer anxiety. They require careful planning and a prudent approach.

Reduce anxiety and mitigate risks by streamlining the migration. Employ techniques such as the 6 R's to create focus and the StranglerApplication to incrementally replace the legacy system without harm. Use NOOP deployments to feel safe and reassured about your first deployment. Build upon.

Distillery is great for packaging Elixir releases but other tools like Ansistrano and systemd must be there to help with the actual deployment on the remote server.

Considering that I'm brand new to this, my first experience deploying an Elixir application was reasonably pleasant. I still feel a few knowledge gaps and I'm also pretty sure that my current setup is too simple for what's coming, but I'm happy with starting with a basic approach and growing it as needed.

Now that I have deployed my Elixir app in production, it's time to think about which parts of the legacy Rails app should be migrated first. This is going to be an important decision that I will leave for the next article, but my brain is already thinking Background Jobs.

I hope that you have enjoyed the 2nd article of the series. I tried to be a little more technical by dropping a few code snippets and diving deeper into tools. I'm wondering if you take this as a positive thing.

As always, feedback is much appreciated 🙏🏻

Thanks for reading ❤️

From Rails to Elixir: Know Your App

JD Costa — Mon, 17 Dec 2018 11:20:58 +0000

This is the first of a series of articles about my journey going from Ruby on Rails to Elixir/Phoenix.
\Part1 From Rails to Elixir: Know Your App
\Part-2 From Rails to Elixir: Safe Rewrites with NOOP Deployments

I've been doing a lot of Ruby on Rails during my career as a Software Engineer, I absolutely love it and I don't necessarily plan to stop doing it.

In fact, Rails has been the de facto choice not only for my professional projects but also for my personal ones. Most of the times I come up with this revolutionary new ideia that will change the world (no doubt about that) and I rush to the implementation because I don't want to lose the excitement. Since I'm usually in a hurry, it seems natural to me to use the most off-the-shelf set of tools out there that allow me to prototype my brain dump as quickly as possible.

The Price Tracker App

That's what I've done, once again, when I decided that I was going to build a price tracker app meant to be compatible with any online shop in the interwebz. Users will mark items as favourites and then receive notifications by email when their price changes.

I wanted people to be able to use a browser extension to save items so I immediately saw trucks loaded with JavaScript coming my way; then I needed some sort of backend to store this information and periodically scrape online pages for updated prices. I also needed something to help me send email notifications.

Here's how my mind works at this point: it needs HTML/CSS/JS to show at least a simple list of items being tracked, an API to get items from the database, a database of course, and perhaps some kind of background processing thing for the scraping and notifications. So...

* I always found that Sidekiq logo hilarous.

Did I get the tech stack right!? Off I go.

I built the browser extension, built the API to get items through both ways, built a background worker that wakes up every two hours and downloads the HTML page for each stored item and built another background worker that parses all these HTML documents searching for updated prices. I even built a small use-and-discard HTTP proxy repository to get away unnoticed with my scraping but I'm saving it for another blog post.

I think I had a working version after one week of coding during my free time (mostly evenings), because Rails ❤️.

Shipping it to Production 💰

I chose to put it all in one server in order to keep things smooth and simple, which means PostgreSQL, Rails, Redis and Sidekiq sharing a single CPU and 1.70GB of memory.

First deploy with just PostgreSQL and Rails went fine, then I added Sidekiq and Redis and the thing just blew up unexpectedly during the deploy - I was using Capistrano. After more than 1 hour of frustration, I finally figured out that my server was out of memory while trying to compile assets and spawn fresh copies of Rails and Sidekiq.

I shouldn't be compiling assets in the production server to start with, but that's how Capistrano works by default. I could also perhaps have spent some time trying to find a way to restart the app in a more memory efficient manner but I felt that I would be fighting against the technology I had imposed on myself. Plus, any time spent on this particular issue felt worthless to me.

So, next up in Google Cloud was a 3.75GB virtual machine which I imagined would be enough for now. As a side note, I was on Google Cloud because I had heard good stuff about it, plus I was planning to use Google Cloud Functions in a later stage and being in the same cloud provider would eliminate network transfer costs.

Everything great so far. My app was running, I was iterating as fast as new JavaScript frameworks are released 😏 and I even managed to convince some friends to start testing it as real users. I was happy!

...until I saw the bill 💰💰💰😐.

Paying as much as USD 24.27 /month (at the time of this writing) to keep alive a low-effort side project was a total waste of money, even more so because the server itself wasn't doing much and sitting still for most of the time. On the other hand, I needed an expensive amount of memory available to accommodate the baseline memory requirements for Puma/Rails and Sidekiq.

It would make sense for a startup company who is committed to a project to pay such a small amount (in these terms) because the compensatory added value in the form of development speed is far superior. There's this controversial article by Rail's creator DHH about Ruby being fast and cheap enough for companies (I agree to a certain extent).

It wasn't the case for me though, I wasn't burning VC money, hadn't committed to any sort of deadline and I didn't have any salaries to pay so I was prepared to bring down my development speed in exchange for a less expensive infrastructure.

Know Your App

Luckily, I've been doing scaling and optimisation on a Rails-based system for the last 2 and a half years as part of my full time job. I learned valuable lessons about knowing my systems before thinking about optimisations by monitoring every workflow, hunt for bottlenecks, deeply understand the underlying workload patterns (are they fundamentally synchronous, can they be concurrent, who depends on who, etc) and finally tracking down who's stealing the most of your CPU and memory resources.

This acquired experience made me look at my price tracker in a sort of zoomed out version of the system, where the specific parts such as classes and objects and how they interact with each other were not important anymore. Instead, what became relevant to me was how the system could be divided into high level concerns, how these concerns were represented and how they were interacting with each other.

In the end I realised that my price tracker could be divided in four separate concerns:

1\ The Web Server's job was to provide a way for users to manage item lists. This component was just a standard stateless HTTP server, in which each request is 100% isolated from other requests. This is very important because it tells us that they are allowed to run concurrently. Since its main job was to insert and retrieve stuff from a database, most of its execution time was being spent waiting on I/O operations.

2\ The Page Downloader is a background process that wakes up every two hours to gather item URLs from the database and download the corresponding HTML pages into disk. I'm again looking at another type of workload that can run concurrently because there's no shared state between downloads.

3\ The Page Parser is another background process that grabs HTML pages from disk, proceeds with parsing them and then navigates through text nodes searching for updated prices. Whether it finds great prices or not, the final result is reported back to the database. Similarly to the aforementioned services, there's no shared state between different pages being processed, meaning they could also run concurrently. Perhaps the main difference between this and the others is the fact that it might tend to be a little more heavy on CPU than memory due to text search.

4\ The Notifications part of the system is not represented for simplicity, but I plan to write a future blog post about how it integrates with the current topology.

By doing this high level analysis, I learned that all four components had the ability to do their work independently from each other - this didn't happen by accident as they were designed to work concurrently in the first place. The Page Downloader could keep downloading pages while the Parser was already processing the first batch that came through. The Web Server was also there on the side letting users manage their item list without blocking the rest of the system.

I also learned that my system was severely I/O bound as most downloads would take several seconds to complete. I should leverage the slack time the CPU was getting by forcing it to do other useful stuff while waiting.

If you're interested in learning more about concurrent workloads, I recommend the first part of this great talk presented by Golang's co-creator Rob Pike.

If I had a real world application with a fair amount of traffic, I would also spend some time studying server and application metrics. It's important to know every hot path on your system so that you can reason about which parts need to be designed with performance in mind - a concept that is generally described in economy as the Pareto Principle.

That's why people keep talking about monitoring, it really is important, and nowadays there are plenty of options out there to do it, both in premises or as a service. I have used New Relic in the past and I got some pretty valuable insights into my platform from it. Totally recommend it.

Make the Decision

Once I knew that I needed 1) a concurrency-friendly environment to work on top of; and 2) a runtime that would consume very low memory; I felt ready to hunt for the right tool for the job.

In 2018, what is it that pops up in your head when the matter is concurrency? Well, I'm sure the answer to this question will differ based on individual's experiences but in my head it's Elixir and Go.

About Go, the syntax scares me personally and I don't really feel comfortable with the way they do error handling. Although they have an elegant way of doing concurrent stuff using Communicating Sequential Processes (CSP), it's easy to mess up because data structures are still mutable. On top of this, as I heard in a recent podcast, I don't feel that it is capable of providing a high enough abstraction for problems like the one I had in hands.

No hard feelings though, others will have different perspectives 👍.

Elixir, on the other hand, felt immediately familiar to me. Maybe it's because the syntax makes it look like Ruby (it's a trap!) but I can also find similarities in the community. Elixir's creator José Valim was a prolific member in the Ruby community, maybe that's why.

Additionally, because it runs on top of BEAM (Erlang's virtual machine), concurrency primitives are deeply embedded in the language and the actual programming model - the Actor Model - is all about concurrency. This felt like a perfect fit for my concurrency hungry platform. You can learn more about how Elixir shines in this field in this article.

Once I started diving deeper into Elixir and the ecosystem, I started to feel that I was advancing in the right direction:

1\ Great tooling. I needed a web framework and Phoenix checked all the boxes; I needed a jobs framework capable of integrating with Sidekiq for the initial migration and Exq was there; I needed an HTTP client library and httpotion was there; I needed an HTML parser and floki was there; finally, I needed deployment gear and distillery looked fine; All these packages were well documented and well maintained just like I was used to in Ruby land.

2\ It had a Stable and Growing Ecosystem, so I wasn't worried about possibly be using an overhyped tool destined to die in a couple of years. Erlang has been around for 20 years now and Elixir is unleashing its potential by making it more accessible. Even Erlang developers are getting excited by this new language and consequently getting involved in the community. This can only be a positive sign.

3\ It checks all the boxes in terms of my Concurrency requirements. I would be able to spread my workloads across the server and run them in a concurrent fashion, maximising CPU and memory usage. I foresee hundreds or even thousands of downloads happening at the same time in a single machine with little effort.

4\ The native concept of Umbrella Projects would allow me to segregate the different concerns of the platform without the risk of engaging in any sort of over-engineering. More info here.

5\ Infrastructure Costs were expected to be lower judging by all the articles you can read online about companies migrating to Elixir and being able to reduce their fleets from hundreds of servers to just a few. Unfortunately, there's little information about how much memory Elixir needed at rest, but Erlang's website states that people have successfully ran BEAM in a system with just 16MB of RAM, which doesn't mean much. If you have valuable information about memory consumption please reach out.

I would have tested my memory assumptions thoroughly if I was on a real world scenario. In my development machine, I could see an average of 40MB of RAM being used by the BEAM after running iex -S mix with just a few loaded dependencies. I don't know how believable this number is.

6\ I can think of at least two friends of mine that will be very happy once they know I'm finally leaning over Elixir. Seriously guys, if you're reading this, please stop annoying me with all your Elixir shenanigans! It's done, I'm Elixiring now.

Check out their contributions to the community: 1) A super fast jobs framework ExJob based on event sourcing and GenStage by @eidge; and 2) a task scheduling system Cronex by @jbamaro.

More about the pros and cons of Elixir in here.

At this point I felt confident enough to go for this rewrite and I have to add that I was very excited too. I wrote a few modules just to test drive the thing and it really caught my attention. All of the sudden I was devouring documentation, podcasts and online talks so I was genuinely interested and having fun with it. This is very important to me. 🤓🤓

Companies

I was working by myself on this stuff with virtually no restrictions apart from wanting to keep infrastructure costs low. My main goal was to learn and have fun building a low-budget toy product.

Had I been in the context of a company, there's a whole lot of other elements I would have to factor in before making such a decision: 1) it's a relatively new technology so I guess it will be harder to find experienced/keen developers; 2) rewrites usually require considerable development effort and might not produce immediate short-term results; 3) new software might introduce new bugs or bring back old ones; 4) the moment of the switch over to the new platform can be disruptive and takes time to plan carefully; 5) younger startups might benefit from using off-the-shelf tools like Ruby on Rails instead to maximise development speed and consequently burn as less time/money as possible. These are just a few factors worth considering off the top of my head, but I'm sure there are others.

Having said that, once certain products mature and reach considerable scale, it might be worth to replace parts of the system with more performant alternatives either to be able to handle more traffic or reduce costs. Or both. These dudes were able to reduce their fleet of servers from 150 to just 5 by replacing their Ruby on Rails monolith with Elixir/Phoenix. Having done my bit of infrastructure management myself, I can only imagine the vast sums of money they're saving each month.

I guess the lesson here is: if you're a company, always assess the worthiness of a platform migration of this kind before jumping off the cliff.

Final Thoughts & Next Up

At the time of this writing I don't know how much money I'll be able to save as I'm not done with the full migration yet. My goal is to document the whole journey in a series of articles, so you can follow me to get notified for the upcoming posts. There's plenty to talk about, all the way from choosing technologies, going over various implementation details, and finally discuss how to switch over to the new platform in production.

I'm planning to release a follow up on how to approach these kind of platform migrations in the next article of the series. I want to dive into the thinking process behind streamlining the whole migration (decide which parts should be migrated first, etc) in order to mitigate eventual risks. I'm going to use the Price Tracker App as an example.

If you find this article too conceptual, I'm planning to get more technical as the series goes, so expect to see some Ruby and Elixir snippets going forward.

Well, thank you for stopping by and stay tuned for the next one.

Feedback is much appreciated 👍