Forem: Louis Sommer

Building the Ultimate Search for Rails - Episode 1

Louis Sommer — Mon, 16 Jan 2023 23:21:46 +0000

During summer 2021, I got lucky enough to cross Twitter paths with Peter Szinek, who introduced me to his team and got me hired at RCRDSHP, Obie Fernandez’s latest Web 3.0 project involving Music NFTs. Being myself a pro musician and music producer, I was thrilled to finally be able to mix my two passions-turned-into-a-living. Surrounded by awesome developers, I learned and built tons of cool stuff, among which a reactive, super performant server-side-rendered search experience using StimulusReflex, ElasticSearch and close to no JavaScript. The feature is still live and pretty much unchanged.

The purpose of this series will be to first reimplement this super friendly UX with basic filters and sort options, along with StimulusReflex. Then, we’ll see how ElasticSearch can allow more complex filters and search scenarios, while improving performance. In the last episode, we’ll try and replace StimulusReflex with the new Custom Turbo Stream Actions and compare implementation/behaviour. If time allows, I might add a bonus episode to show how to deploy all this in production. Let’s dig in.

What are we building?

Remember how back in the day, people used to collect art printed on actual paper ? A bit like NFTs, only physical. Weird, right? Well, let’s picture an app that would allow users to buy and sell limited edition art prints. The prints would include photographs, movie posters, and illustrations of various formats. It should look like that:

Here’s what the DB looks like:

Please note the tags column is of string array type. One might argue that the Listing table, in our case, could easily be skipped. But for the sake of keeping a real-world complexity scenario, let’s say that we’d like to keep the actual Prints separate from their listings (and since the app allows users to sell their prints, we might reasonably think that a print could be listed several times).

OK. Show me the gear

First things first: on the frontend, we’ll use StimulusReflex (a.k.a SR) to build a super reactive and friendly search experience with very little code, and little to no JavaScript. For those unfamiliar:

StimulusReflex is a library that extends the capabilities of both Rails and Stimulus by intercepting user interactions and passing them to Rails over real-time websockets. The current page is quickly re-rendered and morphed to reflect the new application state.

Sounds a bit like Hotwire on paper, though you’ll see how their philosophy greatly differs in the last episode of this series. We’ll also use a sprinkle of CableReady, a close cousin of SR.

On the backend, we'll need a few tools. Apart from the classics (ActiveRecord scopes and the pg_search gem), you’ll see how the (yet officially unreleased but production-tested) all_futures gem, built by SR authors, will act as an ideal ephemeral object to temporarily store our filter params and host our search logic. Finally, we’ll use pagy for pagination duties.

Philtre d'amour

(Please indulge this shitty French pun, the expression philtre d'amour meaning love potion but also sounds like beloved filter)

Let’s start by creating some simple data. We’ll add a few artworks of different kind : photographs, illustrations, and posters. Each will have several tags from a given list and an author. For now, let’s just generate one print per artwork, and a listing for each. Prints can be one of 3 available formats , while listings will be of varying price.
Now let’s list our different features:

Search by name or author
Filter by minimum price
Filter by maximum price
Filter by category
Filter by print format
Filter by tags
Order by price
Order by date listed

Before I started building my feature, my former colleague and friend @marcoroth pointed me to leastbad’s Beast Mode, from which I took heavy inspiration to get going. That’s how I discovered his gem all_futures, which provides us with an ActiveRecord-like object that will persist to Redis. Let’s see how things look like.

# app/controllers/listings_controller.rb
class ListingsController < ApplicationController
  def index
    @filter ||= ListingFilter.create
    @listings = @filter.results
  end
end

# app/models/listing_filter.rb
class ListingFilter < AllFutures::Base
  # Filters
  attribute :query, :string
  attribute :min_price, :integer, default: 1
  attribute :max_price, :integer, default: 1000
  attribute :category, :string, array: true, default: []
  attribute :tags, :string, array: true, default: []
  attribute :format, :string, array: true, default: []
  # Sorting
  attribute :order, :string, default: "created_at"
  attribute :direction, :string, default: "desc"

  def results
    # TODO: Build a query out of these attributes
  end
end

Notice how params are absent from the controller, and nothing gets passed to our ListingFilter object? And how come @filter could potentially be already defined? You’ll see why in a bit, so let’s first look at building the query.

In his approach, @leastbad simply created an ActiveRecord scope for each filter, then very cleverly and neatly, chained them to build his final filtered query, much like this:

# In app/models/listing_filter.rb
def results
  Listing.for_sale
    .price_between(min_price, max_price)
    .from_categories(category)
    .with_tags(tags)
    .with_formats(format)
    .search(query)
    .order(order => direction)
end

You might wonder: “But what if filters are empty and arguments blank? The chain’s gonna break!”. Well, have a look at the scopes declaration:

# app/models/listing.rb
class Listing < ApplicationRecord
  belongs_to :print

  scope :for_sale,        ->{ where(sold_at: nil) }
  scope :price_between,   ->(min, max) { where(price: min..max) }
  scope :with_formats,    ->(format_options) { joins(:print).where(prints: {format: format_options}) if format_options.present? }
  scope :from_categories, ->(cat_options) { joins(:artwork).where(artworks: {category: cat_options}) if cat_options.present? }
  scope :with_tags,       ->(options) { joins(:artwork).where("artworks.tags && ?", "{#{options.join(",")}}") if options.present? }
  scope :search           ->(query) { # TODO } 
end

In ListingFilter, the crucial bit is to make sure that every attribute has a default value. The magic then occurs in the if statement at the end of the scopes expecting an argument: if the lambda returns nil, then it will essentially be ignored, and the collection returned as is. Such a nice trick. Time for some specs to ensure that things actually work:

RSpec.describe ListingFilter, type: :model do
  let!(:photo) { Artwork.create(name: "Dogs", author: "Elliott Erwitt", year: 1962, tags: %w[Animals B&W USA], category: "photography") }
  let!(:poster) { Artwork.create(name: "Fargo", author: "Matt Taylor", year: 2021, tags: %w[Cinema USA], category: "poster") }
  let!(:photo_print) { photo.prints.create(format: "30x40", serial_number: 1) }
  let!(:photo_print_2) { photo.prints.create(format: "18x24", serial_number: 200) }
  let!(:poster_print) { poster.prints.create(format: "40x50", serial_number: 99) }
  let!(:photo_listing) { photo_print.listings.create(price: 800) }
  let!(:photo_listing_2) { photo_print_2.listings.create(price: 400) }
  let!(:poster_listing) { poster_print.listings.create(price: 200) }
  let!(:sold_listing) { poster_print.listings.create(price: 300, sold_at: 2.days.ago) }

  describe "#results" do

    it "doesn't return a sold listing" do
      expect(ListingFilter.create.results).not_to include(sold_listing)
    end

    context "Filter options" do
      it "Filters by price" do
        filter = ListingFilter.create(min_price: 100, max_price: 300)
        expect(filter.results).to match_array([poster_listing])
        filter.update(max_price: 1000)
        expect(filter.results).to match_array([photo_listing, photo_listing_2, poster_listing])
      end

      it "Filters by format" do
        filter = ListingFilter.create(format: ["40x50"])
        expect(filter.results).to match_array([poster_listing])
        filter.update(format: ["40x50", "30x40"])
        expect(filter.results).to match_array([photo_listing, poster_listing])
      end

      it "Filters by category" do
        filter = ListingFilter.create(category: ["photography"])
        expect(filter.results).to match_array([photo_listing_2, photo_listing])
        filter.update(category: ["photography", "poster"])
        expect(filter.results).to match_array([photo_listing, photo_listing_2, poster_listing])
      end

      it "Filters by tags" do
        filter = ListingFilter.create(tags: ["Cinema"])
        expect(filter.results).to match_array([poster_listing])
        filter.update(tags: ["Cinema", "Animals"])
        expect(filter.results).to match_array([photo_listing, photo_listing_2, poster_listing])
      end

      it "Filters by multiple attributes" do
        filter = ListingFilter.create(tags: ["Cinema"], max_price: 300, category: ["poster"])
        expect(filter.results).to match_array([poster_listing])
      end
    end
  end
end

All green. I hope you’ll appreciate how easy it is to test this. Let’s quickly add pg_search to our Gemfile, then take care of the search scope:

# app/models/listing.rb
class Listing < ApplicationRecord
  include PgSearch::Model

  belongs_to :print
  has_one :artwork, through: :print

  scope :search, ->(query) { basic_search(query) if query.present? }
  # Skipping the other scopes...

  pg_search_scope :basic_search,
    associated_against: {
      artwork: [:name, :author]
    },
    using: {
      tsearch: {prefix: true}
    }
  #...
end

Since our listings don’t carry much information, we’ll have to jump a few tables to look where we need, namely the name and author columns of our Artwork model. Unfortunately, pg_search doesn’t support associated queries further than 1 table away, thus the has_one... through relationship we needed to add. Let’s add some tests for the search:

context "Search" do
  it "renders all listings if no query is passed" do
    filter = ListingFilter.create(query: "")
    expect(filter.results).to match_array([photo_listing, photo_listing_2, poster_listing])
  end

  it "can search by artwork name or author" do
    filter = ListingFilter.create(query: "Erwitt")
    expect(filter.results).to match_array([photo_listing, photo_listing_2])
    filter.update(query: "Fargo")
    expect(filter.results).to match_array([poster_listing])
  end

  it "can both search and sort" do
    filter = ListingFilter.create(query: "Erwitt", order_by: "price", direction: "asc")
    expect(filter.results.to_a).to eq([photo_listing_2, photo_listing])
    filter.update(query: "Erwitt", order_by: "price", direction: "desc")
    expect(filter.results.to_a).to eq([photo_listing, photo_listing_2])
  end
end

We run the tests and of course, everything is gr… Oh no. Looks like the last test is acting up:

Apparently, a known problem of pg_search is that it doesn’t play well with eager loading, nor combinations of join and where queries. The recommended workaround (and my usual plan B when ActiveRecord queries start to get ugly) is to use a subquery:

# In app/models/listing_filter.rb
def results
  filtered_listings_ids = Listing.for_sale
    .price_between(min_price, max_price)
    .from_categories(category)
    .with_tags(tags)
    .with_formats(format)
    .pluck(:id)

  Listing.where(id: filtered_listings_ids)
    .search(query)
    .order(order_by => direction)
    .limit(200)
end

Let’s also add some last specs for the sort options and run all this.

context "Sort options" do
  specify "Recent listings first (default behaviour)" do
    filter = ListingFilter.create
    expect(filter.results.to_a).to eq([poster_listing, photo_listing_2, photo_listing])
  end

  specify "Most expensive first" do
    filter = ListingFilter.create(order_by: "price", direction: "desc")
    expect(filter.results.to_a).to eq([photo_listing, photo_listing_2, poster_listing])
  end

  specify "Least expensive first" do
    filter = ListingFilter.create(order_by: "price", direction: "asc")
    expect(filter.results.to_a).to eq([poster_listing, photo_listing_2, photo_listing])
  end
end

Everything’s green… Except for the search and filter option. The error’s gone, but the test still fails; the ordering doesn’t seem to work, despite all the sorting tests being green. After another lookup on pg_search known issues, it appears that order statements following the search scope don’t work. Workarounds include using reorder instead, or moving the order clause up the chain. I opted for the first option, which make all tests pass. Let's move on.

Stairway to Heaven

Now that we know that our backend is working as it should, let’s wire up our stuff. I’m gonna skip on Stimulus Reflex setup and configuration and dive right in. You can easily follow the official setup or, if you use import-maps, follow @julianrubisch’s article on the topic. I also know that leastbad has been working on an automatic installer that detects your configuration and sets everything up for you if you care to try it before the next version of SR gets released.

Once you’re done with that, let’s begin with the sort first. Let’s recap our sorting options and store them somewhere:

class ListingFilter < AllFutures::Base
  SORTING_OPTIONS = [
    {column: "created_at", direction: "desc", text: "Recently added"},
    {column: "price", direction: "asc", text: "Price: Low to High"},
    {column: "price", direction: "desc", text: "Price: High to Low"}
  ]
  #...
  attribute :order_by, :string, default: "created_at"
  attribute :direction, :string, default: "desc"
  #...

  # Memoizing the value to avoid re-computing at every call
  def selected_sorting_option
    @_selected_option ||= SORTING_OPTIONS.find {|option| order_by == option[:column] && direction == option[:direction] }
  end
end

Then in our “Sort by” dropdown, we’ll have something like :

<div class="dropdown">
  <button>
    Sort by:<span><%= @filter.selected_sorting_option[:text] %></span>
  </button>
  <!-- Skipping lots of HTML -->
  <% ListingFilter::SORTING_OPTIONS.each do |option| %>
    <% if option == @filter.selected_sorting_option %>
      <span class="font-semi-bold ..."><%= option[:text] %></span>
    <% else %>
      <a data-reflex="click->Listing#sort"
        data-column="<%= option[:column] %>"
        data-direction="<%= option[:direction] %>"
        data-filter-id="<%= @filter.id %>"
        href="#"
      >
        <%= option[:text] %>
      </a>
    <% end %>
  <% end %>
</div>

Even if you're unfamiliar with StimulusReflex, it should still remind you of the way we invoke regular stimulus controllers. Only here, when our link gets clicked, it should trigger the sort action (a ruby method) from the Listing reflex (a ruby class). Let’s code it:

# app/reflexes/listing_reflex.rb
class ListingReflex < ApplicationReflex
  def sort
    @filter = ListingFilter.find(element.dataset.filter_id)
    @filter.order_by = element.dataset.column
    @filter.direction = element.dataset.direction
    @filter.save
  end
end

And sure enough, it works! So what's going on here? Well, clicking the link invokes our reflex, which gets executed right before our current controller action runs again. It allows us to execute any kind of server-side logic, as well as play with the DOM in various ways, but with ruby code. Then, the DOM gets morphed over the wire.

What we did in our specific case: since our filter object is being persisted in Redis, it has a public id, which we stored as a data-attribute, and later retrieved from our reflex action. Then, we fetched the object from memory and updated it with new attributes. This is why @filter will be already defined by the time we get to that point. By default, not specifying anything more in our action will cause SR to just re-render the whole page before running the controller action. We could be more specific here, and just choose to morph a few elements to save precious milliseconds. But for demo purposes we’ll leave it as is.

Let’s add a filter next. We’ll start with the first one, by minimum price.

<div class="text-sm text-gray-600 flex justify-between">
  <label for="min-price">Minimum Price:</label>
  <span><output id="minPrice">50</output> $</span>
</div>
<input type="range"
  data-reflex="change->Listing#min_price"
  data-filter-id="<%= @filter.id %>"
  name="min-price"
  min="50"
  max="1000"
  value="<%= @filter.min_price %>"
  class="accent-indigo-600"
  oninput="document.getElementById('minPrice').value = this.value"
>

I got lazy and didn’t want to code an extra stimulus controller just to show the price value. But apart from that, we just need to add the new #min_price action:

# app/reflexes/listing_reflex.rb
class ListingReflex < ApplicationReflex
  def sort
    @filter = ListingFilter.find(element.dataset.filter_id)
    @filter.order_by = element.dataset.column
    @filter.direction = element.dataset.direction
    @filter.save
  end

  def min_price
    @filter = ListingFilter.find(element.dataset.filter_id)
    @filter.min_price = element.dataset.value
    @filter.save    
  end
end

And here in action:

I think by now you get the picture. Let’s just do the search and one of the checkbox filters.

In the view:

<!-- Search -->
<input type="search" value="<%= @filter.query %>" data-filter-id="<%= @filter.id %>" data-reflex="change->Listing#search">

<!-- Format Filter -->
<% Print::FORMATS.each_with_index do |format, index| %>
  <div class="flex items-center">
    <input data-reflex="change->Listing#format" <%= "checked" if @filter.format.include? format %> data-filter_id="<%= @filter.id %>" value="<%= format %>" type="checkbox">
  </div>
<% end %>

Our Reflex actions are starting to be pretty similar to each other, which calls for a refactor. You can’t do any better than leastbad’s approach, especially if you start having more complicated logic going on (like custom morphs or pagination):

# app/reflexes/listing_reflex.rb
class ListingReflex < ApplicationReflex
  def sort
    update_listing_filter do |filter|
      filter.order_by = element.dataset.column
      filter.direction = element.dataset.direction
    end
  end

  def min_price
    update_listing_filter do |filter|
      filter.min_price = element.value.to_i
    end
  end

  def max_price
    update_listing_filter do |filter|
      filter.max_price = element.value.to_i
    end
  end

  def format
    update_listing_filter do |filter|
      filter.format = element.value
    end
  end

  def search
    update_listing_filter do |filter|
      filter.query = element.value
    end
  end

  private

  def update_listing_filter
    @filter = ListingFilter.find(element.dataset.filter_id)
    yield @filter
    @filter.save
    # Add custom morphs here or any logic before the controller action is run
  end
end

And so on with the other filters. We can now combine search, filters and sort options with no page refresh.

Ambrosia on the cake

Let’s enhance the UX a bit. Right now there’s no pagination. Straight after adding pagy, clicking any page link will navigate, causing the params to reset. Let’s fix this by overriding pagy’s default template and wire links to our Reflex instead:

 <!-- views/listings/_pagy_nav.html.erb -->
<% link = pagy_link_proc(pagy) -%>
<%#                            -%><nav class="pagy_nav pagination space-x-4" role="navigation">
<% if pagy.prev                -%>  <span class="page prev"><a class="text-indigo-400" href="#" data-reflex="click->Listing#paginate" data-filter-id="<%= filter.id %>" data-page="<%= pagy.prev || 1 %>">Previous</a></span>
<% else                        -%>  <span class="page prev text-gray-300">Previous</span>
<% end                         -%>
<% pagy.series.each do |item|  -%>
<%   if    item.is_a?(Integer) -%>  <span class="page"><a class="text-indigo-400 hover:text-indigo-600" href="#" data-reflex="click->Listing#paginate" data-filter-id="<%= filter.id %>" data-page="<%= item %>"><%== item %></a></span>
<%   elsif item.is_a?(String)  -%>  <span class="page page-current font-bold"><%= item %></span>
<%   elsif item == :gap        -%>  <span class="page text-gray-400"><%== pagy_t('pagy.nav.gap') %></span>
<%   end                       -%>
<% end                         -%>
<% if pagy.next                -%>  <span class="page next"><a class="text-indigo-400 hover:text-indigo-600" href="#" data-reflex="click->Listing#paginate" data-filter-id="<%= filter.id %>" data-page="<%= pagy.next || pagy.last %>">Next</a></span>
<% else                        -%>  <span class="page next disabled">Next</span>
<% end                         -%>
<%#                            -%></nav>

# Add the paginate method to ListingReflex
def paginate
  update_listing_filter do |filter|
    filter.page = element.dataset.page
  end
end

# And update the controller
def index
  @filter ||= ListingFilter.create
  @pagy, @listings = pagy(@filter.results, items: 12, page: @filter.page, size: [1,1,1,1])
  # Can sometimes happen over navigation when collection gets changed in real time
rescue Pagy::OverflowError
  @pagy, @listings = pagy(@filter.results, items: 12, page: 1, size: [1,1,1,1])
end

Another issue is that at the moment, updating filters and our sorting options don’t update the URL params; refreshing the page clears everything, and we’re not able to save or share the result of our search to someone. Let’s take care of that as well. What we want is for our URL to always reflect the current state of filters on one hand, then be able to load our filter params from the URL on the other hand.

First step is made easy by the mighty cable_ready library, namely its push_state operation. Not only is it almost magical, but it is ready to use in any Reflex. Have a look at all you can do with it. Here is what our main action needs to do what we want:

# reflexes/listing_reflex.rb
def update_listing_filter
  @filter = ListingFilter.find(element.dataset.filter_id)
  yield @filter
  @filter.save
  # Updating URL with serialized attributes from our filter
  cable_ready.push_state(url: "#{request.path}?#{@filter.attributes.to_query}")
end

Now if you change any filter, type any query, change page or switch sorting option, the URL will update itself, including every filter attribute. Our last step is to load these attributes from the params on initial page load:

class ListingsController < ApplicationController
  include Pagy::Backend

  def index
    @filter ||= ListingFilter.create(filter_params)
    @pagy, @listings = pagy(@filter.results, items: 12, page: @filter.page, size: [1,1,1,1])
  rescue Pagy::OverflowError
    @pagy, @listings = pagy(@filter.results, items: 12, page: 1, size: [1,1,1,1])
  end

  private

  # Don't forget to update this list when adding filter options
  def filter_params
    params.permit(
      :query,
      :min_price,
      :max_price,
      :page, 
      :order_by, 
      :direction,
      category: [],
      tags: [],
      format: []
    )
  end
end

It’s starting to get pretty nifty. One last issue UX-wise : since we can no longer refresh to clear it all, we lack a Clear All button. Just add a link, then wire it to a Reflex action such as:

def clear
  ListingFilter.find(element.dataset.filter_id).destroy
  @filter = ListingFilter.create
  cable_ready.push_state(url: request.path)
end

And here you are, as close as ever to eternal bliss in search paradise. You can have a look at the live app here. UPDATE: Unfortunately fly.io's Upstash Redis doesn't play well with ActionCable, so connection gets lost after a while, preventing SR to work properly. While I tackle this issue, feel free to clone the repo. Not so heavenly after all.

Behold the afterlife

Let’s recap what we learned. Thanks to StimulusReflex , we learned how to build a super reactive search and filter interface with clean and extendable code, great performance, and almost no JavaScript. We saw how cable_ready could provide some sprinkle of magic behaviour on top of StimulusReflex. We were able to cleanly and temporarily persist, then update our search data thanks to all_futures. We also learned how to chain conditional scopes in a safe manner.

Unfortunately, good things rarely last forever. In our next episode, we’ll see how new requirements and a bigger set of records will party poop our not-so-eternal dream. You'll get to see how ElasticSearch can save the day and allow us to build the ultimate search engine.

Thanks for reading folks, and see you on the other side!

Resources

Your first ruby gem 💎

Louis Sommer — Sun, 20 Jun 2021 21:58:28 +0000

I remember reading many times over this piece of advice given to new, aspiring developers : « Go ahead! Read some source code, then contribute to open source projects! ». Truth be told, hearing this right after attending a 2-month coding bootcamp kind of felt like being pushed to try and climb a mountain with slippers and a swimsuit.

In spite of this, barely three years forward and a few apps later, a good friend of mine and I just released our very first Ruby gem, abyme. From the moment we rolled up our sleeves, up to our first release on rubygems, we just learnt so much along the way we felt we ought to share this experience, in the hope of empowering others to do the same.

Why you should build a gem too 👊

Before we dive in, let's quickly review a few reasons as to why we think building a gem is good for everyone - including you.

An obvious first reason would simply be to give back to the community. After all, most tools we use are open source : not only are they free, but for the most part, they're being actively maintained by contributors like you and me on their free time. As much as we could, sharing the fruit of our labor to help others feels pretty legit.

Another pretty obvious reason is the amount of knowledge you're likely to acquire by doing it. It's one thing to build applications using tools built by others for us ; it's a whole other thing to build the actual tools we use everyday ourselves. Among other things, gems involve quite a bit of metaprogramming, a rigorous file structure and file loading patterns ; you'll often have to resort to all kinds of tricks to make things work, and you will probably have to dig deep into ruby's closures and advanced OOP patterns. Finally, if you code a gem for Rails, you will no doubt have to read through lots of source code to get a grasp of how things actually work behind the scenes. All this becomes priceless knowledge when it comes to using and debugging these tools in your day to day job.

Finally, let's not forget another aspect of building stuff : whether it will get used or not, you'll probably get a nice ego boost out of your achievement. Making something work will make you proud and more confident in your programming duties. And if it does get used, a little fame doesn't hurt either.

Now that you know why you're here, let's go over the journey that awaits.

⚠️ Disclaimer : the rest of this article is not intended as a step by step tutorial on how to make a gem, but rather as a kind of retrospective on the different steps involved in building a gem, their associated challenges, and how we approached them as first-timers. If you're fine with that, read on !

Entrepreneur 101 💼

Like most journeys, you'll have to have some goal in mind. Think of your future gem like you would any other product : start with a pain. Something that you wish existed, or something that you find yourself coding again and again and that you feel could be DRYed up. Even better : try to identify in your actual work some cool behaviour that could be extracted and still hold its value.

Our first motivation to try and build something came from the frustration that often arise when resorting to nested attributes in Rails, and the absence of built-in Javascript behaviour to dynamically add fields on the fly (more details on the subject here). Solutions do exist, but they're either impractical, dated, or require quite a bit of configuration. Therefore our goal was simple : offer a plug-n-play approach to dealing with nested attributes in Rails.

Here's the behaviour we had in mind : a form inside a form, inside a form...

First implementation 📝

Once you know what you're building, just build it. Scaffold some demo app, and implement the behaviour you need as you would in a normal application. Once things work as they should, and if you haven't done TDD (I won't judge), you should write tests, so as not to break behaviour in your coming experiments.

In our case, nested attributes were already implemented in Rails through different modules (namely ActiveRecord and ActionView) so the basics were easy to setup. The JavaScript part though is not part of Rails's core, but many tutorials exist. One of those is this great Drifting Ruby episode, which we took as a base to build our demo application. In just a couple of hours of pair programming, our application was behaving the way we wanted.

While we successfully achieved our main goal front-end wise, the way it worked at that point in the back-end was still a bit tedious : for each new nested association, configuration needed to be copy-pasted in every model, every controller, every form. Plus, there was no way to easily control where the fields would appear, nor to make a difference between fields for already existing associated items or fresh ones created through our JS behaviour. Time to enhance our development experience.

DevX is your new UX 🦾

If we think of a gem as we would a product on the client perspective, our goal is to make sure it's intuitive, efficient and flexible. Think of it like the UX/UI aspect of a website ; only this time your clients are developers like you.

This is probably the most fun part : it all revolves around how we want to interact with our code : what should we call this method ? Should we pass arguments or a block to it ? Should it be an instance method of something ? To what extent should we allow customization ? Where will we put the configuration ? It also means anticipating a few different use cases, potential generators, while as much as possible, respect conventions, and give it a feel as close as possible to native Ruby/Rails code.

An interesting part of this process for us was the naming of our core method : the one that generates the form fields for the nested association. Just so you can have a feel of how it evolved, here's what the first iteration looked like :

<%= form_for @project do |f|
    <%= abymize(:tasks, f) %>
<% end %>

After a few months of use in real conditions, we decided it was really not intuitive, and it lacked the native Rails feel too much to our taste. We decided we wanted something like this instead :

<%= form_for @project do |f|
    <%= f.abyme_for :tasks %>
<% end %>

This had a much nicer feel. Problem was, it was much, much more difficult to implement, since it no longer meant having a simple view helper method. That was one of the toughest obstacle we had to overcome : it meant digging into what f actually stands for, find where it was originally defined, and how to add behaviour to it. The only way out was to - you probably guessed it - dig into some source code.

We eventually directed our search towards the famous simple_form gem source code, since we knew that it was fiddling with the same mysterious f object - which turned out to be an instance of ActionView::Helpers::FormBuilder class. It gave us a few hints (most notably in the way we should register our method) but it wasn't enough to make it work ; it was only after reading through Rails class definition (particularly line 2241 and line 2629) that we found the solution to our problem : we just had to call our existing helper method on the instance variable called @template. Happy devs we were.

Be mindful though : just like with any product, there's an infinite way of providing satisfaction, so you may want to just stick with some basic functions at first (like an MVP !). Open source will also mean that anyone will be able to contribute and add some other behaviour you would not have thought of, so it can actually be a good thing to voluntarily leave some blank space.

Extraction ⛏

After coding all the features we wanted came the million dollar question : how the hell were we going to turn this into a gem ?

You probably know that real rubies need to be extracted from actual rock, which is a tedious and dangerous process for the stone. While not particularly dangerous (as long as you have tests), the software version of the gem extraction remains a pretty delicate exercise. A good starting point is this excellent Drifting Ruby episode. Dave Kimura takes us through the first steps of generating the gem repository and moving the code piece by piece from inside the application to the lib folder, then to where it belongs in the gem’s own repository.

It only goes so far though ; chances are your gem will need to extend some native behaviour, and it will be your job to find how to do that. Luckily, thousands of gems exist in the wild, so your best bet will be to, again, get inspiration from what others did before you. My main education consisted in music harmony, and during my jazz years, I was taught that there was no better education than learning from the best : until today, I've spent countless hours transcribing solos from my favorite players by hand (I even wrote a small application on my free time during lockdown to allow people to freely share their own transcriptions).

In our case, we just went and looked for gems that looked remotely similar to ours, and studied their code. Since our gem required code to run in models, controllers, and views, we actually had to turn it into a Rails engine, which offers several benefits over a simple gem (most notably when it comes to testing, since an engine allows you to have a dummy app in your test directory for - take a guess - test duties). Then loading the different pieces was as easy as :

module Abyme
  class Engine < ::Rails::Engine
    isolate_namespace Abyme

    config.after_initialize do
      ActiveSupport.on_load :action_view do
        include Abyme::ViewHelpers
      end
      ActiveSupport.on_load :action_controller do
        include Abyme::Controller
      end
    end
  end
end

You probably wonder where we include the Abyme::Model part ; this may very well change in the future but we decided not to load it by default, so that users include it manually in their model (unless using our generators) like so :

class Task < ApplicationRecord
  include Abyme::Model

  has_many :comments, inverse_of: :task, dependent: :destroy
  abymize :comments
end

Finally, we shamelessly copied SimpleForm 's way of registering their simple_fields_for method on Rails's original FormBuilder class :

module Abyme
  module ActionViewExtensions
    module Builder
      def abyme_for(association, options = {}, &block)
                # Call to our #abyme_for view helper, defined in Abyme::ViewHelpers
        @template.abyme_for(association, self, options, &block)
      end
    end
  end
end
# The registration takes place below
module ActionView::Helpers
  class FormBuilder
    include Abyme::ActionViewExtensions::Builder
  end
end

Packaging 📦

Your efforts are about to be rewarded. Everything works, your tests are all green, and you're proud as a peacock. All you need to do now is just ship the damn thing.

Fortunately, this comes down to a very simple command (documented in Bundler's documentation) : rake release . You'll just have to update your gem version appropriately before running it, and it will take care of releasing the gem for you (you'll need to register an account on rubygems though). In a few seconds, you'll be ready to just bundle add your gem !

We need to talk about JS

I have to admit we kind of struggled with the JavaScript part of the gem, which is really poorly documented in the context of Rails gem building. Being responsible for the packaging duties, I actually broke the gem several times trying to make it work.

After doing some research it turned out there is no real way of including Javascript code through a gem. The JS code needs to be packaged into a npm or yarn package of its own. However, nothing holds you from using your gem repository as your yarn package repository as well, which is the approach we took. Naming the package like the gem, it allows installation with bundle add abyme and yarn add abyme which makes for a nice symmetry 👩🏾‍🤝‍👩🏼

On the packaging side, I learnt the hard way that you can't just leave some JavaScript in a file and expect it to work right off the bat. I had to try different packaging strategies before making it work (mainly because of Babel-related stuff that is outside the scope of this post). Just remember you'll need to use a packaging tool such as Microbundle to allow your ES6 (or newer) syntax to be compiled correctly by Webpack.

Hello World ! 🌎

At this moment, you may feel like you finally reached the top of the mountain. Unfortunately there's still quite a lot of stuff to be done before introducing your baby to the community. After taking care of packaging your gem and shipping it to rubygems, your to-do list should contain the following things :

Have and maintain maximum test coverage
Write some good and thorough documentation
Build a demo app to showcase your gem and its usage
Write a step by step tutorial (optional, but quite valuable)
Setup continuous integration tools
Grab a few badges to inspire confidence (code coverage, maintainability, tests passing...)
Generate a CHANGELOG to keep track of all changes and code additions

I can assure you from our recent experience that these things take time. Probably even more time than writing the gem itself. You may have written the most revolutionary tool, but if documentation is lacking or if the README sucks, chances are no one will bother to even try using it. Which kind of defeats the purpose of a community-driven endeavour.

Wrapping up 🎁

Time to get a little personal. Since writing abyme, I can really feel how it changed my way of approaching new features or refactoring in my applications. Without giving much thought to it, I now tend to better identify recurring and reusable pieces of logic, then implement them in a way that will make future extraction as straightforward as possible - when appropriate of course. I also have a better feel for when little bits of metaprogramming can sometimes really make things much more tidy and flexible. I think I have a much better understanding of how certains parts of Rails work, and know how to safely override or extend some gems behaviour when I need to. Overall I just feel much more confident as a developer.

Technical skills aside, it also revealed how hungry I was for such challenges. I love building tools, sharing knowledge, and open source represents such an infinite pool of both resources. I don't think I could envision my life without a reasonable time dedicated to this kind of contributions anymore. Coming from a creative industry, I naturally feel at home when giving life to fresh ideas that I think are worth sharing. And I happen to have a few ones coming 😎

Last but not least, I think the most rewarding thing is to observe that other junior/mid developers around me have started to adopt a mindset of I can do it too. And that's what I think open source should be about : share with others the will to just try and contribute.

Resources 📚

I'll keep this list updated with worthy additions.

Bundler doc
Rubygems guide
Rails engines documentation
Drifting Ruby (the gem series)
Eloquent Ruby (particularly for the chapters about DSL and metaprogramming)
Ruby closures

About us 🤓

Romain and I are two alumni from Le Wagon, a famous coding bootcamp which you might have heard of. We both attended batch #177 during summer 2018 in Paris, and are now teaching there while working freelance as fullstack web developers.

Cover picture by Jason D