Forem: Nick Sutterer

Unit tests in Trailblazer: less code, more coverage.

Nick Sutterer — Mon, 10 Feb 2025 09:16:40 +0000

by Nick Sutterer @apotonick

Writing and maintaining automated tests for code is the most hated part for every developer. You might find yourself nodding to this as you're reading.

It's not only the pain to set up the environment needed for testing a specific case, but also the amount of actually written lines of code to make sure you're covering "everything" that could be affected by the new chunk of code you introduced.

In this post, I want to focus on the second part of this bold statement and quickly show you how the just released trailblazer-test gem allows writing very concise unit tests for Trailblazer operations. A lot of work has been put into it to make covering edge cases as simple as possible.

What's that "operation"?

Trailblazer provides a service object called operation. It's the place where you put the business logic for a particular use case you are working on. With it's very simple DSL, logic can be organized in chunks and executed step-wise.

module Memo::Operation
  class Create < Trailblazer::Operation
    step :check_data
    step :validate
    step :save
    # ...

    def save(ctx, params:, **)
      ctx[:model] = Memo.create(params[:memo])
    end
  end
end

In short, when invoking the Memo::Operation::Create operation, the three exemplary steps will be executed in the order you defined them. If a step fails (by returning false), the remaining steps are skipped and the operation terminates, indicating a failure.

If you're keen to learn more about operations and the internal railway model, check our extensive docs or, much better, simply watch a 5 minute video.

It's good practice to have full coverage of your operations as this is the place (in a Trailblazer-driven app) where the business code lives.

Minitest or RSpec?

In the examples used to illustrate the new gem, we're using Minitest, as we think it's much more readable than RSpec. RSpec on the other hand is providing a great toolset but, whatsoever, is putting too much effort into a test DSL that's extremely verbose.

As a matter of fact, this is a matter of taste - no holy wars here: we also have RSpec support.

Asserting success

Now, to show you a very simple test case, let's create a test file, configure it, and run the Create operation with a particular input to assert that it terminates successfully.

# test/operation/memo_test.rb

require "test_helper"

class MemoOperationTest < Minitest::Spec
  Trailblazer::Test.module!(self) # install our helpers.

  it "passes with valid input" do
    input = {params: {memo: {content: "Stock up beer"}}}

    assert_pass Memo::Operation::Create, input
  end
end

Using Test.module! you include the #assert_pass assertion into the test class.

#assert_pass, in its purest form, takes the operation constant, and an input hash. Internally, the assertion now runs the operation with the specified input and then tests if the outcome was successful. This roughly translates to the following snippet, something I've seen throughout many TRB projects.

it "passes with valid input" do
  # ...
  result = Memo::Operation::Create.(input)

  assert_equal result.success?, true
end

There is nothing wrong with doing the above manually, but our assertions bring a (hopefully!) much better developer experience that we're about to discover.

Debugging? You're welcome!

A typical issue for developers when writing or changing tests is that an operation supposed to be passing actually fails. Most of the times, this is due to validation errors. Given that you're using a contract with an errors object, a failing #assert_pass will automatically print out the validation errors.

Within a second, you know that your input passed to the operation is not satisfying the validations. If that is not enough, you can simply add a question mark to the assertion.

it "passes with valid input" do
  # ...
  assert_pass? Memo::Operation::Create, input
end

This gives you the highly popular #wtf? trace on the console

Those two incredibly helpful features for debugging have been suggested by several TRB users over the years, as both checking the contract errors as well as turning on tracing (#wtf?) are the first things immediately done manually by many developers when hitting an issue.

Testing the model

While checking if an operation ran successfully is a great thing to do, bringing joy and happiness to the team, the product managers, and the clients, a good test needs to do a bit more.

In most cases, an operation produces or alters a model, which is usually found under ctx[:model]. After running, you may want to check if model attributes match your high expectations.

One way would be to use the block style and do the testing yourself.

it "passes with valid input" do
  # ...
  assert_pass Memo::Operation::Create, input do |result|
    assert_equal result[:model].content, "Stock up beer"
    # ...
end

The block simply yields the result objectand it's up to you what's done inside.

Alternatively, you can use the built-in attributes test of #assert_pass.

it "passes with valid input" do
  # ...
  assert_pass Memo::Operation::Create, input,
    content:    "Stock up beer",
    persisted?: true,
    id:         ->(asserted:, **) { asserted.id > 0 }
end

Your new best friend #assert_pass takes keywords as its third argument. Those are automatically matched against result[:model]. As you can see for :id, even dynamic assertions are possible.

The combination of the block style and the built-in model assertions provides a rich interface for testing any successful outcome of your operations.

When things go wrong

Testing the scenarios where an operation passes is a wonderful thing to do. However, probably even more important is testing when things don't play and operations are supposed to actually fail. To test a failing operation, we got #assert_fail - you already guessed that method name, right?

In many cases, an operation will fail if its validations aren't met. You can simply check if the operation terminated on the failure terminus by using the new assertion with one argument, only.

it "fails with invalid input" do
  invalid_input = {params: {memo: {}}}

  assert_fail Memo::Operation::Create, invalid_input
end

In rare cases, this might be a sufficient test, but most of the times you want to assert errors more detailed.

Testing error messages

Given that you're using a [contract in the operation], you can ask #assert_fail to check for specific validation error messages.

it "fails with invalid input" do
  # ...
  assert_fail Memo::Operation::Create, invalid_input,
    [:title, :content] # erroring fields.
end

The assertion will now check if the internal contract errors object contains the erroring fields you provided, resulting in a manual test that could look like so.

it "fails with invalid input" do
  # ...
  assert_equal result["contract.default"].errors.messages.keys,
    [:title, :content]
end

To write an even stricter test, you can provide the error messages as an additional constraint.

it "fails with invalid input" do
  # ...
  assert_fail Memo::Operation::Create, invalid_input,
    {
      title: ["must be filled"],
      content: ["must be filled", "size cannot be less than 8"]
    }
end

When expected error messages do not match the actual ones, the assertion automatically shows you the latter.

Again, optimizing your experience and shortcutting ways to help you debug.

What about extendability?

Both assertions shipped with trailblazer-test provide the block syntax and return the result, in case you need to add more test code.

Also, keep in mind that the assertions described here are what we needed to minimize time, code and brain when writing tests. Feel free to ping us for discussing any extensions of the gem.

Suite: Minimizing code

The assertions described so far are designed to take away pain in your testing, but they require you to repeat arguments over and over again. The "Suite" mode targets defaulting, so the written code is even less.

Imagine you're testing our Create operation and you want to make sure that all validations are actually working, each one in a separate test case. Here's how that could look using the suite feature.

# test/memo/operation_test.rb
class MemoOperationTest < Minitest::Spec
  Trailblazer::Test.module!(self, suite: true)

  describe "Create" do
    # insert defaulting here, see below...

    it "{content} works" do
      assert_pass({content: "chill beer"}, {content: "chill beer"})
    end

    it "{tag_list} is converted to array" do
      assert_pass(
        {tag_list: "fridge,todo"},      # input
        {tag_list: ["fridge", "todo"]}  # model value.
      )
    end
  end
end

In suite mode, assertion arguments such as operation, the incoming ctx and expected attributes on the model can be set (and overwritten!) on the class and describe level.

Defaulting over verbosity

You can default arguments by simply defining special-named let() blocks on any level.

# test/memo/operation_test.rb
class MemoOperationTest < Minitest::Spec
  # ...
  describe "Create" do
    let(:operation) { Memo::Operation::Create }
    let(:default_ctx) do
      {
        params: {
          memo: { # Note the {:memo} key here!
            title:   "Todo",
            content: "Stock up beer",
          }
        }
      }
    end
    let(:expected_attributes) { ... }
  end

Instead of having to repeat those values, the suite-enabled assertions will use and accordingly merge arguments for you. A desired side-effect is that #assert_pass always checks all attributes on the model as it merges expected_attributes with the second hash you provided.

Check the docs to dive into this simple yet helpful feature.

Stop mocking me!

While it's usually good practice to test the entire stack of logic, meaning your tests also cover complex system parts like external services, sometimes it's necessary to stub a component.

Replacing a particular step can easily be done using #mock_step. You are correct when objecting that this method should be named #stub_step, but that's too close to dub_step and we haven't added an alias, yet. Note that a "step" could be an entire, nested operation using Subprocess(), anything modeled as a step can be stubbed.

it "runs fine" do
  stubbed_create = mock_step(Memo::Operation::Create, path: [:save]) do |ctx, **|
    # new logic for {save}.
    ctx[:saved] = true
  end

  assert_pass stubbed_create, ...
end

The :path option allows targeting either a first-level step sitting directly in Create, or a deeply nested step somewhere 6 levels down in your nested operation graph. Check the docs for some more detailed examples.

The #mock_step helper returns a new operation class which can then be passed to the assertions, or even returned from let(:operation) if using suite mode.

Maybe this post is a good place to mention that the entire stubbing logic is simply using the patch feature of Trailblazer internally - implementing this for the trailblazer-test gem was nothing more but applying the patching mechanics with three lines of code.

RSpec and more

In the next post we're going to introduce the RSpec matchers that are based on this gem.

it "passes with manual attributes" do
  input = {params: {memo: {title: "Reminder", content: "Do not forget"}}}

  expect(run(Memo::Operation::Create, input)).
    to pass_with(title: "Reminder")
end

Being a bit more verbose, they provide the exact same behavior that we ship for Minitest. If you have suggestions or ideas, never hesitate to discuss those with us! Now, have fun testing. Or at least, try to!

GC compaction: Behold your hash keys!

Nick Sutterer — Mon, 11 Nov 2024 10:59:07 +0000

or: The release of trailblazer-activity-0.16.4

Apparently, recent versions of Ruby (>= 3.2) allow for a new feature to save memory in production: garbage collection compaction. According to the bug reports of a bunch of Trailblazer power users, calling the compaction manually - after the application has loaded - is now a thing in the Rails world.

# app has eager-loaded all dependencies.
GC.verify_compaction_references(expand_heap: true, toward: :empty)

Basically, the fragmentation of the stack is minimized after invoking the compaction, objects and their pointers are moved around, and memory is freed.

While this sounds like a great thing to do, an actual bug in Ruby lead to runtime errors coming in from applications using this pattern and Trailblazer.

NoMethodError: undefined method `[]' for nil
      # trailblazer-activity/lib/trailblazer/activity/circuit.rb:80:in `next_for'

Now, at first glance, this looks like a problem in Trailblazer's heart, the activity gem, which implements the runtime object for an operation. Every time you're running a Trailblazer::Operation, the internal activity will execute a particular step and check for the next step to be invoked.

Hash keys do matter

A "step" in an activity can literally be any callable object. In Trailblazer core code, we often use the pattern of method objects to implement step logic.

class MyOperation < Trailblazer::Operation
  step task: Validation.method(:extract_params)
  step task: Validation.method(:validate)

Deep inside the operation's activity (well, it's actually not really deep), a hash is created that looks roughly as follows.

circuit = {
  #<Method: Validation.extract_params> => {
    Right: #<Method: Validation.validate>,
    Left: ...
  },
  #<Method: Validation.validate> => {...}
}

You can see, the gist of Trailblazer is actually a hash of steps pointing to possible outcomes and the "next" step for each outcome. Super simple stuff!

The problem we now faced was that we use Method instances as hash keys. This worked fine until people started using GC compaction, because the compaction erroneously changed those method hash keys, they were now pointing to ...nothing, crashing the running operation with a NoMethodError.

def a; end
hsh = {method(:a) => 1}

hsh[method(:a)]
 => 1 

GC.verify_compaction_references(expand_heap: true, toward: :empty)

hsh[method(:a)]
 => nil

Keep in mind that this problem only arouse when people were deploying the compaction "trick". I personally didn't even know about this new GC feature.

It's all your fault!

As if that runtime error wasn't enough, bringing us several dramatic bug reports, people now started challenging our code design, asking "why are you using a method instance as a hash key, that's not good style!" or something along that.

However, it turned out that this is really a bug in Ruby and will be fixed in Ruby 3.2.7, 3.3.7 and 3.4.0. While we're at it, I'd love to thank the Ruby core team for their swift responses and their efficiency at fixing this. It took Peter Zhu only a few hours, I wish I was as motivated as this gentleman.

Quick fix

If you happen to be stuck with a Ruby version that's not fixing the compaction problem, you can update to trailblazer-activity-0.16.4 and include our temporary fix.

# For Ruby <3.2.7, <3.3.7, <3.4.0 
# initializers/trailblazer.rb
require "trailblazer/activity/circuit/ruby_with_unfixed_compaction"
Trailblazer::Activity::Circuit.prepend(Trailblazer::Activity::Circuit::RubyWithUnfixedCompaction)

Enjoy the runtime-error free time, and don't forget to join our Zulip chat!

PS: Also, shoutouts to Tiago Teixera, who actively spotted and isolated this gnarly bug!

Fetching models using the new Model::Find() macro

Nick Sutterer — Thu, 27 Jun 2024 06:43:20 +0000

Trailblazer comes with predefined steps we call "macros" that help you with common tasks such as validating a form object or finding an existing model using ActiveRecord (actually, any other ORM you approve of).

The newly released Model::Find() macro is a replacement for Model() that, over the years, turned out to be helpful but a bit hard to customize, as it wouldn't really allow you to change how things are performed in the background.

class Update < Trailblazer::Activity::Railway
  step Model::Find(Song, find_by: :id, params_key: :slug)

You are going to need trailblazer-macro 2.1.16 for this goodie.

Extracting the ID

The new macro provides options such as :params_key and :column_key to configure how the ID is extracted. If you want to do it yourself, simply use a block.

step Model::Find(Song, find_by: :id) { |ctx, params:, **|
  params[:song] && params[:song][:id]
}

A bunch of discussions with long-term users lead us to the decision that overriding ID extraction should be straight-forward since this is more than just an edge case.

Customizing the query

Once the ID is extracted, it's now very simple to customize how the query is performed (e.g. find_by: id vs find(id)). Nevertheless, the new key feature is the :query option that allows you to write that code manually.

step Model::Find(
  Song,
  query: ->(ctx, id:, current_user:, **) { where(id: id, user: current_user) }
)

Note how the query logic can access ctx and keyword arguments, just like a real step. The extracted ID is available in the variable :id.

Not_found terminus

If one of the two steps weren't successful, you can instantly go to a new terminus not_found in your business operation, indicating to the outer world that this particular step failed. With the release of trailblazer-endpoint this will become interesting as the endpoint code could, for instance, automatically render a 404 page for you.

Discussion

The code of this macro is nothing special. In fact, it simply creates a tiny nested activity behind the scenes with two steps, one to extract the ID, and one to actually fetch the model.

Anyhow, we strongly recommend sticking with this macro instead of writing your own, for three reasons.

Well, code we write and maintain is less work for you. Keep in mind that we also provide documentation.
Features like the not_found terminus we added with forward-compatibility in mind: they will save you code once endpoints are becoming a thing.
Debugging Model::Find() is a matter of using our internal tracing. In the trace, you can see which part failed.

Please give us some feedback about what's missing or what you like about this simple addition to our stack. Have fun!

Trailblazer 2.1.3: How we made tracing +4x faster

Nick Sutterer — Fri, 15 Dec 2023 07:19:50 +0000

(PaaS providers hate this trick!)

With the release of Trailblazer 2.1.3 not only does #wtf? tracing receive a notable performance boost, we also paved the way for a web-based debugger by refactoring tracing internals.

Tracing faster by factor 4-10

When you run a (deeply nested) operation with #wtf?, the tracing code uses the runtime taskWrap to inject tracing before and after an actual step is called. BTW, we also uploaded new docs for this whole thing "taskWrap", if you're interested in extending TRB (for example, to add logging or APM), check out the new chapter: https://trailblazer.to/2.1/docs/activity#activity-taskwrap

The reason the old tracing code is super slow was because every time our tracing is run, a snapshot of ctx was taken. That was literally just a ctx.inspect.

def default_input_data_collector(wrap_config, ((ctx, _), _))
  {
    # ctx: ctx.to_h.freeze,
    ctx_snapshot: ctx.to_h.collect { |k,v| [k, v.inspect] }.to_h,
  } # TODO: proper snapshot!
end

You can see the # TODO: proper snapshot!, right? Well that's what we were working on the past weeks! Instead of "photographing" the entire ctx over and over with #inspect - which is tremendously slow once the ctx grows - we now compare its variables and only snapshot those that have changed.

Benchmarks

The entire refactoring is summarized in PR 42, which, by its sheer name, should be answering all your questions. And yes, there are still # TODOs after this refactoring, but tracing a moderate ctx is now 4-10 times faster and consumes a lot less memory.

Calculating -------------------------------------
        inspect-only     10.862  (± 9.2%) i/s -     54.000  in   5.066396s
            snapshot     61.024  (± 8.2%) i/s -    305.000  in   5.032931s

Comparison:
            snapshot:       61.0 i/s
        inspect-only:       10.9 i/s - 5.62x  slower

This benchmark was run in a huge client application and made my day.

Another win in this refactoring is that we can now configure a per-object snapshooter, so if you don't want your ActiveRecords to run arbitrary queries, a value_snapshooter will be your friend. Big thanks to @richardboehme for his support here.

Added `#left` alias

And for all of you who have a problem with Operation.fail because your IDE doesn't like the name, we introduced #left:
https://trailblazer.to/2.1/docs/activity#activity-strategy-railway-left

Trailblazer 2.1.2 bringing the Each() macro.

Nick Sutterer — Sun, 18 Dec 2022 08:49:10 +0000

Recently we released a bunch of updated Trailblazer gems and along with those the trailblazer framework 2.1.2.

Loops in Operation

It felt great to close several pull requests that were asking for (or even adding) support to iterate over datasets in TRB, basically an each do ... end for operations.

Integrating the features added by our fellow users with internal advanced mechanics such as tracing or patching took its time, that's why the "simple" PR sat there quite a while.

module Song::Operation
  class Cover < Trailblazer::Operation
    step :model
    step Each(dataset_from: :composers_for_each) {
      step :notify_composers
      # more steps here or Subprocess()
    }
    step :rearrange

    def composers_for_each(ctx, model:, **)
      model.composers
    end
    # ...
  end
end

The Each() macro allows you to specify where the data set is coming from, and will then run the provided block (or another operation) for each item at runtime.

Assuming we had three composers in the data set when we invoke the operation, the nested code is run subsequently as if there were three dedicated notify_composers steps.

class Cover < Trailblazer::Operation
  step :model
  step :notify_composers.0
  step :notify_composers.1
  step :notify_composers.2
  step :rearrange

We added features such as collecting values from the iterated block or input/output filtering, all described in the newly added docs.

Speaking of, we moved the macro docs to an entirely new section, and rewrote docs for Wrap() as they were pretty basic before.

One thing I particularly love about this new macro is that it integrates nicely with our debugging API. You can even see in which iteration an error occurred! Check the trace below.

Using Ruby's "native" each, this'd be a PITA to debug.

Your input on Each() is crucial - feel free to chat to us at any time.

Composable input/output

In former versions of Trailblazer, we had the :input and :output option to define what comes in and goes out of a step.

class Create < Trailblazer::Operation
  # ...
  step :validate,
    input: [:params],  # pass :params into the step
    output: [:result]  # only let :result out!
end

Variable mapping - as this is called in Trailblazer - is, after nesting, the most commonly used feature of the framework.

Good news: we are not dropping support for :input and :output! However, this is now superseded by "composable" variable mapping. On first glance, nothing much has changed.

step :validate,
    In() => [:params], 
    Out() => [:result]

Well, we replaced the :symbol option with In(), Out(), and Inject(). However, this comes in handy if you want to add or alter variable mapping that has been defined formerly, for example in a macro.

step Validate(), # adds In() and maybe Inject()
  In() => [:current_user] # we can add this on top!

You can even override and alter inherited variable mapping in a subclass.

class Upsert < Create
  step :validate_for_upsert, 
    replace: :validate,  # override inherited #validate step.
    inherit: [:variable_mapping], # inherit filters from Create.
    Inject() => [:current_user] # add this on top.

The entire approach with composable filters is extremely flexible and helps reducing bulky input/output code.

Introspect your filter chains

While this might look like a syntax touch-up, and, yes, anoooother API change, this brings us not only better structural features, but we can now introspect filter chains with the developer gem.

puts Trailblazer::Developer.render(Create, :validate)

This renders the taskWrap and along with it the input and output filter chain. We even combine that with our upcoming IDE.

Note that this is not the final introspect API, though!

The code for variable mapping sits in the dsl-linear gem and got some love, again. It is much easier to follow (and about 1/3 faster!) so maybe have a peek and add whatever you're missing?

You can find brand-new docs on our website.

Debugger and IDE

When working on Reform 3 and Each() I used Trailblazer's debugging tools frequently and while benefiting from wtf? and friends, we decided to improve the internal debugging API.

This turned into a bare-bones IDE that allows you to traverse through the trace of a deeply nested operation.

You can trace the incoming and outgoing ctx for each step, and even inspect its compiled taskWrap - where you can see the variable mapping chain!

The IDE will be a service launched by us in 2023.

What's next

Don't get me started, I could keep talking for hours! Anyhow, let's wrap this up.

Reform 3 is coming, we're pretty close to a presentable version. It is entirely built with Trailblazer, so tracing, variable mapping, extendability and all things TRB come for free!
The endpoint gem will get a stable release, promised!
More Trailblazer Tales are planned, sorry for the long interruption! Next topics will be nesting, variable mapping and the Wiring API.

We're looking forward to your feedback! I deleted Twitter and live on Mastodon now. Cheers and happy holidays!

Five first episodes of TRAILBLAZER TALES

Nick Sutterer — Thu, 13 Jan 2022 10:17:20 +0000

Finally, I started producing short 6 minutes episodes on Youtube to learn about Trailblazer. This is actually more fun than I expected!

The first five episodes focus on refactoring a messy Rails controller action to a Trailblazer operation. Here is the link to our channel.

Let us know what you think, and what is missing! In the following episodes we are going to discuss Reform, macros, nesting operations, and the new dependency injection features.

Don't mix Forwardable and ActiveSupport::Delegate

Nick Sutterer — Thu, 02 Dec 2021 11:47:13 +0000

A few days ago I released the disposable gem minor line 0.6.0. The goal was to remove unnecessary coupling to the old uber gem that was once planned to hold a set of abstractions we need in all Trailblazer gems.

What I mainly worked on was removing Uber::Delegate and supersede it with Ruby's built-in Forwardable. We simply need to delegate a bunch of methods from one object to another, and Forwardable does just that.

So what the new code basically did was to extend the Disposable::Twin class and use def_delegators to create automatic delegations.

class Disposable::Twin
  extend Forwardable # this used to be Uber::Delegate

  def_delegators :model, :title, :title=
  def_delegators :model, :email, :email=

Obviously, removing gem dependencies, even if they're your own, feels good, so I pushed this new gem after running tests in two projects and leaned back in my pricey office desk.

Here comes another delegate!

A day later we received bug reports from several users. All of them were using Reform form objects, the problem was thrown from Reform::Form classes they had in their app.

ArgumentError:
   wrong number of arguments (given 2, expected 1)
 # ruby/3.0.0/forwardable.rb:133:in `instance_delegate'

The developers reported that before the upgrade to Disposable 0.6.0, code as the following worked fine.

class CreateForm < Reform::Form
  delegate :assignment, to: :model
end

After the upgrade, this would break with the ArgumentError illustrated above. My first suspicion was that this is some code not ready for Ruby 3, as this kind of exception is often observed in older code that runs with Ruby 3.

Delegate or delegate?

An hours going through the Forwardable code and playing with the broken example app provided by some nice user didn't bring any progress.

At some point I realized that what the users were actually using to create delegations was the delegate method not from Forwardable but from ActiveSupport::Delegate. The docs enlightened me that this module mixes a method #delegate into the class - the method that was used by all users reporting the error.

Order matters!

It feels really stupid in hindsight, and it took at least two hours to understand the problem.

Here's the chain of sparks that lead to fixing the problem.

A Reform::Form class inherits from Twin::Disposable. This means that each Form class gets Forwardable methods mixed in. Of course, only after upgrading disposable to 0.6.0 where we automatically included Forwardable into Twin.
Before 0.6.0, the former delegation from Uber::Delegate only mixed in one method called #delegates (mind the s).
The Forwardable module includes a method called #delegate.
Our nasty ActiveSupport::Delegate module also has a method called #delegate - just like Forwardable!
In some Rails apps, delegation from ActiveSupport is automatically included into all classes.

The problem we were facing was that ActiveSupport and its delegation code was included automatically into the Reform::Form class before the disposable gem included Forwardable and hence the latter overriding the #delegate method you are expecting.

In other words, Forwardable killed ActiveSupport's #delegate method as it was loaded and included later.

Problem solved

We decided to removed Forwardable from disposable for now and fixed the obscure issue. When using delegate in a form it now refers to ActiveSupport's implementation.

Official Trailblazer 2.1 Release Notes

Nick Sutterer — Thu, 02 Dec 2021 10:57:56 +0000

(Not that there's an unofficial one anywhere, but yeah...)

This was posted on 27 Feb 2021 on the TRB website.

Note: We have fully documented the migration path to 2.1 while upgrading several complex apps. It worked.™

After around 3 years of silence, Trailblazer is back with its 2.1 release. Lots of improvements on the core gems come hand-in-hand with a quite flat migration path, so don't ya worry! We're also proud to tell you all about our new upcoming projects in this post, as well as communicating a looot of background about what's been going on in those past few years.

Here's a quick summary for those who want to get back to coding.

API CHANGE The call API has slightly changed from two arguments to just one.
AUTOLOADING We removed the trailblazer-loader gem just like Apple removed the headphone jack from the iPhone 6. This brings you faster startup and consistency with Rails autoloading.
DOCUMENTATION We have a new website.
TRACING We heard your cries. You can now see what path an operation took and use exception tracing for a beautiful dev experience.

Good to see you, again!

It almost feels unreal finishing up this release post. It's been so long! The work put into Trailblazer 2.1 has been tremendous, it could easily have been TRB 3.0, or even TRB III, since Roman version numbering turns out to be quite a fancy thing to do. However, as much as the internals have been improved, as little has changed on the public APIs of Trailblazer, so we decided to go with a minor release.

With the introduction of another new callback after_save_commit in Rails 6 it becomes obvious that we do have a problem in software engineering: how do we structure business code? We do have a few dozen database abstractions in every language, well written and designed, but, yeah, how do I structure the actual code of my domain, of my application's needs? What's the best way to put together application functions, put them in order, prevent other functions from being executed when it's not the time, and so on?

Trailblazer is an architectural style coming as a framework to implement our patterns, trying to answer that one question: "How do I implement business processes?". It gives you strong conventions along with patterns to structure your code flow better, handle errors, and integrate those components into bigger flows - or processes.

Having said that, I let you decide whether or not you join the ride! There is nothing wrong with building your own "service layer", and many companies have left the Traiblazer track in the past years due to problems they had and that we think we now fixed.

Speaking of problems, the craziness of the past years working on Trailblazer 2.1 can be visualized with the following screenshot.

Very often, in order to fix one function A in a commercial app, I had to rewire a track in the endpoint gem, which would only work if I add another DSL option in the dsl gem, which couldn't work straight away because the normalizer in the context gem needed to be extended, which was only possible if you could finally extract the start event via the activity gem whose test suite was currently broken because I needed to document feature x first, which couldn't be documented because before I had to allow dynamic params in the macro gem and... I think you got my point.

Luckily, I got help from a band of great people, I took it easy on the conference front, and for some other reasons I still don't understand all this is now finished and here's TRB 2.1.

About the Design

A big change from TRB 1.1 and even 2.0 is that we apply functional principles in most places. We barely keep state in instance variables. Rather, data is passed around from operation to operation, from step to step. We use OOP and inheritance solely for compile-time configuration. You define classes, steps, tracks and flows, inherit those, customize them using Ruby's built-in mechanics, but this all happens at compile-time. At runtime, no structures are changed anymore, your code is executed dynamically but only the ctx (formerly options) and its objects are mutated. This massively improves the code quality and with it, the runtime stability, which Ruby is famous for *cough.

I extracted a lot of smaller libraries from the original trailblazer gem. Most gems use the activity gem itself, which is the core of an operation and provides the runtime object that executes your steps in a certain order. It's so simple that I sometimes wonder why it took years to develop it! But all those little interfaces, structures and principles that you see as "common sense", they took a while to figure out.

A lot of work went into the dsl-linear gem that provides a DSL to create activities (or operations) using your old friend #step. It's not even a DSL since it doesn't create code at run-time, it doesn't even create code: it creates runtime objects, activities, that are executed when your operation is run. You could create those objects manually without the DSL, or even write your own DSL, which we're hoping for!

So, whenever you hear the medieval argument "Trailblazer is just a nasty DSL!", forgive your opponent, you now know better. The entire framework is based on small, clean Ruby structures that can be executed programmatically.

Also, the more I use Trailblazer in projects or even in Trailblazer itself, I feel how needed those new abstractions are. Yes, you can write everything with your own code, you don't need abstractions for flow control and automatic error handling, which makes me wonder why you're not programming in assembler since Ruby is also an "unnecessary abstraction" on top of a processor. We need abstractions, unless you want to program like we did 30 years ago.

Ok, enough of this - let's jump into the things that actually help you.

Loader / Startup Speed

Startup speed and loading of operations has been a never-ending problem.

To make it short: we returned to the Rails Way™, lowering our heads in shame, and adhere to the Rails file and class naming structure for operations. What used to be named Memo::Create is now Memo::Operation::Create, which reflects its file location concepts/memo/operation/create.rb (still the same!) and works with Rails' autoloading.

The big benefit is: we could remove the trailblazer-loader gem. The alternative loader was a naive attempt by me to combine "our" naming style with the Rails file structure. After endless fire-fighting against broken development reloading, inconsistencies, slow startup times in develpment mode, more incompatibilities with Rails' autoloader and some clarifying discussions with the great Xavier Noria himself (author of Rails loading and zeitwerk), we ditched our loading mechanism.

If you want to take advantage of the fast server startup times and (almost) seamless hot reloading of classes, you need to rename your operations.

This change is not super dramatic at all, given that in the future, you won't be calling operations yourself all too much anymore. With the upcoming workflow and endpoint gems, those abstractions will invoke the business logic for you, so don't worry too much.

New website

In case you haven't noticed: you are reading this release blog post on a newly designed website! Not only does it come with the most beautiful, romantic and inspiring hero backdrop ever (featuring a James-Cameron-worthy parallax effect, too!), it also attempts to provide all the old documentation while bringing you the hottest off-the-press docs for our new gems.

We try to keep the "information architecture" - a word I wouldn't have learned without the inspiring Alex Coles - as simple as possible: so far, we got a handful of pages accessible through the top navigation, and then the documentation behind the DOCS link. Here, the right sidebar helps you to navigate within the chapter.

After all, we're a nerdy developer's library and don't need no million hipster landing pages! One is more than enough and took months of collaboration between us, the design and web team in Argentina and our awesome comic artist Josh Bauman who also drew the epic Trailblazer book illustrations in 2016.

The documentation has been designed to make coherent information as quickly accessible as possible. We even added a search feature - something our beloved user base has asked for many times. Just hit the / key (or Shift+7) in the docs.

API CHANGES

The new 2.1 version comes with a few necessary but reasonable changes in method signatures. As painful as that might sound to your Rails-spoiled ears, we preferred to fix design mistakes now before dragging them on forever.

New call API

In versions before 2.1, the automatic merging of the params part and the additional options was confusing many new users and an unnecessary step.

# old style
result = Memo::Create.(params, "current_user" => current_user)

The first argument (params) was merged into the second argument using the key "params". You now pass one hash to call and hence do the merging yourself.

# new style
result = Memo::Operation::Create.(params: params, current_user: current_use )

Your steps use the existing API, and everything here is as it used to be before.

class Memo::Operation::Create < Trailblazer::Operation
  step :create_model

  def create_model(options, params:, **)
    # ..
  end
end

The new call API is much more consistent and takes away another thing we kept explaining to new users - an indicator for a flawed API. Believe it or not, but not a single time has this topic come up again on our support forum.

<%= h4 "Symbol keys for ctx" %>

The ctx (or options) object now behaves like a HashWithIndifferentAccess, a pattern used for Rails' params object. In other words, you can now access ctx[:current_user] instead of ctx["current_user"] and conveniently use the keyword argument version of it, too.

Another handy addition to ctx is aliasing. Before 2.1, you had to refer to, say, the main contract via ctx["contract.default"]. While I designed that name "back in the days" with multiple contracts per operation in mind, it turned out to be quite clumsy since you couldn't access it as a keyword argument.

You're able to set aliases now for the ctx object, allowing magical things to happen, amongst those an aliasing from "contract.default" to :contract.

class Memo::Operation::Create < Trailblazer::Operation
  step :inspect_contract

  def inspect_contract(ctx, contract:, **)
    puts contract #=> same as ctx["contract.default"]
  end

The ctx object is now documented in its own section.

Developer Experience

Debugging deeply nested operations before 2.1 sucked. Even if there was no nesting applied, an exception thrown from a random step was hard to find, the native Ruby stacktrace didn't really help, the TRB and Rails framework noise made it a painful experience to spot the source of an exception.

This was one of the major arguments in many companies against Trailblazer, besides it "being too complex" while it seemed much easier to program pure Ruby as we did in the 90s. And we liked it that way!

To tell you the truth, the new tracing feature was the original reason why I decided to write 2.1 and make you sit and wait in agony for years. Nevertheless, tracing is simply blowing my mind. I can't count how many hours and angering rushs of adrenaline I've saved since the introduction of the wtf? method and its helpful higher-level stack trace.

Not only does tracing show you the path the execution took in a successful invokation, it also shows you where an exception happenend. And not "in Ruby", it's "in Trailblazer", in the actual step where things broke.

With the new developer gem, you can trace executions, find exceptions, observe certain variables in the ctx throughout the flow, render operations and activities to visually understand what's going on, and much more.

As this new gem turns out to be a massive time saver for everyone, yes, even for us, we're focusing on extending and adding functions such as visualizing nested activities.

It might sound a bit overly sentimental or even cheesy, but I personally think that this is lifting the dev experience to a new level that will be hard to come by in pure Ruby. Yes, Trailblazer is adding new abstractions and concepts and they are different to the 90s-Ruby, but now, at the latest, it becomes obvious how this improves the developing process. We're no longer talking in two-dimensional method stack traces or byebug hoops, the language and conception is changing to the actual higher level code flow, to activities sitting in activities structured into smaller step units.

Trailblazer::Developer.wtf?(Memo::Operation::Create, [ctx])

Once you've used the wtf? method, you will feel it!

Unlimited Wiring

Over the past years, harnessing the flow control and error handling mechanics we added in 2.0, one thing became obvious pretty quickly: users want more wiring possibilities. The linear railway and "fast track" concepts have been a helpful tool for structuring code flow, but people need to be able to model arbitrary flows.

Needless to say that 2.1 provides you just that. We added the concepts of outputs to steps that can be added or "re-wired" using the Wiring API.

You may add additional outputs, rewire, go back, error-out in additional termini, go nuts, and all that using a super simple DSL.

class Execute < Trailblazer::Activity::Railway
  step :model, Output(:failure) => End(:not_found)
  step :validate
  step :save,  Output(:retry) => Id(:model)
end

This works for both Activity subclasses and operations (which are just special-flavored Activity::FastTracks).

While you're still wondering whether we've completely lost it, we already use those new wiring mechanics to build everything from simple edge cases in code flows to long-running, complex business workflows.

A major improvement here is the ability to maintain more than two explicit termini. In 2.0, you had the success and the failure termini (or "ends" as we used to call them). Now, additional ends such as not_found can be leveraged to communicate a non-binary outcome of your activity or operation.

Using a terminus to indicate a certain outcome - in turn - allows for much stronger interfaces across nested activities and less guessing! For example, in the new endpoint gem, the not_found terminus is then wired to a special "404 track" that handles the case of "model not found". The beautiful thing here is: there is no guessing by inspecting ctx[:model] or the like - the not_found end has only one meaning!

Check the endpoint gem if you want to see that in action.

Core team

In the past 1 ½ years something weird happened: a real core team formed around the Trailblazer gems. I say "real" because in the past 15 years of OSS, I've had people come and go, being of great help but never staying and taking over long-term responsibilities - which I found to be the pivotal element of a core team.

Eventually, those kids convinced me to start the Trailblazer organization on Github and move over all "apotonick gems". Over the course of time, I saw myself giving away that aforementioned responsibility with a smile on my face, adding owners and collaborators to gems, yes, even giving away entire gems, letting people work on documentation and just trusting someone and their skills.

I have no words to describe how good that feels!

For me, a dream has come true. I work with crazy geniuses who share many of my opinions (not all, and that's good). I learned to "let go" and simply trust others to maintain certain gem suites. Messages like "don't worry, I'll do it" combined with a pull requests minutes later - things I literally dreamed of a few years ago, are now part of my daily routine.

Here are those kids, in the order of appearance: <3

Celso Fernandes who's been part of TRB since the 12ths commit or something. He's currently focusing on the business side, consulting, and is the magical sysop of our infrastructure.
Emanuele Magliozzi worked for many years as a TRB consultant and is now the master of reform and its 2.x line and interrupting our important daily work with Aussie bantering.
Abdelkader Boudih who's still unaware that I secretly call him "The Machine" since I'm not sure he's an AI bot trained to fulfill basically any request an OSS author might have.
Pedro Visintin, besides keeping me busy talking about punk rock guitars, is the person behind the PRO editor and our sharply-dressed agile marketing director.
Kamil Milewski has been contributing to TRB for years both intellectually and with many code additions and even more deletions.
Krzysztof Piotrowski has been tirelessly helping with writing example applications and turned out to be the refactoring expert on board.
Adam Piotrowski is resisting my deceiving calls to work on the code side and rather manages our processes - so we gladly let him do just this!
Yogesh Khater is now working full-time for Trailblazer, working on all 48 repositories at the same time (plus our private ones!), releasing gems when I sleep, leaving me jobless and peacefully napping in the sun.

In addition to the organically formed core team, I started Trailblazer GmbH 4 years ago with my relocation from Australia back to Europe. One of our consulting clients is the central police department of a German state that has kept me busy for more than three years now. And yes, at TRB GmbH, we do pay people to work on OSS!

License and PRO

Around 2 years ago I decided to end the experiment of "TRB PRO" as I felt I didn't provide enough value to paying users. In the end, we had around 150 companies and individuals signed up, which was epic and a great funding source for more development.

We're now relaunching PRO, but instead of a paid chat and (never existing) paid documentation, your team gets access to paid gems, our visual editor for workflows, and a commercial license.

The editor (that mostly plays along with the workflow gem) is already in use for a bunch of commercial projects that we consult, but will be launched in its stable version somewhere this year.

The core gems of Trailblazer have been re-licensed to LGPLv3 to raise awareness for this. For 99% of all our users, this has no legal implication at all - go use TRB for free if you don't feel like paying money for OSS.

With all this "monetization" happening around Trailblazer, we will also make sure that all free and paid parts of the project grow adult and maintan an LTS - or long-term support - status. Those are good news to all you users out there having been scared to use gems of this project, not knowing whether or not they're being maintained, breaking code in the future or making your developers addicted to and then cutting off the supply chain. Trailblazer 2.1 onwards is LTS, and the last 1 ½ years of collaboration have proven that.

Upcoming gems and plans

We use Trello boards now to organize our work. Yeah, you can teach old dogs new tricks! Not only does it help to structure myself, it also manifests how many ideas and improvements we got lined up for the near future.

COMPILE-TIME PERFORMANCE While we're pretty good on the runtime performance, compiling all those activities takes more time than it could. We already started improving the compilation process and will ship updates in the next months. Optimization in this case is nothing crazy, just something I neglected while designing the framework.
TRAILBLAZER-TEST The official stable release is only weeks away bringing you a bunch of new assertions that drastically reduce coding effort for tests! Of course, Minitest and RSpec will both be supported.
TRAILBLAZER-STORY will follow as it turned out to be inevitable for setting up application state for tests. Instead of fumbling around with factories and traits in your tests, you "tell a story" about what to create in which order, easily customizable, and all written using activities. Currently, I'm working on designing the interfaces and it's real fun!
TRAILBLAZER-WORKFLOW is another dream 'o mine come true. It allows creating long-term processes (or state machines) based on BPMN diagrams that can be modeled using our editor. I'm pretty confident that you, coming from AASM or your own state machine engine, might find this approach highly interesting. We're using this gem in several commercial apps already, so the first release should be out this summer (I didn't say which hemisphere).
TRAILBLAZER-ENDPOINT I've been promising for many years and it turned out I couldn't have fully designed it without the tools we do have now. Endpoint is the missing link between your routing (Rails, Hanami, ...) and the "operation" to be called. It provides standard behavior for all cases 404, 401, 403, etc and lets you hook in your own logic like Devise or Tyrant authentication, again, using TRB activity mechanics.
TYRANT is an authentication framework fully written in TRB activities, interacting with Rails, workflows and your database. It's highly customizable using the TRB mechanics and definitely less rough to work with than Devise. Looking forward to the release here!

<%= img "/2.1/blog-ep-draft.png" %>

Documentation

Writing documentation for the new website has been fun. Yes, fun! Mainly because we have a helpful little framework behind the page rendering that pulls code snippets from real tests out of the actual gems!

Many users have asked whether there will be an updated version of the Trailblazer book. While it's really cute and humbling to still have around $10 per month income from the now free book, keep in mind it was published end of June 2016, that's pretty much four years ago!

What this means is: I better refrain from writing a new book and we rather focus on more and better docs. Writing examples and technical docs as a team has been working out pretty well recently and we will add more Trailblazer tutorials in the next half year. A comprehensive workflow tutorial demonstrating long-running processes and all things Trailblazer is also in the works! The legendary cfp-app will become a Rails-to-TRB refactoring tutorial.

We decided against paid documentation, so all will be freely available on our shiny new website.

Staying up-to-date

All new publications, changes, ideas, conferences and parties will be posted regularly on our Facebook page. Yepp, that's true. We found it much easier to maintain and follow than most other social media platforms.

And if you have anything to talk about with us, use the new Zulip chat. We're looking forward meeting you! See you on the trail!

Nick Sutterer, @apotonick

We improved the trailblazer-rails gem!

Nick Sutterer — Sun, 29 Aug 2021 11:07:35 +0000

While working on an example Rails application for the new Trailblazer tutorial series I figured out that trailblazer-rails is quite inconvenient to use when you need to customize calls.

Controller-wide cells options

When using the Cells gem for rendering views along with a layout cell, you formerly had to pass the :layout option manually in every controller action.

class SongsController < ApplicationController
  def new
    # ...
    render cell(Song::Cell::New, ctx, layout: App::Cell::Layout)
  end

People started monkey-patching internal methods from trailblazer-rails to get the layout (and other options) automatically injected into every #cell call.

This is built-in now, just define #options_for_cell on the controller.

class SongsController < ApplicationController
  private def options_for_cell(model, options)
    {
      layout: App::Cell::Layout
    }
  end

The hash returned from this method is passed as a defaults hash to #cell. It's documented, too!

Runtime variables for operations

Another clumsiness I noticed was that it's impossible to inject variables into the operation context when using #run. You can simply provide those as keyword arguments now.

class SongsController < ApplicationController
  def create
    run Song::Operation::Create, 
      current_user: current_user, cookie: session do |ctx|
      # ...
    end

This will add :current_user and :cookie to the ctx passed into the Song::Operation::Create operation. Here are the docs.

Trailblazer-Rails docs moved!

In the process of the refactoring we also rewrote trailblazer-rails docs. The old docs are still available on the old website. Why would you need them, though, with new docs waiting here?

The new gem version is 2.2.0. 🔥