Forem: Jon Lauridsen

Perfect Elixir: Testing Shell Scripts with Bats

Jon Lauridsen — Sat, 05 Apr 2025 15:41:35 +0000

Automated testing is crucial for reliable software development, yet shell scripts are frequently left untested, despite managing critical tasks like deployments, database migrations, and CI pipelines. Neglecting tests here can lead to subtle bugs and fragile workflows. Today, let's make sure all our code is testable, treating shell scripts with the same discipline we apply to application code.

ℹ️ BTW test automation doesn't mean every line absolutely must be tested; rather, it's about ensuring we can test. There are cases where skipping tests can make sense — but we can never do so out of laziness.

Table of Contents

Shell Script Testing with Bats
Bats Helper Libraries
- Creating a Bats Download Manager
Mocking
- jasonkarns/bats-mock 🛑
- grayhemp/bats-mock ✅
Speeding Up Bats Tests
Integrating Bats Into the Workflow
Conclusion

Shell Script Testing with Bats

Bats, Bash Automated Testing System, is a testing framework for shell scripts, with a nice test-runner and various commands to test what a script does when run. Let's see how it works!

First we need Bats installed, and pkgx makes that easy (to refresh on pkgx please refer to the "Setting Up an Elixir Dev Environment" article):

$ git-nice-diff -U1 .
/.pkgx.yml
@@ -7 +7,2 @@ dependencies:
   postgresql.org: =17.0.0
+  github.com/bats-core/bats-core: =1.11.1

We can now write a test, e.g. here we assert bin/doctor --help prints out usage information:

$ cat test/bin/doctor.bats
@test "--help outputs usage" {  
  run bin/doctor --help  

  [ "$status" -eq 0 ]  
  echo "$output" | grep -Fq "Usage: bin/doctor [options]"  
}

And to run our test-suite:

$ bats -r test
bin/doctor.bats
 ✓ displays help message when --help is used

1 test, 0 failures

It's a great start, but we quickly hit a blocker: How do we test how our script behaves if a tool such as pkgx is missing? We can't mess with the real pkgx because it's core to our entire project, so the solution is dependency injection — inject mocks instead of real commands. This is similar to what we covered in the previous "Automating Tests in Elixir Projects" article, and the Bats community has created mock libraries that promises to make it simple to create mocks. So let's go explore them.

But wait, before we try those helper libraries, we first need to talk about how we download libraries.

Bats Helper Libraries

The Bats ecosystem includes excellent helper libraries, but incorporating these libraries into our project isn't straightforward, with common approaches relying on Git submodules or committing their code into our repository directly.

But Git submodules rely on cumbersome commands, and committing external source files is a hack that goes against how we deal with dependencies. Basically, both add complexity. Especially compared to how simple Elixir handles it with mix deps.get, where well-defined dependencies are simply pulled down.

So, we'll create a lightweight Bats Download Manager to automate the fetching of Bats libraries, so we can use helper libraries without adding new hacky ways of dealing with source code to our project.

ℹ️ BTW simple doesn’t mean easy, e.g. Git submodules are easy to start but add cognitive complexity with their commands and mental model. Rich Hickey’s “Simple Made Easy”) speaks into this. Optimizing for simplicity — even when it's initially harder — pays off long-term.

Creating a Bats Download Manager

A good software principle is to download external files as dependencies, so we're creating our own "Bats download manager" to handle this for us.

Our first goal will be to download the library bats-assert. We can easily download it via the GitHub CLI tool, and since the download will be a tarball archive we'll also need the tar command to unpack it. These are both available via pkgx:

$ git-nice-diff -U1 .
/.pkgx.yml
@@ -8 +8,3 @@ dependencies:
   github.com/bats-core/bats-core: =1.11.1
+  cli.github.com: =2.68.1
+  gnu.org/tar: =1.35.0

To make use of the tools we wrap them in a shell script that takes arguments for a repository and its release-tag:

#!/usr/bin/env bash  
set -euo pipefail  

BATS_DEPS="$(realpath --relative-to="$(pwd)" "$(dirname "$0")/../.bats_deps")"  

usage() {  
  echo "Bats manager to download helper libraries"  
  echo ""  
  echo "Usage:"  
  echo "  batsman <owner/repo> <tag>"  
}  

 "$1" == "--help"  && { usage; exit 0; }  
 "$#" -ne 2  && { usage; exit 1; }  
repo="$1"  
tag="$2"  
destination="$BATS_DEPS/$repo"  
sha=$(gh api "/repos/$repo/commits/$tag" --jq .sha)  

rm -rf "$destination" && mkdir -p "$destination" && cd "$destination"  
gh release download "$tag" -R "$repo" --archive "tar.gz" -O "release.tar.gz"  
tar -xzf "release.tar.gz" --strip-components=1  
rm "release.tar.gz"  

version_info="$repo $tag (GitHub release) ${sha:0:8}"  
echo "$version_info" > .bats_version.txt  
echo "$version_info -> $destination"

ℹ️ BTW this is just the basics of a download manager, it can be extended as far as is needed. E.g. in this later commit security is improved by checking for specific SHAs.

It's now straightforward to download bats-core/bats-assert (which itself depends on the library bats-core/bats-support, so we'll actually download both):

$ bin/batsman bats-core/bats-assert v2.1.0
bats-core/bats-assert v2.1.0 (GitHub release) 78fa631d -> .bats_deps/bats-core/bats-assert

$ bin/batsman bats-core/bats-support v0.3.0
bats-core/bats-support v0.3.0 (GitHub release) 24a72e14 -> .bats_deps/bats-core/bats-support

$ ls .bats_deps/bats-core/bats-assert/
LICENSE  load.bash  package.json  README.md  src  test

Now we can use the nice assert-functions that come with the bats-assert library:

$ git-nice-diff -U1 .
/test/bin/doctor.bats
L#1:
+setup() {
+  load "$(pwd)/.bats_deps/bats-core/bats-support/load"
+  load "$(pwd)/.bats_deps/bats-core/bats-assert/load"
+}
+
 @test "displays help message when --help is used" {
   run bin/doctor --help
-  [ "$status" -eq 0 ]
-  echo "$output" | grep -Fq "Usage: bin/doctor [options]"
+  assert_success
+  assert_line "Usage: bin/doctor [options]"
 }

$ bats -r test
bin/doctor.bats
 ✓ displays help message when --help is used
1 test, 0 failures

Great stuff. And to ensure our entire team downloads these dependencies, we add a simple check to bin/doctor that prompts everyone to run the proper bin/batsman calls:

$ git-nice-diff -U1 .
/bin/doctor
L#56:
   "mix ecto.create"
+check "Check Bats helpers" \
+  "ls .bats_deps/bats-core/bats-assert > /dev/null" \
+  "bin/batsman bats-core/bats-assert v2.1.0 && bin/batsman bats-core/bats-support v0.3.0"

We now have a simple way for the whole team to consistently fetch the needed Bats libraries, mirroring the familiar dependency management we know from mix deps.get.

And now we can return to the original problem: How can we test bin/doctor without actually having it run all the commands it depends on?

Mocking

We have Bats working, we can download helper libraries, now it's time to find an effective way to test shell scripts in isolation from the external commands they call. We need a reliable way to mock out those command calls, and for that we'll explore mocking libraries and see how they can help us achieve this.

jasonkarns/bats-mock 🛑

The bats-mock library by Jason Karns promises straightforward mocking of external commands. Let’s download and integrate it:

$ bin/batsman jasonkarns/bats-mock v1.2.5
jasonkarns/bats-mock v1.2.5 (GitHub release) 24f995f9 -> .bats_deps/jasonkarns/bats-mock

Then we load it, and use its stub command:

$ git-nice-diff -U1 .
/test/bin/doctor.bats
L#3:
   load "$(pwd)/.bats_deps/bats-core/bats-assert/load"
+  load "$(pwd)/.bats_deps/jasonkarns/bats-mock/stub"
 }
@@ -11 +11,12 @@ setup() {
 }
+
+@test "advise to reinitialize devenv if pkgx is absent" {
+  stub which "pkgx : exit 1"
+
+  run bin/doctor
+
+  assert_failure
+  assert_line "Suggested remedy: dev off; dev || source bin/bootstrap"
+
+  unstub which
+}

$ bats -r test
bin/doctor.bats
 ✓ displays help message when --help is used
 ✓ advise to reinitialize devenv if pkgx is absent

2 tests, 0 failures

This initial success is encouraging: we stub which to simulate pkgx is missing, and verify our script responds correctly. Simple and effective!

Unfortunately, the approach quickly breaks down in more complex cases. For example, this stub:

stub psql '-U postgres -c "\q" : echo ""'

should simulate psql producing no output — but the library fails with obscure errors:

$ bats -r test/bin/doctor.bats
bin/doctor.bats
 ✗ advise to create a user when psql fails to connect
   (from function `unstub' in file .bats_deps/jasonkarns/bats-mock/stub.bash, line 46,
    in test file test/bin/doctor.bats, line 40)
     `unstub psql' failed
1 tests, 1 failure

This appears to be a known bug. Rather than get caught up troubleshooting brittle quoting issues, let's acknowledge this limits and move on quickly to try other libraries.

grayhemp/bats-mock ✅

Sergey Konoplev has authored the grayhemp/bats-mock library, which supports creating mocked commands. It doesn't mock existing commands, rather it generates generic mocks and we're responsible for injecting them into our scripts as needed. Let's see how that works.

First, fetch the library:

$ bin/batsman grayhemp/bats-mock v1.0-beta.1
grayhemp/bats-mock v1.0-beta.1 (GitHub release) ac1a4475 -> .bats_deps/grayhemp/bats-mock

And then we change bin/doctor to allow the which command to be overwritten by specifying the environment variable _WHICH:

$ git-nice-diff -U1 .
/bin/doctor
L#34:
+WHICH=${_WHICH:-which}
 section "Running checks…"
 check "Check system dependencies" \
-  "command which pkgx && which erl && which elixir" \
+  "command $WHICH pkgx && $WHICH erl && $WHICH elixir" \
   "dev off; dev || source bin/bootstrap"
 check "Check developer environment" \
-  "which stat | grep -q '.pkgx'" \
+  "$WHICH stat | grep -q '.pkgx'" \
   "dev off; dev"

Finally we create a generic mock and plug it into the _WHICH variable (by exporting it, it automatically becomes part of the environment that run uses):

$ git-nice-diff -U1 .
/test/bin/doctor.bats
L#3:
   load "$(pwd)/.bats_deps/bats-core/bats-assert/load"
+  load "$(pwd)/.bats_deps/grayhemp/bats-mock/src/bats-mock"
 }
@@ -10 +11,14 @@ setup() {
 }
+
+@test "advise to reinitialize devenv if pkgx is absent" {
+  export _WHICH="$(mock_create)"
+  mock_set_side_effect "$_WHICH" "case $1 in pkgx) exit 1;; esac"
+
+  run bin/doctor
+
+  assert_failure
+  assert_line "Suggested remedy: dev off; dev || source bin/bootstrap"
+   "$(mock_get_call_args $_WHICH)" = pkgx 
+}

And all that works:

$ bats -r test
bin/doctor.bats
 ✓ displays help message when --help is used
 ✓ advise to reinitialize devenv if pkgx is absent

2 tests, 0 failures

That's pretty cool. Using the environment variable mechanism to inject commands is a bit more verbose, but it also makes the code nicely explicit. A worthwhile tradeoff.

But we should stress-test this library further, and if we create tests for all the code-branches of bin/doctor it can end up looking like this:

$ bats -r test/bin/doctor.bats
doctor.bats
 ✓ displays help message when --help is used
 ✓ reports healthy system (all stubs are by default configured for success)
 ✓ executes all stubs in their expected order
 ✓ verifies Elixir is available before running mix
 ✓ advise to reinitialize devenv if pkgx is absent
 ✓ advise to reinitialize devenv if erl is absent
 ✓ advise to reinitialize devenv if elixir is absent
 ✓ advise to toggle the devenv if stat is not provided by pkgx
 ✓ advise to start the database server if pgrep cant connect
 ✓ advise to create a user when psql fails to connect
 ✓ advise to install hex if hex is not installed locally
 ✓ advise to run mix setup if mix dependencies are unavailable
 ✓ advise to run mix setup if mix dependencies are outdated
 ✓ advise to run ecto if the database is missing
 ✓ advise to advise download Bats helpers if theyre missing

15 tests, 0 failures

ℹ️ BTW I subsequently refactored the tests to bring down complexity, so each test is kept focused and simple, and delegate into helper-functions. Without this refactor it was grievously complex trying to keep an overview of e.g. the repeated calls to mock_create and mock_set_side_effect. This article drags on if we also cover those details but if you're interested the full /test/bin/doctor.bats is available here.

Speeding Up Bats Tests

We've made great progress on testing shell scripts, with an easy and simple approach to downloading the Bats framework and helper libraries.

But tests have gotten slow to run.

Here I've added a bin/bats-test script to simplify running tests (see commit "Add initial bin/bats-test script" for details), and it shows our current performance:

$ bin/bats-test
Running all tests...
…
36 tests, 0 failures in 30 seconds

30 seconds!? That's slow. Let's try some straightforward optimizations to see what we can do about this.

Bats can run tests in parallel (to follow the code changes for this see commit "Run bin/bats-test tests in parallel"):

$ bin/bats-test
Running all tests (via 14 parallel jobs)...
…
36 tests, 0 failures in 19 seconds

19 seconds is nearly a 40% improvement, so that did help. But it's still slow…

Shell scripts tend to not change as often as application-code, could we skip tests until they actually change? We could persist a checksum when tests pass, and if a new run has the same checksum then skip the tests (to follow this change see commit "Only run bin/bats-test tests if checksums indicate a change"):

$ bin/bats-test
Running all tests (via 14 parallel jobs)...
…
38 tests, 0 failures in 22 seconds

$ bin/bats-test
No changes since last test run, skipping 38 tests.

0 seconds is certainly fast 😂, but of course doesn't alleviate the pain when tests do have to run. We could keep going to optimize even further, but despite our fingers itching to do more our time will be better spent moving on (for this article, at least), because all told we've successfully improved how we work with shell scripts:

Developers can rely on bin/bats-test to run shell tests as fast as possible (even if that speed leaves something to desired, and can be improved incrementally over time by those who wish to optimize).
For the cases where developers are not working on shell scripts the tests take no time, meaning friction is minimized for everyone that works elsewhere in the system.

Integrating Bats Into the Workflow

In the "Streamlining Development Workflows" article we introduced the core workflow of running bin/shipit to quickly push changes, which internally runs mix test to handle Elixir's test automation. We need to extend this mechanism to also run Bats tests.

Adding more test-commands directly to bin/shipit makes it crowded though; let’s create a dedicated bin/test script instead:

$ cat bin/test
#!/usr/bin/env bash
set -euo pipefail
source "$(dirname "$0")/.shhelpers"
step --with-output "Running Elixir tests" "mix test"
step --with-output "Running Shell tests" "$(dirname "$0")/bats-test"

$ bin/test
• Running Elixir tests:
Running ExUnit with seed: 248176, max_cases: 28
.......
Finished in 0.05 seconds (0.02s async, 0.03s sync)
7 tests, 0 failures
• Running Shell tests:
No changes since last test run, skipping 45 tests.

And we rewire bin/shipit to call our new script:

$ git-nice-diff -U0 .
/bin/shipit
@@ -6 +6 @@
-step --with-output "Running tests" "mix test"
+"$(dirname "$0")/test"

With this we have easily integrated shell testing into our workflows, and team members won’t need extra setup; they’ll just automatically benefit from the added tests.

Conclusion

Shell scripts deserve testing just as much as our application code, yet they're often overlooked. In this article, we've shown it's straightforward to integrate robust shell testing into our structured workflows using Bats and dependency injection.

Instead of accepting untested scripts as the status quo, we've chosen to uphold strong engineering principles and committed to simple, decoupled tools and techniques to accomplish that.

Now we can write our shell scripts with the same sense of safety ✅ as our Elixir applications, with isolated and reliable tests ✅ fully integrated into our ultra-fast bin/shipit workflow ✅.

Happy testing! 🚀

Perfect Elixir: Automating Tests in Elixir Projects

Jon Lauridsen — Wed, 05 Mar 2025 21:44:30 +0000

How do we write fast, reliable tests in Elixir? How do we keep modules decoupled and easy to test? Testing is a broad topic with many opinions, and today we’ll explore techniques that will let us have informed discussions about how we want to test. Because good tests aren't just about passing — they’re about helping us move fast, and deploy with confidence.

In this guide, we’ll explore practical strategies for injecting dependencies and using test doubles in ways that keep our code modular, our tests concurrent, and our team productive.

Table of Contents

Why Test?
Getting Started with ExUnit
Dependency Injection Techniques
- Mock (Global Overriding)
- Application Config Injection 🛑
- Scoped Injection with Mox
- Function Argument Injection
- Process Hierarchy Injection (via Process Tree)
Test Double Strategies
- Manual Fakes via Behaviour
- Mox (Simplified Use) 🛑
- Hammox (Type-Safe Mocks)
- Double (Ad-Hoc Mocking)
Conclusion

Why Test?

Automated testing is foundational for continuous delivery and high-performing teams. The question isn’t should we test — it’s how we test in ways that keep code simple, decoupled, and easy to evolve.

The DevOps Research & Assessment (DORA) research shows the same thing again and again: teams with fast feedback loops perform better. Testing well isn’t a nice-to-have — it’s a key capability.

ℹ️ BTW for more on DORA I've written about "Accelerate", the scientific analysis of software delivery.

A core challenge in test automation lies in managing dependencies between modules. Excessive coupling complicates testing and leads to brittle, intertwined systems. Decoupled modules allow for simpler and clearer systems, and often test doubles are used to isolate dependencies.

This topic can be contentious because some developers use test doubles sparingly (only at external interfaces like APIs or databases), while others inject them frequently, even between internal systems. Rather than taking sides, we'll focus purely on how we can inject test doubles, as understanding these techniques lets teams make informed decisions about testing strategies.

ℹ️ BTW if you’re new to terms like "test double", Martin Fowler's Test Doubles and Mocks Aren't Stubs are excellent starting points.

But enough talk, let's get practical about this.

Getting Started with ExUnit

ExUnit is Elixir's built-in testing framework. It's simple and fast:

defmodule Math do
  def multiply(a, b), do: a * b
end
defmodule MathTest do
  use ExUnit.Case
  test "multiplying two numbers returns their product" do
    assert Math.multiply(2, 3) == 6
  end
end

$ mix test
…
1 test, 0 failures

ExUnit is also well-documented and works great out of the box — so let’s skip the basics and talk about the real challenges: how we develop simple, decoupled code and tests.

Dependency Injection Techniques

Dependency injection helps reduce tightly coupled code. Consider:

defmodule Warehouse do  
  def empty?(%Warehouse{inventories: inventories}) do  
    inventories  
    |> Enum.flat_map(&Inventory.all_products/1)  
    |> Enum.empty?()  
  end  
end

(assume both Warehouse and Inventory are complex domains in their own rights, backed by database- and API-calls)

Here, Warehouse directly depends on Inventory, which creates a tight coupling that complicates testing.

To isolate Warehouse tests effectively we should look at how Elixir lets us inject test doubles — and where each approach shines (or stumbles).

Mock (Global Overriding)

Elixir allows for sophisticated runtime module swapping, and that's what the Mock library (built on Erlang’s :meck) enables.

With Mock, the Warehouse tests directly mock the Inventory module:

defmodule WarehouseTest do
  use ExUnit.Case, async: false
  import Mock

  test "warehouse is empty when its inventories have no products" do
    with_mock Inventory, all_products: fn _ -> [] end do
      warehouse = %Warehouse{inventories: [%Inventory{id: 1}]}
      assert Warehouse.empty?(warehouse) == true
    end
  end
end

Here Inventory.all_products/1 is mocked to always return an empty list, and so the warehouse is always empty regardless of how the Inventory domain actually queries and counts its products.

That Mock can just "reach in" and simply swap out a dependency at runtime is very powerful. But it also comes with drawbacks:

🛑 It's Global: Tests cannot run concurrently because the module replacement affects the global module namespace. That's bad for performance, and it's a detail that's easy to forget and then the symptoms will be hard-to-debug issues where tests will fail seemingly at random.
🛑 It's "Magic": Runtime code doesn't show that dependencies are being swapped, which can cause developer confusion. This is a debatable point that depends on the team, but often more explicit code is preferable.

Application Config Injection 🛑

This approach changes the runtime code to read which module to use from Elixir's configuration system (aka Application environment):

git-nice-diff -U1 HEAD~
/lib/warehouse.ex
L#5:
     inventories
-    |> Enum.map(&Inventory.all_products/1)
+    |> Enum.map(&inventory_impl().all_products/1)
     |> List.flatten()
L#9:
   end
+  defp inventory_impl(), do: Application.get_env(:myapp, :inventory, Inventory)
 end

And tests can now override the module to use via put_env/4:

defmodule InventoryStub do
  def all_products(_), do: []
end

defmodule WarehouseTest do
  use ExUnit.Case, async: false

  test "warehouse is empty when its inventories have no products" do
    Application.put_env(:myapp, :inventory, InventoryStub)
    warehouse = %Warehouse{inventories: [%Inventory{id: 1}]}
    assert Warehouse.empty?(warehouse) == true
  end
end

ℹ️ BTW you might have better ideas for creating a stub than this InventoryStub module, but for now we're only focused on how stubs are injected, we'll cover smarter ways to create stubs later.

This makes the override explicit — which is good — but it comes with a difficult tradeoff:****

🛑 It's (Still) Global: The configuration system is global, so tests still cannot run concurrently.
🛑 Maybe a Misuse? Is it appropriate to use the application environment for dependency switching? It's primarily designed for user-defined configuration of an application's features (e.g. database URLs, feature toggles), should it also expose internal dependencies that users are never intended to adjust?

ℹ️ _BTW the Elixir documentation on library configuration advises against using the application environment for anything other than truly global settings. While that guidance is for libraries and so not a perfect match for our context, I think it reinforces the idea that it's surprising to have configuration options that aren't meant to be user-configured.

Scoped Injection with Mox

The widely used Mox is Elixir's most widely used mocking library, and it recommends the Config-Based injection in its examples. And it has a smart feature that scopes mocks to the calling process which makes Mox tests concurrent safe. That's a big improvement!

First step is registering the mock (but note: Mox can only create mocks from behaviour-based modules):

/test/test_helper.exs
@@ -2 +2,4 @@ ExUnit.start()
 Ecto.Adapters.SQL.Sandbox.mode(MyApp.Repo, :manual)
+
+Mox.defmock(InventoryStub, for: Inventory)
+Application.put_env(:myapp, :inventory, InventoryStub)

That makes InventoryStub globally available, and Warehouse tests can now configure it:

defmodule WarehouseTest do
  use ExUnit.Case, async: true
  import Mox

  test "warehouse is empty when its inventories have no products" do
    InventoryStub |> stub(:all_products, fn _ -> [] end)
    warehouse = %Warehouse{inventories: [%Inventory{id: 1}]}
    assert Warehouse.empty?(warehouse) == true
  end
end

That works, and these tests run asynchronously so performance is optimal.

There's still a line being blurred by utilizing user config for internal dependency switching, but as long as a team can commit to doing things "the Mox way" it'll work fine.

🛑 Mocks are globally registered: Because of the configuration system we are forced to come up with names for our inventory keys. In our example the key myapp.inventory is a small accidental complexity because it only exists to facilitate the internal injection.

So far, we’ve explored a few solid techniques. But Elixir gives us even more options — and there's still room for a more elegant solution.

ℹ️ BTW Mox's is not too opinionated on how one should inject its mocks, I think its documentation suggests Config-Based because it's easy to explain and requires no additional tools. The blog that gave rise to Mox includes a hint that one does not have to use the config system: José Valim's "Mocks and explicit contracts". It's a good read whichever way you prefer injecting dependencies.

Function Argument Injection

Injecting by argument is a classic technique:

git-nice-diff -U1 HEAD~
/lib/warehouse.ex
L#3:
-  def empty?(%Warehouse{inventories: inventories}) do
+  def empty?(%Warehouse{inventories: inventories}, inventory_impl) do
     inventories
-    |> Enum.map(&inventory_impl().all_products/1)
+    |> Enum.map(&inventory_impl.all_products/1)
     |> List.flatten()

This way tests can easily pass a test double:

defmodule InventoryStub do
  def all_products(_), do: []
end

defmodule WarehouseTest do
  use ExUnit.Case, async: true

  test "warehouse is empty when its inventories have no products" do
    warehouse = %Warehouse{inventories: [%Inventory{id: 1}]}
    assert Warehouse.empty?(warehouse, InventoryStub) == true
  end
end

Argument injection can be especially useful for modules that are "orchestration-focused", meaning their purpose is to orchestrate dependencies. But be weary of issues such as:

🛑 Argument pollution: It's very problematic if functions that are already doing a thing ends up polluted by also having to specify dependencies. Callers shouldn't have to wire up internal systems unless that's the point of that module.
🛑 Challenging to Scale: It can become cumbersome to keep multiple functions aligned in how they deal with dependencies.

Process Hierarchy Injection (via Process Tree)

Elixir’s process model gives a unique superpower: hierarchical test isolation. Using the ProcessTree library, we can inject dependencies scoped to a test’s process and all its children — without globals or polluting arguments.

ℹ️ BTW for more details about ProcessTree see JB Steadman's blog async: false is the worst. Here's how to never need it again. Great blog, and my thanks to JB for making this library available 🙏.

First, to make it simple and easy to use ProcessTree we'll quickly wrap it in our own module:

defmodule InjectorTree do
  @doc """
  Injects the specified stub such that it gets registered as an override
  in the process hierarchy.
  """
  def inject(module, stub) do
    injector_map = Process.get(:injector_tree, %{})
    Process.put(:injector_tree, Map.put(injector_map, module, stub))
    :ok
  end

  @doc """
  Provides (aka returns) the specified module such that if that module
  has been injected then that override is what gets returned.

  If the module has not been injected the specified module is returned.
  """
  def provide(module) do
    ProcessTree.get(:injector_tree, default: %{})
    |> Map.get(module, module)
  end
end

Then we make the Warehouse runtime code "provide" an Inventory:

git-nice-diff -U1 HEAD~
/lib/warehouse.ex
L#1:
 defmodule Warehouse do
+  import InjectorTree, only: [provide: 1]
+
   defstruct [:inventories]

-  def empty?(%Warehouse{inventories: inventories}, inventory_impl) do
+  def empty?(%Warehouse{inventories: inventories}) do
     inventories
-    |> Enum.map(&inventory_impl.all_products/1)
+    |> Enum.map(&provide(Inventory).all_products/1)
     |> List.flatten()

And tests can now "inject" a stub:

defmodule WarehouseTest do
  use ExUnit.Case, async: true
  import InjectorTree, only: [inject: 2]

  test "warehouse is empty when its inventories have no products" do
    inject(Inventory, InventoryStub)
    warehouse = %Warehouse{inventories: [%Inventory{id: 1}]}
    assert Warehouse.empty?(warehouse) == true
  end
end

✅ This technique offers clear, explicit, and isolated dependency injection, granting us concurrent-safe tests while keeping runtime code clean and explicit. Very cool!

ℹ️ BTW a more sophisticated implementation of InjectorTree could probably compile itself out in production to avoid any theoretical overhead caused by traversing the process hierarchy. But such optimizations are beyond the scope of this article.

Test Double Strategies

Now that we’ve seen how to inject dependencies — let’s talk about how to build test doubles that are simple, safe, and aligned with our code.

The Elixir community offers several robust methods for defining test doubles, we'll examine them in more detail and consider their strengths and potential trade-offs.

Manual Fakes via Behaviour

A straightforward yet powerful approach involves explicitly creating fake implementations using Elixir behaviours. This keeps the fake and the real implementation synchronized, enforced by compile-time checks:

defmodule FakeInventory do
  @behaviour Inventory
  @impl Inventory
  def all_products(%Inventory{} = _), do: []
end

Tests then inject the fake:

defmodule WarehouseTest do
  use ExUnit.Case, async: true
  import InjectorTree, only: [inject: 2]

  test "warehouse is empty when its inventories have no products" do
    inject(RealInventory, FakeInventory)
    warehouse = %Warehouse{inventories: [%Inventory{id: 1}]}
    assert Warehouse.empty?(warehouse) == true
  end
end

This method is powerful because it is explicit and straightforward, and require no external dependencies.

However, the approach can be effort-intensive, as maintaining parallel implementations (real and fake) demands careful designing, and the fake can require its own tests if complexity grows significantly.

Despite the maintenance cost, this is a highly effective and explicit approach for test doubles.

Mox (Simplified Use) 🛑

Yes we're covering Mox again, but this we'll simplify its use by only using it to create a mock (we'll not rely on Mox's "per-process mock scoping" feature):

defmodule WarehouseTest do
  use ExUnit.Case, async: true
  import InjectorTree, only: [inject: 2]
  import Mox

  test "warehouse is empty when its inventories have no products" do
    inject(
      RealInventory,
      defmock(InventoryMock, for: Inventory)
      |> stub(:all_products, fn _ -> [] end)
    )

    warehouse = %Warehouse{inventories: [%Inventory{id: 1}]}
    assert Warehouse.empty?(warehouse) == true
  end
end

The change may appear subtle, but this code is simpler: Mox is used to create a test double, but nothing is hidden in how that test double gets injected. It's all nicely explicit code, where all the steps of creating, injecting, and configuring the mock is expressed right in the test file.

And Mox has validation to ensure mocks align with the defined behaviour, preventing incorrect mocking:

$ git-nice-diff -U0 .
/test/warehouse_test.exs
@@ -10 +10 @@ defmodule WarehouseTest do
-      |> stub(:all_products, fn _ -> [] end)
+      |> stub(:FOO, fn -> "bar" end)

$ mix test
  1) test warehouse is empty when its inventories have no products (WarehouseTest)
     ** (ArgumentError) unknown function FOO/0 for mock InventoryMock

Great! And the Mox documentation is exceptionally good and well worth a read.

Mox, however, does not care about typespec enforcement, but the library Hammox promises to fix that so lets give that a try next.

ℹ️ _BTW this section marks Mox with a red "do not use" sign only because Hammox turns out to be a more powerful drop-in replacement. It's not a sign of criticism, Mox is awesome!

Hammox (Type-Safe Mocks)

Hammox extends Mox by adding typespec validation. It's a drop-in replacement, bringing greater safety and validation:

$ git-nice-diff -U1 .
/test/warehouse_test.exs
L#3:
   import InjectorTree, only: [inject: 2]
-  import Mox
+  import Hammox

$ mix test
…
7 tests, 0 failures

Hammox ensures return values conform to typespecs, e.g. here the mock returns an invalid number-list:

$ git-nice-diff -U0 .
/test/warehouse_test.exs
@@ -10 +10 @@ defmodule WarehouseTest do
-      |> stub(:all_products, fn _ -> [] end)
+      |> stub(:all_products, fn _ -> [1] end)

$ mix test
  1) test warehouse is empty when its inventories have no products (WarehouseTest)
     test/warehouse_test.exs:6
     ** (Hammox.TypeMatchError) 
     Returned value [1] does not match type [String.t()].
7 tests, 1 failure

That's very cool! This added safety makes Hammox particularly recommended for behaviour-based mocking.

Double (Ad-Hoc Mocking)

Double addresses situations where behaviours feel like unnecessary overhead. It specifically allows ad-hoc mocking without behaviours, offering quick and flexible mocks.

$ git-nice-diff -U1 .
/test/test_helper.exs
L#1:
 ExUnit.start()
+Application.ensure_all_started(:double)
 Ecto.Adapters.SQL.Sandbox.mode(MyApp.Repo, :manual)

ℹ️ BTW a lesser known side-effect of behaviours is that implementations do not auto-complete in IEx. That's by design. It's a small extra friction that Double avoids by allowing ad-hoc mocking.

The test can now stub Inventory, even when it's just a plain old Elixir module (no behaviours, etc.):

defmodule WarehouseTest do
  use ExUnit.Case, async: true
  import InjectorTree, only: [inject: 2]
  import Double

  test "warehouse is empty when its inventories have no products" do
    inject(Inventory, Inventory |> stub(:all_products, fn _ -> [] end))

    warehouse = %Warehouse{inventories: [%Inventory{id: 1}]}
    assert Warehouse.empty?(warehouse) == true
  end
end

That's very elegant. And Double also provides correctness-validation, e.g. we can't stub a function that doesn't exist on the target module:

$ git-nice-diff -U0 .
/test/warehouse_test.exs
@@ -7 +7 @@ defmodule WarehouseTest do
-    inject(Inventory, Inventory |> stub(:all_products, fn _ -> [] end))
+    inject(Inventory, Inventory |> stub(:FOO, fn -> "BAR" end))

$ mix test
  1) test warehouse is empty when its inventories have no products (WarehouseTest)
     test/warehouse_test.exs:6
     ** (VerifyingDoubleError) The function 'FOO/0' is not defined in :InventoryDouble326
7 tests, 1 failure

However, Double doesn't verify stubbed return values against typespecs. If pragmatic flexibility and simplicity matter more to your team than strict typespec enforcement, Double is an excellent choice.

Conclusion

We’ve covered a lot — but the takeaway is simple:

✅ Fast tests
✅ Clear dependencies
✅ Concurrent, isolated setups

Use Manual Fakes for clarity, Hammox for strictness, or Double for pragmatic speed. And consider Process Hierarchy Injection for clean, concurrent-safe test wiring.

Ultimately there’s no single best way to test — the focus should be on what lets your team move fast and ship with confidence. Whatever combo you choose — stay explicit, keep things simple, and test fearlessly 💪.

ℹ️ BTW I’ve seen complex “Inversion of Control” frameworks that turned code into spaghetti. Be cautious of cleverness — simplicity scales best.

Perfect Elixir: Simplifying Elixir Developer Onboarding

Jon Lauridsen — Mon, 28 Oct 2024 13:00:00 +0000

Onboarding can be a costly but hidden drag on a team — not just in raw setup time, but also in how easily it becomes a process full of uncertainty and fragility. How developers get started reflects the standards of care a team chooses to uphold, and we should explore how simple and safe we can make this experience.

This article will explore how to streamline onboarding through a simple, semi-automated script — striking a balance between ease of use and maintainability.

ℹ️ BTW I've started on projects where onboarding could take two weeks 😱. Senior devs spent hours debugging others' setups — a time sink repeated with every new hire. We can do better!

Table of Contents

Redefining Onboarding
Beware Global System Changes
Implementing the Onboarding Script
- Building Trust
- Step 1: Install pkgx
- Step 2: Activate Shell Integration
- Step 3: Clone the Repository
- Step 4: Activate the Development Environment
- Onboarding complete!
Conclusion

Redefining Onboarding

When we say onboarding, we mean everything a new developer needs to do to start working on our project.

The traditional approach relies on written guides. But guides come with downsides:

No testability: They can't be validated automatically, easily drifting out of date.
Complexity grows: Old steps tend to stick around "just in case", with new ones piling on.
Hard to maintain: Updates happen under pressure, usually when someone's blocked.

What if we reimagined onboarding as a single command — no dependencies, no guesswork — that guides the developer interactively?

On macOS and Linux systems, the simplest way to run a remote script is:

$ curl -Ssf https://…/script | sh

That's beautifully simple, but it comes with a limit: To onboard we'll want to inspect the user's environment to understand what tools they have configured, and that's not possible when piping to sh (since it spawns a new shell that is not necessarily the same as their working shell).

Instead, using source allows the script to access the user's environment:

$ curl -fsSL https://…/script > /tmp/script && source /tmp/script

That's still incredibly simple, and with this approach, we can move toward a fully guided, low-friction setup. Next, let's look at one of the trickier parts: installing system tools.

Beware Global System Changes

For our project, the developer must have installed pkgx, as introduced in the Setting Up an Elixir Dev Environment article. But how do we do that?

First instinct could be to automatically install it, but installing system tools is a risky proposition — here's why we must not do it automatically:

It's invasive: Developers customize their environments. Global changes can break their system or other projects.
It's brittle: There are a lot of environment variance out there, very difficult to write code that will work reliably across all of them.
It's high-maintenance: Continuously testing all the various supported permutations spirals out of control fast

Luckily, we only need pkgx, because it takes care of the rest of the tooling. And it's easy to install manually — no sudo needed. So we should ask the developer to install it, and keep our script focused on guidance, not installation. That will also make our onboarding script much safer and much easier to test — win-win.

Implementing the Onboarding Script

Building Trust

Our first priority: earn the developer's trust.

Instead of doing anything upfront, our script can clearly explain what it will do — and what it won't:

$ URL="https://raw.githubusercontent.com/gaggle/perfect-elixir/refs/heads/perfect-elixir-5-onboarding/bin/bootstrap"

$ curl -fsSL $URL > /tmp/boot && source /tmp/boot
🚀 Welcome to the Perfect Elixir Onboarding Script! 🚀

This script will help you set up your development environment for our project. 

🔒 Trust and Transparency: 
  - This script NEVER makes changes to your system
  - It ONLY inspects your environment and makes suggestions

Ready to proceed? [y/N]:

🖥️ Terminal

Clear. Transparent. Opt-in.

ℹ️ BTW I'm skipping the underlying code here (it just prints text), but the full script is available on GitHub.

Step 1: Install pkgx

We check if pkgx is installed:

Ok to proceed? [y/N]: y
• Checking for pkgx… x

📦 pkgx is not installed

User action required: Install pkgx
──────────────────────────────────
pkgx is our package manager for handling system dependencies. Learn more at https://pkgx.sh.

Here are quick ways to install it:
1. Via Homebrew:
   $ brew install pkgxdev/made/pkgx
2. Via cURL:
   $ curl -Ssf https://pkgx.sh | sh
Other installation methods at https://docs.pkgx.sh/run-anywhere/terminals.

ℹ️ pkgx installation is simple, and installing via Homebrew does not require sudo.

After pkgx has been installed please source this script again.

So now the power is in the user's hands, and they can choose to install it according to their preferences. After installation, the user re-sources the script and continues.

Step 2: Activate Shell Integration

Next, we check if pkgx's shell integration is active:

Ok to proceed? [y/N]: y
• Checking for pkgx… ✓
• Checking pkgx shell integration… x

🔧 pkgx shell integration is not active

User action required: Activate pkgx shell integration
─────────────────────────────────────────────────────
To activate pkgx shell integration, run the following command:

  $ pkgx dev integrate

Integration writes one line to your .shellrc.

ℹ️ For more information about shell integration see: https://docs.pkgx.sh/using-pkgx/shell-integration.

After shell integration has been activated please source this script again.

Shell integration unlocks pkgx's dev command — our tool for managing project-specific dependencies.

Step 3: Clone the Repository

Now we're ready to check for our repository:

Ok to proceed? [y/n]: y
• Checking for pkgx… ✓
• Checking pkgx shell integration… ✓
• Checking repository is cloned… x

🌿 Repository not found

User action required: Clone the repository
──────────────────────────────────────────
Please clone the repository using your preferred method:

1. Via GitHub CLI:
   $ pkgx gh repo clone gaggle/perfect-elixir
2. Via SSH:
   $ git clone git@github.com:gaggle/perfect-elixir.git
3. Via HTTPS:
   $ git clone https://github.com/gaggle/perfect-elixir.git

After cloning, navigate to the project directory and source this script again.

ℹ️ BTW the first suggestion leverages pkgx's ability to run arbitrary tools, in this case the command would ephemerally run GitHub's CLI. Quite simple!

Step 4: Activate the Development Environment

Finally, we ensure the pkgx development environment is active:

Ok to proceed? [y/n]: y
• Checking for pkgx… ✓
• Checking pkgx shell integration… ✓
• Checking repository is cloned… ✓
• Checking development environment is active… x

🔧 Development environment is not active

User action required: Activate the development environment
──────────────────────────────────────────────────────────
To activate the development environment, run:

  $ dev

This will make project-specific versions of dependencies available within the repository.

After activating, please source this script again.

Onboarding complete!

When all checks pass, the script confirms completion and guides the user to start using our development workflows:

Ok to proceed? [y/n]: y
• Checking for pkgx… ✓
• Checking pkgx shell integration… ✓
• Checking repository is cloned… ✓
• Checking development environment is active… ✓

✅ Onboarding complete!

- To start working on the project, run:

    $ bin/doctor

🖥️ Terminal
Here is the full flow from a factory-reset machine to being ready to work on the project:
Animated gif showing "Running bootstrap script on a fresh machine, pasting in each of the suggested commands until onboarding is complete"
(dev.to fails to inline this .gif, so it's a hyperlink instead)

This clear, step-by-step approach ensures the developer can get fully set up in a few minutes.

ℹ️ BTW when extending onboarding to cover more steps, it's useful to note that past the first pkgx step the entire pkgx ecosystem is available. That means subsequent steps can easily leverage powerful tools such as GitHub CLI, or specific programming languages, and lots more. It's very powerful.

Conclusion

We set out to rethink onboarding, and landed on something radically simple: A single command that guides developers through a safe, low-friction setup. The entire onboarding instruction is:

Run, and follow the steps: $ curl -fsSL https://…/script > /tmp/script && source /tmp/script

It's minimal, extensible, and respectful of the developer’s environment. By avoiding invasive changes, the script stays safe and is easy to test — qualities that matter for teams who care about maintainability.

In our case setup is especially simple because the repository is public, but it works just as well to host the script behind a VPN, on a secure static server — the principles still hold.

ℹ️ BTW when implementing script-based onboarding, keep these security practices in mind:

Host on trusted infrastructure

Consider checksums for verification

Inspect scripts before running them

The real strength of this approach is its flexibility. Once pkgx is available, the script can evolve to include anything — repo access checks, CLI authentication, project prerequisites, even ticketing workflows.

This small investment pays off quickly. A smooth first experience sends a clear message: this is a team that values quality — from first command to first commit.

Perfect Elixir: Streamlining Development Workflows

Jon Lauridsen — Mon, 17 Jun 2024 12:25:00 +0000

Today we'll define and implement our daily development workflows. We'll begin by identifying what makes a workflow truly effective — drawing from cutting-edge research on software delivery practices. Then we'll establish really simple mechanisms for managing code changes so we can move quickly and accurately as a team. And importantly, these workflows must be flexible enough to scale as our team and product grows. Let's dive in!

Table of Contents

A Reflection on Workflows
- Don't Use Branches
- In Defense of Shell Scripting
Doctor
Update
Shipit
Continuous Code Reviewing
Conclusion

A Reflection on Workflows

Let's begin with a simple question: What exactly is a "workflow"?

Most teams have some loose process — clone the repo, open a pull request, get it reviewed. That's a workflow, technically. But we can do better by going back to first principles.

Are some ways of working objectively better than others? According to the DevOps Research and Assessment (DORA) project — absolutely.

DORA has spent nearly a decade studying thousands of engineering teams, using rigorous, peer-reviewed methods. Their research doesn't just find patterns — it finds causal relationships. That means: improving your DORA capabilities is statistically likely to make your team more effective.

Two of the most powerful predictors of performance are:

Minimal lead time — code should go from commit to production in under an hour.
High deployment frequency — ideally, each commit is a deploy.

These aren't abstract ideals — they're backed by real-world data. And they tell us something important:

To improve how we build software, we must build **workflows that support continuous code flow* — fast, safe, and always moving forward.*

Let's find out what that can actually look like.

ℹ️ BTW want to dive deeper into the research? Check out my intro to Accelerate and DORA metrics, or explore the full Software Delivery Performance Model. And if you haven't yet, I can't recommend their Accelerate book enough — it's essential reading.

Don't Use Branches

The DORA metrics forces us to a key realization:

Branches introduce delays in continuously integrating and delivering code.

Here's why:

Branches add time between writing code and running it in production. The fastest path are small changes pushed continuously to main.
Branches often accumulate multiple commits, reducing deployment frequency. The highest-performing teams push small changes straight to main.

To some, that may sound unsafe. But what we're describing is a well-established practice: Trunk-Based Development.

ℹ️ BTW if that sounds unfamiliar or intimidating, I've written a Beginner's Intro to Trunk-Based Development. Also, the DORA research explores this topic in detail.

Still, we need to answer some important questions:

If we don't use branches, we're also not using pull requests. So how do we protect code quality? Where do tests run? What about linting, security checks, and other automations?

The answer: Shift the workflows left.

Move checks to the developer's machine. Run tests, linters, and verifications before pushing to main.

In other words, we need fast, local workflows that feel lightweight — but offer strong safety guarantees. And they need to be simple enough to evolve with the team.

So how do we do that? How do we make sure code is safe to ship without branches or PRs?

That's exactly what this article will explore. Let's get into it.

In Defense of Shell Scripting

Now that we've established the need for fast, reliable workflows, we need need to dig into how we can safely and continuously pull and push code. Thanks to pkgx we can use any language to write our workflows in — so what should we choose?

I suggest we start with humble Shell Scripting. Here's why:

Industry Standard – Shell is the default choice for scripting across virtually all engineering teams.
Least Surprising – Most developers already know it, or can at least read it.
Practical & Low-Maintenance – It's not the prettiest, but it gets the job done with minimal overhead.
Portable & Consistent – With pkgx, we ensure everyone runs the same version of bash, so there won't be "works on my machine"-issues.

Keep in mind we're not here to build beautiful workflow code — we're here to ship products. Scripts can still be important, but they're not where we should do exciting new things that create exciting new problems. So it makes sense to choose the simplest, most boring tool for the job: shell scripts.

This mindset reflects a core principle for any product team: minimize complexity, stay pragmatic, and focus on what matters — shipping value, fast and safely.

Doctor

Let's kick off our workflows with a script to keep development environments healthy across the team by checking vital preconditions like whether the database is running and dependencies are installed.

ℹ️ BTW I like calling this script doctor because it verifies the health of our environment — but you can pick any name that fits your team.

Back in the "Setting Up an Elixir Dev Environment" article, we picked pkgx to manage our dev environment. So first, let's verify it's installed and active:

$ cat bin/doctor
#!/usr/bin/env bash
set -euo pipefail

source "$(dirname "${BASH_SOURCE[0]}")/.shhelpers"

check "pkgx installed?" \
  "which pkgx" \
  "brew install pkgxdev/made/pkgx"

check "Developer environment active?" \
  "which erl && which elixir" \
  "dev"

ℹ️ BTW this sources .shhelpers for utility functions such as check. The full .shhelpers file is available here.

Running doctor now gives us a clean bill of health:

$ bin/doctor
• pkgx installed? ✓
• Developer environment active? ✓

🖥️ Terminal

Next, let's build toward starting our Phoenix app (chosen back in "Building a Basic Elixir Web App"). First up: Make sure the local database is running:

$ git-nice-diff -U1 .
/bin/doctor
@@ -12 +12,5 @@ check "Developer environment active?" \
   "dev"
+
+check "PostgreSQL server running?" \
+  "pgrep -f bin/postgres" \
+  "bin/db start"

ℹ️ BTW this references bin/db, a small script to easily start and stop the database. You can find it here.

If the database isn't running doctor now fails and suggests a fix:

$ bin/doctor
• pkgx installed? ✓
• Developer environment active? ✓
• PostgreSQL server running? x
> Executed: pgrep -f bin/postgres

Suggested remedy: bin/db start
(Copied to clipboard)

And running the suggestion works:

$ bin/db start
• Creating /Users/cloud/perfect-elixir/priv/db ✓
• Initializing database ✓
• Database started:
waiting for server to start.... done
server started
↳ Database started ✓

$ bin/doctor
• pkgx installed? ✓
• Developer environment active? ✓
• PostgreSQL server running? ✓

🖥️ Terminal

This is the doctor pattern: check for issues, suggest a fix. It's easy to understand, easy to extend.

Let's jump ahead to a more complete version — with checks covering everything needed to start the app:

$ bin/doctor

Running doctor checks…
• pkgx installed? ✓
• Developer environment active? ✓
• PostgreSQL server running? ✓
• PostgreSQL server has required user? ✓
• Hex package manager installed? ✓
• Mix dependencies installed & compiled? ✓
• PostgreSQL database exists? ✓

✓ All checks passed, system is healthy

🖥️ Terminal

ℹ️ BTW the full doctor script is here.

And since everything is passing, we can launch our app:

$ iex -S mix phx.server
[info] Running MyAppWeb.Endpoint with Bandit 1.4.2 at 127.0.0.1:4000 (http)
…
Done in 260ms.
iex(1)>

That's it — bin/doctor now ensures all the critical preconditions are met before development begins. It's simple, extendable, and safe.

But… how do we make sure developers remember to run it? Let's cover that next.

Update

Now let's create a script to get the latest code — a replacement for git pull that also ensures any new code is properly applied.

We'll start simple: check we're on main and pull latest code:

$ cat bin/update
#!/usr/bin/env bash
set -euo pipefail
source "$(dirname "$0")/.shhelpers"
check "Branch is main?" \
  "[ \"$(git rev-parse --abbrev-ref HEAD)\" = \"main\" ]" \
  "git checkout 'main'"
step "Pulling latest code" "git pull origin 'main' --rebase"

$ bin/update
• Branch is main? ✓
• Pulling latest code ✓

🖥️ Terminal

After pulling, we need to handle any follow-up steps — like updating dependencies if mix.exs changed and ensuring the local development environment remains valid. So let's extend the script:

$ git-nice-diff -U1 .
/bin/update
@@ -8 +8,4 @@ check "Branch is main?" \
 step "Pulling latest code" "git pull origin 'main' --rebase"
+step "Getting dependencies" "mix deps.get"
+step "Compiling dependencies" "mix deps.compile"
+"$(dirname "$0")/doctor"

Now, running bin/update does more than pull code — it sets up our environment and ensures it's fully healthy, so we're ready to keep working:

$ bin/update
• Branch is main? ✓
• Pulling latest code ✓
• Getting dependencies ✓
• Compiling dependencies ✓

Running doctor checks…
• pkgx installed? ✓
• Developer environment active? ✓
• PostgreSQL server running? ✓
• PostgreSQL server has required user? ✓
• Hex package manager installed? ✓
• Mix dependencies installed & compiled? ✓
• PostgreSQL database exists? ✓

✓ All checks passed, system is healthy

🖥️ Terminal

This is where our scripts begin to interlock. bin/update becomes our go-to command to get the latest code and ensure our environment is sync, fully replacing git pull — a small habit shift that quickly becomes second nature.

ℹ️ BTW bin/update is also where we should apply database migrations, but since we don't have those yet that step doesn't exist for now.

Shipit

Our final workflow script, shipit, is the cornerstone of our Continuous Integration and Delivery (CI/CD) process. It replaces git push by only pushing code after verifying it's in a shippable state by running tests and quality checks.

Let's take a look:

$ cat bin/shipit
#!/usr/bin/env bash  
set -euo pipefail  
source "$(dirname "$0")/.shhelpers"  
"$(dirname "$0")/update"  
step --with-output "Running tests" "mix test"  
check "Files formatted?" "mix format --check-formatted" "mix format"  
step "Pushing changes to main" "git push origin \"main\""  
cecho "\n" -bB --green "✓ Shipped! 🚢💨"

shipit calls update first, to ensure we're testing against the latest version of main. That's how we continuously integrate our changes with the rest of the team's work.

When we run shipit, here's what it looks like in action:

$ bin/shipit
• Branch is main? ✓
• Pulling latest code ✓
• Getting dependencies ✓
• Compiling dependencies ✓

Running doctor checks…
• pkgx installed? ✓
• Developer environment active? ✓
• PostgreSQL server running? ✓
• PostgreSQL server has required user? ✓
• Hex package manager installed? ✓
• Mix dependencies installed & compiled? ✓
• PostgreSQL database exists? ✓

✓ All checks passed, system is healthy
• Running tests:
.....
Finished in 0.07 seconds (0.03s async, 0.04s sync)
5 tests, 0 failures

Randomized with seed 579539
↳ Running tests ✓
• Files formatted? ✓
• Pushing changes to main ✓

✓ Shipped! 🚢💨

🖥️ Terminal

This gives us a simple but powerful daily rhythm: run bin/update when starting the day, and bin/shipit whenever a commit is ready. A lightweight but robust CI/CD flow that minimizes delays and maximizes confidence.

ℹ️ BTW like the other scripts, this shipit is intentionally basic, because this article is about paving a direction for our workflows. An initial simplicity is a good thing, as it makes it easier to build initial trust and encourage team-wide adoption and iteration.

As the project matures, shipit can evolve to include additional quality gates — like linters, security checks, and even performance testing. But the most important thing for now is: build the habit of shipping frequently. It's how we learn fast and deliver real value.

Continuous Code Reviewing

We've established simple but powerful local workflows: bin/update to continuously integrate, and bin/shipit to continuously push — effectively replacing git pull and git push.

But by eliminating branches, we've also removed pull requests. So, what about code reviews? Our scripts automate local flow, but what replaces the second set of eyes?

The answer: Code reviewing must also be continuous.

That may sound radical, but it's backed by research: Asynchronous reviews add delay — often hours or days — as code sits waiting for attention. In a fast-moving team, that latency is a dealbreaker. Instead, we need reviewing to happen immediately.

This shift requires a cultural change:

When a commit is ready, review it right away.
Don't move on — wait for it to ship.
Remember: Code only delivers value in production.

How?

Call a colleague over and review it together.
Or better yet, write the code together from the start.

The goal is a fast, safe, collaborative flow to production — and frequent, small commits are key. When your team ships dozens of changes per hour, that's true continuous integration and delivery 🤩

ℹ️ BTW there are ways of working like this that are as old as programming itself. Pair programming and team programming (or "mobbing") are examples that naturally support continuous reviewing. While poor pairing can be draining, great pairing is joyful and productive 😊.

Further reading:

Pair Programming by Martin Fowler

Dave Farley on Continuous Delivery

Woody Zuill: Mob Programming

GeePaw Hill: Many More Much Smaller Steps

Conclusion

We began by outlining the principles behind fast, low-friction workflows, and landed on something deceptively simple: a few composable shell scripts that help us pull and push changes quickly, safely, and together.

By stripping away latency — like branches and pull requests — we've created a workflow optimized for high-trust teams that value speed, clarity, and continuous improvement. The simplicity of the scripts is not a limitation; it's an invitation to the team to evolve them as the team grows.

And yes — it's a little ironic that in an article series about Perfect Elixir, we've barely touched Elixir. But that's the point: perfect Elixir isn't just about Elixir. It's about designing the environment that lets great Elixir happen. The workflows we've explored today are broadly applicable, grounded in research, and shaped for the realities of modern, high-velocity product teams.

Perfect Elixir: Building a Basic Elixir Web App

Jon Lauridsen — Sun, 14 Apr 2024 22:00:00 +0000

Today we'll change gears a bit: It's time to make a web app. We'll need a project to work on for later articles in order to explore how to work with Elixir code, and so for this article we'll dive into some of the different ways to build a web app in Elixir. Our goal will be a simple example of a web app, and ideally it should make a data database call to be representative of real-world projects. Let's see what solutions we can find.

Table of Contents

Nothing At All
Cowboy
Plugged Cowboy
Bandit
- Connecting To Database
Phoenix Framework
- What Phoenix Gives Us
Conclusion

Nothing At All

What if we don't choose a solution? It's technically possible to serve HTML directly using the Erlang TCP/IP socket module, and although this isn't a viable solution it might be illuminating to see what it takes to work without abstractions.

The basic idea is to first listen on a TCP socket and wait for traffic, then send data when a request arrives. Here's a script that does that, full of comments to explain the details:

defmodule SimpleServer do
  def start(port) do
    # Listen on a TCP socket on the specified port
    #   :binary       - Treat data as raw binary, instead of
    #                   being automatically converted into
    #                   Elixir strings (which are UTF-8 encoded).
    #                   It'd be unnecessary to convert, as the
    #                   HTTP protocol uses raw bytes.
    #   packet: :line - Frame messages using newline delimiters,
    #                   which is the expected shape of HTTP-data
    #   active: false - Require manual fetching of messages. In
    #                   Erlang, active mode controls the
    #                   automatic sending of messages to the
    #                   socket's controlling process. We disable
    #                   this behavior, so our server can control
    #                   when and how it reads data
    {:ok, socket} = :gen_tcp.listen(port, [
      :binary, packet: :line, active: false
    ])
    IO.puts("Listening on port #{port}")
    loop_handle_client_connection(socket)
  end

  defp loop_handle_client_connection(socket) do
    # Wait for a new client connection. This is a blocking call
    # that waits until a new connection arrives.
    # A connection returns a `client_socket` which is connected
    # to the client, so we can send a reply back.
    {:ok, client_socket} = :gen_tcp.accept(socket)
    send_hello_world_response(client_socket)
    :gen_tcp.close(client_socket)

    # Recursively wait for the next client connection
    loop_handle_client_connection(socket)
  end

  defp send_hello_world_response(client_socket) do
    # Simple HTML content for the response.
    content = "<h1>Hello, World!</h1>"

    # Generate the entire raw HTTP response, which includes 
    # calculating content-length header.
    response = """
    HTTP/1.1 200 OK
    content-length: #{byte_size(content)}
    content-type: text/html

    #{content}
    """
    :gen_tcp.send(client_socket, response)
  end
end

SimpleServer.start(8080)

We can now start the script in one terminal and probe it with curl in another to see HTML being returned:

$ elixir simple_server.exs    | $ curl http://localhost:8080
Listening on port 8080        | <h1>Hello, World!</h1>%

This works, if by "works" we mean it technically returns HTML.

Is this actually useful? No, not in any practical sense, but it gives a small hint at what happens underneath all the abstractions that the libraries we'll explore today provides.

ℹ️ BTW the full code to this section is here.

Cowboy

Cowboy is a minimalist and popular HTTP server, written in Erlang and used in many Elixir projects.

To try Cowboy, we scaffold an Elixir project with mix:

$ mix new cowboy_example
* creating README.md
* creating .formatter.exs
* creating .gitignore
* creating mix.exs
* creating lib
* creating lib/cowboy_example.ex
* creating test
* creating test/test_helper.exs
* creating test/cowboy_example_test.exs

Your Mix project was created successfully.
You can use "mix" to compile it, test it, and more:

    cd cowboy_example
    mix test

Run "mix help" for more commands.

Then we add Cowboy as a dependency and install dependencies:

$ cd cowboy_example

$ git-nice-diff -U1
cowboy_example/mix.exs
L#23:
     [
+      {:cowboy, "~> 2.11"}
       # {:dep_from_hexpm, "~> 0.3.0"},

$ mix deps.get
Resolving Hex dependencies...
Resolution completed in 0.043s
New:
  cowboy 2.12.0
  cowlib 2.13.0
  ranch 1.8.0
* Getting cowboy (Hex package)
* Getting cowlib (Hex package)
* Getting ranch (Hex package)
You have added/upgraded packages you could sponsor, run 
`mix hex.sponsor` to learn more

ℹ️ BTW git-nice-diff is just a small script that works like git diff but simplifies the output to make it easier to show diffs in this article. You can find it here if you're curious.

And then we use Cowboy in a module:

defmodule CowboyExample do
  def start_server do
    # Set up the routing table for the Cowboy server, so root 
    #requests ("/") direct to our handler.
    dispatch = :cowboy_router.compile([{:_, [
      {"/", CowboyExample.HelloWorldHandler, []}
    ]}])

    # Start the Cowboy server in "clear mode" aka plain HTTP
    #   options - Configuration options for the server itself
    #             (this also supports which IP to bind to,
    #             SSL details, etc.)
    #   `env`   - Configuration map for how the server
    #             handles HTTP requests
    #             (this also allows configuring timeouts,
    #             compression settings, etc.)
    {:ok, _} =
      :cowboy.start_clear(
        :my_name,
        [{:port, 8080}],
        %{env: %{dispatch: dispatch}}
      )
    IO.puts("Cowboy server started on port 8080")
  end
end

defmodule CowboyExample.HelloWorldHandler do
  # `init/2` is the entry point for handling a new HTTP request
  # in Cowboy
  def init(req, _opts) do
    req = :cowboy_req.reply(200, %{
      "content-type" => "text/html"
    }, "<h1>Hello World!</h1>", req)

    # Return `{:ok, req, state}` where `state` is
    # handler-specific state data; here, it's `:nostate`
    # as we do not maintain any state between requests.
    {:ok, req, :nostate}
  end
end

And now we can get an HTML response from the server:

$ iex -S mix                  | $ curl http://localhost:8080
Generated cowboy_example app. | <h1>Hello World!</h1>%
Erlang/OTP 26 [erts-14.2.2] [ |
source] [64-bit] [smp:12:12]  |
[ds:12:12:10] [async-threads: |
1] [dtrace]                   |
                              |
Interactive Elixir (1.16.1) - |
 press Ctrl+C to exit (type h |
() ENTER for help)            |
iex(1)> CowboyExample.start_s |
erver                         |
Cowboy server started on port |
 8080                         |
:ok                           |
iex(2)>                       |

We have moved up an abstraction level compared to before, with Cowboy now handling the sockets. The code to make Cowboy work is perhaps a bit low-level in how we end up writing raw HTML responses code, and how we have to learn some Erlang-specific syntax and patterns. But it works, which is certainly pleasing.

ℹ️ BTW the full code to this section is here.

Plugged Cowboy

Actually, Cowboy is most commonly used with Plug, which is an Elixir library to make it easy to write HTML-responding functions. In fact, the two are put together so often there's a library just for that purpose: plug_cowboy, which combines both Cowboy and Plug dependencies.

To try it out we generate a new project, but this time we'll also generate a supervisor (the --sup flag) because Plug manages Cowboy servers in sub-processes:

$ mix new --sup plugged_cowboy
* creating README.md
* creating .formatter.exs
* creating .gitignore
* creating mix.exs
* creating lib
* creating lib/plugged_cowboy.ex
* creating lib/plugged_cowboy/application.ex
* creating test
* creating test/test_helper.exs
* creating test/plugged_cowboy_test.exs

Your Mix project was created successfully.
You can use "mix" to compile it, test it, and more:

    cd plugged_cowboy
    mix test

Run "mix help" for more commands.

Then, add the dependency:

$ cd plugged_cowboy

$ git-nice-diff -U1
/plugged_cowboy/mix.exs
L#24:
     [
+      {:plug_cowboy, "~> 2.0"}
       # {:dep_from_hexpm, "~> 0.3.0"},

$ mix deps.get
Resolving Hex dependencies...
Resolution completed in 0.104s
New:
  cowboy 2.10.0
  cowboy_telemetry 0.4.0
  cowlib 2.12.1
  mime 2.0.5
  plug 1.15.3
  plug_cowboy 2.7.0
  plug_crypto 2.0.0
  ranch 1.8.0
  telemetry 1.2.1
* Getting plug_cowboy (Hex package)
* Getting cowboy (Hex package)
* Getting cowboy_telemetry (Hex package)
* Getting plug (Hex package)
* Getting mime (Hex package)
* Getting plug_crypto (Hex package)
* Getting telemetry (Hex package)
* Getting cowlib (Hex package)
* Getting ranch (Hex package)

Write a request-handling plug:

defmodule PluggedCowboy.MyPlug do
  import Plug.Conn

  # Plugs must define `init/1`, but we have nothing to configure so it's just a no-op implementation
  def init(options), do: options

  # `call/2` is the main function of a Plug, and is expected to process the request and generate a response
  def call(conn, _options) do
    conn
    # Both functions below are part of `Plug.Conn`s functions, they're available because we imported it
    |> put_resp_content_type("text/plain")
    |> send_resp(200, "Hello, World!")
  end
end

Configure and register a Plug.Cowboy child process, which will take care of the underlying Cowboy server:

$ git-nice-diff -U1
/plugged_cowboy/lib/plugged_cowboy/application.ex
L#10:
     children = [
+      # Connect  Plug.Cowboy plug handler
+      {Plug.Cowboy, plug: PluggedCowboy.MyPlug, scheme: :http, options: [port: 8080]}
       # Starts a worker by calling: PluggedCowboy.Worker.start_link(arg)

And… that all just works:

$ iex -S mix                  | $ curl http://localhost:8080
Erlang/OTP 26 [erts-14.2.2] [ | Hello, World!%
source] [64-bit] [smp:12:12]  | 
[ds:12:12:10] [async-threads: |
1] [dtrace]                   |
Interactive Elixir (1.16.1) - |
 press Ctrl+C to exit (type h |
() ENTER for help)            |
iex(1)>                       |

Plug definitely offers us several useful simplifications, by bringing our code into pure Elixir and abstracting over both the Cowboy server details and the raw HTML details in the response function.

ℹ️ BTW the full code to this section is here.

Bandit

Bandit is a modern Elixir web server that integrates well with Elixir's concurrency model, offering great performance.

ℹ️ BTW there's humor in the naming: Bandit is an alternative to Cowboy, and internally Cowboy uses a dependency called Ranch to orchestrate sockets, and Bandit decided to call their socket-wrangling dependency Thousand Island 😂.

Let's generate a new project and add dependencies:

$ mix new --sup bandit_example > /dev/null
$ cd bandit_example
$ git-nice-diff -U1
/bandit_example/mix.exs
L#24:
     [
+      {:bandit, "~> 1.0"},
+      {:plug, "~> 1.0"}
       # {:dep_from_hexpm, "~> 0.3.0"},

$ mix deps.get

ℹ️ BTW the > /dev/null just silences the mix new command.

Then write the code:

$ git-nice-diff -U1
/bandit_example/lib/bandit_example/application.ex
L#10:
     children = [
+      {Bandit, plug: BanditExample.MyPlug, port: 8080}
       # Starts a worker by calling: BanditExample.Worker.start_link(arg)
/bandit_example/lib/bandit_example/my_plug.ex
L#1:
+defmodule BanditExample.MyPlug do
+  import Plug.Conn
+
+  def init(options), do: options
+
+  def call(conn, _options) do
+    conn
+    |> put_resp_content_type("text/plain")
+    |> send_resp(200, "Hello, World!")
+  end
+end

And that just works:

$ iex -S mix                  | $ curl http://localhost:8080    
                              | Hello, World!%
Erlang/OTP 26 [erts-14.2.2] [ |
source] [64-bit] [smp:12:12]  |
[ds:12:12:10] [async-threads: |
1] [dtrace]                   |
                              |
Interactive Elixir (1.16.1) - |
 press Ctrl+C to exit (type h |
() ENTER for help)            |
iex(1)>                       |

This turned out very similar to Plugged Cowboy, and that's no accident: Bandit is designed as a drop-in replacement to Cowboy. So, since this wasn't a challenge, how about we move ahead and connect to our database?

Connecting To Database

Add the Postgres adapter Postgrex:

$ git-nice-diff -U1
bandit_example/mix.exs
L#25:
       {:bandit, "~> 1.0"},
-      {:plug, "~> 1.0"}
+      {:plug, "~> 1.0"},
+      {:postgrex, ">= 0.0.0"}
       # {:dep_from_hexpm, "~> 0.3.0"},

$ mix deps.get > /dev/null

Initialize the database:

$ mkdir -p priv/db 
$ initdb -D priv/db
$ pg_ctl -D priv/db -l logfile start

We also need a user and a database:

$ createuser -d bandit 
$ createdb -O bandit bandit

And register the Postgrex process in our application's supervisor tree:

$ git-nice-diff -U1
bandit_example/lib/application.ex
L#10:
     children = [
-      {Bandit, plug: BanditExample.MyPlug, port: 8080}
+      {Bandit, plug: BanditExample.MyPlug, port: 8080},
+      {Postgrex,
+        [
+          name: :bandit_db,
+          hostname: "localhost",
+          username: "bandit",
+          password: "bandit",
+          database: "bandit"
+        ]
+      }
     # Starts a worker by calling: BanditExample.Worker.start_link(arg)

Now we can add a database query to our HTML response:

$ git-nice-diff -U1
bandit_example/lib/my_plug.ex
L#6:
   def call(conn, _options) do
+    %Postgrex.Result{rows: current_time} =
+      Postgrex.query(:bandit_db, "SELECT NOW() as current_time", [])
+
     conn
     |> put_resp_content_type("text/plain")
-    |> send_resp(200, "Hello, World!")
+    |> send_resp(200, "Hello, World! It's #{current_time}")
   end

The response we get reflects the database query the server performs when generating the response:

20:04:22.598 [info] Running B | $ curl http://localhost:8080
anditExample.MyPlug with Band | Hello, World! It's 2024-03-17
it 1.3.0 at 0.0.0.0:8080 (htt |  19:04:24.547992Z%
p)                            | 
Erlang/OTP 26 [erts-14.2.2] [ |
source] [64-bit] [smp:12:12]  |
[ds:12:12:10] [async-threads: |
1] [dtrace]                   |
                              |
Interactive Elixir (1.16.1) - |
 press Ctrl+C to exit (type h |
() ENTER for help)            |
iex(1)>                       |

We've climbed up from brutal raw socket handling to using powerful high-level web- and SQL-abstractions, so we can once again claim everything works 🎉

ℹ️ BTW the full code to this section is here.

Phoenix Framework

We can't skip Phoenix, the dominant framework for Elixir web development. Phoenix is a powerful, flexible, and highly ergonomic framework for writing very scalable web applications.

Generate and bootstrap a Phoenix sample app:

$ mix local.hex --force && mix archive.install hex phx_new --force
$ mix phx.new my_app  # answer yes when prompted
$ cd my_app
$ mkdir -p priv/db && initdb -D priv/db && pg_ctl -D priv/db -l logfile start && createuser -d postgres
$ mix deps.get && mix ecto.create

ℹ️ BTW if you get database errors, you might need to reset Postgres:

$ lsof -ti :5432 | xargs -I {} kill {}; rm -rf priv/db # Kill all processes on Postgres' default port
$ rm -rf priv/db # Delete the local DB data folder

Now start the app with iex -S mix phx.server and visit http://localhost:4000:

What Phoenix Gives Us

The Phoenix generator connects to Postgres using Ecto which is a library designed to efficiently create database queries.

And Ecto actually uses Postgrex under the hood; if we read config/dev.exs we see the same pattern of hardcoded database credentials that we saw in the Bandit section:

$ cat config/dev.exs
…
# Configure your database
config :my_app, MyApp.Repo,
  username: "postgres",
  password: "postgres",
  hostname: "localhost",
  database: "my_app_dev",
…

Let's also make our Phoenix example use the database:

$ git-nice-diff -U1
my_app/lib/my_app/query.ex
L#1:
+defmodule MyApp.Query do
+  import Ecto.Query
+
+  alias MyApp.Repo
+
+  def get_db_time do
+    # The SELECT 1 is a dummy table to perform a query without a table
+    query = from u in fragment("SELECT 1"), select: fragment("NOW()")
+    query |> Repo.all() |> List.first()
+  end
+end
my_app/lib/my_app_web/controllers/page_controller.ex
L#6:
     # so skip the default app layout.
-    render(conn, :home, layout: false)
+    db_time = MyApp.Query.get_db_time()
+    render(conn, :home, layout: false, db_time: db_time)
   end
my_app/lib/my_app_web/controllers/page_html/home.html.heex
L#42:
   <div class="mx-auto max-w-xl lg:mx-0">
+    <h1 class="text-lg">Database Time: <%= @db_time %></h1>
     <svg viewBox="0 0 71 48" class="h-12" aria-hidden="true">

We now display the database query result on the page:
.

ℹ️ BTW the full code to this section is here.

But there is much more to Phoenix than just getting it working: Phoenix comes with a lot of powerful tooling to create advanced live-updating server-side rendered pages that match or exceed the experience of fully client-side rendered web apps. But it doesn't seem effective to dive into those things here as Phoenix has many amazing guides that introduce its capabilities better than what this article can.

Conclusion

It's been a lot of fun trying out various solutions, but there's no escaping Phoenix ends up as the most realistic choice for our needs: It's a simple and supremely scalable framework with impressive industry-leading features, and it comes with really great documentation and lots of articles and forum posts to draw inspiration from.

That's not to discount the alternatives, e.g. Bandit: For simple or very specialized needs there wouldn't be anything wrong with choosing Bandit, as it does a great job at providing the essential web server. That is, after all, why Phoenix uses Bandit, and it is delightful how these libraries manages to interact and support each other via Elixir's mature and welcoming community.

Phoenix is the obvious fit for this article series , and I will continue to use it in future articles.

Perfect Elixir: Setting Up an Elixir Dev Environment

Jon Lauridsen — Mon, 18 Mar 2024 13:05:25 +0000

We need Erlang and Elixir installed, which might sound simple, but there are trade-offs to consider for a shared team environment. We'll also add a PostgreSQL database to keep our explorations relevant to real-world scenarios.

Let's explore different approaches and discuss their pros and cons.

Table of Contents

Just… Install the Dependencies?
asdf
mise
Nix
pkgx
In Conclusion

Just… Install the Dependencies?

Why not just install the dependencies as suggested by each tool's website?

I’m on macOS and erlang.org, elixir-lang.org, and postgresql.org all recommend installing via Homebrew.

So, first we install Homebrew:

$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

🖥️ Terminal

Then install our tools:

$ brew install erlang elixir postgresql@15

🖥️ Terminal

And… we’re done!?

But there are critical issues with this approach:

No versioning - There's no control over what versions we just installed, because Homebrew is not designed for versioning. It leads to a totally unpredictable mix of versions as different developers will install their tools at different times.
Globally installed - Homebrew tools are globally installed, meaning they affect all other projects. That leads to unpredictable behavior as one project requires an upgrade that ruins another project, and Homebrew doesn't offer a way to switch between versions.

So no, “just installing” isn’t viable at all, we must find a solution that installs exactly the right versions just for our project, across all developer machines and environments. Let's go explore some tools that are designed to do that.

asdf

asdf is a version manager with plugins for Erlang, Elixir, and Postgres. The installation guide suggests first installing some system dependencies via Homebrew and then cloning the asdf repository:

$ brew install coreutils curl git
…
$ git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.13.1
$ echo '. "$HOME/.asdf/asdf.sh"' >> ~/.zshrc

🖥️ Terminal

ℹ️ BTW it's quite odd to install asdf via git clone. Although it can be installed via Homebrew the asdf guide recommends using Git so git clone it is…

Next, add plugins:

$ asdf plugin add erlang https://github.com/asdf-vm/asdf-erlang.git
$ asdf plugin add elixir https://github.com/asdf-vm/asdf-elixir.git
$ asdf plugin add postgres

🖥️ Terminal

And then we need to go through each plugin's GitHub repository's documentation to derive a list of additional dependencies that are needed:

$ brew install autoconf openssl@1.1 openssl libxslt fop gcc readline zlib curl ossp-uuid
$ echo 'export KERL_CONFIGURE_OPTIONS="--without-javac --with-ssl=$(brew --prefix openssl@1.1)"' >> ~/.zshrc

🖥️ Terminal

ℹ️ BTW it's really unnerving how each plugin requires their own set of Homebrew-installed system dependencies, because that seems to throw all the versioning right out the window! But let's keep going…

Then, create a .tool-versions file to specify the tools and versions:

$ cat << EOF >> .tool-versions
erlang 26.2.1
elixir 1.16.0
postgres 15.5
EOF

And then the last command is to install the specified tools:

$ asdf install
…
$ which erl
/Users/cloud/.asdf/shims/erl
$ which elixir
/Users/cloud/.asdf/shims/elixir
$ which psql
/Users/cloud/.asdf/shims/psql

🖥️ Terminal

We now have all our tools installed 🎉

`direnv`

But just having the tools installed isn't quite enough: Developers will have to manually run asdf install to stay in sync with the specified versions, it's not taken care of automatically. Can we automate this part?

direnv is a common tool for keeping developer environments in sync, because it can trigger commands upon entering a folder. So let's install and configure direnv:

$ asdf plugin add direnv
$ echo direnv 2.30.0 >> .tool-versions 
$ asdf install
$ asdf direnv setup --shell zsh --version 2.30.0
$ asdf direnv local
$ echo "use asdf" > .envrc

🖥️ Terminal

Now, if we simulate a change to .tools-versions by updating the Erlang version, we'll see direnv automatically prompts to re-install dependencies:

$ cd perfect-elixir/
direnv: loading ~/perfect-elixir/.envrc
direnv: using asdf
direnv: Creating env file /Users/cloud/.cache/asdf-direnv/env/1510633598-1931737049-390094659-716574907
direnv: erlang 26.2.2 not installed. Run 'asdf direnv install' to install.
direnv: referenced  does not exist
$ asdf direnv install
Downloading 26.2.2 to /Users/cloud/.asdf/downloads/erlang/26.2.2...
...
$ which erl
/Users/cloud/.asdf/installs/erlang/26.2.2/bin/erl

🖥️ Terminal

We now have an workflow backed by asdf that automatically keep our environments in sync, even as our team upgrades tool versions 🎉

We can compare tools later, for now let's move on to try the next tool to see how it's different.

mise

Mise is a recent replacement for asdf, leveraging all the existing asdf plugins but promising to dramatically simplify the steps to get everything work. So let's check it out.

Install Mise via Homebrew:

$ brew install mise

Activate it for your shell (assuming zsh):

$ echo 'eval "$(mise activate zsh)"' >> "${ZDOTDIR-$HOME}/.zshrc"
$ source ~/.zshrc

Create a .mise.toml file to specify dependencies:

$ cat .mise.toml
[tools]
erlang = '26.2.1'
elixir = '1.16.0'
postgres = '15.5'

Install the dependencies:

$ mise install
mise ⚠️ postgres is a community-developed plugin – https://github.com/smashedtoatoms/asdf-postgres
Would you like to install postgres? Yes
…
mise elixir@1.16.0 ✓ installed

🖥️ Terminal

And just like that every tool is available:

$ which erl
/Users/cloud/.local/share/mise/installs/erlang/26.2.1/bin/erl

$ which elixir
/Users/cloud/.local/share/mise/installs/elixir/1.16.0/bin/elixir

$ which psql
/Users/cloud/.local/share/mise/installs/postgres/15.5/bin/psql

And the tools are automatically only activated inside the folder:

$ cd ..
$ which erl
erl not found

$ cd perfect-elixir
$ which erl
/Users/cloud/.local/share/mise/installs/erlang/26.2.1/bin/erl

That's slick! 🎉

Nix

Nix is a tool "for reproducible and declarative configuration management", available for macOS and Linux. Let’s give it a try!

First, install Nix:

$ sh <(curl -L https://nixos.org/nix/install)

🖥️ Terminal
Installing nix.gif

ℹ️ BTW The installer requires sudo, and it creates a new “Nix Store” drive-volume and 32 hidden new users. I immediately find that really intrusive, it sure feels like a lot just to install some system tools…

Nix uses a custom pseudo programming language for specifying dependencies. I've struggled greatly to understand any available Nix guides and tutorials but I think we have to enable some experimental features and create a flake.nix file:

$ mkdir -p ~/.config/nix && echo "experimental-features = nix-command flakes" > ~/.config/nix/nix.conf

$ nix flake new .
wrote: /Users/cloud/Documents/nix/flake.nix

$ cat flake.nix
{
  description = "A very basic flake";
  outputs = { self, nixpkgs }: {
    packages.x86_64-linux.hello = nixpkgs.legacyPackages.x86_64-linux.hello;
    packages.x86_64-linux.default = self.packages.x86_64-linux.hello;
  };
}

ℹ️ BTW I don't understand why a new flake references "legacy" packages, or why they point to Linux packages when I run this on a Mac… but these are just minor confusions in the journey to get Nix working properly.

We then edit flake.nix to specify our (Mac) dependencies:

$ cat flake.nix
{
  description = "A flake";
  outputs = { self, nixpkgs }: {
    devShells.x86_64-darwin = {
      default = nixpkgs.legacyPackages.x86_64-darwin.mkShell {
        buildInputs = [
          nixpkgs.legacyPackages.x86_64-darwin.erlangR26
          nixpkgs.legacyPackages.x86_64-darwin.elixir_1_16
          nixpkgs.legacyPackages.x86_64-darwin.postgresql_15
        ];
      };
    };
  };
}

Now we can activate the flake by running nix develop:

$ nix develop
...
$ which erl
/nix/store/49qw7cw30wszrfn3sa23qnlskyvbnbhi-erlang-26.2.2/bin/erl
$ which elixir
/nix/store/rr6immch9mp8dphv1jvgxym35za4b7jy-elixir-1.16.1/bin/elixir
$ which psql
/nix/store/v5ym92k3kss1af7n1788653vis1d6qsc-postgresql-15.5/bin/psql

🖥️ Terminal

And if we exit the shell, the tools are no longer available:

macOS-14:perfect-elixir cloud$ which erl
/nix/store/49qw7cw30wszrfn3sa23qnlskyvbnbhi-erlang-26.2.2/bin/erl
macOS-14:perfect-elixir cloud$ exit
exit
$ which erl
erl not found

🖥️ Terminal

We now have a reproducible specification of our environment!

`direnv`

But just as with #asdf we would like the tools to automatically be available upon entering the folder. Let's once again automate it with direnv.

First, install direnv via Nix:

$ nix-env -iA nixpkgs.direnv;
$ echo 'eval "$(direnv hook zsh)"' >> ~/.zshrc
$ source ~/.zshrc

🖥️ Terminal

And activate the flake with direnv:

$ echo "use flake" > .envrc
$ direnv allow
direnv: loading ~/Documents/nix/.envrc
direnv: using flake
…
$ which erl
/nix/store/rp1c50s0w039grl22q086h0dyrygk0p2-erlang-26.2.1/bin/erl
$ which elixir
/nix/store/66f9b1d1c4fmhz6bd3fpcny6brjm0fk7-elixir-1.16.0/bin/elixir
$ which psql
/nix/store/zhk6mf2y5c07zqf519zjkm3fm2nazmvj-postgresql-15.5/bin/psql

🖥️ Terminal

Now, our Nix environment automatically activates when we enter the folder 🎉

ℹ️ BTW direnv requires running direnv allow whenever the .envrc file changes to prevent malicious code from executing. Always review .envrc before allowing.

pkgx

pkgx is a new package manager, and is actually actually a collection of small composable tools that can be used and composed to solve a variety of dependency-related use cases. Our needs is covered by their dev tool, which describes itself like this:

dev uses pkgx to create “virtual environments” consisting of the specific versions of tools and their dependencies you need for your projects.

Let's give it a go!

First, we install and activate the dev tool:

$ brew install pkgxdev/made/dev
$ dev integrate

Create a .pkgx.yml file to specify dependencies:

$ cat .pkgx.yml
dependencies:
  erlang.org: =27.3.4.2  
  elixir-lang.org: =1.18.4
  postgresql.org: =17.2.0

Activate the dependencies:

$ which elixir
elixir not found

$ dev

$ which elixir
/Users/cloud/.pkgx/elixir-lang.org/v1.18.4/bin/elixir

And… err, that’s it?! The tools are automatically only available when inside the folder:

$ cd ..
-erlang.org=27.3.4.2 -elixir-lang.org=1.18.4 -postgresql.org=17.2.0
no devenv detected

$ which elixir
elixir not found

$ cd perfect-elixir

$ which elixir
/Users/cloud/.pkgx/elixir-lang.org/v1.18.4/bin/elixir

Hard to get any simpler than that! And worth noting pkgx also supports all manner of core system tools such as bash, grep, etc., which is super useful.

In Conclusion

asdf seems to be a popular choice judging from an abundance of articles mentioning it, but I found it quite cumbersome and old-fashioned to use. I don't mean to be rude, and I'm sure asdf has helped developers for decades, but I think asdf is probably popular more for historical reasons than for how it compares to its present-day peers.

I would not recommend using asdf for a new project.

Mise dramatically simplifies the asdf experience, removing the major ergonomic painpoints of asdf. It's really remarkably simple to use, and it deserves praise for that. But also worth mentioning: Mise does not support system tools such as bash, grep, etc., and having different versions of those are very common sources of errors because e.g. MacOS' grep is very different from GNU grep and some projects often end up requiring one or the other.

As a result Mise-based projects are highly likely to end up also maintaining a list of Homebrew dependencies that developers must install, which causes the issues of lack of versioning we saw in the Just... Install the Dependencies section.

Ultimately Mise provides a nice experience, but cannot offer versioning of a wide enough set of tools.

Nix is clearly powerful, but also very hard to learn. Like, way over the top hard, complete with lacking documentation and hard to grasp jargon. It offers unmatched control over dependencies, but it also requires such a significant learning curve it stands in strong contrast to our needs of just wanting a handful of system tools installed, and it ends up being really intrusive compared to the task we're looking to solve.

I'm sure Nix is a great tool for sophisticated needs such as specifying all dependencies for an entire operating system, but for installing Elixir and Bash? Nix is a huge overkill for that purpose.

You should carefully consider its learning curve and reach into everyone's systems before adopting it, including how it will impact every person who will ever work on this project, including future hires.

🥇 pkgx's dev tool is impressively simple, and very easy to use: Easy to install, easy to configure, and easy to use. It's crazy simple all the way: No need for sudo, the installer automatically handles integrating with the shell, and it's just enormously convenient how dev just works correctly right out of the box.

Furthermore, the pkgx registry includes all manner of packages, including system-tools such as bash and grep. Don't underestimate the value of that, because projects tend to accumulate Bash scripts and it's a common source of errors to have Bash scripts fail in various ways because the team has not synced to the same versions.

Ultimately the best tool must depend on your specific needs and preferences; only you can choose based on the requirements you face. To me pkgx is by far the most user-friendly and comprehensive option available so it's an easy recommendation, and it is the tool I'll use going forward in this article series.

Exploring the Perfect Elixir Setup

Jon Lauridsen — Mon, 18 Mar 2024 13:00:00 +0000

In this series, we'll explore the most effective ways of working with Elixir projects, examining the various tools, techniques, and processes that are available in Elixir. There are several topics to explore, and each of them has various trade-offs we'll need to navigate and discuss. Our ultimate goal is to identify specific answers and practical tools that unlock the most effective and simplest ways of working, such that the solutions can start simple and stay useful as a team and their product grows.

Table of Contents

Elixir: A Brief Overview
Exploring Elixir?
Topics

Elixir: A Brief Overview

For those new to the language, Elixir is a dynamic, functional programming language created in 2011, with syntax inspired by Ruby. Similar to how Java uses the Java Virtual Machine, Elixir runs on the Erlang Abstract Machine (BEAM). The BEAM, originating in the mid-80s, is known for powering critical and highly fault-tolerant distributed systems at scale.

It's a very interesting language because it manages to be simultaneously easy to pick up, has a standard-library with superpowers that are just out of this world, and its community is friendly and welcoming. But this isn't the series to deep-dive into Elixir itself, as there are great resources for that already. For example:

Official Elixir Website and Documentation: The official Elixir guide is a great starting point, with thorough documentation and great getting started guides.
Thinking Elixir podcast: An informative and friendly podcast covering recent developments and community insights.
The Elixir Forum: An excellent place to ask questions and engage with the community. It's known for being welcoming to newcomers.

Exploring Elixir?

Now back to exploring. This series is an open learning journey: I am myself skilling up in Elixir, so this is genuinely a dive into the unknown to explore what's possible in Elixir projects. We are very explicitly not rushing in to write Elixir application code, rather our focus will be the many details and root-decisions that surrounds the application. We'll focus in particular on the many decisions that are often made in a hurry when projects are just getting started, that can end up scaling horribly and cause all manner of irritations and slowdowns as a project grows.

Topics

2. Setting Up an Elixir Dev Environment

We need Erlang and Elixir installed, which might sound simple, but there are trade-offs to consider for a shared team environment. We'll also add a PostgreSQL database to keep our explorations relevant to real-world scenarios.

This article explores asdf, mise, Nix, and pkgx, to set up a reproducible development environment.

3. Building a Basic Elixir Web App

Today we'll change gears a bit: It's time to make a web app. We'll need a project to work on for later articles in order to explore how to work with Elixir code, and so for this article we'll dive into some of the different ways to build a web app in Elixir. Our goal will be a simple example of a web app, and ideally it should make a data database call to be representative of real-world projects. Let's see what solutions we can find.

This article explores Cowboy, Plug, Bandit, and Phoenix to set up a web app.

4. Streamlining Development Workflows

Today we'll define and implement our daily development workflows. We'll begin by identifying what makes a workflow truly effective — drawing from cutting-edge research on software delivery practices. Then we'll establish really simple mechanisms for managing code changes so we can move quickly and accurately as a team. And importantly, these workflows must be flexible enough to scale as our team and product grows. Let's dive in!

This article explores how to write shell scripts to set up effective and efficient workflows.

5. Simplifying Elixir Developer Onboarding

Onboarding can be a costly but hidden drag on a team — not just in raw setup time, but also in how easily it becomes a process full of uncertainty and fragility. How developers get started reflects the standards of care a team chooses to uphold, and we should explore how simple and safe we can make this experience.

This article explores streamlining developer onboarding via semi-automated scripting.

6. Automating Tests in Elixir Projects

How do we write fast, reliable tests in Elixir? How do we keep modules decoupled and easy to test? Testing is a broad topic with many opinions, and today we’ll explore techniques that will let us have informed discussions about how we want to test. Because good tests aren't just about passing — they’re about helping us move fast, and deploy with confidence.

In this article, we'll set up automated tests in Elixir, explore dependency injection, and compare tools like Mox, Hammox, and Double to keep tests clean and scalable.

7. Testing Shell Scripts with Bats

Automated testing is crucial for reliable software development, yet shell scripts are frequently left untested, despite managing critical tasks like deployments, database migrations, and CI pipelines. Neglecting tests here can lead to subtle bugs and fragile workflows. Today, let's make sure all our code is testable, treating shell scripts with the same discipline we apply to application code.

This article explores the Bash Automated Testing System (Bats) to properly integrate shell scripts into our codebase.

…and more articles to come as I find time to write them. If you have suggestions for topics please leave a comment.

Event-Driven Architecture: A Focused Primer

Jon Lauridsen — Wed, 10 Jan 2024 18:00:00 +0000

Event-Driven Architecture (EDA) is a design pattern that helps systems remain resilient and scalable under complexity and high load. It works by decoupling inputs from side effects and letting operations run asynchronously. EDA naturally supports fault tolerance, parallel processing, and graceful handling of service degradations. This primer covers some essential strengths of EDA and when it can make sense to use it.

Commands and Events

EDA systems accept commands: requests to perform an action, such as "cancel an order", "change item quantity" or "play a song". A command represents intent — it expresses what a user or system wants to happen, but does not guarantee that it will.

Processing a command means validating it, applying business rules, and then deciding what events (if any) to emit in response. Events are the true core of EDA — they are immutable, durable records of something that happened. Once emitted, they form the historical record of the system.

Commands are not limited to any protocols, they may be ingested via HTTP, queues, or any transport, but what makes a system event-driven is that events are the durable, traceable outcome of the processing of commands.

Decoupling and Resilience

Unlike synchronous APIs that return results immediately, EDA systems can balance which parts to handle inline and which to defer. For example, suppose a "change item quantity" command includes a business rule that requires a geo-IP lookup. That lookup might be slow or unreliable. Instead of stalling the entire command, an EDA system can emit an event like ItemQuantityChangeRequested, and let another subsystem handle the lookup and emit QuantityChangeRejected or QuantityChanged later.

This separation improves throughput and robustness. Even if the downstream geo service is offline or slow, command ingestion continues. Systems like this are naturally resilient to back-pressure and failure, since only the minimal ingestion surface needs to stay online.

In very high-throughput cases, command ingestion can become entirely asynchronous: just schema validation and queueing. That allows for immense scale and high availability.

Error Handling and Graceful Degradation

Systems built with EDA can also handle transient failures more gracefully. If a "cancel order" command is received but the underlying order system is temporarily offline, the command handler can emit a CancelOrderRequested event, and retry later when the system is available.

Crucially, the event provides an audit trail. Even if that order becomes ineligible for cancellation by the time the cancellation event is processed, the timestamp on the event proves the user acted in time. This enables fair outcomes — for example, issuing a refund even though the cancellation was delayed.

This pattern generalizes well to many domains: decoupling validation from processing lets systems make forward progress even in degraded states.

Event Replay and Correctness

Durable events also make it possible to recover from logic bugs. Suppose a page is showing only cancelled orders due to a filtering bug. If events are stored durably — e.g. in an append-only log or a multi-region database — the page's read model can be reset, the bug fixed, and the full stream of past events replayed to rebuild correct state.

This pattern provides a very powerful safety net for evolving business logic without losing historical context.

And replay isn't limited to debugging — it also supports migrations, rebuilding read models with new fields, or syncing newly onboarded services.

Observing Outcomes

When a command is processed, the emitted events become the system's response — not just to the caller, but to any interested party. Clients may subscribe to event streams, poll read models, or observe real-time updates via web-sockets.

This model enables loose coupling not only internally, but across systems. For example, one service might emit OrderConfirmed, and many others can listen — fraud detection, shipping prep, analytics — all triggered without tight integration.

EDA enables systems to react and evolve independently, along well-defined and stable events. That makes the systems more adaptable and maintainable over time.

Conclusion

EDA is a powerful architectural model for building systems that can handle scale, failure, and change. By treating commands as intent, events as facts, and work as asynchronous and composable, it enables systems to be more resilient and observable.

It’s not a silver bullet, it does require more moving parts, and success can depend on careful event design, durable infrastructure, and clear failure strategies. The simplicity of traditional REST-style APIs are still valid and should not be discarded, but EDA gives more leverage for building systems that evolve and survive in the real world — where slowness, downtime, and complexity are the norm.

ElixirScript, a Github Action to run Elixir code in workflow steps

Jon Lauridsen — Wed, 10 Jan 2024 15:00:00 +0000

Hi everyone,

I’ve released this new GitHub Actions called Elixir Script, because I wanted a way to script various tasks in my GitHub Actions workflows and wished for a way to use Elixir to do it.

Thus: ElixirScript, a Github Action to run Elixir code in workflow steps

It works by taking a script as input, which gets run, and then the script’s return is available in outputs.result for later steps to use. Like this:

.github/workflows/example.yml:

- uses: gaggle/elixir_script@v0
  id: script
  with:
    script: |
      defmodule Greeter do
        def greet(name), do: "Oh hi #{name}!"
      end
      Greeter.greet("Mark")

- name: Get result
  run: echo "${{steps.script.outputs.result}}"
  # Echos "Oh hi Mark!"

And the script is run with bindings to the GitHub Actions context that caused the run, so it’s easy to react to run-specific information. For example, :

  script: |
    "🚀 Pushed to #{context.payload.repository.name} by @#{context.actor}!

I think it’s a pretty easy, very low-friction way of getting to write some of the devopsy glue-code automations in Elixir. And it was a lot of fun to write as I got to learn about Elixir and GitHub Actions 🙂.

It also comes with a pre-authenticated GitHub client via Tentacat, so it’s also easy to write scripts that interact with the GitHub API. Here I get a list of stargazers:

- uses: gaggle/elixir_script@v0
  with:
  script: |
    {200, stargazers, _} = Tentacat.Users.Starring.stargazers(client, "gaggle", "elixir_script")
    IO.inspect(Enum.map_join(stargazers, ", ", & &1["login"]), label: "Stargazers")

Or I can can cause changes too, here I star a repository:

- uses: gaggle/elixir_script@v0
  with:
  script: |
    {204, _, _} = Tentacat.Users.Starring.star(client, "gaggle", "elixir_script")
    :ok

Happy to hear any thoughts, feedback, suggestions, etc.

The actual marketplace page is here: Elixir Script · Actions · GitHub Marketplace · GitHub

Effectively Handling MacOS Screenshots

Jon Lauridsen — Tue, 09 Jan 2024 14:00:00 +0000

(cover image: midjourney)

Screenshots can get messy: If you copy to clipboard then you first have to hope where you want to put it accepts pasting, otherwise you’ll need to somehow get the clipboard saved to a file. And if you want to take multiple screenshots? You’ll have to grab, paste, grab, paste, switching context a bunch of time. And if you suddenly have need for a screenshot again 10 minutes later, then, well too bad it’s gone.

To simplify and streamline all that, here's a small tweak that’s easy to set up so screenshots fall into an easy and simple workflow:

Open screenshot.app (e.g. SHIFT-CMD-5 shortcut, or via Spotlight/Launchpad)
Click Options, then “Other Location”
And then go to your Documents folder and create a new folder called “Screenshots” and select that as your screenshot location
Then close screenshot.app (hit ESC)
Open Finder, go to Documents, and drag the new Screenshots folder to your dock
Then right-click the folder in the dock and select Sort by Date Added

Now all new screenshots will instantly appear at the top of that folder, which is easy to click on from any app and thus easy to drag screenshots in no matter which app you're in. Now it’s easy to take several screenshots in one go and trust they’ll all end up in that folder where they can be easily accessed later. And now they also persist, so you can refer back to them later. And they’re easy to modify by just clicking on them, since that opens Preview.

That's it. Just a tiny trick to reduce a small friction.

Don’t Make Conditional GitHub Actions Jobs

Jon Lauridsen — Tue, 09 Jan 2024 13:00:00 +0000

(cover image: midjourney)

A conditional GitHub Actions job can look something like this:

jobs:
  deploy-o-matic:
    if: github.ref == 'refs/heads/main'

But there’s a better way.

GitHub Actions what?

So, GitHub Actions are based around workflows which are yaml files with GitHub Actions syntax. And workflows consist of jobs which gets assigned to (virtual) GitHub Actions machines. And jobs consists of steps which that machine runs one after another.

And it is a very common pattern to see jobs that are conditional, meaning they are only run under certain conditions and are otherwise skipped. This is often done to have a single workflow that runs both in branches and on commits to the main branch, but only the main branch should deploy or create a release.

That is a sensible requirement, but it does not need to be implemented as a conditional job, because that has several negative implications:

All that conditional code isn't run as regularly, and so can accrue mistakes and rot until a deployment happens, and only then does everything explode.
They're inherently very difficult to debug, precisely because the code only gets run on main commits. That makes them difficult to maintain, and that causes them quickly become legacy code no-one dares touch. It's a real problem!

So let's not make jobs conditional.

Instead, here are three alternative patterns to consider:

Call Simple Scripts

This is a generally applicable tip for all pipelines, but it's quite important as much pipeline logic as possible is extracted into simple scripts that can also be run locally. I put this first because some pipelines are absolutely full of various nested logic, where complex inline scripts form a web of behavior that can be very difficult to change or even make sense of. That means no-one has confidence changing it.

So step one is to ensure simplicity: Workflows should ideally just be high-level orchestrators of scripts, not themselves expressions of complex logic. Take as much complexity into scripts, because those can be tested and debugged locally, which makes them much simpler to change and iterate on compared to pipelines that can only run on CI.

Conditionally dryrun

With a pipeline optimized for simplicity that calls out to a number of scripts, now we should identify the actual steps that must only be run on commits to the primary branch. Which exact step is the one that does the deploying? Can that script get invoked conditionally with something like a —dryrun argument? E.g. like this:

 

steps:
  - name: Run deploy script
    run: |
       if [ "${{ github.ref }}" = "refs/heads/main" ]; then 
        ./deploy.sh
       else
         ./deploy.sh --dryrun
       fi

This way all pipeline-code gets fully exercised even in branches, but with —dryrun passed when the commit isn’t on the main branch which the script could implement as outputting useful information about what it would’ve done but without causing any actual side-effects.

This is a lot better, because now there isn’t any code that lies dormant, all the code is always exercised!

ℹ️ I do find it can look a little “dangerous” to have a job named “deploy” that now shows up in the workflow overview, and then only by looking into its details can one see it actually dryruns. It can cause people who test in a branch to fear if they’re suddenly about to actually deploy 😱

To remedy this, and to more clearly express that the job is actually running in dryrun mode, I like to set the job name dynamically:
jobs:
 deploy:
   name: ${{ github.ref == 'refs/heads/main' && 'Deploy' || > 'Deploy (dryrun)' }}
It’s just a small quality-of-life improvement, but it takes away anxiety because now the job will clearly show up as Deploy (dryrun) in the overview.

Conditionally run a step

If —dryrun isn’t really an option, then at least the conditional check should only be applied to the specific steps that mustn’t get run rather than the whole job. Like this:

steps:
  - name: Prepare deploy
  - name: Deploy
    if: github.ref == 'refs/heads/main'
    run: ./deploy.sh

This way at last most of the pipeline still runs all the time, even though yes it does leave the script itself vulnerable to rot. But certainly better than making the entire job conditional.

And if there are any complex data-transformations that prepares any data that a conditionally run step ingests, then strongly consider extracting those transformations into a separate step so the transformations can be easily tested in branches. And then pass the transformed data directly into the step that must be conditionally run. Like this:

steps:
  - name: Generate release description
    run: |
      echo "BODY=**Full Changelog**: …" >> $GITHUB_ENV
  - uses: softprops/action-gh-release@v1
    if: github.ref == 'refs/heads/main'
    with:
      body: ${{ env.BODY }}

Here I have a separate step that calculates the “release description”, which makes it easily debuggable in branches because it always runs. And then its result is passed into the subsequent conditional step.

In conclusion

There are many good patterns to follow that minimizes risk of ending up with an unmaintainable and complex pipeline. If you have any to share I'd be happy to hear 'em in the comments.

Are You Listening to Your Code? Here's Why You Should Start

Jon Lauridsen — Sat, 06 Jan 2024 13:30:54 +0000

In the rush to deliver features and meet tight deadlines, many programmers move from one story to the next without pausing. But there's great value in injecting moments of tranquility before rushing off to do the next ticket.

The Art of Listening to Code

Just like a book or a piece of music, code has its own rhythm and flow. By setting aside some unstructured time to listen to it, you can uncover insights that might otherwise go unnoticed.

Taking a leisurely stroll through your codebase means setting aside a responsible amount of time to sense the code at all its levels:

Trace the Logic: Follow the pathways of your functions and methods. Do they lead logically from one to the next? Would a new-joiner find them understandable, and see how they all fit together cohesively?
Identify the Structure: Are there areas where files and modules feel cluttered or are overly complex? Do multiple concerns clump together, making it difficult to maintain an overview? Imagine explaining this to someone new to the code, would it be a natural story?
Imagine the Alternatives: What could certain systems look like if they were reimagined? Could they be expressed more ideally? What's a small investment that would move the code towards that more ideal state?

By doing this, perhaps you'll find tests that are overly complex and it's time to clean them up. Maybe you'll notice abstractions that are leaking their implementation details. Or you might discover solutions that are difficult to maintain and would benefit from being reimagined in a way that better solves their problem. Most codebases are in need of various kinds of improvement work, and only by taking the time to do it can they be transformed from a tangled mess into a harmonious system.

Personally, I find that playing some of my favorite music helps soften my focus and brings the "feel" of the code to the forefront. I'll allocate a bit of time, sometimes just half an hour, and try to dive in without external distractions to just explore and experiment with code structures.

The outcomes might be a refactor, but just as valuable can be sketching out how certain components interact to surface hidden complexities, or bringing alternative designs back to the team for them to react to. It's even positive just trying out some ideas and then reverting them in the end, because you walk away with more knowledge of how the code actually functions.

These acts of pruning and refining are very valuable, because it lets you focus on the architecture and robustness of the code. This doesn't just make the current code better — it lays a stronger foundation for future development, clarifying and unifying existing patterns. Clean, well-organized code is more maintainable, scalable, and a pleasure to work with.

So next time you wrap up a story, resist the urge to jump straight into the next task. Instead:

Explore: Wander through the new code you've added.
Reflect: Consider how it integrates with existing systems.
Optimize: Look for opportunities to enhance clarity and efficiency.

By investing just a small amount of time, you can significantly improve code quality, enhance maintainability, and find greater satisfaction in your work. Try to purposefully set aside the rigid flow of tickets for a moment, and just… listen to the code, imagine what could be, and let your mind wander as you take a leisurely stroll.