Forem: Patrick Wendo

Simulating Boids with ECSx and liveview

Patrick Wendo — Mon, 09 Mar 2026 10:06:24 +0000

You can view the live simulation here

You can check the code here

Entity Component Systems for Simulations

Entity Component Systems are a new concept I have come to learn as I am learning more about Video Game development. I was then presently surprised to find that a developer Andrew Berrien had built a library, ecsx for developing entity component systems in Elixir. Naturally I had to play around with it.

In this article, I will cover what entity component systems are, how we can use them to build a boid simulation(after all, what is a game but an interactive simulation) and how we can learn to optimize simulations. We also explore ECSx and ECSx Dashboard. All references used are at the end.

What are Entity component systems??

They are an architectural pattern often used in video game development. They allow for re-usability by separating data from behavior. It is a pattern that obeys the "composition over inheritance principle"

ECS is an architecture for building real-time games and simulations, wherein data about Entities is stored in small fragments called Components, which are then read and updated by Systems.
~Mike Binns, ecsx README.md

Entities

These represent a thing in a video game. A distinct object representing an actor in a simulated space. For instance, if we are playing flappy bird, the bird would be an entity.

Entities have no actual data or behaviors.

Components

They are data types consisting of unique behavior assigned to an entity. They are re-usable modules that are attached to entities, providing behavior, functionality and appearance forming an entity.

In our flappy bird example, some components would be a y_coordinate. When attached to a bird entity we know how high on the screen the bird is.

Systems

Systems provide the logic that operates on the components.

For instance, in our flappy bird example, a jump system would update the y_coordinate of the bird when a click occurs.

Putting it all together

System : This listens to outside events and publishes updates to the components.
Components : They listen to the system events and update their state
Entities: They gain behavior through changes in their associated components.

ECSx

The best intro I could give you for ecsx would be for you to watch Andrew Berrien's talk at ElixirConfUS 2023.

The highlights are as follows:

Entities within ecsx are conceptual.

Components and Systems have generators you can use

mix ecsx.gen.component component_name component_type
mix ecsx.gen.system system_name

There is a manager that controls initialization components, and which components and systems are known to the program
Tags are a special type of component that do not have a type.

Boids and our simulation.

Craig Reynolds introduced the Boid algorithm in 1986. It is a computational model that simulates flocking behavior based on 3 rules; separation, alignment and cohesion. It have since been used widely in computer graphics and robotics to model flocking and swarming behavior.

To design a boids simulation with ecsx, we need to define our components and systems. We will have 4 systems, separation, cohesion, alignment and movement. With regards to components, we need to consider that we will require to do a lot of 2d vector manipulation. As such, most of our components will have an x component and a y component. For instance, Velocity will be split into 2 components XVelocity and YVelocity.

The reason for this split is that ecsx only allows a small subset of types for the components.

From the internal documentation of the library

The type of value which will be stored in this component type. Valid types are: :atom, :binary, :datetime, :float, :integer

But this is ok, because we can build helpers that make it easier to work with these separate components.

Further, we also define a small utility to assist with some common calculations. We will call this utility Vector, because it helps us with Vector calculations.

Designing the systems

We have 4 systems defined, movement, cohesion, separation and alignment. The systems cohesion, separation and alignment systems will update the x and y components associated with those systems. For instance x_cohesion and y_cohesion are updated by the cohesion systems and so forth.

The movement of a boid is defined by the sum of the result of these systems. The purpose of the movement system is to that sum these results and update the x and y position components.

Alignment System

Alignment force helps the boids coordinate their direction with nearby boids. It encourages the boids to match the average velocity of their closest neighbors ensuring the group moves in a unified manner.

First we check if there are any neighbors within the alignment radius, if not the force is zero
Sum all the neighbors' velocity vectors and average them to determine the group's general movement direction
Instead of directly setting the velocity to the average, we subtract the boid's current velocity from it, creating a smooth transition instead of an abrupt change.

Cohesion System

The cohesion force ensures that boids stay close to their group, preventing them from drifting apart. It calculates the center of mass of the nearby boids and applying a force that pulls the boids towards this center

We check if there are any neighboring boids. If not the force is set to a zero vector
Average the position of all nearby boids to find the center of mass of the local group
The boid then moves toward this center helping it stay within the flock.

Separation System

Prevents boids from colliding with each other by making them steer away from nearby neighbors. We calculate the this by tracking the closest boids within a certain radius and calculate a repulsion force based on their distances.

The closer a neighboring boid is, the stronger the repulsion force should be.
The separation_intensity is a parameter that follows a hyperbolic function of the form f(x) = a/x which means that as the distance decreases, the force increases.
The repulsion forces from all the neighbors are summed and averages ensuring smooth movement instead of abrupt changes in direction.

Movement System

We define weights to the 3 other forces, cohesion, separation and alignment. These allow us to provide greater control over their combined effect. After weighting, each force is added to the boid's velocity vector.

Once the velocity is updated, the boid's position is recalculated. It's heading is adjusted to align with the velocity vector. We compute the orientation angle using atan2.

The ECSx Manager.

Our manager is responsible for the startup state of the boids. According to the docs

This module holds three critical pieces of data - component setup, a list of every valid component type, and a list of each game system in the order they are to be run.

Our manager startup function will look like this

def startup do
    for _boid <- 1..@number_of_boids do
      entity = Base.encode16(:crypto.strong_rand_bytes(16), case: :lower)

      Boids.Components.BoidEntity.add(entity)
      Boids.Components.XCoord.add(entity, Enum.random(100..1400) * 1.0)
      Boids.Components.YCoord.add(entity, Enum.random(100..900) * 1.0)
      Boids.Components.Width.add(entity, 7.5)
      Boids.Components.Height.add(entity, 15.0)
      Boids.Components.Heading.add(entity, 360.0)
      Boids.Components.XVelocity.add(entity, Enum.random([-1, 1]) * 1.0)
      Boids.Components.YVelocity.add(entity, Enum.random([-1, 1]) * 1.0)
      Boids.Components.XCohesion.add(entity, 0.0)
      Boids.Components.YCohesion.add(entity, 0.0)
      Boids.Components.XAlignment.add(entity, 0.0)
      Boids.Components.YAlignment.add(entity, 0.0)
      Boids.Components.XSeparation.add(entity, 0.0)
      Boids.Components.YSeparation.add(entity, 0.0)
    end
  end

We randomize the starting position to be random within our screen viewport. We also randomize our velocities.

Showing the boids

To render our boids, we will use phoenix liveview with SVG graphics.

def render(assigns) do
    ~H"""
    <div class="w-full p-4 bg-gray-800 h-screen border rounded-md border-black">
      <div class="w-full flex items-center">
        <p class="w-full text-white text-center text-2xl">BOIDS</p>
      </div>

    <!-- Render the Boids ViewFrame -->
      <div class="flex items-center justify-around">
        <svg version="1.1" width="1500" height="900" xmlns="http://www.w3.org/2000/svg">
          <rect width="100%" height="100%" fill="black" />

          <%= for {height, l_position, r_position} <- @boids do %>
            <% [{h_1, h_2}, {l_1, l_2}, {r_1, r_2}] = [height, l_position, r_position] %>
            <polygon
              points={"#{h_1},#{h_2} #{l_1},#{l_2} #{r_1},#{r_2}"}
              fill={random_fill()}
            />
          <% end %>
        </svg>
      </div>
    </div>
    """
  end

Each component height, l_position and r_position all have an x,y component. These will be used to render the 3 vertices of a triangle. The boids will also have random colors for now.

Performance optimizations

As it stands, there are 2 optimizations I am considering.

Hash Partitioning
Using Nx to perform vector math.

Hash Partitioning

Instead of calculating the forces separation, cohesion and separation for the show system, we can divide the viewport into quadrants and calculates forces for the boids in those quadrants.

Using Nx

Numerical Elixir is optimized for faster mathematical computation in elixir. Right now I am doing Vector math with my own small vector module

Nx would do the math faster, however, in testing, I found that it actually increases system latency.

Benchmarking

For benchmarking, we will render 50 boids on a view-port of 1500x1000 px. We use ECSx-live-dashboard to check performance of the systems.

First we look at performance without hash partitioning and without using Nx.

Next we check how the boids perform with hash partitioning. We use 81 partitions

Finally we check performance with Nx and no hash partitioning. The Nx utility used can be found here

Discussion

Well, none of my optimizations performed better, which is OK. It just means there is room for improvement. The first implementation appears to be the best performing one.

I find that the hash partitioning makes the boids roughly align along the partition edges

Further, I think the reason for terrible performance when using Nx, is that I am not using the correct functions. Perhaps using broadcast capabilities of Nx might be a better fit.

Another optimization might be combining both Nx and hash partitioning.

Conclusion

There is a lot more work to be done to make the system performance manageable.

I think ECSx is very very neat for what it offers. It is simple to setup and understand. Further, Nx, while primarily for machine learning, is still an amazing library fr when you need high performance calculations.

Liveview is still a work of art every time I use it. It is simple and gets more intuitive with practice.

In the github repo, I have separate branches for hash_partitioning and nx optimisations.

Let's build a test runner in Elixir.

Patrick Wendo — Mon, 03 Nov 2025 20:15:46 +0000

I have not interacted much with metaprogramming in Elixir, but I thought it was time to check it out.

For this article, we will be building a test runner to explore the topic.

So let's begin.

Quote

First, we need to understand how ExUnit tests are internally represented on the AST.

We use Elixir's quote and unquote methods. Quote allows you to get an internal representation of your expression as the AST. For intance:

iex> quote do: 1+2

{:+, [context: Elixir, imports: [{1, Kernel}, {2, Kernel}]], [1, 2]}

The first element, :+, is an atom denoting the function call, or another tuple, representing a nested node in the AST.
The second element, [context: Elixir, imports: [{1, Kernel}, {2, Kernel}]], represents metadata about the expression
The third element is a list of arguments for the function call.

Interestingly, this AST representation {:+, [context: Elixir, imports: [{1, Kernel}, {2, Kernel}]], [1, 2]} is exactly what the compiler sees. Other languages, such as flavours of Lisp will have you write the source directly as an AST. Elixir however allows you to convert from high-level source to low-level AST with a simple quote invocation.

Next thing we need to understand about ASTs in Elixir is that some literals will have the same representation as high-level literals. Namely, atoms, numbers, strings, lists and tuples

iex> quote do: :atom #atoms
:atom
iex> quote do: 123 #numbers
123
iex> quote do: 3.14 #numbers
3.14
iex> quote do: [1, 2, 3] #lists
[1, 2, 3]
iex> quote do: "string" #strings
"string"
iex> quote do: {:ok, 1} #tuples
{:ok, 1}
iex> quote do: {:ok, [1, 2, 3]} #tuples
{:ok, [1, 2, 3]}

Unquote

The unquote Macro allows values to be injected into the AST that is being defined. These allows the outside bound variables, epxressions and blocks to be injected into the AST. For instance:

iex> number = 5
iex> ast = quote do
...>    number * 10
...> end
{:*, [context: Elixir, import: Kernel], [{:number, [], Elixir}, 10]}

Here we see that the value of number was not injected into the AST. We instead got a local reference for number provided. However,

iex> number = 5
iex> ast = quote do
...>    unquote(number) * 10
...> end
{:*, [context: Elixir, import: Kernel], [5, 10]}

Here because we called unquote(number) the value 5 gets injected into the AST.

Quote and Unquote allow you to build ASTs without fumbling with the AST by hand.

Building the test runner.

Now let's start from the bottom. What is the basic building block of a test? I think it is the assert call. This checks the equivalency of your function being tested and the expected result.

assert MyFun.sum(2,5) == 7

I think we should build an implementation of this function.

Side Note: I am a big proponent for TDD, so we start by builidng specs for our assert function.

We define a function assert/1 which takes one argument that must be an expression checking equivalency.

iex> Tahinix.assert(2 == 2)
:ok

In our tests, we check the assert cases where we use :!= and :==

defmodule TahinixTest do
  require Tahinix

  use ExUnit.Case

  test "assert returns non-zero value when tests fail equivalence" do
    assert Tahinix.assert(2 == 4) == {:error, "Assertion failed: 2 == 4"}
  end

  test "assert returns non-zero value when tests fail non-equivalence" do
    assert Tahinix.assert(2 != 2) == {:error, "Assertion failed: 2 != 2"}
  end

  test "assert returns 0 when tests pass equivalence" do
    assert Tahinix.assert(2 != 4) == :ok
  end

  test "assert returns 0 when tests pass non-equivalence" do
    assert Tahinix.assert(2 == 2) == :ok
  end
end

Now to make our tests pass, we start by implementing the code for :== function first. We use pattern matching in the atributes in order to get our left hand side(lhs) and right hand side(rhs)

  #{:==, _, [lhs, rhs]}
  defmacro assert({:==, _, [lhs, rhs]}) do
    quote do
      lhs = unquote(lhs)
      rhs = unquote(rhs)
      result = lhs == rhs

      case result do
        false -> {:error, "Assertion failed: #{lhs} == #{rhs}"}
        _ -> :ok
      end
    end
  end

This will match expressions of the form lhs == rhs, eg: 2 == 2. By having lhs and rhs in the head, we bind these values to the variable names, which we then pass to unquote.

We then use case to get our return value. If the result if false, we return a tuple {:error, reason_for_error}. Otherwise we retun :ok

Similarly for the assert :!= we pattern match in the function head to get

#{:!=, _, [lhs, rhs]}
  defmacro assert({:!=, _, [lhs, rhs]}) do
    quote do
      lhs = unquote(lhs)
      rhs = unquote(rhs)
      result = lhs != rhs

      case result do
        false -> {:error, "Assertion failed: #{lhs} != #{rhs}"}
        _ -> :ok
      end
    end
  end

When we run our tests, we find that they are all green now.

The question now remains, how complex an expression could we include in the left hand side? Let us right some tests and see what we can add in.

test "Assert returns :ok when tests pass with a complex lhs function" do
    assert Tahinix.assert(5 * 2 == 10) == :ok
end

test "Assert returns :error when tests fail with complex lhs" do
    assert Tahinix.assert(5 * 2 == 12) == {:error, "Assertion failed: 5 * 2 == 12"}
end

Finally, can we test assertions on the output of functions? For instance,

    iex> Tahinix.assert(MyMod.say_hello == :hello)
    :ok

Test blocks

The next thing we would need to figure out is how to implement a test block in our runner. Let us inspect the AST that ExUnit test case returns when we pass it into quote.

iex> test_ast = quote do
        test "sum test" do
            assert 2 + 2 == 4
        end
    end

{:test, [],
 [
   "sum test",
   [
     do: {:assert, [],
      [
        {:==, [context: Elixir, imports: [{2, Kernel}]],
         [
           {:+, [context: Elixir, imports: [{1, Kernel}, {2, Kernel}]], [2, 4]},
           6
         ]}
      ]}
   ]
 ]}

Remember the breakdown we did earlier on the parts of the return value of quote?

Here we see that the function name is test with no metadata(2nd arg), and its arguments are a do block. All we need to do for our test runner is build a function that returns a similar AST.

We start by writing our tests as usual:

describe "testing test blocks." do
    test "A test block of a passing equivalence test" do
      assert Tahinix.test("equivalence of 2 and 2", do: Tahinix.assert(2 == 2)) ==
               {:ok, ["equivalence of 2 and 2"]}
    end

    test "A test block of a passing non-equivalence test" do
      assert Tahinix.test("non-equivalence of 2 and 2", do: Tahinix.assert(2 != 2)) ==
               {:error, ["non-equivalence of 2 and 2", "Assertion failed: 2 != 2"]}
    end

    test "A test block of a failing equivalence test" do
      assert Tahinix.test("equivalence of 2 and 3", do: Tahinix.assert(2 == 3)) ==
               {:error, ["equivalence of 2 and 3", "Assertion failed: 2 == 3"]}
    end

    test "A test block of a failing non-equivalence test" do
      assert Tahinix.test("non-equivalence of 2 and 3", do: Tahinix.assert(2 != 3)) ==
               {:ok, ["non-equivalence of 2 and 3"]}
    end
end

Now we need an implementation to make the tests green.

defmacro test(test_name, do: block) do
    quote do
      test_name = unquote(test_name)
      block = unquote(block)

      case block do
        :ok -> {:ok, [test_name]}
        {:error, reason} -> {:error, [test_name, reason]}
      end
    end
  end

This function will execute whatever is passed in the do block (in our case we will assume it is an assertion). If the block runs without error, we return an {:ok, [test_name]} tuple. Otherwise, we return a {:error, [test_name, reason]} tuple.

And now we can write blocks of code like

Tahinix.test "Adding 2 numbers" do
    Tahinix.assert(MyFun.sum(2,1) == 3)
end

And it will return whether or not our test has passed.

Conclusion

With all this, we already have a sample test runner with the basics working. Further improvements might be adding in describe blocks, accumulating the test that pass and fail and showing them to the users, and adding in other important functions like refute

And now we also have a better understanding of how metaprogramming in Elixir can help us write utilities like test runners. But remember, the first rule of writing Macros, Don't write Macros

The full repo can be found here

References:

Metaprogramming in Elixir by Chris McCord

Convention over configuration in the age of AI. ( Happy accident ? )

Patrick Wendo — Sun, 13 Jul 2025 18:58:27 +0000

Convention over configuration is a dogma touted in the ruby and rails ecosystem. What does this mean??

Often when working with rails you find that a lot of the libraries focus on building in line with the Rails conventions. This is great! Makes it easier to get up to speed with new libraries in Ruby/Rails.

But now we have LLMs. And while LLMs are great they tend to provide code that doesn't always follow convention. Not always, but it happens.

I read this post for the contribution policy on AI for the Ash Framework in Elixir and it got me thinking. With RAG(Retrieval augemnted generation) trying to control(somewhat) the results AI gives us, does that not mean that systems with built convention would tend to be more correct than those without? Especially because all the code these LLMs would be trained on would have (for the most part) followed convention?

I think that the Rails community was onto something with convention over configuration, not even because of AI. But because it helps developers understand code faster. That AI would learn from this would be a great boon.

What do you think?

Exploring elixir processes using merge sort

Patrick Wendo — Mon, 14 Apr 2025 22:41:51 +0000

In this article, I use the word parallel and concurrent interchangeably. I need the readers to know that they are not the same. Parallel means solving smaller problems of the same chunk of work at the same time. Concurrent means solving multiple problems in what appears to be the same time.

Sorting.

A classic problem in computer science. We always need things in the right order for a variety of reasons. There are many sorting algorithms each with their merits and demerits and some made for the fun of it(I'm looking at you Bogosort). A popular algorithm is merge sort. This algorithm uses a divide and conquer approach, subdividing the list into individual elements and then merging those elements in the correct order. This is a great problem to teach yourself how to get around the recursive nature of functional languages like Elixir or Erlang.

Elixir & its processes.

Elixir runs on the Erlang VM, known for creating low latency, distributed, and fault-tolerant systems. Elixir Docs

A cool thing that Elixir inherits from Erlang are its processes. All Elixir code runs inside processes that are isolated from each other, run concurrently and communicate via message passing. This allows Elixir developers to build distributed and fault tolerant processes. Of importance is the fact that Elixir processes are not Operating system processes. Instead, they are lightweight in terms of memory and CPU, which allows a programmer to run tens or even hundreds of thousands of processes running simultaneously.There is a limit to how many processes you can spawn though(definitely not foreshadowing)

Light weight processes and merge sort

Well, if we have the ability to spawn light weight processes, a functional language and a sorting algorithm which we may be able to parallelize, the obvious next step is to absolutely misuse all of these ingredients until we have a working algorithm, right?

Let's cook

Ok, if I recall from my multicore programming classes, the first thing you want to do when attempting to parallelize a problem that is sequential is to identify which parts of the algorithm can be done in parallel. So we look at the parts of merge sort

Parts of the merge sort algorithm

Find the mid-point of an array and divide the array into two halves, left and right
Perform Mergesort on the left
Perform Mergesort on the right
Merge the two sorted halves(left and right)

Implementation 1

What if in our first solution, every time we want to perform merge sort on a sub_array, we spawn a new process and let it handle that sorting for us? That sounds like a great idea. Let us do this:

  def psort([]), do: []
  def psort([x]), do: [x]

  def psort(list) do
    {sub_arr1, sub_arr2} = Enum.split(list, floor(length(list) / 2))
    parent = self()

    child1 = spawn(fn -> send(parent, {:sorted, self(), psort(sub_arr1)}) end)
    child2 = spawn(fn -> send(parent, {:sorted, self(), psort(sub_arr2)}) end)

    sorted_sub_arr1 = receive_sorted(child1)
    sorted_sub_arr2 = receive_sorted(child2)

    merge(sorted_sub_arr1, sorted_sub_arr2)
  end

  defp receive_sorted(child_pid) do
    receive do
      {:sorted, ^child_pid, sorted_list} ->
        sorted_list

      # {:die} ->
      #   Process.exit(self(), :normal)
    end
  end

  def merge(l1, l2) do
    new_list = has_elements(l1, l2)
    new_list
  end

  def has_elements([], l2, c), do: Enum.reverse(c, l2)
  def has_elements(l1, [], c), do: Enum.reverse(c, l1)

  def has_elements([hd1 | r1], [hd2 | r2], c \\ []) do
    if hd1 > hd2 do
      has_elements([hd1 | r1], r2, [hd2 | c])
    else
      has_elements(r1, [hd2 | r2], [hd1 | c])
    end
  end

Ok, that's a lot of code, but let us focus on this part:

    {sub_arr1, sub_arr2} = Enum.split(list, floor(length(list) / 2))
    parent = self()

    child1 = spawn(fn -> send(parent, {:sorted, self(), psort(sub_arr1)}) end)
    child2 = spawn(fn -> send(parent, {:sorted, self(), psort(sub_arr2)}) end)

We split the list into 2 sub_arrays and then spawn 2 processes to implement the merge_sort on those sub_arrays. `{:sorted, self(), psort(list)} is a message pattern that the parent process will expect to receive. You can read more about sending and receiving messages for processes in Elixir here

Benchmarking implementation 1 vs a sequential sort algorithm.

You only ever see the benefits of parallelization when you get some benchmarks and comparison. So let us compare our implementation with a sequential merge sort algorithm.

In our benchmark, we do 40 runs of random lists with varying sizes for both the sequential and parallel implementations.

Wait, that's not what we want. Our parallel execution is consistently slower, regardless of the number of elements. We wanted to see some form of efficiency. Why?

Processes are not a free lunch

Typically, parallelization will incur some overhead. Spawning process takes some time. Message passing takes some more time. All the while the sequential algorithm just chugs along undisturbed(so to speak).

The way to use processes, it to use them when they would offer a clear benefit over a a sequential execution. So how do we do that?

Implementation 2

What if we kept the implementation 1 method, but only do it depending on the size of a list passed. This means that is we want to sort a list of 100 elements, we could do this sequentially, but if the list has 1_000_000 elements, perhaps we could split the load among multiple processes?

We add a simple check before splitting the array and instruct it to use our sequential merge sort implementation

if length(list) <= @list_threshold do SequentialMergeSort.seq_sort(list) else ... end
I set the threshold at 1000 elements in a list.

This gives us the following results.

What we notice is that all we end up with a similar trend to the sequential algorithm. Which means that we are not really getting any speed up. And if we are, it is practically negligible. Interestingly though, the initial parallel implementation fails when we get to 1 million elements in a list because we spawn too many processes. But the simple solution allows us to go up to a million with no ill effect. So there is some benefit, but there has to be a better improvement somewhere.

I will be working on this a bit more to find out more performance optimizations where I can find them. Feel free to checkout and contribute to the github repo.

Using regex named capture groups to process lines in a CSV file in ruby.

Patrick Wendo — Sun, 16 Mar 2025 07:07:50 +0000

Imagine you are processing a CSV file with information from weather stations. single line might look like this

Hamburg;22.0,Berlin;18.45,Tokyo;11.23,New York;4.20\n

The regex library in ruby allows for named captures. What this means is that you can specify a name for the text being matched. For instance

/(?<city>[\w\s]+)/

The ?<city> is the name of the capture group w+(\s\w+).

If we run the code

"Hamburg;22.0".match(/(?<city>[\w\s]+)/).named_captures

This will return a ruby Hash that looks like this

{"city"=>"Hamburg"}

We can use multiple named captures like this. However, if we want to return all matched captures in the string, we use the .scan method instead. This will return an array of arrays, with each internal array being a matched capture.

For instance:

"Hamburg;22.0,Berlin;18.45,Tokyo;11.23,New York;4.20\n".scan(/(?<city>[\w\s]+);(?<temp>\d+(\.\d+)?)/)

will return

[["Hamburg", "22.0"], ["Berlin", "18.45"], ["Tokyo", "11.23"], ["New York", "4.20"]]

This makes it markedly easier to process the data.

Named captures are pretty cool.

Reference:
Regex Named Captures
Rubular (Regex playground for Ruby)

Using meta-programming in Ruby to build a REST API from a JSON file

Patrick Wendo — Mon, 21 Oct 2024 17:47:50 +0000

So I asked ChatGPT, as you do, for stuff I could do with meta-programming in Ruby. If you don't know Ruby has amazing meta-programming support. And meta-programming is basically code that can modify itself while it's running. if you want to get technical with it, meta-programming is a computer programming technique in which computer programs have the ability to treat other programs as their data. Sometimes the program being used as data is the program running. Ruby excels at this.

In this post I would introduce you to AutoAPI. The goal is that you could just write your endpoint specifications in a JSON file and then the program starts a sinatra server that then has all the endpoints. As of now, the program only works for GET endpoints and I will update it as I go. It also returns either JSON or static HTML files. Further updates would be to serve other MIME types. Let's begin.

First of all, I wanted to run this as a shell script (cause I want to practice my scripting) so we start that shebang #!/usr/bin/env ruby at the top of our file. Because I wanted it to be a shell script that would run from any folder, we have the script looking for a json file called endpoints.json in the current folder.

# Get the file and parse the entire file into a ruby hash
file = File.read("#{Dir.pwd}/endpoints.json")
endpoints_hash = JSON.parse(file)

We use JSON to parse the file into a ruby Hash. This provides us with quality of life methods that we could use later, should we need to.

Now because we are using Sinatra as our server, we would need a way to dynamically define new endpoints from the file. Sinatra is a DSL for quickly creating web applications in Ruby with minimal effort.

Let's first think about the method send or public_send. What these methods do is that they basically perform a method call to a class instance. For instance, if we have a class

class Class
    def hello
        puts "hello"
    end
end

If we did class.send(:hello) our result would be hello.
You could also pass parameters to send if the method takes in any arguments.

To create a GET endpoint in Sinatra, we would write

get '/hello' do
  'Hello world!'
end

we could dissect this and find that :get is the method name, '/hello is the path name and everything in the do block we shall call the do_block. We could therefore also mentally re-write this as

get('/hello') do
    'Hello World'
end

send(:get, '/hello', &block)

where &block is is the do_block passed to the send method.

All this to say, we need to define a method that will read the endpoint names and their associated methods to define new routes. It looks like this

def create_endpoint(method, name, &block)
  Sinatra::Application.instance_eval do
    name = "/#{name}" if not name.start_with?(/\//)
    send(method, name, &block)
  end
end

It takes in the REST method(GET, POST, PUT, DELETE), endpoint name and a code block. It then uses instance_eval to create a new route on the running instance of Sinatra. I added a check to ensure that the the endpoint name is preceded with a forward-slash because I ran into this random bug where the route would seem to be defined, but not accessible because it does not start with a forward-slash. finally, we just send the method as shown above. Simple, right?? Honestly, it's that simple.

Now we should specify that the endpoints.json file has a specific structure.

{
    "GET": {
        "json_response": {
            "header": {"Content-Type": "application/json"},
            "response": {
                "content": { "message": "Hello AutoAPI"},
                "file": false
            }
        },
        "json_response_file": {
            "header": {"Content-Type": "application/json"},
            "response": {
                "file": true,
                "content": "endpoints.json"
            }
        },
        "html_file": {
            "header": {"Content-Type": "text/html"},
            "response": {
                "file": true,
                "content": "test.html"
            }
        }
    }
}

This is a sample of the endpoints file. We shall go through each endpoint one at a time. So all the REST verbs act as keys, that way all the GET routes are in the GET values etc etc. Our first endpoint is the json_response. This one was supposed to test if I could get a hard coded response. The header is a nested object containing what you would expect in a typical HTTP GET request. Here we are passing the Content-Type only. For the response, we have a file key that specifies if the endpoint should send a file, or if it should send the value of content as JSON. This file should also be in the same folder as the script when running it. In this example, the second endpoint actually just returns the endpoints.json file. The third endpoint has a different content type and returns a HTML file.

To process these endpoints, this is the code I wrote

endpoints_hash.each do |method, paths|
  paths.each do |path, params|
    create_endpoint(method.downcase.to_s, "#{path.to_s}") do
      content_type :json if params["Content-Type"] == "application/json"
      content_type :html if params["Content-type"] == "text/html"
      if params["response"]["file"]
        send_file "#{Dir.pwd}/#{params["response"]["content"]}"
      else
        params["response"]["content"].to_json 
      end
    end
  end
end

All it does is iterate through all the endpoints and paths and defines routes for them. Note that I have to specify the content_type and I just use the passed headers to figure that out. The code might be a bit breakable and is a work in progress, but thus far, it works as seen below

For all 3 defined routes, we manage to get a response. And all with about 29 lines of code.

All code is available at my github

Random VS Code finds

Patrick Wendo — Fri, 11 Oct 2024 08:55:04 +0000

Code OSS, (or VS Code) as an IDE are both really powerful when you get to leverage their available extensions. If you follow the VS Code tiktok you always find helpful tips and extensions that come in handy. In this series I will try and highlight some of my favourite extensions.

Thunderclient

If you develop APIs and you want to test as you go, Thunderclient is a great alternative to Postman or Curl. With it being within your IDE you don't have to jump between applications whenever testing is required. I came across it when I was building a rails backend for a react application I was working on. Their documentation is easy enough to understand. If you are in any way versed with Postman, you will feel right at home with Thunderclient.

Todo Tree

I write a lot on paper when I am working on a problem. The problem is usually that when I need to move around my work always gets messed up. I have also been in meetings where I add all my todos in a todo.txt file. Todo Tree however is a bit of a saving grace. By just adding a comment starting with the word todo for instance

// TODO write tests for this function

I can then check the todo side tab and see all the todos left for me to do. Just don't forget to clear your todos before pushing to prod :)

Drawio Integration

During my college days I used to use Drawio to draw wireframes and flowcharts. When I found that there is a VS Code extension that allows me to do it in the IDE it was a no brainer. I have found it is also useful whenever I am screen sharing to use it as a whiteboard during meetings. All you have to do is create a new file with the .drawio extension and you're off to the races. You can then export to .svg and .png as you see fit. However, it is an unofficial extension, so you should beware.

Tailwindcss Intellisense

This was a recommendation from a friend. Tailwindcss Intellisense provides insight into which CSS properties your tailwind code affects. Great for anyone who does a lot of frontend with tailwindcss.

What are some fun or scary vulnerabilities you heard about recently??

Patrick Wendo — Mon, 12 Feb 2024 11:18:40 +0000

Rails is Pretty Intuitive once you get used to it.

Patrick Wendo — Mon, 21 Aug 2023 10:47:53 +0000

I am currently working with background jobs in Rails through Active Job. And I needed some code to run before the job starts. So I thought,

Typically when you are working with the controllers and you want to run some code before an action you have a before_action callback. In stimulus reflex, we have the before_reflex callback. It should be safe to assume that Active Job must have a before_perform callback

So I just set that up real quick and what do you know it run. I checked the docs just to make sure I was not having some vivid hallucination and true enough there is a before_perform callback, as well as a couple other useful ones.

This may not seem like much, but the ability to guesstimate what type of functions may be defined on a class is one of the few things I love about rails and ruby in general. It's like how the Enumerable class from which arrays, sets and hashes all inherit allows you to know that there are some methods that will always exist between these classes. And how a function ending in a question mark will return a boolean while one ending in an exclamation mark will do some potentially destructive action on whatever object calls it.

Rails and Ruby is pretty awesome when it comes to the developer experience.

Fly.io: Just 2 commands and the app is live.

Patrick Wendo — Sun, 30 Jul 2023 15:54:07 +0000

I am a rails developer. I started out my developer life as a rails developer. I mention this because ruby on rails helps you build a lot very fast. If you are looking for the fastest way to move from idea to MVP, rails is almost always one of the top suggestions. When I wanted to pivot to mobile dev, I started with React Native, but I felt a lot more comfortable working with flutter. Then now, I was working on the ruby on rails hackathon this past weekend and one of the goals was to try out fly.io as a hosting provider. And well, fly.io gave me the same level of comfort that rails and flutter do. That comfort I think comes from the adherance, at least somewhat, to the doctrine of convention over configuration.

Ruby on rails works hard to scaffold a lot for you. Similarly to flutter compared to React Native at least. And now, I find that Fly does the same thing. When I was using heroku and render to host my projects, there is often a lot of jumping between my terminal, and the browser in order to set up a postgres database, a redis server and to actually set up the rails server. Fly.io just gives you the cli tool flyctl and gets you up and running in no time flat. Let me explain.

Authorize the login for `flyctl`

After you download the tool and install it you would need to login using the command flyctl auth login. If you have an account, it will open up your browser and log you in, and this is the last time you may need to visit your browser until your deployment is done. (I promise)

Let's get deploying.

So now we stay within the terminal. Make sure you're in the working directory of your project. In this case it is a rails project. First command we run is fly launch. This command will detect the type of app within the directory, be it rails, Phoenix, Django or Laravel. The following are a list of prompts it will ask you.

If a configuration file(fly.toml) is found, it will as if you want to copy the configuration to the new app
The name of your app.
If the app already exists, it will ask if you want to launch into that app.
The region in which you want to deploy your app
Whether or not you want to create a postgres database for your app
Whether or not you want to create a redis server for your app

And that's about it for that command. Flyctl tool will do the rest.

Once that part is done, the next command we run is fly deploy
The fly deploy command gets the app name from the fly.toml file. Then flyctl will start the process of deploying your application to the Fly Platform and return you to the command line when it’s done. If this command runs successfully, then the app has been successfully deployed. The last command you can run is fly open and this will open your app on your your browser.

So that's it. No really, that's it. 2 commands and your app is up and running. Idk about you, but I think it's pretty amazing.

Conclusion

This is the first time I have used fly.io and I think that the developers over there are putting in a lot of work to help with the developer experience of deploying. The great thing is, a lot of the configuration files are also copied to your local machine allowing you to make changes as you see fit, but if you are looking to get from idea to MVP, I think that using fly.io for hosting is such a win.

To the team at fly.io, if you're reading this, ABSOLUTELY STUNNING WORK!!!.

Convergent Evolution in Code.

Patrick Wendo — Fri, 09 Dec 2022 19:32:07 +0000

Convergent evolution is when two different and separate organisms evolve the same mechanism to solve a similar problem. Like wings in insects and birds, or fins in Whales and fish.

Recently I was explaining to my friend, a JavaScript developer, about a helper function I wrote in Rails that would auto-generate rows on a table given the correct data. Their response was,

"Congratulations, you have built your first react component"

Now, I have used react before, but it did not occur to me at all that by building this helper in Rails was akin to building a component in react. So my response was,

" Did I build a react component, or is it convergent evolution? "

Regardless, it made me wonder, what other pieces of code or way of doing things have we as developers all converged upon, regardless of language or framework?

Resist the urge to push and move on immediately.

Patrick Wendo — Mon, 21 Nov 2022 15:27:16 +0000

It never hurts to read through your code one more time. But how you read through it is fairly important too. I have gotten used to the mentality of "make sure it works then optimize later", which is fairly ok for the most part. But sometimes, I will write a piece of code that works, tell myself I will optimize it later, then start working on something else entirely and I forget to optimize my code. Which would leave me having to try and explain why I did not do a rather obvious optimization.

So for the past couple weeks, I have decided to commit my code, but not push until I have re-checked my work. How does this look like? Well, say I am implementing feature A today. I will write the tests, and try to make them pass. Once the tests are passing, I will commit and walk away from my code( By walk away I mean literally not even look at anything remotely associated with the codebase.) I'll watch a youtube video or go outside for a well deserved break. The goal here is to make sure that when you go back to your code, you have a fresh pair of eyes on the issue. A hack I was taught by @jeremyf , is to just walk through a door way. No idea why it works, but it does.

Coming back I will ask myself, "What does this line/block do? Is it critical?" for each line or block of code I had previously written. This helps me remove those pesky bits of dead code or random logs. Similarly, I will go through the tests and start thinking about the random edge cases. If it is a UI issue, I will ask myself, "is this the best look for the system?" "Can it perhaps be made to look better than this?", and other questions pertaining to UX.

Making sure your code works correctly seems like a pretty obvious thing to do, but it's always good to remind yourself the goals that you are striving for in your code. Makes for much better and well thought out code.

Forem: Patrick Wendo

Simulating Boids with ECSx and liveview

Entity Component Systems for Simulations

What are Entity component systems??

Entities

Components

Systems

Putting it all together

ECSx

Boids and our simulation.

Designing the systems

Alignment System

Cohesion System

Separation System

Movement System

The ECSx Manager.

Showing the boids

Performance optimizations

Hash Partitioning

Using Nx

Benchmarking

Discussion

Conclusion

Let's build a test runner in Elixir.

Quote

Unquote

Building the test runner.

Test blocks

Conclusion

Convention over configuration in the age of AI. ( Happy accident ? )

Exploring elixir processes using merge sort

Sorting.

Elixir & its processes.

Light weight processes and merge sort

Let's cook

Parts of the merge sort algorithm

Implementation 1

Benchmarking implementation 1 vs a sequential sort algorithm.

Processes are not a free lunch

Implementation 2

Using regex named capture groups to process lines in a CSV file in ruby.

Using meta-programming in Ruby to build a REST API from a JSON file

Random VS Code finds

Thunderclient

Todo Tree

Drawio Integration

Tailwindcss Intellisense

What are some fun or scary vulnerabilities you heard about recently??

Rails is Pretty Intuitive once you get used to it.

Fly.io: Just 2 commands and the app is live.

Authorize the login for flyctl

Let's get deploying.

Conclusion

Convergent Evolution in Code.

Resist the urge to push and move on immediately.

Authorize the login for `flyctl`