Forem: bitcrowd

Getting React on Rails to work with Turbo Streams

Max — Thu, 15 Jan 2026 09:30:54 +0000

On a recent client project, redesigning a large Rails app, we found ourselves having to get Rails, Hotwire's Turbo and React on Rails to play together nicely. Each of the tools comes with its own amounts of magic™, so integrating them turned out to be a bit of a juggle.

Scenario

On this project, we were implementing the new design for a particular page. At it's core, it shows two lists, each with a "Load More" button to asynchronously fetch and append more items to it. The first list is just a stack of items, the second one also includes a grouping of the items. Imagine it roughly like this:

" button below. List B is divided into groups Group A, Group B and Group C"/>

In "List A", clicking "Load More" should append another batch of 5 items to the bottom of the list until the bitter end. For "List B", clicking "Load More" should also append another batch of 5 items, where some may still belong to "Group C" and others will form new groups, each with their respective group header. The list items are heterogeneous and contain complex markup with graphics, etc.

Pagination

We decided to implement the pagination logic using Turbo Streams, as they seem to be a good fit for our use case. They also integrate nicely into the Rails ecosystem without being too tightly coupled (e.g. to a specific version of Rails). From Rails 7 on, Turbo also replaced Rails' UJS (Unobtrusive JavaScript) functionality and is the recommended tool for sprinkling asynchrony and SPA (Single Page Application)-feelings. From the Turbo Stream docs:

Turbo Streams deliver page changes as fragments of HTML wrapped in <turbo-stream> elements. Each stream element specifies an action together with a target ID to declare what should happen to the HTML inside it. These elements can be delivered to the browser synchronously as a classic HTTP response, or asynchronously […]

And further on:

They can be used to surgically update the DOM after a user action such as removing an element from a list without reloading the whole page […]

This sounds like a match for what we are doing!

By having our server respond with turbo_stream templates for the pagination requests, we are able to use the same View Components and partials we already have in place from the initial server-rendered page. This is important to us because we're already using our component library throughout the whole project. Reimplementing the same components in a different framework or language just for this single page would be unnecessary duplication, not to mention the constant risk of the two implementations diverging.

Example

For the sake of simplicity, let's focus on the first list, "List A". The initially rendered page would have a markup like this for the list:

<h2>List A</h2>
<ul id="list-a">
  <li><abc-item/></li>
  <li><abc-item/></li>
  <li><abc-item/></li>
  <li><abc-item/></li>
  <li><abc-item/></li>
</ul>
<turbo-frame id="load-more-a">
  <load-more-button/>
</turbo-frame>
<!-- ... -->

When the user clicks the "Load More" button, we want to retrieve the next batch of items, append them to the list and replace the button with one to request the next batch. If there is no next batch of items, we remove the "Load More button.

In the controller taking care of the pagination request, we can render a response template of the format turbo_stream. A simplified version could look like this:

<%= turbo_stream.append 'list-a' do %> <%= render
AbcItem.with_collection(@items, as: :item) %> <% end %> <% if @pagy.next %> <%=
turbo_stream.replace 'load-more-a', method: :morph do %>
<turbo-frame id="load-more-a">
  <%= render LoadMoreButton.new(path: '/', params: { page: @pagy.next }) %>
</turbo-frame>
<% end %> <% else %> <%= turbo_stream.remove 'load-more-a' %> <% end %>

Note that the example is View Components and the pagy pagination library. In the template, we're appending items to "List A" and replace the "Load More" button with a new one. If there aren't any items left the "Load More" button gets removed.

The response from the server would then look roughly like this:

<turbo-stream action="append" target="list-a">
  <template>
    <li><abc-item/></li>
  </template>
</turbo-stream>
<turbo-stream action="remove" target="load-more-a">
</turbo-stream>

It appends one item to the list and removes the button to load more.

Problem

Our <abc-item/> element contains a React component, MyComponent in its markup, rendered via React on Rails' react_component helper. In the final HTML those React components appear as three elements each. Two script tags with context and props and one wrapper element for the rendered HTML of the component:

<script type="application/json" id="js-react-on-rails-context">
  …
</script>
<div id="MyComponent-react-component-eaca5b56-0732-4674-b9da-f858c224d410">
  <!-- MyComponent markup -->
</div>
<script
  type="application/json"
  class="js-react-on-rails-component"
  data-component-name="MyComponent"
  data-trace="true"
  data-dom-id="MyComponent-react-component-eaca5b56-0732-4674-b9da-f858c224d410"
>
  …
</script>

For any <abc-item/> element added to the page via Turbo Streams, those three elements are also added to the DOM, but the wrapping <div> stayed empty: the component wasn't actually mounted! 🙀

<script type="application/json" id="js-react-on-rails-context">
  …
</script>
<div
  id="MyComponent-react-component-eaca5b56-0732-4674-b9da-f858c224d410"
></div>
<script
  type="application/json"
  class="js-react-on-rails-component"
  data-component-name="MyComponent"
  data-trace="true"
  data-dom-id="MyComponent-react-component-eaca5b56-0732-4674-b9da-f858c224d410"
>
  …
</script>

Turbo is injecting markup into the already fully rendered page and none of the DOM events React on Rails is hooking into to execute the JavaScript to mount React components on the page are being fired.

Options

React on Rails already has support for Turbo Streams, but only as part of their paid pro plan. Going "pro" wasn't an option for the project as they are on a tight budget and only using small parts of React on Rails in isolated areas of their app. So we're not really qualifying for "pro" here…

Server-side rendering the React components would also have been an option to tackle the problem. But this again was out of scope for our project and we did not want to run JavaScript on the server.

So what now?

JS to the rescue

With the help of some AI coding agents, we explored the internals of the react-on-rails and @hotwired/turbo-rails NPM package to see what was actually happening. The rough idea was that, if any events were emitted by Turbo when Stream elements were injected and rendered, we could hook into them and then manually call React on Rails to mount all the newly inserted components on the page.

In Turbo, we found the StreamElement. It has a render function and a target pointing to the ID of the DOM-element it's rendered to. Combining these two things, the plan was to extend the render function to also take care of mounting any React elements within the updated target element. This means, when the StreamElement contained any React components, we want to make sure they are correctly mounted after the rendering of the StreamElement itself was done.

/**
 * Gets the current target (an element ID) to which the result will
 * be rendered.
 */
get target() {
  return this.getAttribute("target")
}

So we wrote our own tiny JS module to patch Turbo with some additional logic for our React on Rails integration:

// First we get hold of the original render function of Turbo's StreamElement
const originalRender = Turbo.StreamElement.prototype.render;

// Then we override it to extend it with our own additional logic:
Turbo.StreamElement.prototype.render = function () {
  // First we call the original render function
  originalRender.call(this);

  // Then we get hold of the target element
  const target = document.getElementById(this.getAttribute("target"));

  // Then we wait for the next frame, to make sure that the DOM is fully loaded
  // and we call a function to have React on Rails mount all components within
  // the newly inserted DOM snippet.
  if (target) {
    requestAnimationFrame(() => mountReactOnRailsComponents(target));
  }
};

The custom mountReactOnRailsComponents function is pretty much a manual re-implementation of what's happening when React on Rails is client-side rendering components:

function mountReactOnRailsComponents(domNode) {
  domNode
    // First we get all React on Rails components within our node
    .querySelectorAll("script.js-react-on-rails-component")
    .forEach((script) => {
      // Next we get the name of the component and the DOM ID of the placeholder
      const name = script.dataset.componentName;
      const { domId } = script.dataset;
      const shouldHydrate = script.dataset.hydrate === "true";

      // Then we put together the component props
      let props = {};
      try {
        props = JSON.parse(script.textContent || "{}");
      } catch (error) {
        console.error("ReactOnRails props JSON parse error for", name, error);
        return;
      }

      // Now we need the placeholder element
      const placeholder = document.getElementById(domId);
      if (!placeholder) return;

      // We check whether the component was already mounted
      // in this case we don't want to re-render it.
      const hasMounted = placeholder.dataset.reactOnRailsMounted === "true";
      if (hasMounted) return;

      // Then we call React on Rails' render function to mount the component
      ReactOnRails.render(name, props, domId, shouldHydrate);

      // Finally, we mark the placeholder as mounted, so that we don't accidentally
      // re-render it.
      placeholder.dataset.reactOnRailsMounted = "true";
    });
}

With this in place, our Turbo Streams based pagination works like charm. Any React component newly added to the DOM is mounted as normal 🎉.

Conclusion

Drilling around like this in the internals of Turbo and React on Rails of course yields its risks. Now we're maintaining additional code which may break any time we update our Turbo or React on Rails dependences. But as we're only using the module in a small isolated area of the application, it's a risk we can accept for now. Our test coverage of the feature will inform us early enough.

Using open source technologies, we have the chance to peak under the hood of the tools and can if necessary adapt them to our needs. And working with legacy codebases and legacy Rails apps, their may be situations with unconventional needs…

We hope this post was helpful and inspires you to explore the details of the tools you're using the next time you hit a wall with them. Migrating to an entirely different tool is often a way bigger operation, especially in large monoliths.

Loading Chrome extensions for development in 2025

Max — Wed, 03 Dec 2025 08:25:24 +0000

For loading a development build of our open source browser extension Tickety-Tick into Google Chrome during local development, we used to use Chrome's handy --load-extension flag. It allowed to load a local unpacked build of the extension via the command line. As this feature was recently removed from all branded builds of Chrome, we needed to update our workflow to use the-new-way™ - "remote debugging pipes".

Background

Tickety-Tick is on the nichier side of tech within our fleet of open source projects: it's a browser extension for establishing conventions on naming branches and writing commit messages in Git. And while we're slightly leaning towards team Firefox among us developers, Tickety-Tick is of course also supporting Google Chrome (and others…). For this, we as developers have to be able to run a local build of the extension in Chrome during development.

The old way

For local development, we had a yarn open:chrome command which would open Chrome with the current local build of the extension already installed, ready to click around. Under the hood, we utilized Google's chrome-launcher package to start a fresh, unpersonalized instance of Chrome for us. The code looked about like this:

const chromeFlags = ChromeLauncher.Launcher.defaultFlags()
  .filter((flag) => flag !== "--disable-extensions")
  .concat(["--no-default-browser-check", "--load-extension=/dist/chrome"]);

ChromeLauncher.launch({chromeFlags}).then((chrome) => console.log("Chrome running")

The important part is the --load-extension flag which points to the local Chrome build of the extension.

Some day, this just stopped working. Chrome would start as usual, but without the browser extension loaded.

What happened?

Some debugging and internet research brought us to this post by the Chrome team from March 2025:

RFC: Removing the --load-extension flag in branded Chrome builds

Turns out the --load-extension flag was commonly abused by malicious parties and the Chrome team was looking into ways to further restrict this. It lead them to removing this feature from all branded Chrome builds (so Chromium would still have it…) and instead promote other means for loading extensions for testing and development.

What now?

To figure out the new™ way to load an unpacked, locally build extension in Chrome, we just naively searched for the --load-extension flag on GitHub, hoping to find hints how other people updated their code. A promising hit was in the chrome-launcher package itself. If somebody knew how to do it, they should be the ones. The searching pointed us to a really helpful code comment in their tests:

// Note: --load-extension in chromeFlags used to be the primary method of
// loading extensions, but this is removed from official stable Chrome builds
// starting from Chrome 137. This shows the officially supported way to load
// extensions, with --remote-debugging-pipe.
// See: "Removing the `--load-extension` flag in branded Chrome builds"
// https://groups.google.com/a/chromium.org/g/chromium-extensions/c/aEHdhDZ-V0E/m/UWP4-k32AgAJ

As this was in a load-extension-test.ts file, we figured we'd just try to adopt what they are doing in their test setup for our open:chrome script. And as the comment indicates, we need to utilize the --remote-debugging-pipe flag, something we already knew from ChromicPDF our PDF rendering library for Elixir based on headless Chrome.

Following pretty much exactly what the loadExtension via remote-debugging-pipe test was doing, we ended up with an updated open-in-chrome script which would:

Start Chrome with the respective flags

   const chromeFlags = launcher.Launcher.defaultFlags()
     .filter((flag) => flag !== "--disable-extensions")
     .concat([
       "--remote-debugging-pipe",
       "--enable-unsafe-extension-debugging",
       "--no-first-run",
       "--no-default-browser-check",
     ]);

   const options = {
     chromeFlags,
     ignoreDefaultFlags: true,
     startingUrl: url,
   };

   const chrome = await launcher.launch(options);

Get hold of the debugging pipes

   const pipes = chrome.remoteDebuggingPipes;
   if (!pipes) {
     throw new Error("Chrome did not expose remoteDebuggingPipes");
   }

Put together the command to load the extension

   const requestId = Math.floor(Math.random() * 1e6);
   const request = {
     id: requestId,
     method: "Extensions.loadUnpacked",
     params: { path: dir },
   };

Send the request and listen on the pipes

   const firstResponse = new Promise((resolve, reject) => {
     let buffer = "";

     pipes.incoming.on("error", reject);
     pipes.incoming.on("close", () =>
       reject(new Error("Pipe closed before response")),
     );

     pipes.incoming.on("data", (chunk) => {
       buffer += chunk;
       let end;
       while ((end = buffer.indexOf("\x00")) !== -1) {
         const message = buffer.slice(0, end);
         buffer = buffer.slice(end + 1);
         try {
           const parsed = JSON.parse(message);
           if (parsed.id === requestId) {
             resolve(parsed);
           }
         } catch {
           // ignore non-JSON noise
         }
       }
     });
   });

   pipes.outgoing.write(JSON.stringify(request) + "\x00");

Check that the request was successful

   const response = await firstResponse;
   if (response.error) {
     throw new Error(`Failed to load extension: ${response.error.message}`);
   }

With that a fresh Chrome should be running with our extension loaded! 🎉

Our script

In our final script, we also added some emoji-powered checks and debugging statements, so that the CLI output aligns with our other tasks and the webpack build. Here's our final script, maybe you can draw inspiration from it for your own usecase:

#!/usr/bin/env node

// usage: open-in-chrome [extension-dir] [starting-url]

import * as path from "path";
import * as launcher from "chrome-launcher";

const dir = process.argv[2] || path.join(__dirname, "..", "dist", "chrome");
const url = process.argv[3] || "https://github.com/bitcrowd/tickety-tick";

async function launchChrome() {
  const chromeFlags = launcher.Launcher.defaultFlags()
    .filter((flag) => flag !== "--disable-extensions")
    .concat([
      "--remote-debugging-pipe",
      "--enable-unsafe-extension-debugging",
      "--no-first-run",
      "--no-default-browser-check",
    ]);

  const options = {
    chromeFlags,
    ignoreDefaultFlags: true,
    startingUrl: url,
  };

  const chrome = await launcher.launch(options);

  if (chrome.port !== 0) {
    console.warn(
      "⚠️ Expected remote-debugging-pipe mode on port 0, but got a debug port.",
    );
  }

  const pipes = chrome.remoteDebuggingPipes;
  if (!pipes) {
    throw new Error("Chrome did not expose remoteDebuggingPipes");
  }

  console.log("🚀 Chrome launched with remote-debugging-pipe.");
  console.log(`📂 Loading extension from: ${dir}`);

  const requestId = Math.floor(Math.random() * 1e6);
  const request = {
    id: requestId,
    method: "Extensions.loadUnpacked",
    params: { path: dir },
  };

  // --- Send command and wait for response
  const firstResponse = new Promise((resolve, reject) => {
    let buffer = "";

    pipes.incoming.on("error", reject);
    pipes.incoming.on("close", () =>
      reject(new Error("Pipe closed before response")),
    );

    pipes.incoming.on("data", (chunk) => {
      buffer += chunk;
      let end;
      while ((end = buffer.indexOf("\x00")) !== -1) {
        const message = buffer.slice(0, end);
        buffer = buffer.slice(end + 1);
        try {
          const parsed = JSON.parse(message);
          if (parsed.id === requestId) {
            resolve(parsed);
          }
        } catch {
          // ignore non-JSON noise
        }
      }
    });
  });

  pipes.outgoing.write(JSON.stringify(request) + "\x00");

  const response = await firstResponse;
  if (response.error) {
    throw new Error(`Failed to load extension: ${response.error.message}`);
  }

  console.log(`✅ Extension loaded (id: ${response.result.id})`);
  console.log(`🌐 Opening: ${url}`);

  chrome.process.on("exit", () => {
    console.log("💨 Chrome closed.");
    process.exit(0);
  });
}

launchChrome().catch((err) => {
  console.error("❌ Error:", err);
  process.exit(1);
});

What we learned

Comments in code can make other people's lives significantly easier. Think about it the next time you're hesitating to write one!

And: searching GitHub for something as specific as a Chrome flag, can yield helpful results in any kind of file, even in tests.

Meet Tickety-Tick

Max — Mon, 04 Nov 2024 11:01:57 +0000

Tickety-Tick is a tool so essential and at the same time so natural and obvious for our work at bitcrowd, that we actually never really introduced it much. So here is a very late “meet Tickety-Tick”, a tool we love, take for granted and whose usage is part of the collective muscle memory at bitcrowd since what feels like forever ❤️.

Basics

Tickety-Tick is quite an exotic member of the fleet of open source projects at bitcrowd: itʼs a browser extension to generate Git branch names and commit messages for the ticket one is working on. It allows us to align branch and commit message naming patterns throughout all projects and have a fixed and easy to follow relationship between the ticket - may it be a bug description or a feature - and its corresponding change in the source code, regardless of the ticketing system used and its possibilities or impossibilities to integrate with code repositories.

The idea is based on the “Convention over Configuration” design pattern which used to be super popular in the Ruby community where bitcrowd hung out too in 2015 when the project was born. You agree upon a convention on how to name branches and commit messages and then have a tool in place to take care of enforcing this convention without you ever having to think about it or alternative ways to name branches and commit messages ever again.

If your into history, there is also a slidedeck Bodo, the original author of the browser extension, used for a lightning talk at Ruby Conf Portugal in 2016. I was in the audience back then and immediately sold on the idea 🥰.

Convention

At bitcrowd, branch names follow the format of <type>/<id>-<title> and the subject of the first commit message on that branch would be [#<id>] <title>:

type: the type of ticket

This may be feature, bug, task, chore or whatever else the ticketing system uses.
id: the identifier of the ticket in the ticketing system

This may be TT-5045 in Jira, a numeric ID like on GitHub, a UUID on Notion, etc.
title: the title or subject of the issue

On GitHub, this would for instance be the issue title, something like “Add support for Safari”. When used for the branch name, the title gets lowercased and dasherized to add-support-for-safari.

So for an imaginary feature ticket about adding support for Safari, Tickety-Tick would generate the branch name feature/tt-504-add-support-for-safari and a commit message subject [#TT-504] Add support for Safari, when used with the default configuration.

Origin

The convention was derived from existing workflows at bitcrowd. And while having aligned branch names already makes sense for collaboration alone, the aligned commit message subject is a bit more special. Itʼs useful for us at bitcrowd, because we squash-merge 99.9% of our pull requests. This means each merged pull request is represented by a single commit in the Git history. Respectively, each ticket we work on results in a single commit and the history can be read as something like this:

[#504] Add support for Safari
[#503] Fix installation instructions for Chrome
[#501] Remove cookie banner
…

We also use the generated subject as the title for our pull or merge requests and for time-tracking. But more on that later…

Workflow

Letʼs walk though a possible workflow together.

Setup

First we need to add the Tickety-Tick web extension to our browser. Install it via Firefox Add-ons, the Chrome Web Store or follow the manual installation instructions for Chrome, Opera, Firefox or Safari. Then we need an ticket to work on. Tickety-Tick supports a constantly growing range of ticketing systems from GitHub to YouTrack.

Usage

Letʼs say weʼre working on Tickety-Tick itself. We have the project checked out locally, are on the latest changes of the main branch and have a terminal window open in the projectʼs root directory. Then we browse the open issues on GitHub:

We want want to work on the first issue to migrate the ESLint configuration (itʼs already assigned to me…) and click on the issue to read what needs to be done:

The description is only pointing to a series of blog posts from the ESLint team explaining the new default configuration format. Letʼs assume we read it and are ready to start working on the ticket. So we open the Tickety-Tick browser extension either by clicking on the ticket icon in our browserʼs tool-bar or, if weʼre already among the power users, use the nifty ctrl t keyboard shortcut to open the extension popup:

It offers three buttons: one to copy the generated branch name to the clipboard, one to copy the generated commit message and one to copy the “command”. We click the last button, and paste into our terminal:

git checkout -b 'feature/427-migrate-eslint-config-to-flat-format' && git commit --allow-empty -m '[#427] Migrate ESLint config to flat format

https://github.com/bitcrowd/tickety-tick/issues/427
'

Power users / keyboard lovers can also tab through the buttons of course. No need to reach the popup window with the mouse.

Broken down into pieces, this will:

Check out a new branch with the name feature/427-migrate-eslint-config-to-flat-format
Create an empty commit with this message:

   [#427] Migrate ESLint config to flat format

   https://github.com/bitcrowd/tickety-tick/issues/427

Check our post on commit messages for more info about commit messages, subject vs. body, formatting, etc.

With that, we can start coding away. Whether we decide to later amend the initial commit, add our changes and write a proper commit message body, or if we just keep the empty commit as a starting point for a series of smaller commits afterwards, that depends on the use case and personal preference.

I personally only amend the empty commit with my changes if the changes are super small. Otherwise, I prefer to have a series of commits which follow my thought process and just keep the empty commit as a starting point for my pull request.

Done?

Once done with the implementation, we can open a pull request with our changes. If our changes are small enough to fit into a single commit, GitHub will even pre-fill the pull request title with the commit messageʼs subject [#427] Migrate ESLint config to flat format and the description with the commit message body we provided.

We at bitcrowd reuse the first commit messageʼs subject for the pull request title. It has proven to make browsing the list of open pull requests easier to grasp for our colleagues. So again a convention…

Habit

For pretty much every dev team that we introduced to Tickety-Tick as part of our consulting work, using the tool became muscle memory quite fast. Especially the ctrl t shortcut plus tab quickly wired up as the go-to move whenever people would start working on a ticket or want to check out branches of a currently worked on ticket from another team member. Having the naming convention established and enforced, frees the developers from having to spent any attention or brain power onto them. Pick a ticket, read and understand it and paste a command to check out a branch for the ticket to start implementing it.

When adapting Tickety-Tick as a team with already established rules around branch names and commit messages, it may make sense to look into the advanced configuration of Tickety-Tick. It allows to adjust the naming patterns to the teamʼs needs, so that the tool can be used to enforce already existing conventions. The configuration options are powerful enough to cover any naming pattern your team may have come up with. Yet they are simple and limited enough to not have you spent hours on them.

bitcrowd can help you to optimize your workflows and team organization! Just hit us up and we see what we can do for you!

Why use it?

So if you already have a ticketing system and a place to host your repositories, why should especially a browser extension between them be of so much value for you as a developer and/or your team? Arenʼt you already keeping track of your work and juggling around too many tools?

In that regard, Tickety-Tick is not another tool™ in the garage of tools youʼre already using. Itʼs more like a loose bandage to tie your tools together, so that they donʼt drift apart. As a side-effect, it may also loosely bandage your team members together, so that they donʼt fully drift apart concerning naming conventions…

Enforce conventions

Adapting Tickety-Tick helps to enforce conventions, which frees developers from having to think about these naming questions themselves day to day. This reduces the cognitive load when working with ticketing systems and Git. And this freed head-space is something both, a dev team or an individual developer on their own can benefit from.

Having the naming done automatically via an agreed upon set of rules, also makes the project easier accessible for new people. Plus, it helps to keep things nice and tidy on the long run, without having to constantly have an eye on it.

Traceability

Another huge and probably the most important benefit of using Tickety-Tick is the connection between ticket and code contribution that it helps to establish. Combined with a habit of squash-merging pull requests, the team has a linear and easy to follow Git history: changes in code point to their commit and the commit message points to the ticket on the tracker. Looking at a commit message then allows to quickly dive into further detail about the change on the ticketing system, to find out where a feature originated from, etc. The quality and usefulness of this connection of course also depends on the quality of ticket descriptions, but thatʼs a whole different topic…

Having this fixed connection between ticket and source code and having all people in the team know about it, turned out to be extremely beneficial for pretty much every project or team weʼve seen so far. Itʼs an additional mean for knowledge sharing, makes onboarding newcomers on the codebase faster and is generally a trusted reference for any member of the team, newcomer to long term contributer. Itʼs an additional source of documentation for the code with a level of detail which would not be possible within the code itself.

Further usage

The encoded information about the connection between a ticket on the tracker and the code which later was written for it, can be used for further applications, like timetracking, work reports, etc.

A whole new information heaven can be reached once RAG (retrieval-augmented generation) comes into play. Looking at both, the ticket description and the resulting code contribution, a RAG system can learn a lot about the domain of a project and how the team communicates. Teams who adapted Tickety-Tick early enough in their work, now have a huge individual dataset of how they work right in front of them! This opens up quite some possibilities for knowledge sharing, analysis, automation and what not. We are super excited about the topic ourself and are currently doing a series of blog posts on it!

Try it!

After all the praise, the only thing left to say is: Just try it out yourself! Tickety-Tick works in all major browsers people use nowadays (Firefox, Chrome, Opera, Safari). It supports a lot of different ticketing systems and if yours is not in the list yet, we can help you to put together a pull request for it. Weʼre always curious to find out which ticketing systems and workflows people use.

If you have more questions about browser extensions or workflows, just hit us up!

Designing Beautiful and Accessible UX: Colour Contrast

Ed Eva — Thu, 29 Aug 2024 14:27:38 +0000

At the beginning of 2025, new EU laws for web accessibility will come into force, requiring industries and public bodies to ensure their websites are accessible to all users. This brings up a common question: how can we stick to by accessibility rules while still creating a good-looking product? In this post, we will focus on colour contrast, which is not only one of the simplest and most effective ways to enhance accessibility in design but also one of the most common accessibility fails.

Sustainablewebdesign.org uses high contrast colourful design to great effect.

Quick introduction to accessibility

The Web Accessibility Initiative (WAI) defines accessibility as the practice of making websites usable by people of all abilities and disabilities. This means designing and developing digital content that can be easily navigated and understood by everyone, regardless of their hardware, software, language, location, or ability.

What is Colour Contrast and Why It Matters

Colour contrast refers to the difference in light between text (or any other foreground element) and its background. High colour contrast improves legibility, which is crucial for users with visual impairments, such as those with low vision or colour blindness.

The Web Content Accessibility Guidelines (WCAG) set the (AA level) standards for colour contrast:

The minimum contrast ratio for regular text is 4.5:1.
For large-scale text (18 point or 14 point bold), the minimum ratio is 3:1.

You don’t need to memorise these ratios; there are plenty of tools available to help you check and adjust colour contrast, which we will mention later.

So who is this for? Consider that in Europe alone, an average of 1 in 30 people experience sight loss, which means over 30 million individuals. Globally, millions more live with varying degrees of visual impairment. It's important to note that visual impairment exists on a spectrum: It can be permanent, affecting a relatively small number of people. It can be temporary or curable, as in the case of cataracts. And it can even be momentary, such as when someone forgets their glasses or struggles with bright sunlight. Ensuring adequate colour contrast can make a significant difference in the user experience for all these groups.

The advantages of good colour contrast extend beyond this group to benefit the user experience of everyone. Here’s how it helps all your users:

Legibility: Clear, legible text increases users' ability to absorb information quickly, reducing cognitive load. (This is especially beneficial for users with conditions affecting short-term memory or focus.)
Trust: Hard-to-read text can make users suspicious as it is reminiscent of the kind of fine print found on dubious products and contracts.
Bright Conditions: High contrast is crucial for readability in bright environments, a common scenario for mobile users.
User Retention: Easy-to-read content helps retain users, as they are less likely to become frustrated and leave the site.
Error Prevention: Using colour contrast effectively can help users distinguish between different types of content and actions, reducing errors.

Information is absorbed quicker when it is clear and legible.

Accessibility & Aesthetics

Creating accessible designs doesn't mean compromising on aesthetics. In fact, integrating accessibility considerations can enhance the overall beauty and functionality of your design. By thoughtfully incorporating colour contrast and other accessibility principles, you can create visually appealing designs that are inclusive and user-friendly. Here are some practical tips and tools we use to achieve this balance:

Start Early: Consider contrast ratios during the early wireframing stages. Use appropriate shades of grey (or whatever colour you wireframe with) to distinguish elements.
Palette Selection: Choose colours that include a dark, mid, and light shade with good contrast against each other. Using these tonal sets can help your colours sing. Tools that help find complementary, analogous, or triadic colour schemes can also be beneficial - Adobe Color.
Using Colour to Highlight: Even though colour contrast refers to the contrast between any background and foreground elements, text is the key feature that needs to be legible. When a text colour fails a contrast check, consider using it on another element to achieve the same effect.
Avoid Trends that Hinder Accessibility: While low-contrast, greyed-out text may be trendy, it often compromises readability. When torn between accessibility and aesthetics, there's likely a better solution.
Keep Checking: Regularly test your designs for contrast at every step and for every design decision. Better to spend a little time doing this early than making changes during implementation. Use plugin tools in your design software such as the Figma plugin Colour contrast or extensions to check within your browser.

Example of a colour contrast Figma plugin inspecting colours of a button.

Follow Your Eyes as Well as the Numbers

While contrast ratios provide a numerical guideline, it's also important to consider the overall visual comfort, harmony, and emotive effect of your colours. Look at your designs and ensure your colours sit well together without causing visual strain or "vibrating" effects. Do they give off the right impression? Do they make you feel calm? Make you hungry? Research into colour psychology and utilising colour theory can help with this. Tools that implement complementary, analogous, or triadic colour schemes can be a good shortcut for creating aesthetically pleasing and accessible designs.

Even readers with perfect vision would struggle with the blurb of this book.

Conclusion

With the upcoming EU law changes, ensuring colour contrast and accessibility in web design will become a legal requirement for many companies. This shift is ultimately beneficial for everyone, as prioritising accessibility creates more inclusive, user-friendly designs that enhance the experience for all users. The law change will necessitate audits and updates for many companies, but it also offers an opportunity to build trust and improve user satisfaction. Embrace these changes, and you'll be ahead of the curve in creating inclusive, effective digital products.

A RAG for Elixir

Max — Wed, 26 Jun 2024 13:50:49 +0000

Abstract

This is the second part of a series of blog posts on using a RAG (Retrieval Augmented Generation) information system for your codebase. Together we explore how this can empower your development team. Check out the first post for an introduction into the topic if you haven't already.

In this episode we explore how we can adapt our RAG system for Ruby codebases from the first episode to read and understand Elixir code. We will take a look at LangChain and text "splitting" and "chunking".

Let's dive right into it.

Background

Our RAG system was built with the idea to discover Ruby codebases. In order to have conversations about Elixir codebases as well, we need to make sure our LLM "understands" Elixir code. This is where LangChain comes into play.

LangChain is a toolkit around all things LLMs, including RAG. We use it to parse our documents or codebase and generate a vector database from it. In our simple RAG system, we specify which file endings (.rb) and which programming language (Ruby) our documents have.

The ingestion of programming source code into an LLM with LangChain was initially only supported for Python, C and a few others languages. Then this issue proposed the usage of a parser library like Tree-sitter to facilitate adding support for many more languages. The discussion is worth reading.

Finally, this pull request introduced support for a lot more languages, including Ruby, based on this proposal. It was a school project:

I am submitting this for a school project as part of a team of 5. Other team members are @LeilaChr, @maazh10, @Megabear137, @jelalalamy. This PR also has contributions from community members @Harrolee and @Mario928.

Our plan is to use this as a starting point to enable basic parsing of Elixir source code with LangChain. With that, we should be able to have conversations with our RAG system about Elixir codebases as well.

Splitting / Chunking text

To read our Elixir codebase, the parser needs some rules on where to split the provided source code files at. Generally for RAG, when ingesting (reading in) a text file, PDF, etc. it will try to split it into chunks, ideally along it's semantic meaning. In text documents, the meanings are often grouped along:

chapters
paragraphs
sentences
words

If your embedding model has enough context capacity, you would try to split along chapters or paragraphs, because human readable text often groups meanings that way. If those are too big, you would try to break between sentences, and, as a last resort, words. One would generally try to avoid splitting inside words. Take for instance "sense" and "nonsense", which carry quite a different meaning.

Splitting / Chunking code

Embedding code is a bit underdeveloped, but the strategy is to break the code into pieces by inserting new lines so that it looks a bit more like natural text, and then let the embedding model deal with the task of making sense (inferring meaning) from it. Interestingly, the models trained on that task do that surprisingly well.

As said, LangChain has dedicated document loaders for source code and a guide on how to add new ones based on Tree-sitter. So we went ahead and implemented a document loader and parser for Elixir source code in LangChain. It only covers the core basics of the language, but it was already enough for our proof-of-concept RAG application. With LangChain now supporting Elixir out of the box, people can use the parser in a variety of different scenarios and will come up with ways to improve it to fit more use cases. Our implementation is only the ground work. You can have a look at the PR if you're interested in what's necessary to add parsing support for another programming language in LangChain. Spoiler: not much if you can utilize Tree-sitter.

The core of LangChain's programming language parsers based on Tree-sitter is their CHUNK_QUERY. For our Elixir parser it looks like this:

CHUNK_QUERY = """
    [
        (call target: ((identifier) @_identifier
            (#any-of? @_identifier "defmodule" "defprotocol" "defimpl"))) @module
        (call target: ((identifier) @_identifier
            (#any-of? @_identifier "def" "defmacro" "defmacrop" "defp"))) @function
        (unary_operator operator: "@" operand: (call target: ((identifier) @_identifier
              (#any-of? @_identifier "moduledoc" "typedoc""doc")))) @comment
    ]
""".strip()

We are using Tree-sitter's own tree query language here. Without diving into the details, our query makes sure to distinguish top level modules, functions and comments. The document loader will then take care of loading each chunk into a separate document and split the lines accordingly. The approach is the same for all programming languages.

Test drive

Let's take this for a spin in our RAG system scripts from episode one of this series.

Just as a refresher: the idea is to have a RAG system for your team's codebase using LLMs locally without exchanging any data with third parties like OpenAI and the like. It includes a conversational AI built with Chainlit, so that members of the team can "chat" with the LLM about the codebase, for instance to get information about the domain or where to find things for the ticket they are working on.

For testing purposes we will use our RAG system on a popular open source Elixir package, the Phoenix Framework.

Get the RAG ready

First we need to get our local RAG system ready for operating on an Elixir codebase. It needs to know:

Where is the code?
Which programming language is it?
Which suffixes have the source code files?

We provide this information via environment variables in a .env file:

OLLAMA_MODEL="llama3:8b"
CODEBASE_PATH="./phoenix"
CODEBASE_LANGUAGE="elixir"
CODE_SUFFIXES=".ex, .exs"

We just cloned the current state of the Phoenix Git repository right next to our RAG code. We also keep using Meta's Llama3 model, and instruct the document loader to look at Elixir files.

ℹ️ Info

At the time of testing our PR on LangChain was not released yet. So we were pointing to our fork's local code for the langchain, langchain-community and langchain-text-splitter Python packages from the requirements.txt file.

For the sake of simplicity we assume the project documentation to be in Markdown and hard-coded this information into our code ingestion logic.

With that, we can set up our vector database for the Phoenix codebase:

python ingest-code.py

Then we start the chat bot:

chainlit run main.py

Chatbot:

Hi, Welcome to Granny RAG. Ask me anything about your code!

Now we are ready to have a conversation about the codebase.

Ask questions

Let’s take an original PR from Phoenix to evaluate the quality of our solution, similar to what we did with Ruby in the previous post.

The pull request fixes a small issue in the recycle/1 function of the Phoenix.ConnTest module at phoenix/lib/phoenix/test/conn_test.ex:

Currently, if you set remote_ip during a test when building a conn, calling a dispatch call e.g get will run the conn through recycle which will lose this information. This PR changes that so it is maintained similar to host.

We reset our clone of the Phoenix repository (and our vector database) to the state right before the PR was merged and then ask the RAG system for help with the issue:

Understand the problem

We:

Please help me understand the following briefing:
Currently, if you set `remote_ip` during a test when building a `conn`,
calling > a dispatch call, e.g `get`, will run the `conn` through `recycle`
which will > lose this information. Instead, we want `the remote_ip`
maintained, similar to `host`.
Can you see why remote_ip is not preserved in test cases where recycle is used?

Chatbot:

The briefing explains that when running a test with Phoenix's Conn and Endpoint frameworks, the remote_ip information is lost during the test. This happens because of the way Phoenix's Conn.recycle/1 function works.
Here's what's happening:

In your tests, you create a new conn with a specific IP address (e.g., remote_ip = "192.168.0.100").

You then dispatch an HTTP request to this connection using something like get("/").

When you do this, Phoenix's Conn framework calls the Conn.recycle/1 function behind the scenes.

This recycle function is designed to clean up and reset the connection for subsequent requests.

However, as part of this cleanup process, it loses the previously set remote_ip information.

The reason why host is preserved but not remote_ip is that Phoenix's Conn framework has a special handling for the host attribute. When you set the host attribute, it gets stored in the connection's internal state. This means that when you recycle the connection, the host information is still retained.

On the other hand, remote_ip is not preserved because it's not explicitly stored in the connection's internal state. Instead, it's just a temporary value set during the test execution.
To maintain the remote_ip information across tests, you can create a custom Conn module that overrides the default behavior of Conn.recycle/1. This would involve storing the original remote_ip value and restoring it after recycling the connection.

In summary, remote_ip is lost in test cases because Phoenix's Conn.recycle/1 function doesn't explicitly preserve this information. To keep remote_ip intact, you'll need to create a custom Conn module that overrides the default behavior of recycle.

Sources:

/Users/joshuabauer/code/rag_time/phoenix/test/phoenix/test/conn_test.exs,

[…]

Implement a solution

This would give us a hint on how to solve the issue in our app if we were "using" the Phoenix framework. But let's say we want to fix the issue "for good" in the framework itself. We now know the Conn.recycle/1 function is what we're interested in. So we look at the listed sources, search for the recycle/1 function in the code and find ConnTest/recycle/1 as the test-conn equivalent to Conn.recycle/1. This is pretty much already be the "custom Conn module" the RAG was referring to and it's just used for tests. And after all, we only want to change the behavior for tests. So let's ask the RAG what to do with it:

We:

Given the following code from phoenix/lib/phoenix/test/conn_test.ex, what would I need to change to persist conn.remote_ip in the same way as conn.host?
def recycle(conn, headers \\ ~w(accept accept-language authorization)) do
  build_conn()
  |> Map.put(:host, conn.host)
  |> Plug.Test.recycle_cookies(conn)
  |> Plug.Test.put_peer_data(Plug.Conn.get_peer_data(conn))
  |> copy_headers(conn.req_headers, headers)
end
Chatbot:

A Phoenix ConnTest question!

To persist conn.remote_ip in the same way as conn.host, you would need to add a line to the recycle/2 function to copy the remote IP address.

Here's how:
def recycle(conn, headers \\ ~w(accept accept-language authorization)) do
  build_conn()
  |> Map.put(:host, conn.host)
  |> Map.put(:remote_ip, conn.remote_ip)  # Add this line
  |> Plug.Test.recycle_cookies(conn)
  |> Plug.Test.put_peer_data(Plug.Conn.get_peer_data(conn))
  |> copy_headers(conn.req_headers, headers)
end
Now, when you recycle a connection, both conn.host and conn.remote_ip will be preserved.

Sources:

[…]

Compare solution

Looking at the PR's file changes, this is exactly what the person came up with:

So the conversation with our Elixir RAG was quite helpful for guiding us through the code and finding an adequate solution to the problem.

Try it yourself!

It is really easy! Just clone our repo, follow the README and tell the script where to find your codebase:

CODEBASE_PATH="./path-to-my-elixir-codebase"
CODEBASE_LANGUAGE="elixir"
CODE_SUFFIXES=".ex, .exs"

We kept the scripts basic, so that they are easy to understand and extend. Depending on your codebase, the results might not always be perfect, but often surprisingly good.

Outlook

In this post we saw how we can extend a simple off-the-shelf system to better fit the needs of our dev team. We enabled our RAG system to read and understand Elixir code! Text splitting and chunking is just one possible example of where to start when it comes to adjusting a RAG system for your specific needs. What we got is already quite useful, but it's definitely still lacking precision.

We will explore possibilities for further improvements and fine tuning in the next episodes of this blog post series.

Or, if you canʼt wait, give the team at bitcrowd a shout via granny-rag@bitcrowd.net or book a consulting call here.

How even the simplest RAG can empower your team

Max — Wed, 26 Jun 2024 13:34:01 +0000

Abstract

This is the first part of a series of blog posts on using a RAG (Retrieval Augmented Generation) information system for coding. Find out how this can empower your development team.

In this episode, we will discuss at a very simple RAG system for Ruby made with LangChain, JinaAI embeddings and a very light local LLM served via Ollama.

If you donʼt know what any of this means, this blog post is for you. You will see how to benefit from the technology without uploading/donating your codebase to external providers like OpenAI.

It is an introduction, the coding follows in the next episodes.

Naming is hard, and I wanted something that goes against the flashy-ai-y. So letʼs call this a Granny RAG.

The next parts will then move on to the following topics:

examine different embedding models
splitting and chunking strategies
RAG for and with Elixir

Letʼs start.

A Use Case

Imagine you are developer and you just started a new job. The code base looks well maintained, the team is nice. Could be bitcrowd. You grab your first ticket, and it looks like this:

Uff. No description. Normally, you would reach out to your onboarding-buddy and ask for advise. This ticket is clearly written with a developers who knows the application in mind.

But ... you could also quickly investigate. Your team does have a Granny after all!

You go to your local granny. Granny RAG is in fact just some light off-the-shelf application wrapper around a simple RAG system, but you are used to call her Granny RAG, because she is patient, kind and knows all the stories.

Letʼs ask her:

Hy Granny, I need to show the end date of the subscriptions in the “subscription list”. What is a subscription in the application?

Granny: To display the end dates of subscriptions in the list of subscriptions, you can modify the COLLECTION_ATTRIBUTES constant in your SubscriptionDashboard class to include the active_until field. Hereʼs an updated version of the SubscriptionDashboard class with the modified COLLECTION_ATTRIBUTES:
class SubscriptionDashboard < ApplicationDashboard
 ATTRIBUTE_TYPES = {
   course: Field::BelongsTo.with_options(order: :name),
   user: Field::BelongsTo.with_options(order: :name),
   user_id: Field::Number,
   id: Field::String,
   active_until: Field::DateTime,
   deactivated_at: Field::DateTime,
   active?: Field::Boolean,
   product_type: EnumField,
   created_at: Field::DateTime,
   updated_at: Field::DateTime
 }.freeze

 COLLECTION_ATTRIBUTES = %i[
   course
   user
   id
   active?
   product_type
   active_until
 ].freeze

Wow, this is magic!

Without knowing the application too well, you came from a non-workable ticket to a plan of action. You can now chose to update the ticket description, or create a PR and use that for the discussion.

ℹ️ Note

This is a real life ticket from a client.

This ticket came from the work at our client MedGurus. When I tried out my Granny RAG, I browsed through the tickets in progress and found this one. I compared the solution Granny RAG proposed with the PR that came out of this ticket. I was actually thrilled how well this works.

My conversation with Granny RAG was actually a a bit longer. Here is the full conversation with Granny RAG.

How does it all work?

Granny RAG is a RAG system. That is short for Retrieval Augmented Generation. If you are looking for a quick intro, here is the nice video by Marina Danilevsky.

In essence, RAG improves the quality of LLM responses by enriching user prompts with relevant contextual information. It retrieves this information from an efficiently searchable index of of your entire project, generated with the help of an embedding model.

Embedding models

Itʼs not easy to say something simple about the embedding process without being incorrect. Embedding models are models that generate a representation of the “meaning” sequence of text. This “meaning” is represented as a vector called “embedding”. It is a long array of numbers that represent semantic meaning within the given context.

Tokens with a similar meaning in the source document get embedding vectors “close to each other” by some distance measurement.

A suitable model will place expressions with similar meaning in similar spaces of its vector space. So subscription will be next to activation and active_until.

You can think of the process as hashing with hashing function that understands the input.

Retrieval

Instead, when the user asks a question, we throw it into the same embedding function to get an index for it. With that, we do a lookup what sequences of text occupy a similar space in the memory.

There are multiple strategies for this similarity criteria. We will explore similarity in more depth in the second post of this series. For now, letʼs assume we found entries “close” to the index we got for the search term.

Each of those entries carries a piece of text and some metadata. The metadata tells us more about the source, e.g. which file it came from. Until now, we have build a more intelligent search function. It finds active_until even if you searched for end date. Something, a classic fulltext index would not find.

In an “old fashioned” information system, we would output those magical pieces of text and leave it to the reader to go through them, understand their meaning and evaluate their relevance.

“But wait”, you say, “are there not these new cool kids on the block, The LLMʼs™, that are brilliant at exactly that?”. You are right, this is exactly what RAG systems do.

Context

Attention: We will be simplifying heavily. If you would like to get a l ightweight intro head over to this huggingface course, or this series of videos from three blue one brown.

It boils down to this: When LLMs generate, they find the next word, or gaps in a text. They take this a step at a time, a bit like friends finishing each otherʼs sentences.

Then, they look at the text created, including the new word, and compile the next word, and the next. Put differently, they try to find the piece of text or the character that is most likely to make sense in the previously generated context.

Here is an example for a prompt that uses RAG:

You are an assistant for question-answering tasks. Use the following pieces of
retrieved context to answer the question. If you donʼt know the answer, just
say that you donʼt know.
Use three sentences maximum and keep the answer concise. # (1)
--
Question: “What would I need to change to show the active_until date in the list
of subscriptions?” # (2)

Context: {context} # <- The RAG magic happens here

Answer: # (3)

ℹ️ Info

A system prompt tells the LLM what is expected from it (1), then a question is specifying the task (2) and the “please fill in your answer here”-part (3) is what LLMs are used to work with.

LLMs do so, again, based on vector representations. Starting from a seed, often the system prompt, and the userʼs instructions.

The idea of RAG is that if you include facts that you researched into your prompt, the context for the generation is narrowed down significantly compared to a prompt that does not include those facts. Retrieval Augmented Generation is an effective countermeasure against hallucinations. It does not stop them, but makes them less likely.

💡 Tip

Outline the Important Findings in your follow up

All LLM based systems hallucinate at some point. RAG helps to avoid that, but as you can see in 5. Follow Up, even Retrieval Based Systems stray from the truth at times. You can detect that because the information in 5. Follow Up does not align with the previous answers.

If this happens, it helps to outline the previous facts in the next prompt, as I did in 6. Follow Up:

...

“You said before that the COLLECTION_ATTRIBUTES are responsible for the list of Subscriptions (...)”

...

Mentioning previous findings in the new prompt amplifies them in the context. This steers the conversation in the direction you like and helps helps the LLM to let go of the hallucinations.

The Code

You can find the code on bitcrowdʼs GitHub, which is probably easier to read than going through the inline version here.

We will use LangChain, which is a Python library. There are also implementations of it in JavaScript and Elixir. We use Python to start with, because it is the common language for machine learning, so prototyping is easy. Once we know what we want, we will explore the Elixir way. So please make sure you have Python installed.

We use JinaAI embeddings, which is a very capable model that can be used locally under the Apache 2 licence.

Finally, you need Ollama, or any other tool that letʼs you run a model and expose it to a web endpoint. In the example, we use Meta's Llama3 model. Models like CodeLlama:7b-instruct also work. Feel free to change the .env file and experiment with different models.

Test Drive

You need some Ruby library that we can try out here. Letʼs go for Discourse.

Weʼre interested in the logic Discourse uses prevent users from re-using an old, expired password and which files in the code are relevant for this. So we ask it:

How does Discourse keep track of which passwords a user already used? So that when changing their password, a user does not use the same, expired password again.

Granny RAG answers:

According to the context you provided, Discourse keeps track of which passwords a user already used by storing the old password hash and algorithm in the user_passwords table. This is done through the following mechanisms:

The user_passwords table stores the userʼs previous passwords along with their corresponding salts and algorithms.

When a user changes their password, Discourse generates a new password hash using the new password, salt, and algorithm.

Discourse then checks if the new password hash is already present in the user_passwords table for that user. If it is, it means the user has used this password before, and therefore, itʼs considered expired.

By storing the old password hashes and algorithms in the database, Discourse can easily verify whether a user has used a particular password before and prevent them from using an expired password again.

This approach ensures that users cannot reuse the same password twice, even if they change their password multiple times.
…

Without knowing all the details of Discourseʼ codebase, this seems a perfectly valid answer. It gives us a nice summary of the feature an idea on where to go looking for the implementation in Discourse otherwise overwhelmingly big codebase.

Wrap up

For our InfoServer Granny RAG we created a sophisticated search index for our codebase. This index allows us to infer meaning. We can now search for an answer by infering the meaning of the question and retrieve relavant information by looking what is close in the vector space.

We fed this information to an LLM to understand and prioritize the sources. We made the LLM create answers to our questions, based on its training data and the information retrieved.

So with a few, off-the-shelf, open source components, we manage to get expert knowledge about our code base, and get it delivered to us by our friendly LLM companion, hosted on our own premises.

How is this different from Copilot, Duo and friends?

If you ask Granny RAG a question, it can draw information from the whole codebase. It is able to incorporate all kinds of documents, and can be extended to incorporate additional data-sources. Granny RAG operates on resource effective, local LLMs.

No data needs to leaves your control.

The scripts that ingest and embed your data and code can be specific to your needs - as is your codebase. That way, you can even specify what should, and what should not, find its way into your RAG knowledge base.

Copilot and GitLab Duo have a much narrower angle of vision. Their context is primarily the opened files of the editor, or the PR. That means, once you know where to look, they can be helpful. Both to you and their creators, which can (and probably will) use some data to improve their models. Even if, per contract, your data and code should not be shared with GitLab or Microsoft, you lost all control once your data leaves the premises.

If you set theses concerns aside, you still have little control about what makes its way into the LLMs that are hosted on remote servers.

Here again, Granny RAG is different. You can collect data from usage and reactions, and you can use that data to train both, LLM and embedding model, on your data and needs.

That way, new arrivals in your dev team get an assistant that is steadily improving. Granny RAG can integrate into a Slack channel to provide a first opinion, and take feedback from the more seasoned developers to improve.

All in all, Granny RAG is a concept that can (and should) be adopted to your use-case and needs. Itʼs not a subscription you buy, but a technique your team learns to master. You invest in consulting or learning time, and you get control and excellent knowledge about the core or your business logic.

Try it yourself!

It is really easy! Just clone our repo, follow the README and tell the script where to find your codebase:

CODEBASE_PATH="./path-to-my-codebase"
CODEBASE_LANGUAGE="ruby"

We kept the scripts basic, so that they are easy to understand and extend. Depending on your codebase, the results might not always be perfect, but often surprisingly good.

Outlook

In this introductory post, we saw what a little off-the-shelf system can achieve. Itʼs already impressive, and it only uses local models, namely Llama3 and JinaAI Code.

You will find that this off-the-shelf solution is lacking precision in some use cases. To improve this, we will explore how changes in the parsing, chunking and embedding strategies will change performance in the next episodes of this blog post series.

Or, if you canʼt wait, give the team at bitcrowd a shout via granny-rag@bitcrowd.net or book a consulting call here.

Idempotent seeds in Elixir

bitcrowd — Thu, 14 Mar 2024 09:36:37 +0000

A standard Phoenix app contains a priv/repo/seeds.exs script file, which populates a database when it is run, so that developers can work with a conveniently prepared environment.

At Bitcrowd, we like our seeds to be idempotent. In practice, this means that running mix run priv/repo/seeds.exs multiple times will not create more rows each time, but rather upsert the existing data. While it is always possible to drop the local database with mix ecto.reset, we might want to keep the current state of our development database.

Upsert all the things!

Our strategy here is to use insert! with a conflict_target on the :id. Let's take a classic blog app with Post and Comment schemas for the sake of the example. In our seeds.exs file we add this little helper:

def insert_idempotently(schema) do
  Repo.insert!(schema, on_conflict: :replace_all, conflict_target: :id)
end

And instead of inserting a seed record like so:

Repo.insert!(%Post{title: "Hello World!"})

We wrap our insert in the helper:

@id = "64af9d13-0f60-45fc-971f-07e6b490c059"
insert_idempotently(%Post{id: @id, title: "Hello World!"})

Nice 🎉

Next time we run the seeds, we will not get a second Post row in the database if a post with this :id already exists. This keeps our development environment neat and clean.

Idempotency for `has_many` associations

While the previous example is fairly simple, hardcoding UUIDs has its limitations when seeding has_many associations. For example, let's say we want to insert 50 Comments associated to the Post:

@id = "64af9d13-0f60-45fc-971f-07e6b490c059"
post = insert_idempotently(%Post{id: @id, title: "Hello World!"})

Enum.each(1..50, fn index -> 
  insert_idempotently(%Comment{
    id: ???, 
    message: "Comment #{index}", 
    post_id: post.id
  })
end)

Of course one could add 50 lines of hardcoded UUIDs… Or could we generate deterministic UUIDs from the index value? Yes we can ⚡️!

Deterministic UUID v4 from a string

Our deterministic UUID generator should take a string as an argument, and always return the same UUID for the same argument. We first need to hash our string, and then to extract the number of bits that we need. UUIDs have a consistent structure: "64af9d13-0f60-45fc-971f-07e6b490c059": one group of 8 characters, then three groups of 4, and finally a group of 12, all separated by a -.

This is how it works:

def deterministic_uuid4(string) do
  # Hash the string and extract the 128 bits,
  # and match on the length of our characters group
  <<a::size(32), b::size(16), c::size(16), d::size(16), e::size(48), _rest::binary>> =
    :crypto.hash(:sha256, string)

  # Override some bits (necessary to create valid UUID v4)
  c = bor(band(c, 0x0FFF), 0x4000)
  d = bor(band(d, 0x0FFF), 0x8000)

  # Glue all of the chunks together and turn it into a string
  Enum.map_join([a, b, c, d, e], "-", &Base.encode16(:binary.encode_unsigned(&1)))
end

Let's see it in action:

iex(1)> MyApp.Seeds.deterministic_uuid4("foo")
"2C26B46B-68FF-468F-899B-453C1D304134"

iex(2)> MyApp.Seeds.deterministic_uuid4("foo")
"2C26B46B-68FF-468F-899B-453C1D304134"

iex(3)> MyApp.Seeds.deterministic_uuid4("bar")
"FCDE2B2E-DBA5-4BF4-8860-1FB721FE9B5C"

Finally, let's validate that our generated UUID is valid with the uuid utility:

iex(1)> UUID.info("2C26B46B-68FF-468F-899B-453C1D304134")
{:ok,
 [
   uuid: "2C26B46B-68FF-468F-899B-453C1D304134",
   binary: <<44, 38, 180, 107, 104, 255, 70, 143, 137, 155, 69, 60, 29, 48, 65,
     52>>,
   type: :default,
   version: 4,
   variant: :rfc4122
 ]}

Let's rewrite our seeds to make use of our brand new function:

@id = "64af9d13-0f60-45fc-971f-07e6b490c059"
post = insert_idempotently(%Post{id: @id, title: "Hello World!"})

Enum.each(1..50, fn index -> 
  insert_idempotently(%Comment{
    id: deterministic_uuid4("comment-#{index}"), 
    message: "Comment #{index}", 
    post_id: post.id
  })
end)

Amazing! We won't get 50 new rows of Comment each time we run the seeds script. Our development database is clean and we made our developers happy ☕️.

UUID Version-5

To ruin the party, deterministic UUID generation is exactly what UUID v5 is designed for. And since Ecto does not validate UUIDs against their specs, you might as well use uuid again and do:

iex(7)> UUID.uuid5(:nil, "foo")
"aa752cea-8222-5bc8-acd9-555b090c0ccb"

iex(8)> UUID.uuid5(:nil, "foo")
"aa752cea-8222-5bc8-acd9-555b090c0ccb"

But what's the fun in that 🤷‍♀️.

Additional resources

If you enjoyed reading this, you might be interested in working with Elixir at bitcrowd. Check our job offerings!

Git, GitHub and your work email

Max — Mon, 09 Aug 2021 16:35:55 +0000

A possible setup for separating private and work-related contributions on GitHub: associate commits with your "private" vs. your "work" email.

GitHub being such a convenient place to collaborate on software, it's likely that both, work and "leassure coding" take place on the same platform. Do you need two separate accounts for both activities? Eventually yes… But to spare the hassle of switching accounts back and forth and remembering to actually do that, a lot of people - including myself - prefer to use a single account (as long as this is legally aligned with your work place). However, aside from keeping that yin-yang of a work-life-balance, fully blending in your work-related coding into your private ramblings on GitHub and vice versa feels wrong from multiple angles, mainly the legal one. In which role did you contribute this piece of code, who "owns" this contribution now? You, your employee, a client?

So a bit of separation can come in handy. But how to get there without juggling accounts? Say hi to your old friend from the 80s: email 📧

Different to what one would expect on first glance, GitHub does not directly associate contributions with accounts. It's rather just "connecting" the contribution to the account that happens to own the email address associated with a Git commit ^¹. Listen to the GitHub Ate My Résumé episode of The Bike Shed podcast for some wild situations this can bring…

It's probably save to assume most people have a dedicated email address for work. Using your private email address for work-related communication uncomfortably stresses the yin-yang and should be avoided. And since we're already there, why not apply the same email hygiene to your commits? Use your work email address for work contributions and your riotcat1991@smthng address for your private coding adventures.

Here are some tips how I set this up for myself. Disclaimer: people are different. While this works fine for me, other solutions may eventually work better for you - find what feels comfy 🧘.

Git Config ⚙️

The setup mainly lives in the gitconfig files in my home directory:

~/
├── .gitconfig
├── .gitconfig_personal
└── .gitconfig_work

Gitconfig

In .gitconfig, set a name via user.name and deliberately not configure any email address in user.email. I combine this with setting user.useConfigOnly to true:

Instruct Git to avoid trying to guess defaults for user.email and user.name, and instead retrieve the values only from the configuration

Recent versions of Git don't allow to commit without having a name and email address configured for the repository. The email address is an integral part of a commit. By not configuring one globally and instructing Git to not magically figure one out itself, I can avoid accidentally committing with the "wrong" email address.

With this configuration, I now have to manually set user.email for every repository. For convenience, I have two alias set up for this:

[alias]
  personal = "config user.email 'me@private.internet'"
  work = "config user.email 'me@work.internet'"

Inside a Git repository, running git work configures the email to be used for commits in that repository to my work address.

Directories

In most cases, it's obvious which email address I want to use. On disk, my projects are organized in a folder structure like this:

~/code/
├── work/
│   ├── client_a
│   │   ├── projetc_a
│   │   └── projetc_b
│   ├── internal
│   │   ├── project_c
│   │   └── project_d
│   └── client_b
└── personal/
    ├── project_a
    └── project_a

In my .gitconfig file I include private or work-related additional configuration depending on the directory path of a repository:

[includeIf "gitdir:code/personal/"]
  path = ~/.gitconfig_personal
[includeIf "gitdir:code/work/"]
  path = ~/.gitconfig_work

These personal and work sub-configurations only configure user.email. For example:

# .gitconfig_personal
[user]
  email = me@work.internet

This spares manually running git work or git personal in every repo. Cloning a new project at work into a subdirectory of ~/code/work, the configuration already tells Git to use my work email.

GitHub 🐙

Your work email needs to be added to your GitHub account: https://github.com/settings/emails

GitHub UI

Merging pull requests in the GitHub UI or accepting change suggestions also creates a commit. Similar to Git itself, GitHub doesn't associate commits with your account, but associates the account via the email address the commit is made with 🤯. For UI interactions, this is a bit tricky to get right.

On the first UI interaction in a repository, GitHub shows a dropdown to select the email address to use for the resulting commit. It then stores this email address for that repository and continues to use it for all UI interactions without showing a dropdown to change it ever again. This means, after having used the wrong email once, it is a bit hard to get out of this again.

I already reached out to GitHub support for this problem, but they could not help me. For now, the only "hack" I found to switch the email again is to:

Go to the respective repository
Click on the 🖊 icon to edit a file (e.g. README.md) in the web editor
You can choose your email again from a dropdown when commiting
GitHub will continue to use this email from now on for this repo

Disclaimer: this email juggling in the UI is what worked for me back then, it may already be different now.

References

1: https://docs.github.com/en/github/setting-up-and-managing-your-github-user-account/setting-your-commit-email-address#about-commit-email-addresses

Script up your projects

Max — Tue, 02 Feb 2021 18:08:51 +0000

At bitcrowd, we love conventions. Being an agency, we jump projects, codebases, frameworks and languages quite a lot - to a point where it's impossible to keep all the details about every project one's head. It's not necessary to actually have this knowledge in practise though. Some people knowing the details of some projects is enough. They can onboard others who then slowly forget the details of their previous projects… A well-established, semi-stable cycle of knowledge transfer with some silos here and some information lost there - but overall working.

Still, there is this nice and fluffy feeling of coming to a new project, seamlessly setting it up and immediately finding your way around. Taking developers by their hands upon entry, these hygge codebases appear nice, friendly and motivating. And developer happiness aside, even on the dark and cold business side of things, people coming to these "welcoming" codebases are probably likely to be more productive with the head start they get there.

How can we get to this shiny place?

Conventions ⚖️

Notorious self-optimization and reaching for ultimate "productivity" are neoliberal derailments. But decluttering daily tasks from unnecessary cognitive overhead, can actually have a calming, decelerating effect on our personal work life: it gives us a cleaner, virtual "desk", less stress and easier time to set details aside and focus on the important things.

In software development, conventions can be a useful tool to reduce the cognitive effort involved when switching between projects. Backed by a known outline of what to expect where, developers can, for the moment, put aside the nitty-gritty details of which version X of framework Y a project uses and instead focus on things like domain or datamodel to easier wrap their head around the new problem space. Utilizing agreed upon patterns, we can take load off peoples' shoulders for the ultimate "hygge" developer experience.

Situation 🚣

In agency work, with its fast and frequent project switches, the "onboarding" and "getting started" phase on a codebase are especially crucial. But product teams shouldn't overwhelm new members with days of fiddling with different package managers either.

First Contact 🐣

Coming to a new project. What are the first steps you always take?

You most likely have a look at the README first (hopefully it's a good one) 🔦
You search the README for installation and setup instructions (the classic "getting started") 🔎
You spend a few ~minutes~ hours copy-pasting things from the README into your shell to get to a state where you can finally run the project 🚜
You nag your colleagues for the steps which are missing, outdated or simply not documented 🕵️

All in all you pretty much spend around half a day setting up the project and already lost the joy of contributing to it. Time, effort and motivation you could have spent on getting to know the domain better and that will be missing in later phases of the project.

A sad situation for a group of people aiming to utilize computer to "solve problems". Projects with a bad setup and onboarding experience are likely to distract and discourage people. Coming out of a rough setup safari, one may feel insecure and as if one knew "nothing" about the project.

Day-in day-out 🌚🌝

Once actually working on a project, one has to run certain development tasks now and then: database migrations, the test suite, managing translations, etc. One may want utilize all possible brain capabilities on memorizing those for each possible framework or alternatively just search up and down the shell history for the one command one at some point managed to successfully copy-and-paste. But what if you don't even know what to search for?

A classic example: You come back to the project after a few days on a different one. You fetch the latest changes on the main branch and then… Run the migrations? Update your JavaScript version? Update packages and then run migrations? Is it npm install or yarn install? A steam of questions and decisions to take before you can even start your editor or run the first test. And there is also the extra cognitive overload of context switches within the project: switching between feature branches, pairing with a colleague on their ticket, etc.

Agreeing on conventions, on a common way to do and approach things across projects may reduce this overload and give people an easier time thought the day 🐖.

Scripts to the Rescue 🚑

People at Github made an attempt to fix this situation: scripts to rule them all. The idea is to have common set of executable scripts for common developer tasks in a script/ directory in the root of every project:

A consistent bootstrapping experience across all our projects reduces friction and encourages contribution.¹

While every project may use different tools or languages, the script directory consistently following the same pattern everywhere gives developers something to hold on to. Scripts as an anchor in the sea of cognitive overload after checking out a codebase:

script/bootstrap to install/update dependencies
script/setup to set up a project for the first time
script/update to update a project to run at its current version
script/test to run tests
script/console to opens a console
…

Technical setup instructions can of course live as 100+ copy-pasteable steps in the README. Putting them into a script has roughly the same effort but a huge benefit: it can do the work for future developers 🤖. Scripts can work well as a runnable documentation for a project's tooling setup. Imagine switching between Python and JavaScript projects. How to run migrations here, what are the default CLI flags there? Instead of infinitely searching the shell history, simple scripts in the project can wrap and document common developer tasks.

The selection of files in your script directory of course depends on every team's individual situation and workflow. Just go with Marie Kondo, get rid of scripts which don't "spark joy" and add others where you feel pain.

Recommendations 🛒

We adapted this pattern for our own workflows at bitcrowd. Here are some loose recommendations on what we find useful:

script/test-e2e

On a lot of projects, high level end-to-end tests are slower. So it's nice to be able to run them with a separate command. With that, script/test can focus on running unit tests only, finish faster and provide important feedback earlier.

script/lint

We are obsessed with linters. Running them as part of the test suite does not fit our way of working. For us it makes sense to run tests and linters separately as they hint at different problems in your code. Maybe one wants to not care about linting until one has a working implementation or tests and prefers to do linting and code cosmetics later?

script/format

Some ecosystems support formatters. If machines can do the formatting for us, why not accept their help?

Your own 🎨

Add your own scripts, like script/deploy, script/psql, etc., there are no limits. Come up with whatever suits your organization and team. The only thing: stay consistent and have conventions. That's where the real benefits lie.

For instance script/manage to facilitate the manage.py utility in Django projects:

#!/usr/bin/env bash

# Run manage.py tasks
# Usage: scripts/manage [...args]

set -o errexit
set -o pipefail
set -o nounset

SCRIPTDIR=$(cd "$(dirname "$0")"; pwd)

cd "$SCRIPTDIR/.."

exec poetry run python manage.py "$@"

Or on a project which requires a specific version of a PostGIS database, using a utility script to to conveniently run the database in a Docker container. Developers may run script/db start to start the database and script/db stop to stop it:

#!/usr/bin/env bash

set -o errexit
set -o nounset
set -o pipefail

CONTAINER_NAME=project_x_db

find_container() {
  docker ps \
    --all \
    --quiet \
    --filter name="$CONTAINER_NAME"
}

start_db() {
  if [ "$(find_container)" = "" ]; then
    exec docker run \
      --interactive \
      --tty \
      --name "$CONTAINER_NAME" \
      --env LC_ALL=C.UTF-8 \
      --env POSTGRES_USER="$POSTGRES_USER" \
      --env POSTGRES_PASSWORD="$POSTGRES_PASSWORD" \
      --env POSTGRES_DB="$POSTGRES_DB" \
      --publish 5432:5432 \
      postgis/postgis:9.6-2.5-alpine \
      postgres
  else
    exec docker start \
      --interactive \
      "$CONTAINER_NAME"
  fi
}

stop_db() {
  exec docker stop "$CONTAINER_NAME"
}

print_usage() {
  echo "Usage: db [<option>]"
  echo ""
  echo "Options:"
  echo "  start   Start database."
  echo "  stop    Stop database."
  echo "  --help  Show this message and exit."
}

if [ $# -lt 1 ]; then
  print_usage
  exit 1
fi

case $1 in
  start)
    start_db
    ;;
  stop)
    stop_db
    ;;
  *)
    print_usage
    exit
    ;;
esac

How to do scripts? 🚌

Who would ever blindly run a script on their machine? Better take a look at the contents first. If written, formatted and documented with care, scripts work as a great way to consistently document development workflows. And since we never get things right the first time: continue to update your scripts (e.g. when new people are onboarded and run into issues). Integrating your scripts into your daily workflows and running them regularly also helps to iron out rough edges. We for instance usually run script/test, script/test-e2e and script/lint as part of our CI pipeline.

When it comes to writing scripts, pick whatever suits your usecase and what you feel comfortable with. For the sake of compatibility, Bash can be a good choice. But be aware, it can be tricky at times… ⚠️:

The weird thing about shell scripts is that even strong advocates of good practices gladly forget all they know when it comes to shell scripting.²

Linting your shell scripts with Shellcheck can improve the situation a lot though. It is also a great resource and opportunity to learn about best practices.

Here are some resources we found useful for leveling up our scripting:

Conclusion 🍼

Scripting is not a silver bullet, but it can make developer lives a bit easier. And it's a great rabbit hole to get lost in 🕳

References

Prepending Modules to the Rescue

Max — Thu, 26 Nov 2020 14:26:25 +0000

Ruby's ability to "overlay" default implementations of constants, methods and variables via the prepend or prepend_features method on Module can be a helpful tool when dealing with gems overriding setter methods for attributes.

Scenario 🧫

On a Ruby on Rails project, we were recently migrating from the attr_encrypted to the lockbox gem for encrypting database fields. For live applications which cannot accept hours of downtime, the migration path is a multi-step process which includes a period where both gems are used side by side until all data has been migrated from the old columns to the new columns.

Both gems integrate into Rails models via their own macro-style methods one is supposed to add to the models' class definitions:

class User < ApplicationRecord
  attr_encrypted :email, key: key
  attr_encrypted :phone, key: key

  encrypts :email, :phone, migrating: true
end

Under the hood, both gems then dynamically generate the respective getter and setter methods for the attributes (email, email=, phone and phone= in this example).

In addition to that, we were also overriding the setter methods email= and phone= ourselves to do some normalization on the provided values before assigning them. Combining this with the generated setters from attr_encrypted introduces a lot of fuzz: in what order are the implementations called - if at all - and what does super mean in which context? In order to eliminate all this confusion from the start, we previously decided to make our own implementation the "source of truth" and instead of relying on super calls, just integrate the respective parts of attr_encrypted's implementation into our own:

def phone=(value)
  normalized_number = Normalizers::PhoneNumber.normalize(value)
  self.encrypted_phone = encrypt(:phone, normalized_number)
  instance_variable_set(:@phone, normalized_number)
end

Problem 💥

Now, during the migration phase were we utilize both gems in parallel, our own implementation would of course also need to integrate the internals of both gems in our own overriden implementation. Plus we would need to ensure the attr_encrypted related code is removed again once the migration phase is over 🤯.

This seemed overwhelming and just way too many things to take care of for our own tiny model implementation. Integrating gems should ideally not interfere too much with our own plans to normalize attributes before assigning them. In addition to that, integrating so deeply with a gem, that understanding the code required reading the gems internals beyond the "normal" instructions in the README also comes with a high price on maintainability.

So we needed to find another way to use both gems in parallel while also guaranteeing our values are normalized before assigning and before encrypting them.

Rescue 🚑

A post on the arkency blog describes how Ruby's prepend method on Module can be utilized to override or better "overlay" methods added directly onto the class by a gem. One can prepend an anonymous module inline with the own implementation to either fully override the gem's implementation or just "prepend" one's own implementation and then call super() to still invoke the code generated by the gem.

We simply want to "prepend" our own normalization step before the gems start to do their magic and ideally don't want to get into the details of what they are actually doing. So utilizing super() after the normalization step fits our use case perfectly. In the model class definition, this could look like this:

class User < ApplicationRecord
  attr_encrypted :email, key: key
  attr_encrypted :phone, key: key

  encrypts :email, :phone, migrating: true

  prepend(Module.new do
    def phone=(value)
      normalized_number = Normalizers::PhoneNumber.normalize(value)
      super(normalized_number)
    end
  end)
end

This "overlays" or "prepends" our normalization step before the class' implementation of the setter method, even when it's changed by any of the included gems. So we make sure the value is normalized before it is assigned and the gems' implementation of the setter invoked afterwards without any need for us to fiddle with internal details.

⚠️ Note however, that for the sake of readability and consistency, our model class definition follows Rubocop's Rails Style Guide:

Group macro-style methods (has_many, validates, etc) in the beginning of the class definition.

Following this class layout, our own implementation is sure to be processed after the gems' methods were defined and we are prepending our normalization step before the final definition of the setter method. Ordering the definitions differently in our class would impact the value of super() here.

Cleanup 🧽

While prepending an anonymous module directly inside the class definitions works perfectly fine for our use case here, it still looks very verbose and suspiciously distracting for anyone reading over the model definition. And as we made use of this technique in multiple models within the project, we extracted the boilerplate into a model concern. The goal was to hide the details of the prepending trick while at the same time making the normalization step more visible and explicit to the reader.

We extracted a more general Normalizable concern. We already had multiple normalizer classes in the project. They all follow the same pattern and expose a single normalize class method as their public API. So it just made sense tie the implementation of the Normalizable model concern close to those normalizers:

module Normalizable
  extend ActiveSupport::Concern

  class_methods do
    def normalize(attr, with:)
      prepend(Module.new do
        define_method("#{attr}=") do |value|
          normalized_value = with.public_send(:normalize, value)
          super(normalized_value)
        end
      end)
    end
  end
end

Utilizing this, we can change the previous example to:

class User < ApplicationRecord
  include Normalizable

  attr_encrypted :email, key: key
  attr_encrypted :phone, key: key

  encrypts :email, :phone, migrating: true

  normalize :phone, with: Normalizers::PhoneNumber
end

This hides the internal details of our own implementation of the setter while still making it explicit that we are doing a normalization on the attribute. Similar to the original implementation inlining prepend with an anonymous module, this approach of course still only works as intended if the normalize macro in the class definition is defined after any other setter methods generated by gems are defined. However, in our case it seemed most fitting to place the `normalize calls at the end of the macros section anyways.

Improving callable service objects with private constructors in Ruby

Max — Wed, 25 Nov 2020 10:06:41 +0000

We recently found a nice practical use case for private class methods in Ruby: the constructor methods of what we will refer to here as "callable services" in Ruby on Rails projects.

Callable Services ☎️

First off: what do we mean with "callable services" in Ruby on Rails applications?

Drawers 🗄

In the real world, hardly any Ruby on Rails web application only consists of simple, atomic CRUD (create, read, update, delete) operations on plain resources. Even the famous blog engine built in 15 minutes may at some point incorporate logic beyond the complexity of creating or deleting posts. Take for instance translations, previews, comment moderation, etc. Real world applications model a specific domain and therefore include control flows, procedures and "business logic" tied to that domain. These bits are in fact what makes an app unique and interesting.

When it comes to organizing code, Rails people traditionally aim for "skinny controllers and fat models". But as models packed with responsibilities beyond modeling the domain soon tend to get overweight and hard to maintain, the community started to reach for additional concepts and patterns to organize their domain specific code. One "drawer" one may come across in a lot of Rails applications are "services", also referred to as "service objects" or "procedures". They are usually organized under the app/services/ directory and encapsulate functionality to handle domain-specific logic, such as checking our a cart, registering for the site or starting a subscription. ¹

Services are usually implemented as POROs ("plain old Ruby objects") which, upon a given input, perform a set of operations and return a predictable response. They are easy to unit test and help developers to maintain a sense for the bigger picture by hiding away the internal details in what appears to the outside as a large black-box-function. That is why people started to implement them as classes exposing only a single method, call or run. Depending on its internals and their complexity a service may create a new instance for each call or just utilize a single class method for its work:

class Authenticator
  def self.call(user)
    # complex logic
  end
end

# invoking the service:
Authenticator.call(user)

The same service implemented to use a new instance for each call:

class Authenticator
  def initialize(user)
    @user = user
  end

  def call
    # complex logic, operate on @user
  end
end

# invoking the service:
Authenticator.new(user).call

Conventions 🧘

At bitcrowd we - as you know - love conventions and therefore usually strive for a common API for our classes in app/services in Rails projects. So even if we don't know what's in the box, we at least know we're dealing with a box… Picking up the previous example, this could look like this:

class Authenticator
  def initialize(user)
    @user == user
  end

  def call
    # complex logic
  end

  private

  attr_reader :user

  def private_helper_method
    # some bits of logic, can operate on user
  end
end

So within the project all services follow the same general structure:

a constructor takes all the core input data the service needs to do its work
a single exposed call method invokes the service to perform the actual work

Simplifications 🧹

The functionality encapsulated within a service is seen as one operational unit on the outside. So we probably won't interfere between initializing the service and calling it. If the data needed to be manipulated between new and call, we should probably rather think about drawing the boundaries between our objects differently instead. But having things clearly encapsulated, we could also simplify the service' API to Service.call(<input-data>) and hide the implementation details of it using a new instance for each call inside of it. Since Ruby 2.7, we also make use of the 3 dots argument forwarding syntax ² - also referred to as "forward everything" - for the .call class method:

class Authenticator
  def initialize(user)
    @user = user
  end

  def self.call(...)
    new(...).call
  end

  def call
    # complex logic
  end
end

This API makes our service more streamlined and predictable. The service is easier to read and less verbose on the outside, hiding implementation details inside the class itself. We're also less likely to accidentally sneak code between new and call when invoking the service. And while it's aesthetically pleasing on the eye, it also allows us to write less verbose expectations in our unit tests for code which interacts with the service:

# Before
let(:service_instance) { instance_double(Authenticator) }

it 'calls the service' do
  expect(Authenticator).to receive(:new).with(user).and_return(service_instance)
  expect(service_instance).to receive(:call)
  # ...
end

# After
it 'calls the service'
  expect(Authenticator).to receive(:call).with(user)
  # ...
end

While we previously needed two expectations to ensure both, the service being initialized with the right data and then being invoked, we can now do both in one step.

Communication 📢

Even with the new simpler API, our services can of course still be called in "the old" way, initializing and calling the service in two steps. The API for this approach is public and aside from examples in the code or documentation, we don't have anything at hand to ensure the service is used as intended. People may still happily do Authenticator.new(user).call and use instance doubles in their tests… The new API only gives a hint on how to use services in the project, it does not actually encourage or enforce one unified style.

Private Constructors to the Rescue 🚒

Turns out we can make use of Ruby's private_class_method method on Module to hide the constructor and make our intentions on how to use the services more obvious:

Makes existing class methods private. Often used to hide the default constructor new.

Adapting our example:

class Authenticator
  def initialize(user)
    @user = user
  end

  private_class_method :new

  def self.call(...)
    new(...).call
  end

  def call
    # actual logic
  end
end

With this we introduce a new problem though: the visual overhead of the additional boilerplate code makes the actual service implementation harder to read and understand. One way to circumvent this and allow the readers to stay focussed on the actual business logic, would be extracting the boilerplate into a concern:

module Callable
  extend ActiveSupport::Concern

  included do
    private_class_method :new
  end

  class_methods do
    def call(...)
      new(...).call
    end
  end
end

We can then shorten the service to:

class Authenticator
  include Callable

  def initialize(user)
    @user = user
  end

  def call
    # actual logic
  end
end

This ensures our service is now used as intended. Trying to call its parts separately as Authenticator.new(user).call results in an error:

NoMethodError: private method `new' called for Authenticator:Class

Doing so, we can a clean and concise outer API for our services while not sacrificing readability on its internals. Invoking a service with its single exposed call class method ensure we pass the initial data to set up the state it needs to perform its work and then immediately trigger the actual "work" part.

TL;DR

Use private_class_method to hide the initializer of your service objects for a clearer API and less boilerplate in tests 💅.

References

Folding sections of Markdown in Vim

Max — Mon, 25 May 2020 07:22:45 +0000

TIL: Vim is able to fold sections or Markdown on their headings 💡

Modern version of Vim and Neovim support folding .md Markdown documents on their section headings #, ##, ### etc. out of the box, even without installing the additional vim-markdown-folding plugin.

What is folding 🤔

Vimʼs documentation describes “folding” as:

Folding is used to show a range of lines in the buffer as a single line on the
screen.  Like a piece of paper which is folded to make it shorter:

    +------------------------+
    | line 1         |
    | line 2         |
    | line 3         |
    |_______________________ |
    \            \
     \________________________\
     / folded lines       /
    /________________________/
    | line 12        |
    | line 13        |
    | line 14        |
    +------------------------+

The text is still in the buffer, unchanged.  Only the way lines are displayed
is affected by folding.

The advantage of folding is that you can get a better overview of the
structure of text, by folding lines of a section and replacing it with a line
that indicates that there is a section.

When editing large documents, I personally find it handy to temporarily fold and “hide away” certain parts I currently donʼt care about.

Folding Markdown 🗂

Out of the box, Vim and Neovim currently wonʼt know “how” to fold Markdown sections. Their included default vim-markdown filetype and syntax plugin however offers an undocumented setting to enable just this:

let g:markdown_folding = 1

There is an open pull request include this in the official documentation, but it is not merged yet.

Flicking this switch turns my .md buffers into something like this:

+--- 32 lines: ## What is folding 🤔···········································
+--- 23 lines: ## Folding Markdown 🗂···········································

Pressing zR will recursively unfold all sections again. From there on you may continue to happily fold and unfold. Drew Neil from Vimcasts put together a nice and short overview over the most important folding commands:

command	effect
`zo`	open current fold
`zO`	recursively open current fold
`zc`	close current fold
`zC`	recursively close current fold
`za`	toggle current fold
`zA`	recursively open/close current fold
`zm`	reduce `foldlevel` by one
`zM`	close all folds
`zr`	increase `foldlevel` by one
`zR`	open all folds

First look 👀

With Markdown-folding enabled, Vim will default to the fully “folded” view when opening a Markdown buffer. I personally prefer to start with the “full” document though. According to this Stackoverflow thread there are a few ways out of this behavior.

I went for setting a custom foldlevelstart value in my configuration. It instructs Vim how or if a buffer should be folded when one starts editing:

'foldlevelstart' 'fdls' number (default: -1)
            global
    Sets 'foldlevel' when starting to edit another buffer in a window.
    Useful to always start editing with all folds closed (value zero),
    some folds closed (one) or no folds closed (99).
    This is done before reading any modeline, thus a setting in a modeline
    overrules this option.  Starting to edit a file for |diff-mode| also
    ignores this option and closes all folds.
    It is also done before BufReadPre autocommands, to allow an autocmd to
    overrule the 'foldlevel' value for specific files.
    When the value is negative, it is not used.

I went for starting with all folds open in Markdown buffers:

au FileType markdown setlocal foldlevel=99

Happy folding 🗺

Forem: bitcrowd

Getting React on Rails to work with Turbo Streams

Scenario

Pagination

Example

Problem

Options

JS to the rescue

Conclusion

Loading Chrome extensions for development in 2025

Background

The old way

What happened?

What now?

Our script

What we learned

Meet Tickety-Tick

Basics

Convention

Origin

Workflow

Setup

Usage

Done?

Habit

Why use it?

Enforce conventions

Traceability

Further usage

Try it!

Designing Beautiful and Accessible UX: Colour Contrast

Quick introduction to accessibility

What is Colour Contrast and Why It Matters

Accessibility & Aesthetics

Follow Your Eyes as Well as the Numbers

Conclusion

A RAG for Elixir

Abstract

Background

Splitting / Chunking text

Splitting / Chunking code

Test drive

Get the RAG ready

ℹ️ Info

Ask questions

Understand the problem

Implement a solution

Compare solution

Try it yourself!

Outlook

How even the simplest RAG can empower your team

Abstract

A Use Case

Wow, this is magic!

ℹ️ Note

How does it all work?

Embedding models

Retrieval

Context

ℹ️ Info

💡 Tip

The Code

Test Drive

Wrap up

How is this different from Copilot, Duo and friends?

Try it yourself!

Outlook

Idempotent seeds in Elixir

Upsert all the things!

Idempotency for has_many associations

Deterministic UUID v4 from a string

UUID Version-5

Additional resources

Git, GitHub and your work email

Git Config ⚙️

Gitconfig

Directories

GitHub 🐙

GitHub UI

References

Idempotency for `has_many` associations