Forem: Josh Branchaud

Reduce a JSON Object to just Entries of a Specific Type with jq

Josh Branchaud — Tue, 10 Jan 2023 21:28:48 +0000

A large JSON object can be hard to work with when it has tons of top-level fields that you have to wade through. One way to make that data exploration more approachable is to reduce what is there — filtering by type is a great starting point. Here is how I do that with the jq utility.

Let's say I want to start with a view of the JSON that is restricted to just the field whose values are of type array.

To do this, I need to use a couple different jq helper functions.

The select function produces the input value as the output if the given boolean expression is true. This is where I can do the type check. In this case, does the type of .value match "array".

select(.value | type | match("array"))

Combining that select with map, I can filter an array to just those values that match my condition.

map(select(.value | type | match("array")))

Because map expects an array and I'm starting with an object, I need to open up with a to_entries call. This turns the object into an array of .key and .value pairs. Hence, the .value that appears inside the select. Putting a from_entries on the other end of the map will take the reduced array of key-value pairs and turn it back into an object.

some_obj | to_entries | map(...) | from_entries

Putting this all together:

jq '. | to_entries | map(select(.value | type | match("array"))) | from_entries' data.json

There is a less verbose way to do the above. The to_entries and from_entries can be collapsed into a with_entries that replaces the map call.

jq '. | with_entries(select(.value | type | match("array"))' data.json

with_entries(foo) is a shorthand for to_entries | map(foo) | from_entries, useful for doing some operation to all keys and values of an object.

This can be extended to select for multiple types with a little conditional logic like so:

jq '. | with_entries(select((.value | type) == "array" or (.value | type) == "object"))' data.json

That's just a drop in the bucket for how jq can be used to explore JSON data files.

With this tool being so general-purpose and having so many utilities, workflows like this one are where things get most interesting. Share your jq workflows with me in the comments below 👇

If you enjoyed this post, consider joining my newsletter or following me on twitter. If this helped you or you have a question, feel free to drop me note wherever. I'd love to hear from you!

Breaking Down a Complex Mapped Type

Josh Branchaud — Sun, 29 May 2022 22:14:35 +0000

There is a useful generated type pattern in TypeScript called a mapped type. Let's make sense of it by starting from first principles and building up to the full example

While watching Matt Pocock's video on Using DECLARE GLOBAL for amazing type inference, I was pretty stuck trying to make sense of the type of event in the GlobalReducer.

There are a lot of pieces to this type and, at least for me, none of them are particularly intuitive. For me to make sense of it, I needed to break it down into smaller pieces first.

Here is a walkthrough of that process.

For starters, here is a GlobalReducerEvent that we can add and remove reducer types from.

interface GlobalReducerEvent {
    ADD_TODO: {
        text: string
    }
    LOG_IN: {
        email: string
    }
    DELETE_TODO: {
        todo_id: number
    }
}

type ReducerEvent = ... /* what we're about to build */

type GlobalReducer<TState> = (
    state: TState,
    event: ReducerEvent
) => TState

We want our GlobalReducer to do type checking based on what events are included. For that we'll need to use some fun TypeScript features that allow us to map the GlobalReducerEvent into a type that is useful in our user-land code.

// this is our target type
type ReducerEvent = ({
    type: "ADD_TODO";
    text: string;
}) | ({
    type: "LOG_IN";
    email: string;
}) | ({
    type: "DELETE_TODO";
    todo_id: number;
})

Building up to the ReducerEvent (the type of event) is our end goal here.

Starting with keyof, we can get a union type of all the events.

// What are our different types of events?
type EventUnion = keyof GlobalReducerEvent;
/*
type EventUnion = 'ADD_TODO' | 'LOG_IN' | 'DELETE_TODO'
*/

That is a crucial piece of the puzzle as it is used in two different places in the end result.

The next few steps are going to be building up an intermediate type of sorts. A type that will help us map the original type into our target type.

With in, we can create an index signature over the values from EventUnion (which remember is keyof GlobalReducerEvent). For the moment, we'll type each of those keys as any.

// Gather up the types of events
// (we'll replace the `any` in a moment)
type EventTypes = {
    [EventType in EventUnion]: any
};
/*
type EventTypes = {
    ADD_TODO: any;
    LOG_IN: any;
    DELETE_TODO: any;
}

same as doing:

type EventTypes = {
    [EventType in keyof GlobalReducerEvent]: any
}
*/

Now instead of typing them as any, let's key them to an object. This object is going to start building toward the values we are trying to map to.

Each event should have a type whose value matches its name. EventType is a reference to the name of the event, so we can use that in the object type ({ type: EventType }).

// Event type keyed to itself as an object
// (weird intermediate type, stick with me here)
type EventTypesWithSelf = {
    [EventType in keyof GlobalReducerEvent]: {
        type: EventType
    }
};
/*
type EventTypesWithSelf = {
    ADD_TODO: {
        type: "ADD_TODO";
    };
    LOG_IN: {
        type: "LOG_IN";
    };
    DELETE_TODO: {
        type: "DELETE_TODO";
    };
}
*/

Each event has typings defining the data that needs to go with it. For instance, ADD_TODO requires a bit of text that constitutes the thing to be done. Hence, ADD_TODO: { text: string }. In this next part, we are going to fold in the typings for each event's data.

We can use the intersection type literal (&) to combine each event type ({type: 'ADD_TODO'}) with its data typing ({text: string}).

// Event type keyed to itself and its data
// (still looks weird, but starting to take shape)
type EventTypesWithSelfAndData = {
    [EventType in keyof GlobalReducerEvent]: {
        type: EventType
    } & GlobalReducerEvent[EventType]
};
/*
type EventTypesWithSelfAndData = {
    ADD_TODO: {
        type: "ADD_TODO";
    } & {
        text: string;
    };
    LOG_IN: {
        type: "LOG_IN";
    } & {
        email: string;
    };
    DELETE_TODO: {
        type: "DELETE_TODO";
    } & {
        todo_id: number;
    };
}

which you can think of as:

type EventTypesWithSelfAndData = {
    ADD_TODO: {
        type: "ADD_TODO";
        text: string;
    };
    LOG_IN: {
        type: "LOG_IN";
        email: string;
    };
    DELETE_TODO: {
        type: "DELETE_TODO";
        todo_id: number;
    };
}
*/

Before we get to the last part, let's wrap our heads around indexed access of a type object.

The works a lot like accessing the value for a key from a JavaScript object. Using the type from the previous step, we are able to grab just the 'ADD_TODO' type.

// Let's try an indexed access of EventTypesWithSelfAndData
type AddTodoType = EventTypesWithSelfAndData['ADD_TODO']
/*
type AddTodoType = {
    type: "ADD_TODO";
} & {
    text: string;
}
*/

Where it differs from how JavaScript object access works is evidenced by passing a union type instead of an individual value.

// What if we try to access multiple types at once with a union
type SomeEventTypes = EventTypesWithSelfAndData['ADD_TODO' | 'DELETE_TODO']
/*
type SomeEventTypes = ({
    type: "ADD_TODO";
} & {
    text: string;
}) | ({
    type: "DELETE_TODO";
} & {
    todo_id: number;
})
*/

Notice the resulting type is a union type made up of just the types indexed at the union values.

We can take this a step further by doing an indexed access with keyof GlobalReducerEvent which is a union of all our event types.

// That means we can pass in a union of all our event type names
// to get a union of all the type signatures.
type AllEventTypes = EventTypesWithSelfAndData[keyof GlobalReducerEvent]
/*
type AllEventTypes = ({
    type: "ADD_TODO";
} & {
    text: string;
}) | ({
    type: "LOG_IN";
} & {
    email: string;
}) | ({
    type: "DELETE_TODO";
} & {
    todo_id: number;
})

which, you might remember, is equivalent to this:

type AllEventTypes = ({
    type: "ADD_TODO";
    text: string;
}) | ({
    type: "LOG_IN";
    email: string;
}) | ({
    type: "DELETE_TODO";
    todo_id: number;
})
*/

That last example is equivalent to this full ReducerEvent. We built it up from all the constituent parts. Fun!

Let's take another look at the whole thing again.

type ReducerEvent = {
    [EventType in keyof GlobalReducerEvent]: {
        type: EventType
    } & GlobalReducerEvent[EventType]
}[keyof GlobalReducerEvent]

Hopefully having walked through all the different pieces of this, it is looking less like magic and more like something you can reason about.

If you enjoyed this post, consider joining my newsletter or following me on twitter. If this helped you or you have a question, feel free to drop me note wherever. I'd love to hear from you!

Want to play around with this in more detail? Check it out in the TypeScript Playground.

Cover photo by GeoJango Maps on Unsplash

Speeding Up An Expensive PostgreSQL Query: B-Tree vs. BRIN

Josh Branchaud — Thu, 05 May 2022 20:03:22 +0000

I had an expensive ~20 second query on a massive ~500 million row PostgreSQL table. I brought that query down to about 2 milliseconds. The quick answer is that there was a missing index. In this post, we'll explore how we came to that conclusion and how we determined what kind of index to add.

tl;dr: the query we will look at in this post was slow because of a missing index on a created_at timestamp column. We tried separately adding a BRIN and a B-Tree index. When a BRIN index was added, there was no performance improvement. When a B-Tree index was added, however, we saw a huge performance improvement. The most expensive query dropped from ~20 seconds to ~2 milliseconds. A 10,000x speed up. The average timing of this query is now even a little lower than that.

Read on if you want more of the details.

Note: all the queries I'm about to cover are run against a locally-restored snapshot of production data. Take care when adding indexes like these in production. Do so concurrently and be sure the monitor your app in the process.

The Data Model

First, let's look at a sufficient snapshot of the data model.

We have a massive events table, it contains hundreds of millions of rows. Each of those events is associated with a particular record in the users table.

Users do things and those things are recorded as events in the events table. Users do lots of things, so there are lots of events.

The Slow Query

The query that our monitoring software was flagging as slow was one where we were trying to find the two most recent events for a specific user.

select * from events
where user_id = 123
order by created_at desc
limit 2;

Some users have way more events than others, so the slowness of this query can vary pretty drastically. This particular user has over ~200,000 events. It's instances like this that revealed the bottleneck in this table's design.

Let's explain analyze this query to better understand where the issue is.

explain analyze
select * from events
where user_id = 123
order by created_at desc
limit 2;

Running explain analyze with this query gives us a really good idea of what is going on because it actually executes the query. Here is what the output of that looks like:

                                                                               QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=1066813.50..1066813.50 rows=2 width=114) (actual time=23207.023..23207.024 rows=2 loops=1)
   ->  Sort  (cost=1066813.50..1067596.73 rows=313292 width=114) (actual time=23207.022..23207.022 rows=2 loops=1)
         Sort Key: created_at DESC
         Sort Method: top-N heapsort  Memory: 25kB
         ->  Bitmap Heap Scan on events  (cost=5868.59..1063680.58 rows=313292 width=114) (actual time=74.682..23098.624 rows=222272 loops=1)
               Recheck Cond: (user_id = 123)
               Rows Removed by Index Recheck: 6448758
               Heap Blocks: exact=67292 lossy=128418
               ->  Bitmap Index Scan on index_events_on_user_id  (cost=0.00..5790.26 rows=313292 width=0) (actual time=57.349..57.349 rows=222272 loops=1)
                     Index Cond: (user_id = 123)
 Planning time: 0.095 ms
 Execution time: 23207.502 ms
(12 rows)

The headline number is the Execution time at the bottom of the output -- 23207.502 ms. That's 23 seconds.

The thing I then want to note is where the bulk of those 23 seconds is going. Fortunately in this explain analyze output, it pretty clearly stands out in a single area.

Notice this bit of the output where it is sorting the table based on created_at DESC.

         Sort Key: created_at DESC
         Sort Method: top-N heapsort  Memory: 25kB
         ->  Bitmap Heap Scan on events  (cost=5868.59..1063680.58 rows=313292 width=114) (actual time=74.682..23098.624 rows=222272 loops=1)

We can see the actual time ranges from 74.682..23098.624. That's where this query is spending the vast majority of its time.

Looking at the table indexes and the query we are running, I probably could have guessed the issue was a missing index on created_at. I don't just want to add indexes haphazardly. I want numbers to back it up. This confirms that. We are on track to fixing this.

BRIN vs. B-Tree

So, we know we need an index on created_at, but what kind of index? B-Tree is the default when the index type isn't specified. But I know I've heard of Postgres supporting other kinds of indexes. What about those?

Here is what the Crunchy Data blog has to say about the BRIN index.

PostgreSQL 9.5 introduced a feature called block range indexes (aka BRIN)
that is incredibly helpful in efficiently searching over large time series
data and has the benefit of taking up significantly less space on disk than a
standard B-tree index.

"Large time series data" is exactly what we are dealing with here, and an index that takes up less space on disk sounds great. So that seems worth a try. In fact, let's start there.

With BRIN Index

I'll start by adding the index.

create index index_events_on_created_at
  on events using brin (created_at);

Note: Adding the BRIN index to a ~500 million row table took ~12 minutes on my dev machine.

Now I'll re-run the query from earlier still with explain analyze:

> explain analyze select * from events where user_id = 123 order by created_at desc limit 2;

                                                                               QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=1066813.50..1066813.50 rows=2 width=114) (actual time=23074.657..23074.658 rows=2 loops=1)
   ->  Sort  (cost=1066813.50..1067596.73 rows=313292 width=114) (actual time=23074.655..23074.655 rows=2 loops=1)
         Sort Key: created_at DESC
         Sort Method: top-N heapsort  Memory: 25kB
         ->  Bitmap Heap Scan on events  (cost=5868.59..1063680.58 rows=313292 width=114) (actual time=68.786..22975.087 rows=222272 loops=1)
               Recheck Cond: (user_id = 123)
               Rows Removed by Index Recheck: 6448758
               Heap Blocks: exact=67292 lossy=128418
               ->  Bitmap Index Scan on index_events_on_user_id  (cost=0.00..5790.26 rows=313292 width=0) (actual time=48.569..48.569 rows=222272 loops=1)
                     Index Cond: (user_id = 123)
 Planning time: 2.097 ms
 Execution time: 23075.473 ms
(12 rows)

Unfortunately, that output looks very similar to the one before we added an index. It's hard to know exactly why a BRIN index isn't conferring any benefit here. The important thing is that we measured. We know that at least for this data set, a BRIN index isn't the way forward.

Since we don't want the BRIN index, let's drop it and then give B-Tree a try.

drop index index_events_on_created_at;

With B-Tree Index

create index index_events_on_created_at
  on events(created_at);

Note: Adding the B-Tree index to a ~500 million row table took ~8 minutes on my dev machine.

And again, we'll run our slow query and see how it performs.

> explain analyze select * from events where user_id = 123 order by created_at desc limit 2;

                                                                               QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.57..535.62 rows=2 width=114) (actual time=0.056..1.951 rows=2 loops=1)
   ->  Index Scan Backward using index_events_on_created_at on events  (cost=0.57..83812631.72 rows=313292 width=114) (actual time=0.055..1.950 rows=2 loops=1)
         Filter: (user_id = 123)
         Rows Removed by Filter: 4819
 Planning time: 0.116 ms
 Execution time: 2.000 ms
(6 rows)

Besides the fact that the query returns immediately, I can tell by the change in shape and height of the query plan that we're on to something. The execution time has dropped to 2.000 ms as well. The second line of the query plan shows that the new index is being used.

   ->  Index Scan Backward using index_events_on_created_at on events  (cost=0.57..83812631.72 rows=313292 width=114) (actual time=0.055..1.950 rows=2 loops=1)

That's more like it. This backs up the decision to add a B-Tree index for the created_at column in production. As I mentioned at the beginning of the post, adding an index like this in any production setting, but especially for a table
of this size, it is important to do so concurrently and to monitor your app's health metrics during the process.

If you enjoyed this post, consider joining my newsletter or following me on twitter. If this helped you or you have a question, feel free to drop me note wherever. I'd love to hear from you!

Cover image credit: Kiyoshi on Unsplash

Strong Confirmation Modal with XState

Josh Branchaud — Tue, 31 Aug 2021 17:14:30 +0000

The UI element that I'm calling the Strong Confirmation Modal is a prompt to the user to doubly confirm a destructive action. I'll quickly discuss the idea behind this and then show how I implemented it with XState.

Strong Confirmation

"Yes, you clicked the delete button, but are you sure? Type the name of the thing you want to delete so that we can both be sure."

I've seen this UI element in plenty of places, but the one that stands out to me is GitHub's. Deleting a repository is definitely a destructive action and not one you'd like to do accidentally. Certainly not something you'd like your cat to be able to trigger by stepping on the keyboard. Here is what it looks like.

You have to type the name of the repository you want to delete in order to enable the button that confirms the deletion. It seems like a small thing, but a UI element like this can go a long way in helping users avoid huge headaches.

An XState Implementation

Here is the codesandbox if you want to dive right in. Below I'll talk about some of the XState concepts and features that stand out to me.

Nested States

State machines are hierarchical. They can be made up of simple and composite states. Composite states have sub-states nested within them. (See the stately viz)

Nested states are a great way to represent those concepts that feel like they need multiple booleans. This machine has an open state which is composite. While the modal is open, the machine can be in different sub-states. If the input doesn't match the confirmation text, then it stays in {open: "idle"}. Once the input matches the confirmation text, the machine will transition to {open: "confirmable"}.

Validations

The piece of this machine that I found trickiest to implement was validation of the input. If the input matches some criteria, I want to move to some valid state. If the input doesn't match, then I need to stay in or move to the invalid state.

I achieved this with an invoked service.

{
  services: {
    checkInputConfirmText: (context) => (send) => {
      console.log("Checking input confirm text: ", context.inputConfirmText);

      if (context.doubleConfirmText === context.inputConfirmText) {
        send("REPORT_MATCHING");
      }
    }
  }
}

An invoked service can send an event to the machine that invoked it which seemed like the perfect way to trigger the transition I needed. I also took advantage of the fact that an invoked service that exits cleanly will trigger an onDone action.

Each time this service is invoked, it will check the validation (do the two text strings match?) and then do one of two things.

The validation doesn't pass, it exits, and the onDone internally self-transitions back to the idle state.
The validation does pass, the REPORT_MATCHING event is sent, and the invoking machine transitions to the confirmable sub-state.

External Self-Transitions

The CHANGE event that is sent each time the modal's input value changes triggers an external self-transition to the idle state.

open: {
  exit: ["clearErrorMessage"],
  initial: "idle",
  on: {
    CANCEL: {
      target: "#closed"
    },
    CHANGE: {
      target: ".idle",
      internal: false,
      actions: "assignValueToContext"
    }
  },
  states: {
    idle: { /* ... */ },
    confirmable: { /* ... */ }
  }
}

A transition of { target: ".idle" } would be an internal transition. That would prevent the validation service from being re-invoked. But I want that service to be invoked on each change, so I include internal: false in there to make it an external transition.

Conclusion

There are lots of other interesting bits going on in this machine beyond what I highlighted. It is worth taking some time to read through it and see what stands out.

Implementing a machine like this was fun because it had a real-world use and I learned a lot while figuring it out. I learned new things about XState and I was pushed to think differently about how to model the problem as a state machine.

If you enjoy my writing, consider joining my newsletter or following me on twitter.

1/7 GUI Tasks with React and XState: Counter

Josh Branchaud — Tue, 06 Jul 2021 02:02:24 +0000

The first part of this article explores a couple learnings from implementing a Counter with XState and React. A Counter is the first of the 7 GUIs tasks. The second, lengthier part of this article will walk through a full explanation of my solution.

You'll get a lot out of the first part even if you don't want to read through the full walk through.

A Few Learnings

The Core of a State Machine

The state machine that backs this Counter is one of the most basic XState machines you can build. I find that instructive because it shows me, once I cut away all the other features, what is at the core of defining a functioning state machine.

const countingMachineDefinition = {
  initial: "counting",
  context: { count: 0 },
  states: {
    counting: {
      on: {
        INCREMENT: {
          actions: assign({
            count: (context) => context.count + 1,
          }),
        },
      },
    },
  },
};

The initial state that the machine will be in when it is first turned on.
The starting context that the machine will start with in its initial state. This is the secondary state, all the data beyond the current state itself.
A finite set of states, at least one, that the machine can be in. In this case I just have the counting state.
Each state can have a set of one or more events on which it will respond to with a transition and actions. In this case I just have the INCREMENT event. When this event is triggered in the counting state, it will transition to itself and an assign action will update the count in the context.

Self Transitions

A state's event that doesn't specify a target will implicitly do a self transition. In the state diagram, rather than the an arrow going from this state to another state, the arrow points to itself. This means that when that state receives that event, it will transition right back to itself. A transition always takes place.

Internal Transitions

Because the target wasn't specified at all for counting's INCREMENT event, the self transition will be an internal transition (as opposed to an external transition). This means that on this internal transition, we don't leave the current state node. The implications of that are that the entry and exit actions of that state will not be triggered.

Another, more explicit way to define an internal transition would be to specify the internal option as true.

  states: {
    counting: {
      on: {
        INCREMENT: {
          internal: true,
          actions: assign({
            count: (context) => context.count + 1,
          }),
        },
      },
    },
  },

Another explicit way of doing the same thing here is to say outright that the target is undefined.

  states: {
    counting: {
      on: {
        INCREMENT: {
          target: undefined,
          actions: assign({
            count: (context) => context.count + 1,
          }),
        },
      },
    },
  },

External Transitions

Out of curiosity, let's look at a self transition that involves an external transition.

  states: {
    counting: {
      on: {
        INCREMENT: {
          target: "counting",
          actions: assign({
            count: (context) => context.count + 1,
          }),
        },
      },
      entry: () => {
        console.log("Entering 'counting'");
      },
      exit: () => {
        console.log("Exiting 'counting'");
      },
    },
  },

We include the target option which points to the parent state, counting. To be sure that this brings back the entry and exit actions, I've add a couple logging actions. On each button click, we'll see the exit and then immediately the entry actions be triggered.

That's it... for my learnings from this super small state machine. If you are interested in digging into the full implementation keep reading.

Otherwise, thanks for reading. If you enjoy my writing, consider joining my newsletter or following me on twitter.

Full Implementation Walk Through

The first of the 7 GUIs tasks is to create a counter. This is a classic "Hello, World"-esque challenge for both UI frameworks and state management libraries. In our case, we are using React (a UI framework) and XState (a state management library). So we'll be exercising both aspects of this.

The task description is:

The task is to build a frame containing a label or read-only textfield T and a button B. Initially, the value in T is “0” and each click of B increases the value in T by one.

Counter serves as a gentle introduction to the basics of the language, paradigm and toolkit for one of the simplest GUI applications imaginable. Thus, Counter reveals the required scaffolding and how the very basic features work together to build a GUI application. A good solution will have almost no scaffolding.

The author of 7 GUIs describes the goal of this first task as: "understanding the basic ideas of a language/toolkit."

In that spirit, the very first thing we'll have to understand is the interplay between React and XState.

Let's start by installing both XState and its React bindings into our React application.

$ yarn add xstate @xstate/react

The part that is core to XState is being able to turn a JSON description of a machine into a machine. This is done with the createMachine function which we will import.

import { createMachine } from "xstate";

The React bindings part is when we interpret this machine definition into something that React can interact with the useMachine hook.

import { useMachine } from '@xstate/react';

Let's define a counting machine in a separate machine.js file.

import { createMachine } from "xstate";

const countingMachineDefinition = {
  initial: "counting",
  context: { count: 0 },
  states: {
    counting: {
      on: {
        INCREMENT: {
          actions: 'incrementCount',
        },
      },
    },
  },
};

export const countingMachine = createMachine(countingMachineDefinition);

This machine isn't quite ready, but it introduces most of the pieces we need to get our count on.

Our machine definition is made up, in this case, of initial, context, and states.

initial specifies the state that this machine should start in when it is first interpreted. Our starting state is counting. That's also our only state.
context is where we define an object containing any initial context for our machine. The only piece of context we are keeping track of is count. We will have it start at 0.
states lists the finite set of states that make up this state machine. At any given time, our machine is going to be in one of these defined states. This is an extremely simple state machine that has a single state—counting.

Let's look a little closer at the states definition.

  states: {
    counting: {
      on: {
        INCREMENT: {
          actions: 'incrementCount',
        },
      },
    },
  },

The counting state contains some information about itself. It tells us what events it responds to in the on object. Since we are only counting up, the counting state will only respond to the INCREMENT event.

Often the response to an event will be one or more actions as well as a transition to some other target state. This machine, only having one state, doesn't transition to another state. It implicitly does an internal self transition. It is like it is pointing to itself, but without making a show of it.

When the INCREMENT event is sent, the incrementCount action will be triggered. You might have noticed that there is no function definition for incrementCount.

In fact, if we were to start up this machine and send it the INCREMENT event, we would see the following warning in the console.

Warning: No implementation found for action type 'incrementCount'

We still have to implement that.

We can either replace the 'incrementCount' string with an inline function or we can define a function under that name in an actions section.

The function is small enough that I'll just replace the string.

import { createMachine, assign } from "xstate";

const countingMachineDefinition = {
  initial: "counting",
  context: { count: 0 },
  states: {
    counting: {
      on: {
        INCREMENT: {
          actions: assign({
            count: (context) => context.count + 1,
          }),
        },
      },
    },
  },
};

export const countingMachine = createMachine(countingMachineDefinition);

Notice I imported assign from xstate. It is being used to generate an action handler that will update the machine's context. The only context that needs updating is count. Similar to React, Redux, and other state management libraries, the context value is updated using a function that provides the current context and returns the updated context value.

So, each time the machine receives the INCREMENT event, it will trigger this assign({ ... }) action that increments the count. Each subsequent event, will be working with the newest version of the context which will contain the incremented count.

And that's it, that's the counter machine.

Here is how we can use it (in a React component).

import React from "react";
import { useMachine } from "@xstate/react";
import { countingMachine } from "../../src/machines/counter";

const Task1 = () => {
  const [state, send] = useMachine(countingMachine);

  return (
    <>
      <p>Count: {state.context.count}</p>
      <button onClick={() => send('INCREMENT')}>
        Increment
      </button>
    </>
  );

Each time the button is clicked, the INCREMENT event will be sent to the machine. The count context will be incremented and that value will trickle down to being rendered into the view via {state.context.count}.

If you enjoyed this post, consider joining my newsletter or following me on twitter. If this helped you or you have a question, feel free to drop me note wherever. I'd love to hear from you!

Test ActionMailer `deliver_later` in RSpec Controller Tests

Josh Branchaud — Tue, 29 Jun 2021 00:16:17 +0000

A lot can happen when a Rails controller action gets called. This includes transactional emails getting queued up for delivery. To ensure our controller's behavior stays consistent as our app evolves we can write RSpec tests.

Among other things these tests can ensure that transactional emails get queued for delivery at the appropriate times.

This post documents a couple different methods I've used for those tests.

`ActionMailer::Base.deliveries`

If you have your queue_adapter set to :inline, then a deliver_later will happen synchronously. So, the email will immediately end up in the deliveries box.

describe '#welcome' do
  it 'sends the welcome email to the user' do
    valid_params = { user_id: user.id }

    expect {
      post :invite, params: valid_params
    }.to change { ActionMailer::Base.deliveries.count }.by(1)
  end
end

At this point you could even write an additional test to look at properties of the email that was sent, like who it was sent to and what the subject line said.

`have_enqueued_job`

The behavior is a bit different if your queue_adapter is set to something like :test or async. In this case, the email is going to be queued in the app's job queue. Since it is not immediately being sent, the expectation will have to be about the job queue instead.

describe '#welcome' do
  it 'sends the welcome email to the user' do
    valid_params = { user_id: user.id }

    expect {
      post :invite, params: valid_params
    }.to have_enqueued_job(ActionMailer::DeliveryJob)
  end
end

We can even dig into more specifics about what mailer class and method were invoked, like this:

describe '#welcome' do
  it 'sends the welcome email to the user' do
    valid_params = { user_id: user.id }

    expect {
      post :invite, params: valid_params
    }.to have_enqueued_job(ActionMailer::DeliveryJob)
      .with('UserMailer', 'welcome', 'deliver_now', Integer)
  end
end

docs

Receive Block and Mail Double

This approach mocks the mailer so that we can test that deliver_later gets called. We take things a step further with the receive method by using its &block argument to make assertions about the values passed to the mailer method.

describe '#welcome' do
  it 'sends the welcome email to the user' do
    mail_double = double
    allow(mail_double).to receive(:deliver_later)

    expect(UserMailer).to receive(:welcome) do |user_id|
      expect(user_id).to match(user.id)
    end.and_return(mail_double)

    valid_params = { user_id: user.id }

    post :invite, params: valid_params
  end
end

ActionMailer RSpec Matcher

The previous approach requires a bit of boilerplate setup. If There is a way to go the (instance) double route, without duplicating this setup over and over. That can be achieved with a custom RSpec matcher. I've used some version of the following on many Rails projects.

# spec/support/mailer_matcher.rb
require "rspec/expectations"

RSpec::Matchers.define :send_email do |mailer_action|
  match do |mailer_class|
    message_delivery = instance_double(ActionMailer::MessageDelivery)
    expect(mailer_class).to receive(mailer_action).and_return(message_delivery)
    allow(message_delivery).to receive(:deliver_later)
  end
end

Assuming the spec helper requires support files, this custom matcher will be available in your specs. Here is how to use it.

describe '#welcome' do
  it 'sends the welcome email to the user' do
    expect(UserMailer).to send_email(:welcome)

    valid_params = { user_id: user.id }

    post :invite, params: valid_params
  end
end

These are the approaches I know about and use. If I'm missing an approach to testing ActionMailer, drop a note. I'd love to see how you're doing it.

If you enjoy my writing, consider joining my newsletter or following me on twitter.

References:

Cover photo by Timothy Eberly on Unsplash

Give your Postgres Queries More Memory to Work With

Josh Branchaud — Thu, 10 Jun 2021 23:41:17 +0000

Are you getting notices from Heroku warning that you are "Running out of temp space on your Postgres add-on"?

A potential fix to this is adjusting the work_mem value for the database. This will require some tuning to optimize the database for the particular query load that it experiences. So, some trial and error.

In this article, we'll walk through how to check your database's work_mem value and how to change it to a better value that will make this warning go away.

work_mem "specifies the amount of memory to be used by internal sort operations and hash tables before writing to temporary disk files." (source)

Before getting started, you'll want to take note of the specs of your Postgres instance where these errors occur. You'll be interested in both the available RAM and the number of supported connections.

Making More `work_mem`

The work_mem value tells Postgres how much RAM to let a query consume before starting to write intermediate results to the temporary disk. Giving queries permission to use more RAM is one way of reducing the amount of temporary space that gets written to. This will need to be balanced against how much RAM is being consumed overall. The work_mem value is likely set to the default of 4MB (4 megabytes) -- unless it isn't.

You can check this by running the following query in a psql session connected to the target database:

> show work_mem;
 work_mem
----------
 4MB
(1 row)

You can start by altering it to a slightly higher value. Too high and you could flip from "out of temp space" to "out of memory". Depending on the RAM size and number of connections, you could try going anywhere from 8MB or 16MB up to 64MB. This will be highly dependent on your data and query load, so some trial and error may be necessary.

Here is how you can apply that change from a psql session:

> alter database <database-name> set work_mem = '16MB';
ALTER DATABASE

It has altered the database, but that change will be applied only to new sessions. If you check the work_mem for the existing session, it will appear as if it hasn't changed.

> show work_mem;
 work_mem
----------
 4MB
(1 row)

Exit this psql session and start a new one to check that value again. You show now see that it has been updated to what was specified in the alter statement.

> show work_mem;
 work_mem
----------
 16MB
(1 row)

Unlike some config changes, this is applied immediately to new connections. There is no need to restart the Postgres server. This change will persist between restarts as well.

So, how do you pick the right `work_mem` value for your database and its query load?

Some rough math on picking a value:

If you have 8GB of RAM and you bump work_mem up to 16MB, then it would take 512 connections all maxing out their work_mem space to cause an out of memory error. Whereas if you bumped it to something like 64MB, then you're looking at ~128 connections.

Again, these are really rough numbers and also doesn't account for base RAM needs of the Postgres instance. It will all depend on the query load that the DB experiences. Depending on how this change performs, you can make further adjustments either up or down to dial in a work_mem value that makes most sense.

If you enjoy my writing, consider joining my newsletter or following me on twitter.

Notes

If you are using a managed database service like Heroku, the default setting for your work_mem value may be dependent on your plan. For instance, Heroku's Standard 4 plan has a default work_mem value of 110MB. source

Resources and Further Reading

Cover photo by Harrison Broadbent on Unsplash

Tackle that Big Task

Josh Branchaud — Fri, 04 Jun 2021 00:11:49 +0000

Sometimes a programming task (or really any task) feels like a really big unknown. The thing to do at that point is to break it down into a bunch of smaller unknowns.

Go through them one at a time by exploring, experimenting, and learning so that each one is no longer an unknown. After a bit of this, you may want to revisit the big unknown.

The first thing you'll notice is that it is not so big anymore. The second thing you might notice is that what you now know changes your perspective on the problem. You realize the original trajectory you had when it was one big unknown was a bit off target.

This is an opportunity to adjust. This reframing will lead you to ask new and different questions. It will send you down the path of unlocking another set of small unknowns that propel you forward.

Big, daunting tasks can kill momentum. Breaking those big tasks down and making steady progress creates momentum.

There is no triple-quote multi-line string syntax in Ruby

Josh Branchaud — Tue, 01 Jun 2021 20:32:59 +0000

A lot of languages distinguish between single-line and multi-line strings. They have specialized syntax for declaring one or the other.

Not Ruby.

In Ruby, any string can be a multi-line string if you hit the Enter key a few times.

Nevertheless, there are conventions. When I want a multi-line string, I tend to reach for a heredoc. Another convention I've seen in a few codebases are these triple-quote strings.

Book.where("""
  where publication_year >= 2000
    and page_count > 200
    and page_count < 500
    and author_id = #{author.id}
""")

Like me, you probably look at that triple-quote syntax and think, "yeah, I'm pretty sure I've seen that before."

// Detect dark theme var iframe = document.getElementById('tweet-1398059592555405312-381'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1398059592555405312&theme=dark" }

Fun Side Quest: try finding where in the Ruby docs this syntax feature is discussed.

Don't spend too long searching though. As the title of this post suggests, there is no triple-quote multi-line string syntax.

So, what gives? Why does this work and where did it come from?

Let's start with "why does this work?"

First, let's demonstrate that we can do multi-line strings with the "standard" double-quote syntax.

> puts "
  This is a multi-line string.
"

  This is a multi-line string.
=> nil

It prints out the string, newlines and extra spaces included. And as expected, puts returns nil.

Let's look at another completely different example.

> puts "a" + "b" + "c"
abc
=> nil
> puts "a" "b" "c"
abc
=> nil

These two lines do identical things. A feature of the Ruby parser and interpreter—actually not a feature so much as an obtuse implementation detail adopted from C—is that adjacent strings are concatenated together (source).

Let's see how that plays into our multi-line string example.

> puts "" + "
  This is a multi-line string.
" + ""

  This is a multi-line string.
=> nil

This explicitly concatenates empty strings on to either end of our multi-line string with the + operator.

As we just learned, that is equivalent to the following.

> puts "" "
  This is a multi-line string.
" ""

  This is a multi-line string.
=> nil

This omits the + operators, but has identical behavior.

We can take this a step further by removing those superfluous spaces.

> puts """
  This is a multi-line string.
"""

  This is a multi-line string.
=> nil

Again, the same behavior.

So what appears to be some special triple-quote multi-line string syntax is instead a syntactic trick that exploits a Ruby interpreter feature where adjacent strings are concatenated (source).

On to the other question, "where did this come from?"

The best I can tell is that this syntactic convention was introduced to Ruby developers through Learn Ruby The Hard Way.

I hope that was enlightening or at least cleared up some confusion. For my part, I will continue to reach for heredocs when appropriate and do my best to remove this pseudo-syntax from the Ruby codebases I work in.

If you enjoy my writing, consider joining my newsletter or following me on twitter.

Cover photo by Severin Höin on Unsplash

Upgrade a Rails app from the Heroku-16 to the Heroku-18 Stack

Josh Branchaud — Sun, 30 May 2021 00:44:42 +0000

I'm working with a client to urgently upgrade their Rails app from the Heroku-16 stack to the Heroku-18 stack. Heroku has already end-of-life'd the Heroku-16 stack at the beginning of this month. And in a few days, on June 1st, they will completely turn off support patches and new builds of Heroku-16 apps.

The Heroku-18 stack is supported until April 2023 (source).

The minimum Ruby version supported by the Heroku-18 is Ruby 2.4.10 (source).

Testing the Ruby Upgrade Locally

Install the target Ruby version:

$ asdf install ruby 2.4.10

Switch to that Ruby version:

$ asdf local ruby 2.4.10
$ ruby --version
ruby 2.4.10p364 (2020-03-31 revision 67879) [x86_64-darwin19]

Install an older version of bundler (pre 2.x):

$ gem install bundler:1.17.3

Install all the app's gems with for the new Ruby version using the specific version of bundler:

$ bundle _1.17.3_ install

Start up the server locally (connected to whatever DB makes sense) and try things out:

$ bundle exec rails s

Deploying a Preview App

A Preview App is a great way to get this app up and running in a production-like environment. This is an opportunity to further kick the tires of the app. Are there particularities of the production environment and settings or of Heroku that make this app not work? If so, it is best to find out here rather than in production.

This will involve deploy a branch with a new Ruby version to a Preview app on a newer Heroku stack.

Here is a Heroku guide on upgrading the stack of an app.

The recommended way to do this is to create a branch for your app, alter the stack in app.json and then deploy a preview app (actually called Review Apps) to Heroku. It will use the stack version specified in that JSON file.

{
  "stack": "heroku-18"
}

Deploying the Changes to Production

I initially tried to push the Ruby upgrade changes and the app.json Stack changes directly to production. This led to the Ruby upgrade changes being deployed on the old stack. Despite the changes made to app.json, Heroku doesn't upgrade the stack for an existing app that way.

An existing app’s stack cannot be changed using app.json. The stack specified is only applied to newly created apps that are a Review App, a Heroku CI test run app, or an app created using a Heroku Buttons. source

Fortunately, the Ruby upgrade changes deployed fine to the existing stack. And the app was able to build and deploy the existing stack because we are still 2 days out from the full deprecation.

How I should have done it was to first set the stack for the app using the Heroku CLI.

$ heroku stack:set heroku-18 -a <app name>

Heroku will apply that stack upgrade on the next build. So once that is set, I should have then pushed up the Ruby upgrade changes.

Because I did it out of order, I had to trigger another production deploy by pushing up an empty commit.

$ git commit --allow-empty -m "Upgrading to heroku-18"

Once that empty commit is on the main branch, I then pushed that directly to the remote's main branch (git push origin main). This works because I have GitHub integrated with Heroku so that any changes pushed to the main branch will trigger a deploy. If that wasn't the case, you can trigger a deploy the Heroku way by pushing to the Heroku remote (git push heroku main).

If you enjoy my writing, consider joining my newsletter or following me on twitter.

Cover photo by Charlota Blunarova on Unsplash

Beware The Missing Foreign Key Index: A Postgres Performance Gotcha

Josh Branchaud — Thu, 27 May 2021 16:35:14 +0000

I got absolutely thrashed by some PostgreSQL performance issues this past week. It was painful. And, it was incredibly instructive. I didn't just solve the problem, I now better understand Postgres. I'm better equipped to diagnose the next performance skirmish that comes my way. And I'm paying the lesson forward with this article.

Let me give you an idea of how bad it was. I had a query deleting 50k records on one table in ~100ms. That's fast enough for me. I then had a similar query deleting the same number of records from another table that would run for ~30 minutes. 10,000x slower! 😱 That's a nope.

After diagnosing and addressing the issue, I got that second delete query down to about 1 second. Much better! Toward the end of the post we'll see how to make it even faster than that.

But before I reconstruct a minimal example that reproduces the problem and get into the details, here is the...

tl;dr

Foreign keys are essential for enforcing the shape and integrity of our data. Indexes are there to keep queries fast. We can combine the two to start getting the most out of our database.

I don't tend to think of foreign key constraints as impacting performance. As we are about to see, they do.

Without that index, we're bound to run into some really sneaky performance gotchas.

Let's dig in.

A Minimal Example

This minimal example is also a real-world example. Most software systems have users and those users need to be given a variety of roles. These roles help the system determine the access and permissions of each user.

If you want to skip over the example setup, you can jump right to the details.

The Data Model

First is the users table. It doesn't feature strongly in the example so I'll leave it at the id column.

Then we have the roles table which can contain global roles like admin and support as well as roles tied to specific resources like team_owner and team_collaborator.

Creating a relationship between the two is the users_roles join table. A record in this table tying my user record to the admin role would tell the system that I am an admin. Another record in this table tying my user record to a team_collaborator role for the Red team would tell you that in addition to being an admin, I am a collaborator of the Red team.

Building The Schema

Here is some SQL that will generate our schema for reproducing the performance issue. The specific of how this works are out of scope for this post. If that interests you, let me know and I'll write a follow up.

The full queries are embedded in these code blocks, but you can also check out the code in this repo.

create table users (
  id bigint generated by default as identity primary key
);
create table roles (
  id bigint generated by default as identity primary key,
  name varchar not null,
  resource_type varchar,
  resource_id bigint
);
create table users_roles (
  user_id bigint references users,
  role_id bigint references roles
);

I put this in a file called schema.sql. From a psql session I connect to an empty database where I can safely experiment. I then tell psql to execute the script which will create these three tables and their sequences.

> \i schema.sql
CREATE TABLE
Time: 2.605 ms
CREATE TABLE
Time: 2.615 ms
CREATE TABLE
Time: 1.141 ms
Time: 0.099 ms
> \d
               List of relations
 Schema |     Name     |   Type   |   Owner
--------+--------------+----------+------------
 public | roles        | table    | jbranchaud
 public | roles_id_seq | sequence | jbranchaud
 public | users        | table    | jbranchaud
 public | users_id_seq | sequence | jbranchaud
 public | users_roles  | table    | jbranchaud
(5 rows)

Generating Some Data

Let's fill these tables with some data with help from the generate_series function. Again, I won't go into specifics here. Drop a note if you're interested in a follow up post on this part.

-- create 50,000 users
insert into users
select from generate_series(1,50000);

-- create 100,000 roles
insert into roles (
  name,
  resource_type,
  resource_id
)
select
  (array['admin', 'support', 'member'])[floor(random() * 3 + 1)],
  null,
  null
from generate_series(1,50000);

insert into roles (
  name,
  resource_type,
  resource_id
)
select
  'team_collaborator',
  'Team',
  floor(random() * 1000 + 1)
from generate_series(1,50000);

-- create 500,000 connections between users and roles
-- start with 225,000 random global roles
insert into users_roles (
  user_id,
  role_id
)
select
  floor(random() * 50000 + 1),
  floor(random() * 50000 + 1)
from generate_series(1,225000);

-- then 50,000 for the team collaborator role
insert into users_roles (
  user_id,
  role_id
)
select
  floor(random() * 50000 + 1),
  floor(50000 + g.id)
from generate_series(1,50000) as g(id);

-- then another 225,000 random global roles
insert into users_roles (
  user_id,
  role_id
)
select
  floor(random() * 50000 + 1),
  floor(random() * 50000 + 1)
from generate_series(1,225000);

This creates 50,000 users and 100,000 roles. It then creates 500,000 connections between them in the users_roles join table, with 50,000 of those connection specifically for the team_collaborator role.

With all the data in place, let's find some specific data and delete it. This is where we'll see a reasonable query and a drastically slower query.

Fast Query, Slow Query

In the actual production data set I was dealing with, I had a bunch of data that I needed to clear out of the roles table. It turned out to be about 50k records. Because of the foreign key relationship, I first had to clear dependent records out of the users_roles table.

The following might feel a bit contrived, but it is based on that real scenario.

Here are the 50k roles that we are targeting for deletion.

> select count(*)
  from roles
  join users_roles
    on roles.id = users_roles.role_id
  where roles.name = 'team_collaborator';

 count
-------
 50000
(1 row)

Why are we going to be deleting all the team_collaborator related data? Because that gives us about 50k rows. Like I said, a bit contrived. Nevertheless, we are going to learn some performance things from this.

The Quick Delete

Let's delete the dependent records from users_roles first.

Anytime I'm about to modify or delete data, I like to open a transaction. This makes it easy to safely rollback the changes if anything looks off. That is done with a begin.

> begin;

> delete from users_roles
  using roles
  where users_roles.role_id = roles.id
    and roles.name = 'team_collaborator';

DELETE 50000
Time: 116.313 ms

That looks good. That query appears to have handled the 50k records from users_roles that I wanted deleted. Because I'm in a transaction, I can always dig into the data with a few select queries to be sure. And it was fast enough, clocking in at ~100ms in this instance.

I'm not ready to commit this transaction yet. Next I want to attempt to delete the roles records.

The Slow Delete

With all the foreign key dependencies taken care of, we are clear to get on with our main goal, deleting the roles records.

This query is even simpler than the previous. Typing it out, I was certain it would run just as quickly.

> delete from roles
  where roles.name = 'team_collaborator';

-- ... keep waiting ...

For real though, if you are following along at home, don't hold your breath. This is going to take more or less 30 minutes depending on the specs of your machine.

What gives?

Can We Explain This?

This query isn't all that different than the previous one. It is targeting the same number of rows. What could account for the slow down?

Let's start by taking a look at the explain.

> explain delete from roles
  where roles.name = 'team_collaborator';

                            QUERY PLAN
------------------------------------------------------------------
 Delete on roles  (cost=0.00..1937.00 rows=49990 width=6)
   ->  Seq Scan on roles  (cost=0.00..1937.00 rows=49990 width=6)
         Filter: ((name)::text = 'team_collaborator'::text)
(3 rows)

There isn't a lot going on in this one. The seq scan stands out to me. This means the query planner expects the query will have to look at every row sequentially. For a mere ~50k rows that doesn't seem like it should be an issue.

By contrast, here is the explain for the previous query.

> explain delete from users_roles
  using roles
  where users_roles.role_id = roles.id
    and roles.name = 'team_collaborator';

                                   QUERY PLAN
--------------------------------------------------------------------------------
 Delete on users_roles  (cost=2561.88..11577.43 rows=249950 width=12)
   ->  Hash Join  (cost=2561.88..11577.43 rows=249950 width=12)
         Hash Cond: (users_roles.role_id = roles.id)
         ->  Seq Scan on users_roles  (cost=0.00..7703.00 rows=500000 width=14)
         ->  Hash  (cost=1937.00..1937.00 rows=49990 width=14)
               ->  Seq Scan on roles  (cost=0.00..1937.00 rows=49990 width=14)
                     Filter: ((name)::text = 'team_collaborator'::text)
(7 rows)

This one is more involved. I'm noticing that there is a seq scan of users_roles (500k rows) and nested under that is another seq scan of roles (~50k). In terms of predicted work, this one has a ton more to do, yet it finished in 100ms. Computers can be fast.

Despite that comparison, we know from running the queries that the first (simpler) one is going to take much longer than the second one. Some back of the napkin math says the simpler query plan for deleting from roles is looking to take 10,000x the amount of time of the second query 😱.

So that's the second scream emoji in this post—we've got to figure out what is going on.

What's the deal with this "simple" delete query?

The Mystery Uncovered

> delete from roles
  where roles.name = 'team_collaborator';

The issue with this two-line, 50k row query doesn't have anything to do with the roles table itself. Instead, it has to do with the relationship that the roles table has to other tables.

The users_roles table depends on the roles table—specifically, on its id column. This dependency on roles.id is through the foreign key constraint on its own role_id column.

The foreign key is causing this

Kinda.

To demonstrate that the foreign key constraint is at the center of this performance issue, let's start by removing the foreign key constraint.

> begin;

> delete from users_roles
  using roles
  where users_roles.role_id = roles.id
    and roles.name = 'team_collaborator';

DELETE 50000
Time: 120.267 ms

> alter table users_roles
  drop constraint users_roles_role_id_fkey;

ALTER TABLE
Time: 0.561 ms

> delete from roles
  where roles.name = 'team_collaborator';

DELETE 50000
Time: 54.115 ms

> rollback;

With the foreign key constraint out of the picture, the delete is quite speedy. Notice I immediately rolled these changes back. This was to highlight the impact of the constraint. This isn't the solution.

Keep the Foreign Key Constraint

Foreign key constraints are essential to solid schema design, and they are one of my favorite features of Postgres. They help ensure that the data relationships between two tables are always intact (that's called referential integrity). Because of this assurance, I never have to worry that data is going to be inadvertently orphaned.

How is the foreign key constraint having such an impact on performance?

For every single row you tell Postgres to delete from the roles table, it is going to first check with all dependent tables to make sure it is okay to do that. If the foreign key value isn't being used by any records in the associated table(s), then the delete can proceed. If even just one record depends on that value, the foreign key constraint is going to flag that and abort the delete action.

So, even though we preemptively deleted all the related data from users_roles, Postgres still must check at the moment of the delete if that is still true.

Postgres goes through the users_roles table and does a sequential scan checking every single record. For each roles record, Postgres is going to check 500,000 users_roles records.

And there is the root of the problem.

The Query Planner Needs More Info

Postgres is going to use the information at its disposal to verify that the foreign key constraint. As is, it doesn't have enough info to do anything better than look at every single row. If Postgres had an index though, it would have the info necessary to do a lot less work.

A database index is roughly like the organizational system that a library uses. Books are categorized and then they have a placement relative to other books in their same category. Using this system, you can quickly navigate to where a book should be.

Not having an index on users_roles.role_id is kinda like walking into a library without an organizational system, you have to start on one end and go book by book, perhaps through tens of thousands of books, until you find what you're looking for.

We can cut our delete query down to around 1 second by adding an index to users_roles.role_id. The index provides Postgres with the info it needs to quickly shortcut its way to any dependent rows.

create index role_id_index
  on users_roles (role_id);

CREATE INDEX
Time: 342.591 ms

This adds an index to the users_roles.role_id column. It is a B-Tree index. "The PostgreSQL query planner will consider using a B-tree index whenever an indexed column is involved in a comparison" (with operators like =, <, >, etc.).

With this index, we'll tend to get faster queries, especially with this delete. One tradeoff is that the index will take a little disk space. Since Postgres automatically updates the index on each write to the indexed column, that adds a teeny tiny bit of write time. Both of these tradeoffs are well worth it in the vast majority of cases.

Let's see how the index performs.

Deleting Faster

With that index in place, let's give both of our delete queries another try.

> begin;

> delete from users_roles
  using roles
  where users_roles.role_id = roles.id
    and roles.name = 'team_collaborator';

DELETE 50000
Time: 111.732 ms

> delete from roles
  where roles.name = 'team_collaborator';

DELETE 50000
Time: 590.668 ms

In the blink of an eye it is done. The second delete query went from around 30 minutes to under 600ms. That's a massive difference. That's what indexes can do for us.

That brings me back to the tl;dr of this post.

Foreign keys are essential for enforcing the shape and integrity of our data. Indexes are there to keep queries fast. We can combine the two to start getting the most out of our database.

Without that index, you're bound to run into some really sneaky performance gotchas.

But can we go even faster?!

Deleting Even Faster

Since we are in a transaction and we are doing a ton of deletes, what if we tell Postgres to hold off on verifying the foreign key constraint until we're done with the deletes.

> begin;

> alter table users_roles
  alter constraint users_roles_role_id_fkey
    initially deferred;

ALTER TABLE
Time: 0.455 ms

> delete from users_roles
  using roles
  where users_roles.role_id = roles.id
    and roles.name = 'team_collaborator';

DELETE 50000
Time: 114.574 ms

> delete from roles
  where roles.name = 'team_collaborator';

DELETE 50000
Time: 68.433 ms

Whoa, that made it another 10x faster, coming in at ~68ms.

Instead of checking the foreign key constraint 50,000 times, even with the index, it can check it once at the end. Zoom!

Wrapping Up

After running into this incredibly slow delete query, I went on quite a journey trying to understand what was causing the performance hit. There is a lot to be learned by being willing to really dig into a hard problem and try to understand, even if you can only understand it a little bit at a time. I came out the other end better for it. And I hope you are at the other end of this article better for having read it.

Foreign keys help ensure the integrity of your data. Indexes help keep your queries fast. Combine the two to really take advantage of the speed and power of PostgreSQL.

If you enjoy my writing, consider joining my newsletter or following me on twitter.

Notes

An improvement to this schema would be to add an on delete cascade directive to the role_id foreign key constraint. This would mean we could go right to deleting the roles record and the corresponding users_roles records would be automatically removed. This would still suffer from the same performance issues.
Postgres allows you to add indexes concurrently to avoid locking writes to the entire table. Though adding an index is a pretty quick action, it is a good rule of thumb to add them concurrently. This is especially true when dealing with live tables in a production environment. I chose not to add our index concurrently in this article to avoid tossing one more concept into the mix.

Acknowledgements

A big thanks to several people who read through and provided feedback on earlier drafts of this post. They are awesome. Check out their stuff.

Simon Willison, who is doing awesome work on datasette
Chris Toomey, who does fantastic write ups on Vim and Postgres
Jordan Lewis, who has an engaging weekly stream on Go and databases
Dillon Hafer
Max Rozen

Cover photo by Ryan Johnston on Unsplash

Use an XState Machine with React

Josh Branchaud — Sat, 08 May 2021 22:21:21 +0000

XState gives you the tools to take control over the state of your UI. When you've got it under control, you can build interfaces that provide a predictable and delightful user experience.

Let's look at how to integrate XState into a React app.

There are a bunch of well-constructed XState machines available to directly copy into your project from XState Catalogue. For instance, I can interact with and then grab the Confirmation Dialog machine with the 'Copy' button.

I'll then paste that machine definition into something like confirmMachine.js. XState is framework agnostic, so there is nothing about this machine, on its own, that has anything to do with React or Vue or Svelte or whatever. I do want to use this within a React app, so I then need to grab @xstate/react. XState's React "bindings" come with a useMachine hook.

An Example

Here is what that will look like.

import * as React from "react";
import { useMachine } from "@xstate/react";
import confirmMachine from "./confirmMachine";
import Dialog from "./dialog";

export default function App() {
  const [current, send] = useMachine(confirmMachine);

  return (
    <div className="App">
      <Dialog
        message="Are you sure you want to delete something?"
        {/* other props ... */}
      />
      {/* other stuff */}
    </div>
  )
}

The useMachine call both interprets and starts up the machine service. This hook gives you two values as an array. The current value is everything about the current state of the machine. The send is a function for dispatching transitions between machine states.

The Current State of the Machine

With current I can figure out the current state of the machine to determine whether or not I should be showing the dialog. current.value will tell me what state the machine is in.

I can also get access to any error message that comes from the machine.

import * as React from "react";
import { useMachine } from "@xstate/react";
import confirmMachine from "./confirmMachine";
import Dialog from "./dialog";

export default function App() {
  const [current, send] = useMachine(confirmMachine);

  const showDialog = current.value !== "closed";

  return (
    <div className="App">
      <Dialog
        message="Are you sure you want to delete something?"
        showDialog={showDialog}
        errorMessage={current.context.errorMessage}
      />
      {/* other stuff */}
    </div>
  )
}

Notice I check current.value !== "closed" to determine whether or not the dialog should be showing.

Moving Between States with Send

I can now incorporate the send function into some handlers so that users can interact with the dialog. I'll create a handler for opening, closing, and confirming the dialog.

import * as React from "react";
import { useMachine } from "@xstate/react";
import confirmMachine from "./confirmMachine";
import Dialog from "./dialog";

export default function App() {
  const [current, send] = useMachine(confirmMachine);

  const deleteAction = () => { /* ... */ };

  const showDialog = current.value !== "closed";
  const open = () => {
    send({ type: "OPEN_DIALOG", action: deleteAction });
  };
  const close = () => {
    send("CANCEL");
  };
  const confirm = () => {
    send("CONFIRM");
  };

  return (
    <div className="App">
      <Dialog
        message="Are you sure you want to delete something?"
        handleConfirm={confirm}
        handleClose={close}
        showDialog={showDialog}
        errorMessage={current.context.errorMessage}
      />
      {/* other stuff */}
      <button onClick={open}>Delete Something</button>
    </div>
  )
}

The open handler when called will transition the machine to open.idle using the OPEN_DIALOG event. It also includes an action which will be called if the dialog is confirmed. When triggered, this will cause the showDialog value to evaluate to true. This handler is wired up to some element outside of the dialog, in this case a button.

The close handler is passed to the dialog. When called this sends the CANCEL event to the machine. That will transition the machine back into the closed state. This change will cause the showDialog value to evaluate back to false. Any user action that should dismiss the dialog will trigger this handler.

Once the dialog is open, the user can confirm the dialog's prompt by clicking a 'Confirm' button. This will call the confirm handler which will send the CONFIRM event to the machine. When the machine receives this event it will trigger the action given on OPEN_DIALOG.

Wrapping Up

There are more details to this specific machine. Depending on whether the action's promise resolves or rejects, the machine will take a different course of action. That's an exercise for the reader or the subject of another post.

At this point, we have explored enough of XState in a React context that you can start using the two together. If you'd like you can start by interacting with and remixing the codesandbox example I used for this post.

There are a lot of moving parts when getting started with XState, so if you have questions about what was covered here, feel free to drop me a note on twitter.

If you enjoy my writing, consider joining my newsletter.

Cover photo by Ball Park Brand on Unsplash

Forem: Josh Branchaud

Reduce a JSON Object to just Entries of a Specific Type with jq

Breaking Down a Complex Mapped Type

Speeding Up An Expensive PostgreSQL Query: B-Tree vs. BRIN

The Data Model

The Slow Query

BRIN vs. B-Tree

With BRIN Index

With B-Tree Index

Strong Confirmation Modal with XState

Strong Confirmation

An XState Implementation

Nested States

Validations

External Self-Transitions

Conclusion

1/7 GUI Tasks with React and XState: Counter

A Few Learnings

The Core of a State Machine

Self Transitions

Internal Transitions

External Transitions

Full Implementation Walk Through

Test ActionMailer `deliver_later` in RSpec Controller Tests

ActionMailer::Base.deliveries

have_enqueued_job

Receive Block and Mail Double

ActionMailer RSpec Matcher

Give your Postgres Queries More Memory to Work With

Making More work_mem

So, how do you pick the right work_mem value for your database and its query load?

Notes

Resources and Further Reading

Tackle that Big Task

There is no triple-quote multi-line string syntax in Ruby

Let's start with "why does this work?"

On to the other question, "where did this come from?"

Upgrade a Rails app from the Heroku-16 to the Heroku-18 Stack

Testing the Ruby Upgrade Locally

Deploying a Preview App

Deploying the Changes to Production

Beware The Missing Foreign Key Index: A Postgres Performance Gotcha

tl;dr

A Minimal Example

The Data Model

Building The Schema

Generating Some Data

Fast Query, Slow Query

The Quick Delete

The Slow Delete

Can We Explain This?

The Mystery Uncovered

The foreign key is causing this

Keep the Foreign Key Constraint

The Query Planner Needs More Info

Deleting Faster

Deleting Even Faster

Wrapping Up

Notes

Acknowledgements

Use an XState Machine with React

An Example

The Current State of the Machine

Moving Between States with Send

Wrapping Up

`ActionMailer::Base.deliveries`

`have_enqueued_job`

Making More `work_mem`

So, how do you pick the right `work_mem` value for your database and its query load?