Forem: Timothy Ecklund

Backend Architecture

Timothy Ecklund — Mon, 04 Apr 2022 00:01:24 +0000

I’ve been a backend engineer for a while. Like, a while. Here is an actual picture of me designing my most recent backend architecture:

It's been my experience that almost all architectures come down to a small set of general patterns. In this article I’m going to lay out the various patterns, what problems they solve, and when they are appropriate.

What is a backend system? A backend system is a centralized place where data is processed, stored and made available for access. Does that sound super general? It is! But there are some important details in that description. For one, it's a “centralized place”. It doesn’t happen in your browser, or on your phone or inside a random frog. It happens on a server somewhere at a location that is well known (ie. has an address like google.com). This well-known location processes and stores data - for example, when you send a direct message to someone it goes to a backend system where it gets stored. Then later, the recipient can see that message by logging into the system and looking at their messages.

The most critical thing that backend systems do is care for data. Imagine you are sending someone a saucy DM, and you typed “I never want you to leave,” but the system mangled your message and what they received was “I want you to leave.” Or, maybe worse, your message gets sent 1000 times. Or it says sent, but never was. Or it was sent to the wrong person. Good backend architecture guarantees that these things will never happen.

Pictured: An architect who takes good care of your data

Backend techniques we’re going to cover in this article:

Synchronous processing, Synchronous error handling
Asynchronous processing, Synchronous error handling
Asynchronous processing, Asynchronous error handling
Broadcasting without guarantees
Broadcasting with guarantees
Batching - when and how to do it

Overview

All of the above backend systems are composed of these building blocks:

System input -> processing (stateless service) -> sink (stateful service)

where each sink can become the system input for another process.

Why do we split stateless processing from stateful sinks? Because it makes everything much easier. Let’s talk about what we need to do:

Safeguard the data by making sure something is always responsible for its safety
Handle all the requests that come to us without overloading and crashing too much
Validate data before accepting it

These things can be shockingly difficult to accomplish, especially at scale. It's easy to set yourself up for failure - like maybe you try to batch requests in your stateless service before putting them into a sink for performance reasons. What happens when the service unexpectedly goes down? You told the user that you took responsibility for the data, but then dropped their baby on the floor before handing it off. Poor form. Party foul.

Pictured: An architect who batches data in their API

Or maybe you think to yourself, this is a small little thing, we won’t need to worry about scaling. We’ll just have one server and store state in memory. If it gets wiped no big deal, the user just gets logged out or whatever. Cool cool. Now your service is crashing constantly because people are actually using it and you keep running out of memory. You’re at the biggest server your cloud service provides, it's costing an arm and a leg and now you need to redesign your entire system in the middle of a running production outage. Not fun. Not fun for you and not fun for your users.

Pictured: An architect who did not design for scale

Or maybe you think, it's too hard to get the user to provide us with valid data, we’ll just accept whatever they give us and sort it out later. You have just opened yourself up to an absolute world of pain. Now every time the user posts some new nonsense data you have to comb through it, determine what fields mean what and try to massage it into something you can make sense of. This is a special circle of integration hell, where nothing works and no one knows why. A squishy interface is never OK.

Pictured: A backend engineer manually trying to fix the database because data wasn’t properly validated

We split things out into stateless and stateful services because doing so solves these problems. You have guarantees about data integrity. You can (more) easily scale your system when you inevitably need to. You have a specific place that is responsible for gatekeeping and rejecting bad data before it gets into storage. I cannot overstate this enough - the more guarantees you put at the edge of your system, the simpler all of your downstream systems will be, the fewer bugs you will have and the more people will like you.

Stateless-ness

A stateless service provides only one guarantee - a request is either processed successfully, or it is not. If the service dies in the middle of a request, the client is guaranteed to know that the request failed. The client will not be given a cheery 200 response, then find out later that their request was dropped on the floor.

Stateless services have no memory of what has happened, or what requests they did or did not process. They don’t guarantee that the data will be there later. They don’t guarantee that you will find love. They have one purpose in life, and that is handling requests, telling you if they succeeded or failed, and then immediately forgetting about them. They never (never!) take responsibility for the data. If something takes responsibility for data, it is by definition a stateful service.

Imagine a stateless service as a bank teller. They sit in front of the bank, and they provide access to your money (data), but they don’t actually keep your money (data) in their pockets. They take it from you and put it directly into the bank, or they take it from the bank and give it to you, but they never say, ‘Oh, I’ll just put this in my pocket and put it in the bank later’.

Examples of stateless services are: APIs that store nothing, Lambda functions, Cron jobs and Batch jobs.

Stateful-ness

Stateful services are quite different - they are the bank. Stateful services exist to store and protect your data. They swaddle it up and care for it, making sure it never gets lost or corrupted. Your users should never access it directly for the same reason you should never access the bank directly - that’s what tellers are for.

Stateful services are of course much more difficult to scale. It’s easy to get new tellers, but it's hard to set up a whole new bank. Cloud providers have gotten better about this over time, and if you need to do it, you should use whatever system your cloud provider has set up. If you find that you are scaling a stateful system yourself, re-evaluate your architecture immediately because you’ve done something wrong. (Or you're in the extreme minority of people who legitimately need to run an unmanaged database. In this case, may God have mercy on your soul.)

Examples of stateful services are: Databases, Queues, File systems, APIs that store requests or logs, and External APIs (APIs that you transfer ownership of data to)

Sink/Database Terminology

For the rest of this article, the word “database” is synonymous with “sink”. Databases are most commonly used so that’s what I’ll use in the examples, but really anything that takes responsibility for data could be swapped in.

Architectures

Synchronous processing, Synchronous error handling

This is the most common backend architecture. It takes some input, validates that the input is in the expected shape (number between 1-100, or string less than 10 characters), stores data in the sink and then returns a success.

In the case of failure, for example if the provided input was 300 when it was supposed to be less than 100, an error is returned to the caller.

You might ask, where is logging in all this? As the provider of the service, I want to know when an error happens! Thanks for the easy question that I made up for myself to answer! You are not limited to a single sink, you can write your logs to another database, like Elasticsearch or Datadog or some-such. Something like this:

Look Ma, logs!

But Tim, you say. What if Elasticsearch is down, but I still want my logs. Is ES now a dependency? Shouldn’t I write to disk just in case?

The answer is no. If you write your logs to disk, then this is no longer a stateless service. The logs are State with a capital S, and your service is now responsible for caretaking them. You will lose all of the niceties of a stateless service, and are now faced with the complexity of a stateful service. In other words - don’t do it. You will be sad. Maybe not today, and maybe not tomorrow, but soon.

Pictured: You being sad because you turned your stateless service into a stateful one

If indeed you do not wish to be dependent on your sink being always available, there are other options open to you. Since queues are more reliable than Elasticsearch (way more, ES goes out more often than a narcoleptic sloth) you can always push your logs into a queue, something like:

This is what Kafka was designed for, and it works fine. BUT you might say. ‘Hmm. What if my queue goes down? I’ve just replaced one dependency with another.’ First, I must ask you, what are your requirements here? Is it good enough to just say, ‘well when your queue is down the system is down’? In almost all circumstances you can harden your database and queue such that the likelihood of either of them going down is extremely minimal. You can totally hit your SLAs and everything is copacetic. This architecture is good and fine, and you should use it 99.99999% of the time, because it's simple and fine. It’s fine. For real.

BUT. Let’s say for the moment that either you have a very sadistic boss, or you really do work at a bank or something and any downtime at all is a super big deal. You can invert the pattern like this:

The API in this case ONLY does validation, and then puts the request on a queue for actual processing and storage. (You’ll notice this is actually the “Asynchronous processing, Synchronous error handling” pattern from later in the article.) The benefit of this architecture is that you’ve decoupled “receiving” a request from “processing” a request. Now you’ve limited your possible failures to only the gateway into the system. The gateway can remain simple and pure with as few dependencies as possible, keeping your (apparently very intense) SLAs nice and safe. The tradeoffs are

You also won't be able to return the result of your processing synchronously, which can sometimes be a deal-breaker
Splitting and maintaining a separate validation API and processing worker is a lot more work, which will make your project manager sad.

Of course, if you would like your system to have 100% uptime, I might suggest a different profession. Maybe you can author science fiction.

Getting back to simpler patterns, here is another variation - the polling system:

Notice how this is the same pattern as the first, it's just that the connection is initiated from your cron task instead of from a client. The biggest difference is that you need to provide the method for re-processing on failure since the cron task takes on the role of “caller” as well as processor. There are a couple of ways to address reprocessing. You can maintain a “last successful” watermark, and always get data from after that watermark (works great when the data is timestamped), or you can provide a way to run the cron manually with a range. Always make sure your ingest is idempotent!

Here’s another one, where the data source is a queue that we own:

On failure we put the item back in the queue. After enough failures the item goes into a DLQ (dead letter queue). Failed items like this should be handled manually, since they are likely malformed.

This “Synchronous processing, Synchronous error handling” architecture is most appropriate when your API can validate and process each request quickly. What does quickly mean? It means “your users will not complain that it’s too slow, and your network requests will not timeout.” What that means will depend on your users and your network architecture, but it's usually something between 5 and 10 seconds. We’ll get to what to do when your processing takes too long next.

Asynchronous processing, Synchronous error handling

Alright, so you can’t do your processing synchronously because you need to crunch a lot of numbers, and it's just going to take far too long (like more than 10 seconds). Your users will get antsy looking at a spinner and your network gateways will start severing your long held connections because they think you fell asleep at the wheel. What do?

This do. Split your validation from your processing. That way you can guarantee the client that their request will be fulfilled, even if it isn’t expressly fulfilled right now. Now your queue worker can take its own sweet time fulfilling the request, and the client can relax knowing that it's in the works. If the queue starts to back up with requests, no big deal! The queue workers are stateless so they are easy to scale out during the spike and then scale in when the spike is over.

This architecture is most appropriate when you can validate the request quickly, but it will take too much time to process it for the client to wait. As mentioned in the previous section, this architecture can also be useful for uptime guarantees.

Asynchronous processing, Asynchronous error handling

Sometimes the request is so large that you can’t validate the request at the same time that you receive it. This is usually the case with large files, where you need to go through the whole file and determine if the contents are valid. In this case our stateless service gateway modifies its “success” return value to mean “successfully uploaded.” You will need another mechanism to tell the client if their file was successfully processed or not.

The best way to handle this case is to stash the raw file somewhere (S3 or equivalent), put a ticket into the work queue, then inform the user that the upload is being processed. Once the queue worker is finished you can store the results of the processing in the database. Then you can notify the user that processing has completed via a queue, or just wait for them to poll for status.

Broadcasting without guarantees

Sometimes you need to inform other downstream components that something has happened, or some data has been inserted/updated. This often happens in event driven systems and microservices and is a pretty standard pattern for syncing state across service boundaries. It's a perfectly reasonable architecture - assuming you are OK with losing events sometimes. What happens when the write to the db is successful and the write to the queue fails? In order to return an error to the user you’d have to have some way of rolling back the change that was made to the DB. Not a fun prospect.

Broadcasting with guarantees

There is a way to guarantee that events are processed - put the events in the database at the same time that you write your actual data to the database as part of a single transaction.

This is known as the “Outbox Pattern.” You write both to the standard tables and also an “outbox” table in a single transaction. It’s a fair bit of work on the developers' part, so only do this if you need to. CDC (change data capture) is very similar but automates the second write, reducing the load on the developers.

Batching

Batching is the practice of bundling up multiple requests to be processed at once. It's generally done for performance reasons - writing 10,000 items in a single batch to a database is much faster than writing a single item 10,000 times. We’ve said before that you shouldn’t batch your items in your API, so how should we do it? Like this:

In this architecture, the Queue Worker pulls as many items from the queue as you’d like to batch at once. In this way your API and Queue Worker can remain stateless. Your Queue maintains responsibility for your data until the Queue Worker finishes writing the batch, at which point it releases the data from the Queue. Simple and effective.

If you find that your queue can’t handle a large number of messages, or becomes prohibitively expensive with large numbers of messages (I’m frowning at you, SQS) then you have chosen the wrong queue for your system. If you outgrow SQS, switch to Kafka (or Kinesis if you hate yourself) - don’t try to half-ass it by batching in your API.

Of course as always, if you can, use an off-the-shelf solution like AWS Firehose. Less code is more better.

Conclusion

This has been a whirlwind, high level tour of basic architectures. There is of course much more to say, and many more opinions to sling. If you’re interested I can provide more detail, or go over advanced topics like caching strategies in later articles. Let me know what you’d like to see in the comments!

Code Quality is a Lie

Timothy Ecklund — Wed, 22 Sep 2021 18:23:22 +0000

The phrase "Code Quality" makes me extremely uncomfortable. That phrase gets thrown about frequently and causally, but it is so vague that it's useless at best. At worst it's merely a verbal cudgel to beat someone else down. The phrase is synonymous with "I like that code" or "I don't like that code." (Sometimes it's even synonymous with "I don't like the author of that code.")

But as a metric, that you "like it" is not a measure of somethings quality.

It gets worse. Not only is that phrase ridiculously vague, it also implies that code can be judged against some universal metric - that there is a knowable and linear scale. Some code is simply "better" than other code and you know how to tell the difference.

This is nonsense. A thing can not be "better" or "worse" than another thing in and of itself - it can only be better or worse at something. Let's consider a story of two toasters.

Toaster A costs $15. It has no settings and no manual. It plugs into the wall, you put in the toast and you get the same exact level of toasting out of it every time. Its slots are only wide enough for bread so don't bother with bagels.

Toaster B costs $40. It has a very detailed and thick manual, describing how to maintain and service this toaster. It has extra wide slots for bagels, and a wide variety of settings. It also plays a cheery tune when it's done toasting.

So, which of these toasters is of higher quality? The answer is obvious - neither of them. They have different properties, and those properties come with costs. Which set of properties and costs is appropriate and preferable for you depends on your circumstances.

"Quality" does not exist independent of what your requirements are, and therefore "quality" does not exist, only contextual preference. Toaster A might suit your needs very well, and you might appreciate its durability given how few things there are that might break. Toaster B might be more appropriate for the toast or bagel enthusiast, someone who doesn't mind soldering on new heating elements as the manual suggests.

But Tim, you might object - what if Toaster A and Toaster B had exactly the same features and costs, but Toaster A lasted 50% longer before wearing out? Wouldn't Toaster A be of higher quality?

No. Toaster A has the property of durability, and assuming durability is one of your requirements (maybe you don't care!) then it's fair to say Toaster A is preferable. However, that is only because we care about durability - if we didn't, then we would have no preference.

But Tim, you might object - What if Toaster A and Toaster B are outwardly identical. They have the same observable properties, like features and durability, but Toaster A is made cleanly and simply while Toaster B is convoluted and complex inside. Wouldn't Toaster A be of higher quality?

No. If they have the same observable properties then neither is preferable. There is no magical substance, "Quality-O" that suffuses one and not the other. If one lasts longer or is easier to maintain then those are observable properties - assuming we someday observe them. It's entirely possible that the toaster lives its entire useful life without needing to be serviced. In that case, there were no observable difference and they were equally preferable.

Ah-ha, you might say. But if Toaster A is simpler then it is less likely to break. Your pedantry avails you nothing. The word "quality" is merely a shorthand for how likely something is to fail to meet our requirements. We say "high quality" when we mean "likely to meet our expectations and needs" and "low quality" when we mean "likely to disappoint us." It might be imprecise, but it's useful.

On that, I wish I could agree with you. If the phrase "Code Quality" were used thoughtfully, with the understanding that it was a shorthand for "the properties of this code lead me to believe that it will or will not suit the requirements or business need" then I would not be writing this post.

But even if that were true, "I believe this code won't meet our needs" is still extremely vague. Why won't it meet our needs? What about it exactly is problematic, why are those problems important to the business, and what metric do you expect would change if we did it your way?

If you have a hard time answering those questions, then your shorthand is not useful - it is sophistry.

This leaves us with a couple of problems. First, it challenges us to think through what exactly we mean by "quality" in a different light. Second, if we can't say "quality" then what shorthand can we use to talk about our contextual preferences?

Let's address the usual suspects that people bring up under the umbrella of "quality." When you look at a piece of code and judge it, what exactly are you thinking about?

If you think it needs more tests, why? Are you having a problem with bugs on release?
If you think it needs more documentation, why? Do you have people who need to understand it but can't? What cost are you willing to pay for more documentation?
If you think it needs to be faster, why? What business requirement are you meeting by improving latency? Is the dev time worth the reduction in latency?
If you think it needs to be simpler, why? Are you changing this code frequently, so the complexity is hindering your velocity?
If you think it is insecure, why? Is this public facing? What breaches are you concerned with and what business impact would they have?

Whatever metric you are using, unless it is tied to a specific and business oriented metric then it is vanity.

Alright, let's say for a moment that you've gone through the exercise of orienting your priorities and assessing your contextual preference about some code. It lacks tests, and so it breaks frequently on release, causing headaches for the business. It is overly complex, so changing it takes too long, and the business is suffering for it. On and on, you have nailed down the issues. You have also come up with solutions to those issues, and let's say that your solutions are good ones (fingers crossed).

When talking to someone else, especially someone non-technical, how do you communicate that your code is, or will be, better for the business? How do you communicate the extent to which it will be "better"? It's a tough one, because it's a lot of nuanced information, and usually it's too much information.

I don't have a perfect answer. There isn't a one size fits all solution, communication is a hard thing to do. I will say this however - the exercise will make you a better engineer. Forcing yourself to choose the right metrics, and to judge something against those metrics will make you more thoughtful, slower to judge, and ultimately much more effective.

Then when someone asks you if a codebase is of good quality, you can confidently say, "Maybe. It depends."

Converting Lodash to fp-ts

Timothy Ecklund — Sun, 25 Apr 2021 16:47:42 +0000

TLDR; I have started a project to provide examples of how to convert from Lodash to fp-ts and I could use your help! Please consider chipping in, all PRs are welcome!

Lodash is the single most downloaded package on npm. It deserves its place at the top - it provides a massive suite of functionality that is performant and has a clear, consistent interface. Lodash is, without a doubt, a fantastic Javascript library.

However, Lodash was written before the advent of Typescript and has significant holes when it comes to typed functional programming. Let me hit you with an example:

const log = console.log // just to make things a little nicer to read

const obj = { 'a': [{ 'b': { 'c': 3 } }] }
const result =  _.get(obj, 'a[0].b.c')
const ohno = _.get(obj, 'a[0].b.d')
log(result) // 3
log(ohno) // undefined

What type is result? Why, it's the any type! Not only are we missing type information on the result, we are also missing type information on the path we provided - if someone renames c to d we won't know until it gets all the way to production and explodes. On top of that we have to remember to check for undefined everywhere it might exist. Hope you never forget!

There is a better way! Let's look at how to do this using libraries that were designed from the ground up for typescript (fp-ts and monocle-ts):

import * as Op from 'monocle-ts/lib/Optional'

const getC = pipe(
  Op.id<{ a: readonly { b: { c: number } }[] }>(),
  Op.prop('a'),
  Op.index(0),
  Op.prop('b'),
  Op.prop('c'),
  opt => opt.getOption,
)
log(getC(obj)) // { _tag: 'Some', value: 3 }

Aw yeah. This is a technique known as Optics and it provides type safety through and through. Notice that we are providing a type with id - any calls to prop that don't align with the type will error. Finally, we're safe from Dave a few desks down who is constantly renaming things. We also have a strong return type - Option<number>. Option will force us to remember to add error handling in case our object is malformed, and number because we know that c is a number.

Here's another example:

var mutable = {a: 0, b: 2}
log(_.assign(mutable, {a: 1, c: 3}))
log(mutable)
// { a: 1, b: 2, c: 3 }
// { a: 1, b: 2, c: 3 }

Mutation! Noooooo! :(

Let's try again, this time with a library that is consistently immutable:

import {merge} from 'fp-ts-std/Record'

var mutable = {a: 0, b: 2}
log(merge(mutable)({a: 1, c: 3}))
log(mutable)
// { a: 1, b: 2, c: 3 }
// { a: 0, b: 2 }

Oh thank goodness, we're safe.

In my opinion, the biggest hurdle to widespread adoption of fp-ts is a lack of good examples. Almost everyone is familiar with Lodash - why not provide a set of examples that would help everyone transition?

Well I've started doing just that. I hope as people see the conversion is simple, and the benefits provided are significant, fp-ts will become even more widespread. Wouldn't that be a wonderful world?

Working through all the Lodash functions can take a long time, however, and I am (gasp) sometimes wrong. If you are reading this and have a few minutes, please take a crack at helping me with this project. PRs are very welcome!

Hyper-ts is my Cyborg Brain

Timothy Ecklund — Sun, 10 May 2020 18:16:50 +0000

EDIT: Hyper-ts has updated since this was written, here is an updated version of the sample server

I have a confession. I have been programming professionally for 17 years, but for all my experience and efforts, I am not a good programmer. I make mistakes constantly. My tests have holes. I break production. I get confused. I forget things that seem trivial until they explode into fireballs of doom. I get pulled into various offices to explain why the team needed to spend the weekend scraping data off the walls with a spatula.

Being a good programmer means interpreting the code you are reading in your head as though you were the computer, and ultimately I am just a bag of meat. An attractive and erudite meat bag perhaps, but still not the sort of thing you want sticking its grubby fingers into your multi-million dollar production system.

This is why I love typed functional programming. Typed FP lets me offload much of the interpretation of code to the compiler, so that it can tell me where my meat brain screwed it up. Extracting side-effects means I can have certainty about what the code will do, and even more importantly what the code won’t do. Enforcing completeness at the compiler level means I can’t forget to handle all the various edge cases that make things go BOOM in the night. Composition means I can take small functions that I know work, slap them together and know that the whole complex thing will work. Then the project manager can get off my back and I can go read my book in peace.

This is why I am so freaking excited about hyper-ts. Hyper-ts takes things a step further - it enforces not just type safety, but protocol correctness at compile time. Holy. Freaking. What.

Hyper-ts keeps us safe from:

Incorrect ordering of header and body writing
Writing incomplete responses
Writing multiple responses
Trying to consume a non-parsed request body
Consuming a request body parsed as the wrong type
Incorrect ordering of, or missing, error handling middleware
Incorrect ordering of middleware for sessions, authentication, authorization
Missing authentication and/or authorization checks

This might seem impossible, so it’s time to break it down. Hammertime. Let's talk about State.

In pure FP you can’t just say, hey let’s update this variable. That would be mutation, which is gross, unsafe and terribly uncool - maybe something your Dad who writes VisualBasic would do. Instead, put on your fire aviators and maintain immutability by passing your current state into your functions and have the functions return the new state along with whatever computation they did. It’s easier to understand with an example.

Let’s model a DVD player.

type dvdstate = 'PLAYER_EMPTY' | 'TRAY_OPEN' | 'PLAYER_FULL'

const openTray = () => (state:dvdstate):[undefined, dvdstate] =>
 tuple(undefined, 'TRAY_OPEN')

const loadTray = (dvdTitle:string) => (state:dvdstate):[string, dvdstate] =>
 tuple(
   state === 'TRAY_OPEN' ? `Loaded DVD ${dvdTitle}` : 'EXPLODE',
   state === 'TRAY_OPEN' ? 'PLAYER_FULL' : state)

const startDVD = (dvdTitle:string) => (state:dvdstate) =>
 tuple(`Playing ${dvdTitle}`, state)

const startState = 'PLAYER_EMPTY'
const [_, state] = openTray()(startState)
const [title, state1] = loadTray('Die Hard')(state)
const [movie, state2] = startDVD(title)(state1)
console.log([movie, state2])

This is the most basic way to do state transitions in pure FP. We’ve got a dvdstate, which is what states our DVD player can be in. We’ve got some functions, openTray, loadTray and startDVD that do some state transitions. And finally, you can see here:

const startState = 'PLAYER_EMPTY'
const [_, state] = openTray()(startState)
const [title, state1] = loadTray('Die Hard')(state)
const [movie, state2] = startDVD(title)(state1)

how we thread state through our program. When we run this code we get:

[ 'Playing Loaded DVD Die Hard', 'PLAYER_FULL' ]

Look Ma, no mutations!

But, this is really a hassle. While I was writing this I messed up which state variable went where like five times. If only there were an easier way.

import * as S from 'fp-ts/lib/State'

const playDisk = (movie:string) => pipe(
 openTray(),
 S.chain(() => loadTray(movie)),
 S.chain((title) => startDVD(title))
)('PLAYER_EMPTY')
console.log(playDisk('Elf'))

Well well well, what have we here. Our old friend the State Monad. We can use the built-in State Monad to thread the state through each function automatically. This is exactly the same as what we did before, but we don’t have to make new variables on every function call. And we get:

[ 'Playing Loaded DVD Elf', 'PLAYER_FULL' ]

However, this is still fragile. Imagine that you were up late the night before reading Bridge to Terabithia and having your heart ripped out. Your eyes are bloodshot and you’re not thinking clearly. You forget to open the dvd player before loading the DVD.

const brokenLikeMyHeart = (movie:string) => pipe(
 loadTray(movie),
 S.chain((title) => startDVD(title))
)
const result = brokenLikeMyHeart('Big Trouble in Little DVD Player')('PLAYER_EMPTY')
console.log(result)

What happens?

[ 'Playing EXPLODE', 'PLAYER_EMPTY' ]

Oof. The player tried to start the DVD and exploded. Maybe you can return it and they won’t notice.

Ok, so we have some kinds of safety, but your meat brain can still screw things up. This is where Indexed Monads come in. Indexed Monads are like State Monads, but with guard rails.

The type of the State Monad is State<A, S>, where A is your result and S is your state. But the type of an Indexed Monad is IxMonad<I, O, E, A>. Vowel soup right? Bear with me, I promise it makes sense.

I stands for Input - the input state. O stands for Output - the output state. E is for Error, and A is any type. Here’s what’s so cool. When you use an IxMonad, you specify - statically! - what the input and output state of your function must be. So we could define an IxMonad such that the brokenLikeMyHeart function would break at compile time because it does not adhere to the specified protocol. Sounds like space magic. Let’s see how hyper-ts uses this wizardry to enforce correctness.

import * as E from 'fp-ts/lib/Either'
import { pipe } from 'fp-ts/lib/pipeable'
import * as t from 'io-ts'
import { failure } from 'io-ts/lib/PathReporter'
import { toRequestHandler } from 'hyper-ts/lib/express'
import express from 'express'
import * as H from 'hyper-ts'
import pgPromise from 'pg-promise'
import * as TE from 'fp-ts/lib/TaskEither'
import { IntFromString } from 'io-ts-types/lib/IntFromString'

const pgp: pgPromise.IMain = pgPromise({})
const db = pgp('postgres://tim:password@localhost:5432/test_db');

const QuestionCodec = t.strict({
 name: t.string,
 text: t.string
})

export type Question = t.TypeOf<typeof QuestionCodec>

function badRequest(message: Error) {
 return pipe(
   H.status(H.Status.BadRequest),
   H.ichain(() => H.closeHeaders()),
   H.ichain(() => H.send(message.message))
 )
}

const paramDecode = pipe(
 H.decodeParam('question_id', IntFromString.decode),
 H.mapLeft((e) => new Error(failure(e).join('\n')))
)

const doAPIWork = (id:number) =>
 H.fromTaskEither(TE.tryCatch(() =>
   db.any<Question>('select * from screening.question where question_id = $1', [id]), E.toError ))

const questionHandler = pipe(
 paramDecode,
 H.ichain((id) => doAPIWork(id)),
 H.ichain((questions:Question[]) =>
   pipe(
     H.status<Error>(H.Status.OK),
     H.ichain(() => H.json(questions, E.toError))
   )
 ),
 H.orElse(badRequest)
)

const app = express()

app
 .get('/questions/:question_id', toRequestHandler(questionHandler))
 .listen(3000, () => console.log('Express listening on port 3000. Use: GET /'))

Here we have a little API that lets us find questions in a database. Let's break it down.

questionHandler is where we do the work of handling our request. First thing, we decode the parameter like so:

 H.decodeParam('question_id', IntFromString.decode),

If we’re unable to do the decode, like if someone passed in the string ‘i am not a number’ instead of a number, we generate a human readable error string and wrap it in an error like this:

H.mapLeft((e) => new Error(failure(e).join('\n')))

Next up, we do our actual API work. We hit the database and grab the data. We know for sure that id is a number so we can do:

H.fromTaskEither(TE.tryCatch(() =>
   db.any<Question>('select * from screening.question where question_id = $1', [id]), E.toError ))

Nice and safe with our error cases covered (we could decode the db response with io-ts but...there is such a thing as being too paranoid you know?) So far so good, but the mind blowing magic hasn’t happened quite yet.

H.ichain((questions:Question[]) =>
   pipe(
     H.status<Error>(H.Status.OK),
     H.ichain(() => H.json(questions, E.toError))
   )
 ),

Oh yeah, that’s the stuff. The state transitions defined in hyper-ts are:
StatusOpen -> HeadersOpen -> BodyOpen -> ResponseEnded

These states correspond to the parts of the HTTP response that you are writing:

HTTP/1.1 200 OK <- **STATUS**
Content-Type: text/plain <- **HEADERS**
Transfer-Encoding: chunked 
7\r\n <- **BODY (streaming)**
Mozilla\r\n 
9\r\n 
Developer\r\n 
7\r\n 
Network\r\n 
0\r\n 
\r\n <- **END BODY**

In this case, hyper-ts is enforcing (at compile time) that your code must have written a status, header and body - and in that order.

ichain is just like the chain method from the State example - state is automatically threaded through in the same way. Here's the cool part that's different: if you look at the type of the first line, H.ichain((questions:Question[]), you'll see:

H.Middleware<H.StatusOpen, H.ResponseEnded, Error, void>

The input state is H.StatusOpen. This tells us that when this function is called, we have not written our status line yet. The output state is H.ResponseEnded. That tells us that when this function returns we will have completed writing our entire response. In the case of an error we’ll get an Error, and if everything goes right we’ll get a void. Already we’re in better shape - we know that going into this function we can’t have written anything, and coming out we must have closed the request. Next, let’s look at:

H.status<Error>

the type is:

H.Middleware<H.StatusOpen, H.HeadersOpen, Error, void>

The input state is H.StatusOpen and the output state is H.HeadersOpen, which means that by the end of this function we must have written our status to the response. The next line:

H.ichain(() => H.json(questions, E.toError))

has type:

H.Middleware<H.HeadersOpen, H.ResponseEnded, Error, void>

This combines the header and body writing for convenience, but we must have written the headers and finished writing the body by the time this function returns. And, if you look at the implementation of H.json:

pipe(
    contentType<E>(MediaType.applicationJSON),
    ichain(() => closeHeaders()),
    ichain(() => send(json))
)

That is exactly what happens. Ok, let’s break it.

 H.ichain((questions:Question[]) =>
   pipe(
     H.ichain(() => H.json(questions, E.toError))
   )
 ),

We didn't write a status and what happens? BOOM compiler error!

Type '(ma: Middleware<unknown, HeadersOpen, Error, unknown>) => Middleware<unknown, ResponseEnded, Error, void>' is not assignable to type 'Middleware<StatusOpen, unknown, Error, unknown>'.

H.json has input type H.HeadersOpen, and we are passing H.StatusOpen, which isn’t going to fly.

From now on, I’ll never forget to send a status. I’ll never forget to write a body. I’ll never get my protocol out of order - because the compiler does that thinking for me. Welcome to my new cyborg brain.

fp-ts and Beautiful API Calls

Timothy Ecklund — Thu, 05 Mar 2020 16:53:17 +0000

Last time we visted fp-ts, we made concurrent API calls but didn't spend any time on error handling or keeping it DRY (Don't Repeat Yourself). Well we're a bit older and wiser now, and it's time to re-visit. Let's add some elegant error handling and tighten things up. Here's what we had last time:

const getUser = pipe(
  httpGet('https://reqres.in/api/users?page=1'),
  TE.map(x => x.data),
  TE.chain((str) => pipe(
    users.decode(str), 
    E.mapLeft(err => new Error(String(err))), 
    TE.fromEither)
  )
);

const getAnswer = pipe(
  TE.right("tim"),
  TE.chain(ans => pipe(
    answer.decode({ans}), 
    E.mapLeft(err => new Error(String(err))), 
    TE.fromEither)
  )
)

Bleh, there is a lot of duplication. Also our errors are going to be useless. If we run that the code above we get Error: [object Object]. What the heck is that? Completely useless, that's what. We can do better. First thing, let's make our error messages actually readable.

import { failure } from 'io-ts/lib/PathReporter'

const getAnswer = pipe(
  TE.right("tim"),
  TE.chain(ans => pipe(
    answer.decode({ans}), 
    E.mapLeft(err => new Error(failure(err).join('\n'))), 
    TE.fromEither)
  )
)

The failure method from the io-ts PathReporter takes an array of ValidationErrors and gives back a string. If we run this we get Error: Invalid value "tim" supplied to : { ans: number }/ans: number which is definitely a lot more helpful. Nice.

Ok, next up let's see what we can do to get rid of that gross duplication.

const decodeWith = <A>(decoder: t.Decoder<unknown, A>) =>
  flow(
    decoder.decode,
    E.mapLeft(errors => new Error(failure(errors).join('\n'))),
    TE.fromEither
  )

const getUser = pipe(
  httpGet('https://reqres.in/api/users?page=1'),
  TE.map(x => x.data),
  TE.chain(decodeWith(users))
);

const getAnswer = pipe(
  TE.right({ans: 42}),
  TE.chain(decodeWith(answer))
)

Well that looks way better. decoder.decode takes an unknown and gives back an Either<Errors, A> which is perfect. But getUser is still pretty specific to that url and to that type which is uncomfortable. One more time:

const getFromUrl = <A>(url:string, codec:t.Decoder<unknown, A>) => pipe(
  httpGet(url),
  TE.map(x => x.data),
  TE.chain(decodeWith(codec))
);

Aw yis. Now we can make any API call that we want and the response will be validated against our codec. We can even throw in a TE.mapLeft after httpGet if we want to do something fancy with errors thrown by axios.

Let's put it all together.

import axios, { AxiosResponse } from 'axios'
import { flatten, map } from 'fp-ts/lib/Array'
import * as TE from 'fp-ts/lib/TaskEither'
import * as E from 'fp-ts/lib/Either'
import * as T from 'fp-ts/lib/Task'
import { sequenceT } from 'fp-ts/lib/Apply'
import { pipe } from 'fp-ts/lib/pipeable'
import { flow } from 'fp-ts/lib/function'
import { failure } from 'io-ts/lib/PathReporter'
import * as t from 'io-ts'

//create a schema to load our user data into
const users = t.type({
  data: t.array(t.type({
    first_name: t.string
  }))
});
type Users = t.TypeOf<typeof users>

//schema to hold the deepest of answers
const answer = t.type({
  ans: t.number
});

//Convert our api call to a TaskEither
const httpGet = (url:string) => TE.tryCatch<Error, AxiosResponse>(
  () => axios.get(url),
  reason => new Error(String(reason))
)

//function to decode an unknown into an A
const decodeWith = <A>(decoder: t.Decoder<unknown, A>) =>
  flow(
    decoder.decode,
    E.mapLeft(errors => new Error(failure(errors).join('\n'))),
    TE.fromEither
  )

//takes a url and a decoder and gives you back an Either<Error, A>
const getFromUrl = <A>(url:string, codec:t.Decoder<unknown, A>) => pipe(
  httpGet(url),
  TE.map(x => x.data),
  TE.chain(decodeWith(codec))
);

const getAnswer = pipe(
  TE.right({ans: 42}),
  TE.chain(decodeWith(answer))
)

const apiUrl = (page:number) => `https://reqres.in/api/users?page=${page}`

const smashUsersTogether = (users1:Users, users2:Users) =>
  pipe(flatten([users1.data, users2.data]), map(item => item.first_name))

const runProgram = pipe(
  sequenceT(TE.taskEither)(
    getAnswer, 
    getFromUrl(apiUrl(1), users), 
    getFromUrl(apiUrl(2), users)
  ),
  TE.fold(
    (errors) => T.of(errors.message),
    ([ans, users1, users2]) => T.of(
      smashUsersTogether(users1, users2).join(",") 
      + `\nThe answer was ${ans.ans} for all of you`),
  )
)();

runProgram.then(console.log)

George,Janet,Emma,Eve,Charles,Tracey,Michael,Lindsay,Tobias,Byron,George,Rachel
The answer was 42 for all of you

And if we return erroneous data like:

const getAnswer = pipe(
  TE.right({ans: "tim"}),
  TE.chain(decodeWith(answer))
)

we get

Invalid value "tim" supplied to : { ans: number }/ans: number

Damn that's pretty. With this pattern we can handle any API calls that:

Can error
Return something that needs validation
Run in sequence, in parallel or by themselves

with complete confidence that all of the edge cases are covered. Stay (type)safe out there!

Invert your JavaScript

Timothy Ecklund — Sun, 27 Oct 2019 23:13:42 +0000

The outstanding article Destroy All Ifs does an excellent job describing the mechanism of Inversion of Control in Haskell. Unfortunately while Haskell is a beautiful language, it can often be off-putting for people who want to get things done without all the academic mumbo-jumbo. Since I would also like to get things done I think it’s worth exploring the concept of Inversion of Control in standard JavaScript.

What is Inversion of Control

Inversion of Control is the method by which frameworks are built. It is a mechanism for injecting new behaviors into an existing system. That sounds pretty abstract, so let’s look at an example.

const getWheatBread = (numSlices) => Array(numSlices).fill("wheat");
const getWhiteBread = (numSlices) => Array(numSlices).fill("white");

const makeToast = (isWheat, hasButter, hasJam) => {
  var bread = isWheat ? getWheatBread(1) : getWhiteBread(1);
  bread = bread.map((slice) => slice + " toasted")
  if(hasButter){
    bread = bread.map((slice) => slice + " butter")
  }
  if(hasJam){
    bread = bread.map((slice) => slice + " jam")
  }
  return bread;
};

makeToast(true, true, true)

Here we have defined a protocol for making toast. The protocol is

Get the bread
Toast it
Maybe add butter
Maybe add jam

There is some trouble here. First, what the heck is makeToast(true, true, true)? This is very difficult to read, and very easy to get wrong. Second, it’s not very extensible at all. What if we want to specify raspberry jam, or strawberry? We could add more booleans, but that seems like it would get quickly out of hand. Let’s try out this Inversion of Control thing that everyone is so hot about.

Attempt #1

const makeToast = (isWheat, hasButter, applyJam) => {
  var bread = isWheat ? getWheatBread(1) : getWhiteBread(1);
  bread = bread.map((slice) => slice + " toasted");
  if(hasButter){
    bread = bread.map((slice) => slice + " butter");
  }
  bread = bread.map(applyJam);
  return bread;
};

makeToast(true, true, (slice) => slice + " raspberry jam");

Nice! We've made the application of jam dynamic, so we can add any kind of jam we want. But what if we want to toast up some rye bread, or try out a new buttering technique? Let’s take it a step further and invert the rest of the steps as well.

Attempt #2

const getWheatBread = (numSlices) => () => Array(numSlices).fill("wheat");
const getRyeBread = (numSlices) => () => Array(numSlices).fill("rye");

const makeToast = (getBread, applyButter, applyJam) => {
  var bread = getBread();
  bread = bread.map((slice) => slice + " toasted");
  bread = bread.map(applyButter)
  bread = bread.map(applyJam)
  return bread;
};

makeToast(
  getRyeBread(1),
  (slice) => {
    busyWait(5); // multiply numbers for 5 minutes so the computer will heat up and soften the butter
    return slice + " butter";
  },
  (slice) => slice + " raspberry jam")

Ok great, now we can pass in different behaviors! We've decided that toasting will always work the same way, so we haven't inverted control of it. This is now much more extensible, and it’s much easier to understand what the parameters do. Let’s clean this up a little more.

Attempt #3

const makeToast = (getBread, applyButter, applyJam) =>
  getBread()
  .map((slice) => slice + " toasted")
  .map(applyButter)
  .map(applyJam)

Neat. There is a clear separation between things that can change behavior and things that can't. Let’s take another look at the protocol we defined at the beginning:

Get the bread
Toast it
Maybe add butter
Maybe add jam

Our structure is still in place, but each piece can be customized to how we need it.

Testing

One last thing. Getting our bread might require that we go out to the BreadService. That’s going to be slow, and jeeze who wants to stand up a BreadService just to be able to test our toasting function? What if instead, we injected the getFakeBread function when we're running our tests?

const getFakeBread = (numSlices) => () => ["fake"];

it('should make some toast', async function() {
  expect(makeToast(
    getFakeBread(),
    doButter, 
    doStrawberry)
  ).to.eql(["fake toasted butter strawberry jam"]);
})

Awesome. Let's take stock of what we've gained.

We have defined which things in our protocol can change, and which things can't
We can inject any behavior we want into the protocol, as long as the function conforms to the expected signature
We can easily test our protocol
Our code is much easier to understand and get right, since we are explicit about what behaviors we want

Victory!

fp-ts, sequenceT, and sweet sweet async typed FP

Timothy Ecklund — Sat, 31 Aug 2019 16:56:20 +0000

Recently I found the need to make a set of async calls with different return types. This is a pretty common task, we'd like to make some calls in parallel and collect the results once everything is done. Let's take a look at the docs for fp-ts async tasks.

const tasks = [task.of(1), task.of(2)]
array
  .sequence(task)(tasks)()
  .then(console.log) // [ 1, 2 ]

Hmm. Well that's pretty nice, but what happens when the types are different?

const tasks = [T.task.of(1), T.task.of("hello")]
array
  .sequence(task)(tasks)()
  .then(console.log) // [1, "hello"] I hope?

Uh oh.

Well darn. The type of sequence is (simplified) Array[F[A]] => F[Array[A]], so all the return types would have to be the same.

What do? :/

After some googleing, I ran across the magical sequenceT.

 /** 
 * const sequenceTOption = sequenceT(option)
 * assert.deepStrictEqual(sequenceTOption(some(1)), some([1]))
 * assert.deepStrictEqual(sequenceTOption(some(1), some('2')), some([1, '2']))
 * assert.deepStrictEqual(sequenceTOption(some(1), some('2'), none), none)
 */

Nice! Ok, let's try it out.

import * as T from 'fp-ts/lib/Task'
import { sequenceT } from 'fp-ts/lib/Apply'
import { pipe } from 'fp-ts/lib/pipeable'

pipe(
  sequenceT(T.task)(T.of(42), T.of("tim")), //[F[A], F[B]] => F[A, B] 
  T.map(([answer, name]) => console.log(`Hello ${name}! The answer you're looking for is ${answer}`))
)();

Hello tim! The answer you're looking for is 42

Well that's rad. pipe allows us to chain calls together, so the result of sequenceT is passed in to T.map. T.map destructures the tuple and we can do as we please with some guarantees about our data. But what if our Tasks can fail?

pipe(
  sequenceT(TE.taskEither)(TE.left("no bad"), TE.right("tim")),
  TE.map(([answer, name]) => console.log(`Hello ${name}! The answer you're looking for is ${answer}`)),
  TE.mapLeft(console.error)
)();

no bad

Awesome! Ok, time to get fancy with it. What if we are actually making some calls to an API, and we want to ensure that the results we get from the API conform to the expected schema?

Let's try it out by hitting a dummy REST endpoint with axios, which is a handy http client.

import { array } from 'fp-ts/lib/Array'
import axios, { AxiosResponse } from 'axios';
import * as t from 'io-ts'

//create a schema to load our user data into
const users = t.type({
  data: t.array(t.type({
    first_name: t.string
  }))
});

//schema to hold the deepest of answers
const answer = t.type({
  ans: t.number
});

//Convert our api call to a TaskEither
const httpGet = (url:string) => TE.tryCatch<Error, AxiosResponse>(
  () => axios.get(url),
  reason => new Error(String(reason))
)

/**
 * Make our api call, pull out the data section and decode it
 * We need to massage the Error type, since `decode` returns a list of `ValidationError`s
 * We should probably use `reporter` to make this nicely readable down the line
 */
const getUser = pipe(
  httpGet('https://reqres.in/api/users?page=1'),
  TE.map(x => x.data),
  TE.chain((str) => pipe(
    users.decode(str), 
    E.mapLeft(err => new Error(String(err))), 
    TE.fromEither)
  )
);

const getAnswer = pipe(
  TE.right(42),
  TE.chain(ans => pipe(
    answer.decode({ans}), 
    E.mapLeft(err => new Error(String(err))), 
    TE.fromEither)
  )
)

/**
 * Make our calls, and iterate over the data we get back
 */
pipe(
  sequenceT(TE.taskEither)(getAnswer, getUser),
  TE.map(([answer, users]) => array.map(users.data, (user) => console.log(`Hello ${user.first_name}! The answer you're looking for is ${answer.ans}`))),
  TE.mapLeft(console.error)
)();

Hello George! The answer you're looking for is 42
Hello Janet! The answer you're looking for is 42
Hello Emma! The answer you're looking for is 42
Hello Eve! The answer you're looking for is 42
Hello Charles! The answer you're looking for is 42
Hello Tracey! The answer you're looking for is 42

Heck yeah! We did it! Async Typed FP for everyone! :)