Forem: Szabo Gergely

First experiences with Nix

Szabo Gergely — Tue, 17 Dec 2019 08:13:32 +0000

I've been experiencing with Nix, little by little for some time now, so I thought it's time to summarize my thoughts about it.

I love functional programming, so it was almost inevitable that I would start tinkering with Nix sooner or later. However, the Nix ecosystem is huge, and it tends to be daunting at first.

Step 1 - installing the nix package manager

My first step into the world of Nix was installing the nix package manager on my Mac OS. For a period of time I used it simultaneously with Brew. As a package manager, it works just fine, the commands are a bit strange, and searching for packages tends to be slow. By the way, since my first install, the nix search solved the slow package lookup problem. At this point, nix is not really so much different, from Brew or MacPorts. Under the hood, nix has an interesting approach: it has a storage folder, where all your installed files will be saved, with a hash which represents the exact version, so multiple versions of the same packages can coexist. These packages will then be linked to you nix profile folder, which is added to the PATH environment variable.

By the way, there is a small glitch when installing Nix on Mac OS Catalina, but here is a solution for it: https://github.com/NixOS/nix/issues/2925#issuecomment-564493970

And here is a useful page for Nix beginners: https://nixos.wiki/wiki/Cheatsheet

Step 2 - nix-shell

The true magic starts with the nix-shell. With this, you can switch between packages, that are in store.

Earlier, I used nvm, rvm or asdf like tools, and sometimes even Docker to create a separate development environment with different versions of node or ruby, etc. However, there are a few problems with these tools: every language has its own tool, and of course every tool uses different commands and options. Docker is too heavy-weight, it drains the battery, uses too much memory for such a simple task. And all they can do is to set the version for one dependency, but what happens, when you have both Ruby and JS in a project. It's a mess.

One day I was working on a monolithic Ruby project, with several OS level dependencies and JS. It would never behave the way I wanted, I had to debug the build steps from time to time. Then I got fed up, uninstalled all the dependencies, and wrote my first shell.nix file. Surprisingly, it was rather easy, and it just worked every time I needed it. And the feeling that I don't need to install stuff globally just for that one project is wonderful.

However, it wasn't perfect in from the start... After I updated my nix-channel, the repository for packages, my build wouldn't work anymore. Nix promises reproducible builds, but because the package names in the shell.nix file doesn't contain version numbers, so these can change and break the build. Tip number 1: always pin your nixpkgs.
https://nixos.wiki/wiki/FAQ/Pinning_Nixpkgs

I also experienced some problems with reproducibility when I tried to use the same settings on a different device. The thing is, nix-shell still uses global packages. If you have a library installed globally, your program will see it inside the nix-shell, but of course it won't be there on other computers. So **tip number 2: try running your nix-shell with the --pure flag`. This way, only packages in the shell.nix file will be available.

Step 3 - shellHook

I realized, that Nix could serve as a full replacement for Docker. I mostly use Docker locally to spin up a DB and run my project in the desired environment. I knew how to do the latter, but I still needed a DB instance, but I didn't want to install one globally. So, my next step was not only installing dependencies with nix-shell, but also running them using the shellHook property. And also stopping them, when I exit the shell. As tip number 3:trap commands are useful!

If you haven't done it already, it's a good time to learn the Nix programming language. The syntax looks strange, but it's not at all difficult: https://nixos.org/nixos/nix-pills/basics-of-language.html

Step 4 - overlays

As I dug myself deeper into Nix, it sometimes becomes necessary, to overwrite some packages. I had node.js application, which uses yarn as a package manager and build tool. After I setup the nix-shell, I realized that the node version in yarn is different than the one in npm. The reason is that yarn won't use the global node in the PATH, but the one it is built with - if it wouldn't be like this, the build would not be reproducible. Yarn was using an older version of node. Fortunately, overlays are relatively easy to do, so I could override the build dependency. However, at this point I guess you'll need to understand more about the programming language of Nix. It is not at all complicated, but is a bit different, than what most of us are used to.

Prospects for the future

The world of Nix is way much more, than I could discover, I am just in the doorway. Deployment using Nix, creating packages, building monorepos, the possibilities are endless. Also the nix infrastructure is still growing, so a lot of new tools are appearing, the best practices are still forming.

If you haven't tried Nix yet, I definitely advise you to give it a go. You don't have to go all in, you can still use it as a simple package manager, and incrementally get swallowed by it.

Haskell do notation explained through JavaScript async await - part 2

Szabo Gergely — Wed, 28 Aug 2019 03:26:44 +0000

Hi. This is the second part of my little tutorial about Haskell. In the first part we looked at some simple Haskell examples using simple IO effects, and similar programs written in JavaScript where every IO effect was returning a Promise to discover the similarities between a JS Promise and a Haskell IO monad.

This time I will explore some more complex ideas: how to handle sequential effects.

First of all, let's see a really simple example: we will create a program that

reads a number n from user input
reads n lines of user input into an array of numbers
adds a 100 to all of the numbers.

So, for the input

2
3
5

we expect an output of

103, 105

Here is how it looks like in imperative JavaScript:

process.stdin.setEncoding('utf-8')

const getNumber = () => new Promise(
    resolve => process.stdin.once('data', data => resolve(Number(data)))
)

const main = async () => {
    const n = await getNumber()
    const numbers = []
    for (let i = 0; i < n; i++) {
        const newNumber = await getNumber()
        numbers.push(newNumber + 100)
    }
    console.log(numbers)
}

main()

However this won't work in a purely functional language because it uses mutable variables. We need think in terms of data and how that data flows through our application, rather than instructions given to the computer to process. We also need to restrict ourselves to use immutable values only, and functions like map, fold, etc.

The solution might be a bit counterintuitive for people new to functional programming: we will

generate an array from 1 to n
map and evaluate our effectful getNumber function over this array
print the resulted array to the screen

If this doesn't make sense at first, just bare with me, hopefully the following examples will make it clear.

First, we need to generate our array. Functional languages usually have some powerful utility functions for tasks like generating an array, but in JS we have to implement it ourselves.

We could implement this in a nice functional way using recursion, but its not the point of this article, so I wrote a more hacky JS version:

const range = (from, to) =>
    [...Array(to - from + 1)].map((_, index) => index + from)

Now, we can reimplement our main function.

const main = async () => {
    const n = await getNumber()
    const numbers = range(1, n).map(_ => getNumber())
    const mapped = numbers.map(x => x + 100)
    console.log(mapped)
}

Our range function generates an array from 1 to n, then we map each number to the getNumber function, throwing away the numbers of the original array.

Sweet... Would be, if it would work. But we have a problem: the getNumber returns a Promise, so our numbers variable will be an array of Promises, but we want an array of numbers. We cannot get rid of the Promises, but we can aggregate them to one. JavaScript has a built in function called Promise.all that will do just that. Let's pass our array to Promise.all and put an await before it to get the resolved value out of the Promise.

const main = async () => {
const n = await getNumber()
    const numbers = await Promise.all(range(1, n).map(_ => getNumber()))
    const mapped = numbers.map(x => x + 100)
    console.log(mapped)
}

Voila. Actually, it still has one bug, that has to do with our implementation of getNumber. Our program now resolves all promises on the first user input with the same value. A not-so-functional solution to this:

const queue = []

const getNumber = () => new Promise(resolve => {
    queue.push(input => resolve(Number(input)))
})

process.stdin.on('data', data => {
    const nextResolver = queue.shift()
    nextResolver(data)
})

Now, let's dive into Haskell, with the same approach:

main :: IO ()
main = do
  n       <- getNumber
  numbers <- sequence (map (\_ -> getNumber) [1 .. n])
  let mapped = map (100 +) numbers
  print mapped


getNumber :: IO Int
getNumber = fmap read getLine

Instead of the Promise specific Promise.all, Haskell has a more generic function called sequence. Its type signature says (Traversable t, Monad m) => t (m a) -> m (t a). t and m are type variables, where t must be a Traversable and m a Monad. Traversable and Monad are type classes, so this function is not specific to Lists, but polymorphic on every type in the Traversable type class.

If we substitute the type variables with the concrete types in our program, we get: [IO Integer] -> IO [Integer]. Remember, when we added the Promise.all in our example, we needed to convert our array of promises to a promise of an array. This time we need to convert a list of IO monad to an IO monad of a list.

If you look at the JS and the Haskell example, they look really similar. That is because Promise is a monad, so you already know how to deal with them. This knowledge can really be helpful when you are lost in the jungle of monads in Haskell.

Haskell's IO monad and JS's Promise have a lot in common. When you are working with a Promise, you cannot simply use its value, you have to use either the then method or the async await syntax. Also, once you unwrap a Promise in your function, it will become an asynchronous function itself, it contaminates your function, just like an IO monad in Haskell.

About type classes and polymorphism

Type classes are groups of types that can use the same group of polymorphic functions. Every type in a type class has to implement a few basic functions - if you are familiar with OOP concepts, this idea is very close to implementing interfaces. In the first part of this tutorial, we saw the bind, or >>= function in action. This is one of the basic functions, that every Monad has to implement. sequence uses this function to join the values in the list together.

Just as an example, on how polymorphism works, this is what happens, when you use sequence with Maybe monads:

> sequence [Just 4, Just 5, Just 6]
Just [4,5,6]
> sequence [Just 4, Nothing, Just 6]
Nothing

The sequence function goes from left to right, and uses the implementation of the >>= of the Maybe type to join the values in the list. Once a Nothing appears in the list, the >>= will return a Nothing.

instance Monad Maybe where
    (Just x) >>= k = k x
    Nothing  >>= _ = Nothing

In Haskell many type classes get their names from category theory. Monad is one of them, but there are also classes like Monoid, Functor, Applicative etc. However it is good the know the theory, it is enough to have a shallow knowledge to be able to write Haskell. As you get more and more familiar with the language, naturally you'll learn more about category theory also. For a start, it is good to understand, that every type class has some capability, some polymorphic function it can use: Functors can be mapped with fmap, Monads can be binded with >>=. Also, because every Monad is a Functor, every Monad can also be mapped.

Special map functions for monads

Let's return to our example. It can be further simplified using some utility functions called mapM and mapM_.

The type signature of mapM is (Traversable t, Monad m) => (a -> m b) -> t a -> m (t b) . This one does the same thing as sequence and map together. It will map a monadic function to a list, and collects the results. Here is our simplified main function:

main :: IO ()
main = do
  n       <- getNumber
  numbers <- mapM (\_ -> getNumber) [1 .. n]
  let mapped = map (100 +) numbers
  print mapped

Now that we know how to do a sequence of monads, let's see another example: we want to output our list of numbers one by one.

In JS we can simply use forEach on our array. We will now use our meaningless asynchronous output function we introduced in the first part:

const output = word => new Promise(resolve => {
    setTimeout(() => {
        console.log(word)
        resolve()
    }, 1000)
})

const main = async () => {
const n = await getNumber()
    const numbers = range(1, n).map(_ => getNumber())
    const mapped = numbers.map(x => x + 100)
    mapped.forEach(output)
}

The forEach is the same as the map, but it ignores the return values. It seems to be OK to ignore the returns in some cases, but what if we want to know when the async functions have finished executing. The output function actually returns a Promise<undefined>. We need to collect the return functions, and only resolve our main function, when all of them are resolved. It leads us to the same solution, as the input.

const output = word => new Promise(resolve => {
    setTimeout(() => {
        console.log(word)
        resolve()
    }, 1000)
})

const main = async () => {
    const n = await getNumber()
    const numbers = range(1, n).map(_ => getNumber())
    const mapped = numbers.map(x => x + 100)
    return Promise.all(mapped.map(output))
}

Now, let's attempt to use the same approach in Haskell:

main :: IO ()
main = do
  n       <- getNumber
  numbers <- mapM (\_ -> getNumber) [1 .. n]
  let mappedNumbers = map (100 +) numbers
  mapM print mappedNumbers

We have a type error:

    Couldn't match type ‘[()]’ with ‘()’
    Expected type: IO ()
    Actual type: IO [()]

The main function happens to return a IO [()]. Let's see what is going on: the last line is mapM print mappedNumbers , where the print is a -> IO (). If we substitute the abstracted types of mapM with our concrete types, we get: (a -> IO ()) -> [a] -> IO [()].

We can ignore the return value of the mapM ourselves:

main :: IO ()
main = do
  n       <- getNumber
  numbers <- mapM (\_ -> getNumber) [1 .. n]
  let mappedNumbers = map (100 +) numbers
  _ <- mapM print mappedNumbers
  return ()

We have a simpler version with mapM_ that ignores the return values:

(Foldable t, Monad m) => (a -> m b) -> t a -> m ()

(a -> IO ()) -> [a] -> IO ()

main :: IO ()
main = do
  n       <- getNumber
  numbers <- mapM (\_ -> getNumber) [1 .. n]
  let mappedNumbers = map (100 +) numbers
  mapM_ print mappedNumbers

I hope this part wasn't too daunting. See you again next time!

Haskell do notation explained through JavaScript async await - part 1

Szabo Gergely — Mon, 29 Jul 2019 09:25:05 +0000

This blog is intended to be an introduction to Haskell's IO monad and do notation for programmers familiar with JavaScript. I assume that you have just started learning Haskell and have a hard time understanding what is going on in your main function. I will introduce the idea that Promises in JavaScript have a monadic nature, and if you already use Promises, it can help you understand monads and Haskell in general.

When I first learned Haskell I tried to do just as I would with any other new language: requiring some input from the console, doing something with the given value, and outputting something on the screen. However, in Haskell this isn't that easy.

main :: IO ()
main = do
  putStrLn "Insert your name"
  yourName <- getLine
  let greeting = "Hello " ++ yourName ++ "!"
  putStrLn greeting

At first glance it looks like any other imperative language, but there are two strange things:

do notation - what is it? why do I need it? is it always needed, when I write a function?
left arrow and the let keyword - what is the difference?

To answer the first question, the do notation is a special kind of syntax in Haskell that lets you write imperative-like code. However the true nature of Haskell is not imperative, so it is just a syntactic sugar to hide the the more functional world behind.

So let's step back a bit and think about what makes something imperative or functional. There are keywords, like immutability, pure functions, etc., but what I want to focus on is, functional languages are based on expressions while imperative language are on instructions.

// imperative style
let a = 5
if (b === true) {
    a = 10
}

// functional style
const a = b === true ? 10 : 5

In the above example the first part is using an immutable variable, and giving and instruction to change that variable when a condition is met. The second example does the same things without instructions.

When you write something in JavaScript, you think about instructions you give to your computer, while in Haskell its closer to some kind of data-pipeline. You won't find if statements like the one above (without the else block), or for loops, because we are not using instructions. Everything has to be an expression, or a function that has some input and returns an output, and does nothing else. Functional languages have their own set of tools to achieve the same thing, with these restrictions, like mappers and reducers (or folds) instead of loops. And of course monads instead of arbitrary side effects.

Let's return to our first example. You might already know that any function written in do notation can be also written as an expression:

main :: IO ()
main =
  putStrLn "Insert your name"
    >>= (\_ -> getLine)
    >>= (\yourName -> let greeting = "Hello " ++ yourName in putStrLn greeting)

More crazy things happened! >>= and some anonymous functions appeared. Meanwhile, the left arrow disappeared. Really hard to comprehend this code, that's the main reason of the do notation's existence.

Let's try to break up this into small functions to see all the building blocks. Remember, Haskell is like a LEGO where your functions are small building blocks that click together. (I would not recommend breaking up things so small, I just did it in hope of getting a better view on how these building blocks fit together.)

main :: IO ()
main = giveInstruction >>= getName >>= outputGreeting


giveInstruction :: IO ()
giveInstruction = putStrLn "Insert your name"


getName :: () -> IO String
getName _ = getLine


outputGreeting :: String -> IO ()
outputGreeting yourName =
  let greeting = "Hello " ++ yourName in putStrLn greeting

The giveInstruction will perform IO, but only returns a unit, which is something similar to void in other languages.

We want to pipe the result of the giveInstruction to the getName , so we made it to take a unit as an argument. It is not necessary though, using the >> operator would be nicer, I only used it to make our example resemble more the JavaScript version.

The result of the getName is a String, so it can be easily piped into the last function.

Now, here is a Node.js script that does the same thing:

process.stdin.setEncoding('utf-8')

const output = word => console.log(word)

const giveInstruction = () => output("Insert your name")

const getName = () => new Promise(resolve => process.stdin.once('data', resolve))

const outputGreeting = yourName => {
    const greeting = "Hello " + yourName
    output(greeting)
}

const createGreeting = yourName => "Hello `


const main = () => {
    giveInstruction()
    getName()
        .then(outputGreeting)
}

main()

We need to use a Promise to handle our user input. The Promise wraps up the input value and we can only access it through the then method. Now imagine that for some questionable reason we wanted to delay our output a second. Now the output function returns a Promise.

process.stdin.setEncoding('utf-8')

const output = word => new Promise(resolve => {
    setTimeout(() => {
        console.log(word)
        resolve()
    }, 1000)
})


const giveInstruction = () => output("Insert your name")

const getName = () => new Promise(resolve => process.stdin.once('data', resolve))

const outputGreeting = yourName => {
    const greeting = "Hello " + yourName
    return output(greeting)
}

const main = () => {
    giveInstruction()
        .then(getName)
        .then(outputGreeting)
}

main()

At this point you might see some resemblances with our Haskell code. If you want to use the result of an asynchronous function, you have to use the then method. The then method has the same purpose for a Promise as the >>= also known as bind has to the IO monad. And I dare to say that async await syntax has almost the same purpose as do notation:

const main = async () => {
    await giveInstruction()
    const yourName = await getName()
    await outputGreeting(yourName)
}

We now got rid of the thens, but had to save the result of getName to a variable, so our code lost its pipe-like nature. Also important to know that an async function is just a function that returns a Promise. It's only syntactic sugar, just like do notation.

Let's go a step further and break up the output function, by separating the logic from the IO action. The newly created createGreeting is a pure function, which means it doesn't invoke any side effects, and it doesn't need to be wrapped in any monad. By the way separating pure business logic from the side effects is considered a good practice. This time, I will use the do notation again:

main :: IO ()
main = do
  giveInstruction
  yourName <- getName ()
  let greeting = createGreeting yourName
  outputGreeting greeting


giveInstruction :: IO ()
giveInstruction = putStrLn "Insert your name"


getName :: () -> IO String
getName _ = getLine


createGreeting :: String -> String
createGreeting yourName = "Hello " ++ yourName


outputGreeting :: String -> IO ()
outputGreeting greeting = putStrLn greeting

In JS we would change our program like this:

const giveInstruction = () => output("Insert your name")

const getName = () => new Promise(resolve => process.stdin.once('data', resolve))

const createGreeting = yourName => "Hello " + yourName

const outputGreeting = yourName => output(greeting)

const main = async () => {
    await giveInstruction()
    const yourName = await getName()
    const greeting = createGreeting(yourName)
    await outputGreeting(yourName)
}

main()

This should answer the question about the let and the left arrow. Our JS implementation has await keywords on every line, except before the createGreeting. It is because it is not an asynchronous function.

The same is true for the Haskell code: where we want some value out of an IO function, we need to use the <- but createGreeting function is not a monad, so we use the let binding instead.

I hope this article was helpful. Next time I plan to do some deep dive with some more complex examples.

Some side note

I did not intend to touch this area but as I was writing I thought this part would need some explanation: why monads doesn't need to have a -> in their type signatures, like every other normal function. The giveInstructions :: IO () function is a good example for that. If you look at its signature, it doesn't even look like a function. And in fact, it isn't. It is return value of the effect, wrapped in an IO monad. This means that strictly speaking, our JavaScript would look something like this:

const giveInstruction: Promise<void> = output("Insert your name")

Of course in JavaScript it would run the output function immediately on program start. So in order to delay the function evaluation, we to wrap it up in a function, that takes no argument.

You may know already but Haskell is a lazily evaluated language, which means a function or effect is only evaluated, when it is needed. So if you have an unused value in your code it won't be calculated. And this means that the giveInstruction value is only evaluated, when it is used in the main function.

Continue reading with part 2