Forem: Franco Victorio

Debugging transactions in Ethereum. Part 2: Truffle and Buidler

Franco Victorio — Tue, 17 Mar 2020 13:37:59 +0000

In the previous post we looked at some ways to debug transactions that were mined in a testnet or in the mainnet, using etherscan and eth-cli. In this post we'll move to transactions that we execute in "local" blockchains, like ganache-cli and the Buidler EVM.

I'm going to use the Truffle MetaCoin box for the examples, so if you want to follow along you can start by running npx truffle unbox metacoin in an empty directory. You'll also have to edit the truffle-config.js file, uncomment it, and change the port of the development network from 7545 to 8545.

Truffle debug

If your project is built using Truffle, you can use its debugger to inspect a transaction step by step. This is a command line tool similar to gdb or pdb; if you've used something like that in the past, the experience is very similar.

In our example, if we start ganache-cli in another terminal and then run npx truffle test, we'll see a bunch of transactions being executed. Copy the hash of the last transaction and run:

npx truffle compile # this might be necessary if the build/ directory isn't there
npx truffle debug 0x3e3299b90e92efeccfe806f7b67e4683fb2c6ad2e114cbbe4b1308e4402ce090

(Your hash could be different.)

And you'll get something like this:

Here you can do things like seting a breakpoint with b, stepping over to the next line with o or evaluating an expression with :. Let's step over twice and inspect the value of the amount parameter with :amount:

Check the docs to learn more about this feature.

Debugging tests

Another tool you can use is the global debug() function in your tests. Go to test/metacoin.js and replace this line

-const balance = await metaCoinInstance.getBalance.call(accounts[0]);
+const balance = await debug(metaCoinInstance.getBalance.call(accounts[0]));

and then run your tests with npx truffle test --debug (notice the extra flag). Your tests will be interrupted when that method is called and you'll be able to inspect it:

This is great not only because you don't need to get the hash of some transaction, but also because you can use it for plain calls that don't even send a transaction.

Debugging non-local transactions

What about transactions in a testnet or in the mainnet? As far as I know, truffle debug can be used against any node that supports the debug_traceTransaction JSON-RPC method. And (again, AFAIK; here I am in the "I don't really know what I'm talking about" territory) besides local blockchains, this method is supported mainly by geth when syncing in archive mode. I haven't tested it because this requires syncing more than 2 TB of data.

Are there any other alternatives? Not that I'm aware of. Infura doesn't support debug_traceTransaction calls. I also tried using ganache --fork but this didn't work either (it actually kind of worked once? I think it might if the transaction is recent enough).

Anyway, if anyone knows how to do this, please let me know in the comments.

Buidler

Buidler is a task runner for smart contract projects that comes with a lot of useful tools. You can use it on its own, or you can complement a truffle project with it, which is what we're going to do here. First, install it:

yarn add --dev @nomiclabs/buidler

and then run npx buidler and select the "Create an empty buidler.config.js" option.

console.log

One of the things that come with Buidler is the Buidler EVM, that lets you do things like logging stuff inside your contracts or having stack traces that work across solidity and javascript. Let's start with the console.

First, open the contracts/MetaCoin.sol file and add these two lines:

 pragma solidity >=0.4.25 <0.7.0;

 import "./ConvertLib.sol";
+import "@nomiclabs/buidler/console.sol";

...

    function sendCoin(address receiver, uint amount) public returns(bool sufficient) {
+       console.log(msg.sender, receiver, amount);
        if (balances[msg.sender] < amount) return false;

Next, kill ganache-cli (if it's still running) and execute npx buidler node, which starts a node that uses the Buidler EVM.

Now just run npx truffle test and you'll be able to see the console messages:

You'll notice that here we are logging the same information that is emitted in the Transfer event, so what's the point? Well, first of all, events are not emitted for failed transactions, but the console works just fine. And second, you can also use logs for method calls that don't send transactions.

Stack traces

The Buidler EVM also supports stack traces that look like this:

As you can see, here we get both the line of the contract that failed, and the line of the javascript test that triggered the transaction!

Getting this to work is a little more involved, but worth it. First, install the truffle and web3 buidler plugins:

yarn add --dev @nomiclabs/buidler-truffle5 "@nomiclabs/buidler-web3@^1.2.0"

Then open buidler.config.js and add

usePlugin("@nomiclabs/buidler-truffle5")

to the beginning of the file.

Then create a file called test/truffle-fixture.js:

const MetaCoin = artifacts.require("MetaCoin")
const ConvertLib = artifacts.require("ConvertLib")

module.exports = async () => {
  const convertLib = await ConvertLib.new()
  ConvertLib.setAsDeployed(convertLib)
  MetaCoin.link(convertLib)
  const metacoin = await MetaCoin.new()
  MetaCoin.setAsDeployed(metacoin)
}

This is just some glue code we need so that our contracts are deployed before running our tests with buidler (so far we've been running them with truffle).

You'll also need to remove the solidity test that comes with the MetaCoin box:

rm test/TestMetaCoin.sol

This is necessary because buidler doesn't support Solidity tests yet.

Now everything is ready: run npx buidler test to run your tests in the Buidler EVM. Of course, tests still pass so we won't get a stack trace. Let's modify our contract to trigger an error. Open contracts/MetaCoin.sol and add this line:

  ...

    function sendCoin(address receiver, uint amount) public returns(bool sufficient) {
+       require(amount <= 5);
    ...

and run npx buidler test again. You'll get a stack trace like the one we saw at the beginning.

Both Truffle and Buidler offer some nice functionalities to help make sense of what's happening inside your contracts. The MetaCoin contract is really simple, so this might seem overkill, but you've surely run into issues where you wished you'd had a debugger, a logger or a stack trace. Or, even better, all three of them!

Debugging transactions in Ethereum. Part 1: The Long and Winding Road

Franco Victorio — Fri, 28 Feb 2020 14:00:33 +0000

Do you know that happy feeling when a transaction fails and you are the one that gets to debug it? Of course you don't.

Debugging transactions in Ethereum is still a major pain point. While there's been a lot of progress (and I hope to cover much of it in this series), it remains an unpleasant activity. So the more tools you have at your disposal, the better.

We'll start by exploring the data we get from etherscan. All of the transactions I'll use as examples were executed in Rinkeby, but everything should be valid for Kovan or Mainnet. I think. I hope.

(I'll also use eth-cli for performing some simple tasks. You can install it by doing npm install -g eth-cli, but you don't need to.)

Function selectors

Let's start with a very simple contract:

contract Box {
    uint256 public value;

    function inc() external {
        value++;
    }
}

Now let's say we are interacting with this contract somehow (maybe we are building a dapp for it) and our transaction fails. Check for example this transaction:

There isn't a lot of data here, but we can manage to get some useful information out of it.

If you click on "Click to see More", you'll see more data about this transaction. The part we care about now is the Input Data, a bunch of bytes that make up our transaction.

Input data always starts with a function selector: 8 characters (4 bytes) that identify the method being called. So you can start by checking that you are calling the proper method. In this case, since there's only one function, you can use eth method:hash 'inc()' to find out its selector. Turns out it's 371303c0, but in the failing transaction the 4 bytes are 33da8f9c. So we found the problem!

In this case I artificially made the situation harder by using a weird name (it's incrementPlease(), in case you care). But in other cases it's even easier.

In this transaction, etherscan correctly decodes the method being called.

I don't know exactly when it does this and when it doesn't (maybe it leverages the 4byte directory?), but in any case if the method being called is common enough, then it will probably be automatically decoded here, as in this case where we are calling increment() instead of the correct one, inc().

Revert reasons

Let's add another method to our contract:

contract Box {
    // ...

    function dec() external {
        require(value > 0, "Value must always be positive");
        value--;
    }
}

Now we have a require with a revert reason. In this transaction we get an error, but we can easily guess what happened thanks to the message in the "Status" row.

This helps a lot, so the take-away is simple: always add reason strings to your reverts!

Of course, if you don't include a reason string, like here, then you'll need to try other approaches.

Digression: how revert reasons work

Curious about how to recover the error manually? (Hopefully you will never have to do this.) First, we make an eth_call with the same data, and at the same block number (0x5c471e is 6047518, the block number where the failed transaction was mined)

$ curl -H "Content-Type: application/json" -X POST --data \
        '{"id":1, \
        "jsonrpc":"2.0", \
        "method":"eth_call", \
        "params":[ \
          {"from": "0x9A2015Ed446E7A7450b9175413DEb04Fe4e555c2", "to": "0x23c1fd51DD362D87A9E20F7370B7E9A0CbC40D4f", "value": "0x0", "data": "0xb3bcfa82"}, \
          "0x5c471e" \
        ]}' https://rinkeby.infura.io/

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": "0x08c379a00000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000001d56616c7565206d75737420616c7761797320626520706f736974697665000000"
}

And then we interpret the result as the data of a call to the function Error(string):

$ eth method:decode 'Error(string)' '0x08c379a00000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000001d56616c7565206d75737420616c7761797320626520706f736974697665000000'

[
  "Value must always be positive"
]

Invalid opcodes

Now let's add a third method to our contract:

contract Box {
    // ...

    function divideBy(uint256 x) external {
        value = value / x;
    }
}

Again, this is a very simple code, so you can probably guess what's coming, but let's do it anyway.

This transaction fails and we don't get an error message. Instead, there's an invalid opcode 0xfe.

If you check the list of op codes you'll see that 0xFE means "INVALID". If we decode the data that was sent:

$ eth method:decode 'divideBy(uint256)' 0x1fcc4afb0000000000000000000000000000000000000000000000000000000000000000
["0"]

Then the error is clear: there was a division by zero that triggered this. Another example of an invalid opcode being thrown happens when you try to access an array using and index that is out of bounds.

We had to manually decode the data because the contract is not verified. If it were, then etherscan would've decoded it for us.

Of course, in most real life scenarios your code won't be so easy to read as it was in these examples. In the next post, we'll continue to explore what can be done in those cases.

A comparison of BigNumber libraries in JavaScript

Franco Victorio — Wed, 06 Nov 2019 00:14:23 +0000

If you've ever wanted to handle big numbers in JavaScript, you've probably noticed that there are a lot of different libraries, even for JavaScript standards. This article compares seven of them and hopefully will help you choose one.

I'll start by comparing some of the features that they do or don't support. Then I'll show some snippets of code for each one to give you a feeling of their API. After that I'll make a brief comment on the libraries used in the Ethereum ecosystem, since that's where I work on and it's an area where this kind of libraries is very present. Finally I'll give you my advice on which one to use (spoiler alert: it's big.js).

Feature comparison

The following table shows the libraries I picked and some aspects of each one. There are a lot of other things you might want to consider, like their API, performance, supported operations, etc., but this should give you a place to start.

Library	Integers	Floating-point	Other bases	Scientific Notation
big.js	Yes	Yes	Throws	Yes
bignumber.js	Yes	Yes	Yes	Yes
decimal.js	Yes	Yes	Yes	Yes
bn.js	Yes	Throws	Yes	Throws
BigInteger.js	Yes	Throws	Yes	Yes
JSBI	Yes	Wrong	Yes	Yes
jsbn	Yes	Wrong	Yes	Wrong

Integer values

All of them support integer values, but decimal.js, by design, can lose precision (more on this later).

Both BigInteger.js and JSBI can act as some sort of polyfill for the ECMAScript BigInt proposal, although their approaches differ. Check the Why? section in JSBI's readme to find out more.

Floating-point values

Only the first three support floating point numbers, and they were all developed by the same author. He wrote an explanation on how they differ, but the tl;dr is this:

big.js is a minimalist library. Use it if you don't need a lot of features and/or you care about the size of your dependencies.
bignumber.js and decimal.js are similar, the main difference is that bignumber.js expresses its precision in terms of decimals (appropriate for financial applications, for example) and decimal.js does it in terms of significant digits (better for scientific applications). That's why decimal.js is not a good choice for arbitrary integer arithmetic¹.

The rest of the libraries don't support floating-point numbers, but they have different behaviors when you try to create an instance with, for example, 3.14:

bn.js and BigInteger.js throw an error.
JSBI accepts it, but it parses it as 3.
jsbn accepts it, but it parses it as 314.

Other bases

All of them, except big.js, support inputs in different bases. big.js throws an error if used that way. bn.js and jsbn do support different bases, but you have to be explicit: if you do new BN('0x1f3') it will return 33253 for some reason, but new BN('1f3', 16) works fine. The same comments apply to jsbn.

Scientific notation

Scientific notation works for all of them except bn.js (that throws an error) and jsbn (that, again, returns some very wrong value)².

Show me the code

How do they look like? Let's see how to add 2+2 in each one of them. This is not enough to make a judgement on their API, but it showcases some important details:

// big.js
Big(2).add(2)

// bignumber.js
BigNumber(2).plus(2)

// decimal.js
Decimal(2).add(2)

// bn.js
new BN(2).add(new BN(2))
new BN(2).addn(2)

// BigInteger.js
BigInteger(2).add(2)

// JSBI
JSBI.add(new JSBI('2'), new JSBI('2'))

// jsbn
new jsbn.BigInteger('2').add(new jsbn.BigInteger('2'))

There's a bunch of things you can see here:

Some of them require the use of new, while it's optional for the rest.
The add method in bn.js has to receive a BN instance as its argument. If you want to use a number, you need to use addn. jsbn requires that the argument to add be another instance.
Instances of JSBI don't have methods like add, you need to use the static methods of the library.
JSBI and jsbn require strings as the arguments to their constructors. The other libraries accept both numbers and strings.

Size and popularity

The following table shows the (minified) size of each library and their weekly number of downloads at the time of writing this:

Library	Size	Weekly downloads
big.js	8K	9.272.986
bignumber.js	20K	2.390.156
decimal.js	32K	290.392
bn.js	56K	7.101.573
BigInteger.js	32K	899.179
JSBI	28K	16.508
jsbn	24K	11.648.984

A note on Ethereum

Arbitrary-precision libraries are important in the Ethereum ecosystem because smart contracts can return numbers with up to 256 bits, and JavaScript can't handle that precision. That's why the main client libraries come with some sort of big number library:

web3@0.x uses bignumber.js (actually, a fork of it).
web3@1.x uses bn.js. There is a discussion about changing it again.
ethers.js exposes a custom big number library that uses bn.js under the hood but which also adds some extra functionality.

This means that the most used clients (web3 after 0.x and ethers.js) use a library that doesn't support floating-point numbers. This kind of makes sense, since Solidity doesn't (yet) support them, but it also makes some things harder (e.g., computing some percentage of a value).

Which one should I use?

Which library you'll choose will depend, of course, on your use case, but my advice is that you can't go wrong with big.js. The API is very nice and its feature set should cover most use cases. You can check it out and, if you need a feature that it doesn't support or if it has some behavior that makes life harder for you, then you can check some of the other ones.

For example, if you are using 10 significant digits, then Decimal('22222222222222222222').div(2).toFixed(0) is equal to 11111111110000000000. ↩
I think it has to do with some base inferring, but couldn't find any documentation on this. ↩

A Dictionary of Modern Programming Usage

Franco Victorio — Sat, 16 Mar 2019 15:04:54 +0000

You know the old saying: «There are only two hard things in programming: cache invalidation and naming things», or its tongue-in-cheek variant: «There are two hard problems in programming: cache invalidation, naming things, and off-by-1 errors». They are misguiding, though, because naming things is way harder than the other two problems. Off-by-1 errors are annoying, but they can be solved just by being methodical. Cache invalidation is harder, but at least you have a lot of research around it. But if you are having trouble naming a variable, you are on your own.

Is there something we can do about this? Does it even make sense to think about doing something? I think it does.

Choosing the proper words to express an idea is not a new problem. I bet it exists since writing was invented, and maybe even before that. And there are tools meant to help with this, like dictionaries, thesauruses and usage dictionaries. What exactly are usage dictionaries? Well, I just used one to check if it was OK to use "thesauruses". Here's what Garner's Modern English Usage has to say about it:

thesaurus (= a book or online resource that supplies synonyms) has long formed the standard plural thesauri. But since 1960, thesauruses has climbed in frequency, especially in AmE.

That's not a dictionary. Even if it has short definitions, it also —and mainly— offers commentary about the word: how it's used and how it should be used.

Should? Who is to say how words should be used? Language is organic, you can't stop it from changing, etc. I'm not going to get into the war between prescriptivists and descriptivists. Suffice it to say that I am on the side of (moderate) prescriptivists. Otherwise I wouldn't consult a usage dictionary in the first place. (If this subject interests you, go read David Foster Wallace's essay "Authority and American Usage"; if it doesn't, go read it anyway: Wallace can make any subject interesting.)

Garner's has another usage dictionary called "Garner's Dictionary of Legal Usage", that is limited to the vocabulary and style of legal texts. It makes sense: lawyers have their own dialect, and how they use it is very important.

Sounds familiar?

We as programmers have a similar problem. Using a bad name won't affect a trial, but it will hurt maintainability in the long run. Bad names contribute to technical debt and are mentally taxing. How comes, then, that nobody has written a "Dictionary of Modern Programming Usage"?

Of course, this would get outdated. But so did usage dictionaries from the 1920s. Does that mean they weren't useful when they came out? And yes: programming languages change much faster than natural languages, but I would argue that the way we choose names for variables, functions or classes doesn't change as fast as available syntax features or libraries.

It's interesting to think about the implementation issues that arise from this idea, the first one being who would write it —or rather, how many people. Can this be a one person job? Or do you need a group of people that come from different backgrounds? Or, going further, maybe this should be a crowdsourced effort, like a wiki?

Another question is how do you define the scope of such a thing. This would require serious thought, but I think some things can be said for sure. First, you don't get into anything that is already in an English usage dictionary. Second, you don't talk about domain specific things (like networking concepts, for example), since this would make the work impossible and, besides, it wouldn't make sense to talk about the usage of well-defined technical terms.

What is in scope then? I can think of some examples:

What does qualified means? How it's different from canonical?
What's the difference between formatting, serializing, pretty-printing, etc.? (See this discussion.)
When do you use get, compute, select, fetch?
There are a lot of words for (what I call) maps: maps, dictionaries, associative arrays, hashes (a perl/ruby monstrosity). Can they be differentiated or it just depends on the programming language?

That last point takes us to the third question: should this hypothetical dictionary be specific to some programming language? I'm inclined to say no, since I feel most things would be language-agnostic. But some terms should include clarifications on their different usages among languages.

There's something that should make a project like this easier, or at least more interesting and valuable. The fourth edition of Garner's Dictionary takes advantage of a tool that wasn't available when its previous editions came out: the Google Ngram Viewer. You can imagine how useful this can be for a lexicographer.

Something similar could be used for a programming usage dictionary. For starters, we have GitHub, where we can search any term across millions of repositories. But more complex things could be done. You could use parsers to get the names of every defined function. How many of them start with compute? And then you could sample some of them, or the ones from the most famous projects, to get a better understanding of how they are used and why they are named that way. Again, this runs into the issue of which subset of the existing programming languages should you use. But you can start small, with two or three of the most prevalent ones, and work your way from there.

This doesn't mean that the effort could be automated, not even close. A project like this would require a lot of reading and exploring. But open source and tools that analyze code could amplify the work of programming lexicographers and give more certainty to their judgements.

I'm not aware of the existence of something like this, nor of someone working on it. Maybe it wouldn't be as useful as it seems to me, or maybe the necessary effort is just too large. I don't know. At the very least, I think it would be worth the shot.

Originally published in my newsletter. You can subscribe here.

How to write a plugin for Prettier

Franco Victorio — Wed, 13 Mar 2019 22:50:49 +0000

Originally posted here.

In this post I will show you how to build a very simple plugin for Prettier. You will learn how to structure, test and, of course, write a plugin. I'm far from being an expert on this, but I learned a lot by contributing to the Solidity plugin, and noticed that there doesn't seem to be a lot of material on this subject, except for the official documentation.

We'll be using TOML as the example language. I chose TOML because it has an easy syntax and, as far as I know, there's no Prettier plugin for it. The result won't be a usable plugin, as you'll see, but with any luck you will learn enough to develop a proper one.

Why would you want to learn this? Well, there are two strong reasons. First, you'll be able to create a plugin for any language that is not supported yet (like TOML and Dockerfile, but there probably are many others). And second, you'll be able to contribute to one of the existing plugins. Moreover, since the core of Prettier itself is written using the plugin API, you could even contribute to the main repository if you wanted to.

How Prettier works?

At its core, what Prettier does is very simple: it takes some code (a string), converts it to an AST (Abstract Syntax Tree, a representation of the code) and then prints the code using only the AST. That means that the style of the original code is (almost) completely ignored. You can learn more in the original blog post.

For our purposes, the important part is that we need a parser that transforms the code to an AST, and a function that takes this and pretty-prints it. Our initial setup already has a configured parser, using toml-node, so we only need to worry about the printer function.

Setup

First we'll clone this repository that has all the boilerplate you'll need to start. I'll explain its contents soon. After cloning it, go the top level directory and run npm install to install the dependencies. You should now be able to run the plugin in the example file (example.toml) with this command:

./node_modules/.bin/prettier --plugin . example.toml

There's also a npm script for this, so npm run example should work too, but this shows how to run the plugin in any file you want.

After running the command, you won't see any output, and that's OK. For now, our plugin doesn't emit anything: when the printer function receives the AST, it just returns an empty string.

There's also an initial test that you can run with npm test. Our tests will be written in jest, using snapshots, but since the setup is already there the only thing you have to do is to add new fixtures. This initial test will format the contents of tests/StringAssignements/example.toml and compare the result with the expected output in the snapshot. All of our tests will be like this one: a TOML file and a snapshot with the correct format. This test will fail, of course, but our first goal is to make it pass.

All the code we'll write will be in the src/index.js file. In fact, everything will be inside a single function: printToml. You can take a look at the rest of the file, but don't worry about the details. If you are curious, it's all explained here.

And if you want to read the finished code instead of writing it, just checkout the finished branch.

The printer function

The printToml function is very simple. It takes three arguments:

path, that represents a node in the AST
options, that represents the configuration given to prettier (the combination of .prettierrc and the flags given to the command, among other things)
and print, that is how we call the printer function recursively

Notice that I said that path is some node in the AST, not the root. That's because the function is called recursively. For example, if I have the body of a function, I may want to pretty-print each individual statement separately and then do something with this result. This will become clearer as we continue.

This is the boilerplate of our function:

function printToml(path, options, print) {
  const node = path.getValue()

  if (Array.isArray(node)) {
    return concat(path.map(print))
  }

  switch (node.type) {
    default:
      return ''
  }
}

The first line just extracts the AST node from the path. This is because path has some extra information and logic related to the AST node.

Then we have a strange block that checks if the node is an array. This will only be necessary in the initial call, because the parser that we are using represents the code as a list of nodes, not as a tree of nodes. Don't worry about this, but keep it in mind, because later this will impose some serious limitations to our plugin.

Finally, we have the switch. Here's where we'll spend most of our time. The logic we have is very simple: we check the type of the AST node and act accordingly. Let's start to fill it in.

A simple assignement

If you take a look at our test, you'll see that it contains two key/value pairs. The node that represents the first pair is something like this:

{
  type: 'Assign',
  value: {
    type: 'String',
    value: 'TOML Example',
    line: 1,
    column: 9
  },
  line: 1,
  column: 1,
  key: 'title'
}

(How do we know this? There are a lot of ways of obtaining it: a good old console.log, using the parser in the node REPL, or running the plugin using ndb and inspecting the value.)

There are two interesting things here. First, the type property, that is what we use in our switch. The second one is that, while the key of our pair is a simple string, our value is another AST node, whose type is String.

So the first thing we'll do is to add a clause for Assign nodes:

case 'Assign':
  return concat([node.key, ' = ', path.call(print, 'value'), hardline])

There's a lot to unpack here, but the main idea is easy to grasp: we are telling prettier that an assignment is printed by concatenating four things:

The key. Remember that this is just a plain string
A literal equal sign, padded with spaces
The result of pretty-printing the value of the assignment, whatever that is
And a hardline

What are concat and hardline? They are called builders, and they are functions and values -exposed by Prettier- that we use to build the result we want. We already have imported concat, but we need to add hardline to the list of builders we are using:

const {
  doc: {
    builders: { concat, hardline }
  }
} = require('prettier')

The concat builder is easy to understand: it tells prettier to just concatenate the list of parts it's given. And hardline just means "put a line break", no matter what. You can see the full list of builders here.

What about the path.call(print, 'value') part? This is a prettier idiom and it just means "call the printer function recursively, using the node that is in the 'value' key". Why can't we just do print(node.value)? Well, remember that the printer function expects a path, that is, a wrapped node, not a node. So you have to do it like this.

If we add just this and run our test, it will fail. The diff tells us that the keys and the equal sign were printed, but not the value. This makes sense, since the values are nodes of type String and we don't have a clause for that yet. Fortunately, that clause is very simple. Take a look at the AST sub-node again and see if you can guess it.

Yes, it's that easy:

case 'String':
  return concat(['"', node.value, '"'])

You may have guessed just return node.value, but that would've been wrong, because in that case we would be printing just the content of the string, not the full string. For example, foo = "bar" would've been printed as foo = bar.

If we run our test again, it should pass now.

Adding support for other values

TOML supports other data types besides strings, and we should support them too. If you look at the example in the root directory, you'll see that it has numbers, booleans, dates, and lists.

Numbers and booleans are easy:

case 'Integer':
  return node.value.toString()
case 'Boolean':
  return node.value.toString()

We have to convert them to strings, because that's what prettier expects, but that's it.

Dates are a little trickier and here we'll run into the first limitation of the parser we are using. Here's the AST representation of a date assignment:

{
  type: 'Assign',
  value: {
    type: 'Date',
    value: 1979-05-27T15:32:00.000Z,
    line: 5,
    column: 7
  },
  line: 5,
  column: 1,
  key: 'dob'
}

Look at the value of the date. That is a Date object, a unique representation of a date. But if you take a look at the TOML spec, you'll see that you can specify dates in many different formats. That is lost to us during parsing, so we'll always print dates with the same representation.

case 'Date':
  return node.value.toISOString()

That's not nice at all! But to do it properly, we should know the original representation of the date. We could get it using the position of the node and the original text (that we receive in options.originalText), but it would be even better to have a parser that keeps the original value in the AST. Since our parser doesn't do this, we'll have to settle for this.

Tables

In TOML, we can separate different sections with what the spec calls "Tables", but our parser assigns the type ObjectPath. An AST node looks like this:

{
  type: 'ObjectPath',
  value: [ 'owner' ],
  line: 3,
  column: 1
}

As you can see, the value of the node is not a string but an array. This is because we can have nested sections like [servers.alpha]. We print this with the following clause:

case 'ObjectPath':
  return concat(['[', node.value.join('.'), ']', hardline])

Nothing new here. We join each part of the value with a period and surround everything with square brackets.

Arrays

So far, everything we've done has been very straightforward. Arrays are a little more complex, and we'll have to make some decisions. There are several ways in which an array can be printed, for example:

arr1 = [1, 2, 3]
arr2 = [ 1, 2, 3 ]
arr3 = [1,2,3]
arr4 = [
  1,
  2,
  3
]

Here's what prettier usually does in situations like this: if the array fits in one line, print it in one line. Otherwise, print everything on its own line. So we'll do the arr1 approach when the array fits, and we'll print something like arr4 when it doesn't.

That seems hard, doesn't it? But prettier can help us. This is the clause that does what we want:

case 'Array':
  return group(
    concat([
      '[',
      indent(
        concat([
          softline,
          join(concat([',', line]), path.map(print, 'value'))
        ])
      ),
      softline,
      ']'
    ])
  )

I know this is quite a jump from what we've been doing so far. And the bad news is that a lot of plugin code looks somewhat like this. The good news is that you get used to it quickly.

Let's start from the innermost part of that expression and work our way out.

First we have a path.map(print, 'value') expression. This is similar to the path.call idiom we discussed before, but here we are saying "in the value key of this node I have an array of sub-nodes; call the printer function on each one and give me an array with the results". In other words, it's like doing node.value.map(print), but remember that we can't do that.

So we have an array with the result of pretty-printing each element in our list. The next step is adding our commas. We use the join builder for that. Its signature is join(separator, list), and it just joins the list of parts with the given separator. For example, concat(["1", ",", "2", ",", "3"]) is equivalent to join(",", ["1", "2", "3"]). And we could do that here, right? Just join(",", path.map(print, 'value')). But we want to have a space after the comma when the list fits in one line, and a line break when we split it. That is done with the line builder, and that why we join by concat([",", line]). The documentation is clear:

Specify a line break. If an expression fits on one line, the line break will be replaced with a space. Line breaks always indent the next line with the current level of indentation.

So we print each value separated by a comma and a space if the list fits in one line, and we replace the spaces with line breaks if it doesn't fit. We should be ready, right? Just add the opening and closing square brackets and be done with it. Well, no. Because we want to indent each element of the list when we split it.

We do that by surrounding what we have done so far with indent(concat([softline, ...])). What is going on here? First we put a softline at the beginning of the list. softline is very similar to line, but the difference is that, if everything fits in one line, softline is replaced with an empty string. We also use the indent builder, that just increases the indentation. When everything fits in one line, we won't have line breaks, so indent won't do anything.

Almost there! After that, we surround everything with concat('[', ..., softline, ']'). We are just adding the brackets. We also add a softline before the closing bracket, and since it's outside the indent builder, the ] will have the same indentation we started with. Otherwise our lists would look like this:

arr = [TOML spec
  1,
  2
  ]

And finally we surround everything with a call to group. This is a builder that tries to fit everything inside it in one line. If it doesn't, it will start replacing lines and softlines with line breaks. It's actually a little more complex, but that explanation will do for now. Check the documentation to see the nuances of it.

Again, this seems hard, but you'll get it quickly when you start playing with prettier. All of this also shows how powerful prettier is, if you think about it. We've used just a few building blocks to pretty-print any list. In fact, this will work even with nested list, no matter how deep they are!

Aside: How to experiment

How can you check how builders interact, besides reading the documentation and running your full plugin with some examples? It turns out you can use the node REPL to interact with prettier. First start the REPL and import some stuff:

> const prettier = require('prettier')
> const print = prettier.doc.printer.printDocToString
> const { concat, group, join, line, softline } = prettier.doc.builders

And then you can experiment with the builders:

> print(concat(['foo', 'bar', 'baz']), {})
{ formatted: 'foobarbaz' }
> print(join('|', ['foo', 'bar', 'baz']), {})
{ formatted: 'foo|bar|baz' }

To test things like group, you'll need to specify a printWidth:

> print(group(join(line, ['foo', 'bar', 'baz', 'qux'])), { printWidth: 20 })
{ formatted: 'foo bar baz qux' }
> print(group(join(line, ['foo', 'bar', 'baz', 'qux'])), { printWidth: 10 })
{ formatted: 'foo\nbar\nbaz\nqux' }
> print(group(join(softline, ['foo', 'bar', 'baz', 'qux'])), { printWidth: 20 })
{ formatted: 'foobarbazqux' }
> print(group(join(softline, ['foo', 'bar', 'baz', 'qux'])), { printWidth: 10 })
{ formatted: 'foo\nbar\nbaz\nqux' }

You can learn this way. I know it's not a great user experience, and it would be nice to have something better (maybe a web playground where you can run expressions like this and see the result with different inputs?), but I'm not aware of anything better.

Pending things

If we run our example again, we'll see that we have an equivalent TOML printed as we specified it:

> prettier-plugin-toml@0.0.1 example /home/fvictorio/repos/prettier-plugin-toml
> prettier --plugin . example.toml
title = "TOML Example"
[owner]
name = "Tom Preston-Werner"
dob = 1979-05-27T15:32:00.000Z
[database]
server = "192.168.1.1"
ports = [8001, 8001, 8002]
connection_max = 5000
enabled = true
[servers]
[servers.alpha]
ip = "10.0.0.1"
dc = "eqdc10"
[servers.beta]
ip = "10.0.0.2"
dc = "eqdc10"
[clients]
data = [["gamma", "delta"], [1, 2]]
hosts = ["alpha", "omega"]

But it would be hard to argue that this is prettier. There are too very important things we are not doing, and that we can't do easily with the parser we are using:

We are not preserving blank lines. Prettier's philosophy is to keep them (although if there are two or more blank lines together, they are replaced by a single blank line). This can be done, but to do that we need an easy way to get the start and end indices of the node. As you see in the nodes examples, we only have the starting line and column.
We are not indenting the tables. This would be relatively easy if the representation of the AST would be a proper tree, but remember that we have instead a list of nodes for each line. If under the table objects we'd have a, say, "children" key, we could do something like path.map(print, 'children'), join that by hardlines and indent them, for example.

What's next?

Hopefully you learned enough to start your own plugin or contribute to one. Take a look at the list of plugins: if the language you would like to see prettified isn't there, you can create your own! And if it is, you can jump on and contribute.

A nice thing about prettier plugins is that it's very easy to do TDD with them. If you want to contribute to a plugin, just add a fixture with an example that it's not working and try to make all tests pass. If you are creating a new plugin you can start small: add tests with some simple examples using a subset of the syntax and make them prettier!