Forem: Ethan Arrowood

Multi OS Ansible Playbooks Demo

Ethan Arrowood — Tue, 02 Mar 2021 15:49:40 +0000

Foreword: I am by no means an expert or even experienced IaC developer. This article is summarizes some work I did on a customer engagement as a Microsoft Commercial Software Engineer. This article is meant to serve as a starting point for other IaC developers looking to develop solutions for their multi OS environments.

Purpose

The purpose of this demo is to demonstrate some basic multi-os Ansible playbooks. The patterns demonstrated within the playbooks can be used when establishing a Kubernetes cluster. For example, the main node can generate a cluster join command and set that as a fact. That fact can then be read by the other nodes and executed in their respective shells in order to join that worker host to the Kubernetes cluster.

The source files referenced in this post can be found in my public repo: Ethan-Arrowood/multi-os-ansible-demo.

In the src/ folder there are three files:

inventory.ini - is an Ansible inventory file. It defines hosts, groups, and variables. This demo uses it to group hosts based on their OS and apply specific variables to those groups.
hello-world-playbook.yml - is an Ansible playbook file that demonstrates a hello-world example where each host prints "Hello, World" to their respective shell environment.
fact-playbook.yml is another Ansible playbook file that demonstrates a slightly more complex example of sharing information between hosts using Ansible facts. It requires Python to be installed on all hosts.

Requirements

Azure Virtual Machines are a great way to set up this demo environment.

Three VMs belonging to the same vnet
A linux based VM with public network ssh access enabled (main)
A linux based VM with private vnet ssh access enabled (linux_worker_1)
A windows based VM with private vnet ssh access enabled (windows_worker_1)

Notice: Azure currently does not have automatic OpenSSH Server support for Windows Server VMs. Use a pre-configured instance or follow these guides to set it up.

The main linux VM (configured with public ssh access) should have the necessary ssh keys for accessing the two other VMs (linux_worker_1 and windows_worker_1).
Make sure to sync ssh known_host values for the main host to the worker hosts so that the hosts trust each other prior to executing the ansible playbooks.

Demo Steps:

Securely copy over the /src files from this directory to the main VM using scp
SSH into the main linux VM and install Ansible

  sudo apt-add-repository ppa:ansible/ansible
  sudo apt update
  sudo apt install ansible

Execute the examples from the main VM using:

  ansible-playbook -i ./src/inventory.ini ./src/hello-world-playbook.yml
  ansible-playbook -i ./src/inventory.ini ./src/fact-playbook.yml

Notice: The fact-playbook.yml requires Python to be installed on the Windows Server and the path to the python interpreter added to the list of [windows:vars] in src/inventory.ini as ansible_python_interpreter. This will most likely be set to C:/Python39.

Closing

Ansible is a fantastic tool for Infrastructure automation and management. This article is based off of a project I worked on as a Microsoft Commercial Software Engineer. If this kind of work interests you, CSE is hiring across the world for a variety of roles. Visit https://aka.ms/csejobs for details.

Cover Image by unDraw

Windows OpenSSH Guides

These guides should help in setting up Windows Server 2019 with OpenSSH. Make sure you copy over the same ssh key used by Ansible in this demo. If you're following these guides in context of this demo you'll need to allow public RDP access to the Windows instance in order to enable OpenSSH.

Personal Scratchpad

This section contains a bunch of commands I used for completing the operations listed in the demo. They were written and used on macOS 10.15.7, so they are not guaranteed to work on other platforms.

main_vm_public_ip=<insert main host ip here>

# create a local .ssh folder
mkdir .ssh

# generate an empty passphrase ssh key in the local folder
ssh-keygen -q -m PEM -t rsa -b 4096 -N '' -f ./.ssh/id_rsa <<<y 2>&1 >/dev/null

# copy the contents of the key file into the clipboard; use this key when creating the VMs in Azure Portal
pbcopy < ./.ssh/id_rsa.pub

# copy over the physical keys for ansible main-main and main-worker connections
scp -i ./.ssh/id_rsa -r ./.ssh/ azureuser@"$main_vm_public_ip":~/

# copy over the src files for the demo
scp -i ./.ssh/id_rsa -r ./src/ azureuser@"$main_vm_public_ip":~/

# ssh into main
ssh -i ./.ssh/id_rsa azureuser@"$main_vm_public_ip"

# install ansible
sudo apt-add-repository ppa:ansible/ansible
sudo apt update
sudo apt install ansible

# run the ansible-playbooks
ansible-playbook -i ./src/inventory.ini ./src/hello-world-playbook.yml
ansible-playbook -i ./src/inventory.ini ./src/fact-playbook.yml

TypeScript support for Pino with Fastify

Ethan Arrowood — Mon, 21 Dec 2020 22:42:15 +0000

By default, Fastify ships with a Pino instance as its logger; however, since Pino types are not maintained (yet) by the Pino project itself, Fastify does not ship type support for the complete Pino API. Understandably, this can be frustrating, but in this post I want to show you just how easy it is to add @types/pino to your Fastify app.

Install Pino Types
```
npm i -D @types/pino
```

Augment the FastifyLoggerInstance type in the same file the app is instantiated in (generally index.ts or server.ts)

import type { Logger } from 'pino'
declare module 'fastify' {
    interface FastifyLoggerInstance extends Logger {}
}

This works because in the v3 Fastify types, the FastifyInstance.log property is defined as a generic Logger that is defaulted to FastifyLoggerInstance (ref). Declaration merging the FastifyLoggerInstance type with interface FastifyLoggerInstance extends Logger {}, informs TypeScript to consider the type as an extension of the logger type defined in @types/pino.

HarperDB and WebSockets

Ethan Arrowood — Tue, 01 Dec 2020 15:45:15 +0000

HarperDB and WebSockets

WebSocket is a computer communications protocol, providing full-duplex communication channels over a single TCP connection. Wikipedia

TCP stands for transmission control protocol. It is a reliable protocol; thus, it assures that the data being transmitted will be sent in its entirety (a.k.a. lossless). There are many different higher-level protocols based on TCP, each serving the general purpose of transmitting data from one point to another. HTTP, the main communication protocol of the internet, is an example of a high-level protocol many developers are familiar with. WebSocket operates similar to HTTP, but has its own variety of differences.

One of WebSocket's main principles is the full-duplex communication channels. These channels allow for simultaneous, bi-directional data transfer between the two peers. This means that at any time the systems at either end of a WebSocket connection can receive or send data.

HTTP differs in this principle because generally only the client can make requests to the server. The server completes the request, sends the response back to the client, and then waits to receive another request.

I use "generally" here since new versions of HTTP have enabled things like server push.

Consider a chat application scenario:

All four clients need to be kept in sync. When John sends a message, Chris, Sam, and Jane, should all receive it as quickly as possible. If this app is using HTTP, the send operation is simple, use a POST request to the server with the message. But the receive operation is a bit harder. Client's need to simultaneously be polling the server for new messages. Depending on the polling interval and the number of connected clients, the server could be looking at an alarming number of requests to handle.

With WebSockets, the flow is simplified significantly. All clients establish a connection with the server and begin subscribe to a certain channel. Any client (or the server) can publish a message to the specified channel at any time. When it does so, the server will then process the publish event, and depending on the implementation, will broadcast the new message to all client connections on the channel. This pattern is often dubbed as the pub/sub model. And is used by a plethora of applications, including HarperDB clustering!

Internally, HarperDB clusters replicate data between instances using the bi-directional pub/sub model. Clients can subscribe to the same WebSocket channels HarperDB uses and access all of the updates of a given table. Using this mechanism, developers can build applications with close-to realtime data updates. To make this application development even easier, HarperDB has published a new module, harperdb-websocket-client.

Follow along with the demo below to see it in action.

Demo

The source code for this demo can be found at Ethan-Arrowood/harperdb-websocket-tutorial

This demo will walk you through the creation of a Node.js project powered by WebSockets and HarperDB. The application will be a data entry CLI with an additional display script.

Prerequisites and Set Up

Set your current working directory to a new folder:

mkdir hdb-websocket-demo
cd hdb-websocket-demo

Create a docker-compose.yml file with these contents:

version: '2'
services:
  harperdb:
    image: harperdb/hdb:latest
    environment:
      - INIT_HDB_USERNAME=HDB_ADMIN
      - INIT_HDB_PASSWORD=password
      - INIT_CLUSTER_USER=cluster_user
      - INIT_CLUSTER_PASSWORD=password
      - INIT_ENABLE_CLUSTERING=true
      - INIT_NODE_NAME=docker_node
      - INIT_CLUSTER_PORT=12345
    ports:
      - 9925:9925
      - 31283:31283
      - 12345:12345
    volumes:
      - ./hdb_data/:/opt/harperdb/hdb/

Initialize a new npm project and install two dependencies:

npm i -y
npm i undici harperdb-websocket-client

Create three project files:

touch init.js data-entry.js data-display.js

And finally launch HarperDB using Docker

docker-compose up

init.js

The init script contains some necessary set up operations so the CLI application has a schema and table to query against.

Start off by importing undici, initializing a new undici client, and creating two methods requestFactory and readStream.

undici is a new Node.js http 1.1 client. It is blazing fast and has a really extensible API.

const undici = require('undici')

const client = new undici.Client('http://localhost:9925') // make sure to use the http port for the HarperDB instance

/**
 * This function takes a JSON operation and returns an undici
 * client request with the given operation stringified in the
 * `body` property.
 */
const requestFactory = op => client.request({
    path: '/',
    method: 'POST',
    headers: [
        'content-type', 'application/json',
        'authorization', 'Basic SERCX0FETUlOOnBhc3N3b3Jk' // this token was generated based off of the credential settings in docker-compose.yml
    ],
    body: JSON.stringify(op)
})

/**
 * This is a helper function for processing the request responses.
 * Undici response `body` are returned as streams.
 */
const readStream = async stream => {
    let res = ''
    for await (let chunk of stream) {
        res += chunk
    }
    return res
}

Next, create four database operations using the requestFactory method

const dropSchema = () => requestFactory({
    'operation': 'drop_schema',
    'schema': 'dev'
})

const createSchema = () => requestFactory({
    'operation': 'create_schema',
    'schema': 'dev'
})

const createTable = () => requestFactory({
    'operation': 'create_table',
    'schema': 'dev',
    'table': 'animals',
    'hash_attribute': 'id'
})

const describeTable = () => requestFactory({
    'operation': 'describe_table',
    'schema': 'dev',
    'table': 'animals'
})

Finally, define an immediately invoked function expression (iife) using async/await.

;(async () => {
    await dropSchema()
    await createSchema()
    await createTable()

    // destructure the body property from the request response
    let { body } = await describeTable()
    // process the body stream
    let result = await readSTream(body)

    // sometimes the table creation fails so retry
    if (result.includes('Invalid table')) {
        await createTable()
        // re-destructure and process
        ;({ body } = await describeTable())
        result = await readStream(body)
    }

    // Check the intended output in the next code block
    console.log(JSON.parse(result))

    // close the undici client
    client.close()
})()

Now, with the HarperDB Docker container running, run this script with node init.js. The output should look something like this (but with different times and id):

{
  __createdtime__: 1606254462088,
  __updatedtime__: 1606254462088,
  hash_attribute: 'id',
  id: 'dc42484f-0586-44c7-9e80-9d2f36b6a4ed',
  name: 'animals',
  residence: null,
  schema: 'dev',
  attributes: [
    { attribute: 'id' },
    { attribute: '__updatedtime__' },
    { attribute: '__createdtime__' }
  ],
  record_count: 0
}

data-entry.js

Finally, some WebSocket action!

Start off by importing readline and HarperDBWebSocketClient, as well as initializing both:

readline is a native Node.js library.

const HarperDBWebSocketClient = require('harperdb-websocket-client')
const readline = require('readline')

// The credentials used here are specific to the docker-compose file defined earlier
const hdbClient = new HarperDBWebSocketClient({
    hostname: 'localhost',
    socketClusterOptions: {
        rejectUnauthorized: false,
        autoReconnect: false,
        ackTimeout: 10000,
        secure: true
    },
    port: 12345,
    username: 'cluster_user',
    password: 'password',
    implicitInit: true
})

const rl = readline.createInterface({
    input: process.stdin,
    output: process.stdout,
    // All of the \x1b things are color codes. When this runs in your terminal it should be multicolored!
    prompt: '\x1b[34mEnter new animal record, in the format <type>,<name>,<size>\nExample:\x1b[89m \x1b[36mdog,harper,medium\x1b[89m \n\x1b[91m>\x1b[39m '
})

Next, using the rl interface, create the basic CLI

// kick off the cli
rl.prompt()
rl.on('line', line => {
    // for this demo we are skipping any kind of input validation.

    // trim the input, split it up into an array, and then trim each entry.
    // desctructure into three variables
    const [ type, name, size ] = line.trim().split(',').map(v => v.trim())

    // Some useful debug info
    console.log(`\x1b[33mInserting new animal record: ${JSON.stringify({ type, name, size })}\x1b[89m`)

    // ✨ insert the new record using the websocket client
    hdbClient.insert('dev:animals', [
        { type, name, size }
    ])

    // prompt the user again for subsequent data entry
    rl.prompt()
}).on('close', () => {
    // print out a nice message when the user ends the CLI session
    console.log('\n\x1b[92mThank you for using HarperDB Animal Data Entry CLI\x1b[39m')
    process.exit(0) // exit safely
})

The core to this demo is quite short, hdbClient.insert('dev:animals', [{ type, name, size }]) is all you need to insert records over WebSockets! The client also supports .update and .delete. At this time, the WebSocket client cannot manage things like users, tables, or schemas.

Try running the new script with node data-entry.js. The following image shows what is output in my terminal interface.

Great! The CLI is working, but we can't actually see the new records... onto data-display.js.

data-display.js

Start again by importing and initializing HarperDBWebSocketClient.

const HarperDBWebSocketClient = require('harperdb-websocket-client')

const hdbClient = new HarperDBWebSocketClient({
    hostname: 'localhost',
    socketClusterOptions: {
        rejectUnauthorized: false,
        autoReconnect: false,
        ackTimeout: 10000,
        secure: true
    },
    port: 12345,
    username: 'cluster_user',
    password: 'password',
    implicitInit: true
})

Next, subscribe to the table the data-entry script is inserting records into.

hdbClient.subscribe('dev:animals', data => {
    const record = data.transaction.records[0]
    console.log(`New record ${record.id} inserted`)
})

✨ And just like that, now data-display will log every time a new record is added to the table!

With the Docker container and the data-entry script running, open up another terminal and run node data-display.js.

Go back to the data-entry terminal and enter a new record. You should see a new ID in the data-display terminal.

The following screenshot shows the same output from data-entry as before, but also with the data-display output on the right.

,,". The third line is purple and blue and says "Example: dog,harper,medium". The fourth line is red and white and says "> cat,garfield,large". The fifth line is yellow and says "Inserting new animal record: {"type":"cat","name":"garfield","size":"large"}". The second, third, and start of the fourth line repeat themselves one more time. On the right-side: The first line is my bash profile header styled and says "➜ node-app git:(main) node data-display.js". The second line is in white and says "New record e9923250-e316-4483-9de0-08f25204e320 inserted"."/>

Great work! Now enter as many records as you like and see how the WebSocket connections almost instantaneously output the new records. And for a special trick, try opening two more terminals. Run both scripts again so that you have two terminal instances for each script. Enter a record on the first data-entry terminal and see it output in both data-display process. Enter another record in the second data-entry terminal and see it also output in both data-display processes! WebSockets are pretty awesome.

I hope you enjoyed this article. My name is Ethan Arrowood and I'm a full time software engineer for Microsoft. I'm also an open source contributor to Fastify and Node.js. Come say hi on Twitter (@ArrowoodTech) and GitHub (@Ethan-Arrowood).

TypeScript - Type | Treat Halloween Challenge

Ethan Arrowood — Sun, 01 Nov 2020 22:21:07 +0000

Type | Treat Challenge

This halloween, the TypeScript team has released a series of challenges for us to solve. In this post I'll be going over all of my solutions to the challenges.

If you have any questions about my solutions or want to share your own, comment below!

The day I was planning on publishing this article, I realized the TypeScript team posted their own solutions for each of the challenges. If taken a bit extra time to expand upon my answers and try to provide additional information about the TypeScript patterns used in the solutions.

Challenge 1

Try it for yourself here

Beginner/Learner - Ghosts and Hauntings

The first challenge solution requires replacing the any argument in displayHauntings. The previous method displayGhost uses the displayHauntings method to print out the array of hauntings defined by the GhostAPIResponse. Unfortunately, the hauntings type is inlined with GhostAPIResponse interface and the challenge specifies not to modify GhostAPIResponse.

My first solution is to infer the item type of GhostAPIResponse.hauntings array using a custom utility type:

type ExtractArrayType<Arr> = Arr extends (infer Element)[] ? Element : never

To better understand this utility, lets break it down piece by piece. The type itself has a single generic Arr. The TypeScript interpreter attempts to match the generic with the type (infer Element)[]. This type uses the infer keyword to specify a second generic, Element. The interpreter is smart enough to assign whatever the item type of Arr is to the generic Element, and then since the interpreter has successfully matched the generic Arr to the type (infer Element)[] it returns the inferred element type Element. And if the interpreter is not able to match Arr, the utility returns never; preventing misuse.

Finally, to complete the challenge I replaced any with ExtractArrayType<GhostAPIResponse['hauntings']>, and discovered that there was in fact a type error on line 41 (provemance -> provenance).

Check out my challenge solution playground

Looking back on my first solution, I wouldn't consider the infer keyword something a beginner/learner would know. So, there had to be another solution. This got me thinking more about the Array type.

The Array interface is defined here in lib.es5.d.ts. It has one generic T, and the definition has an index type of [n: number]: T.

Using the index type definition, the solution can be simplified to: GhostAPIResponse['hauntings'][number]. No inference necessary!

Here is the updated solution playground.

Intermediate/Advanced - Trick or Treat

This challenge was a blast for me. One of my favorite math subjects is set theory.

The challenge starts with a union of halloween tricks or treats, ResultsFromHalloween. Each item in the union has either a candy: true property or a trick: true property. The first step in the challenge is to sort the union between tricks and treats.

Luckily, there is a built-in utility type.

Extract<Type, Union> constructs a type by extracting from Type all union members that are assignable to Union.

type AllCandies = Extract<ResultsFromHalloween, {candy: true}>
type AllTricks = Extract<ResultsFromHalloween, {trick: true}>

Under the hood, the Extract type utilizes conditional types:

type Extract<T, U> = T extends U ? T : never

For each discrete item in T, the interpreter checks if it satisfies the conditional T extends U, if it does then that item is added to the resulting union.

The challenge has one more task, create another type that excludes candies with peanuts in them. But, the treat types only describe if they have peanuts in them, not if they don't. Now the solution requires the inverse of extracting, excluding.

More set theory! Yay!

type AllCandiesWithoutPeanuts = Exclude<AllCandies, {peanuts: true}>

Yet again, TypeScript has a utility type for this operation.

Exclude<Type, ExcludedUnion> constructs a type by excluding from Type all union members that are assignable to ExcludedUnion.

This one also uses conditional types:

type Exclude<T, EU> = T extends EU ? never : T

For each discrete item in T, the interpreter checks if it satisfies the conditional T extends U, if it doesn't only then is that item added to the resulting union.

Check out my challenge solution playground

Challenge 2

Try it for yourself here

Beginner/Learner - Pumpkin Patch

This challenge tasks us to make use of the typeof operator. TypeScript is quite smart as long as you provide it with the right information. For the first part of this challenge the solution is to derive the Pumpkin type from the pumpkin object already defined using typeof pumpkin. The second part of the challenge uses a TypeScript utility type ReturnType<F>. The solution here is to pass the derived type of the function to the ReturnType utility: ReturnType<typeof createExamplePumpkin>.

To better understand how ReturnType works take a look at the implementation

type ReturnType<T extends (...args: any) => any> = T extends (...args: any) => infer R ? R : any;

The type specifies a generic T that is constrained to a function with any arguments and any return value (i.e. (...args: any) => any). This generic T is then matched against the type (...args: any) => infer R. Here, the interpreter infers the return value of the function and assigns it the parameter R. If this match is successful, then the type R is returned.

Check out my challenge solution playground

Intermediate/Advanced - Ghosts, Gods, and Demons oh my!

Stay on your guard.

Thank you challenge author, hint received! The solution for this challenge makes use of type guards. The challenge tasks us to create three user-defined type guards areGods, areDemons, and areEctoPlasmic.

My initial thought was to define three new types and the three functions:

type Gods = unknown
function areGods(ghosts: Ghosts[]): ghosts is Gods[] {}

type Demons = unknown
function areDemons(ghosts: Ghosts[]): ghosts is Demons[] {}

type EctoPlasmic = unknown
function areEctoPlasmic(ghosts: Ghosts[]): ghosts is EctoPlasmic[] {}

User-defined type guards can be unsafe if not used correctly. Whenever I create type guards, I try to think in absolutes. For example, the areGods guard must only return true if all of the ghosts are gods. This is true for other type guards as well. Since the ghosts argument is an array, we can use an ES6 method Array.prototype.every to validate that every item in the array satisfies a condition. In this case, the condition is the existance and truthiness of the god, demon, or ectoplasmic properties. Additionally, similar to the previous challenge, the three types can be defined using Extract and the Ghosts type union.

type Gods = Extract<Ghosts, { god: true }>
function areGods(ghosts: Ghosts[]): ghosts is Gods[] {
    return ghosts.every(ghost => "god" in ghost && ghost.god)
}

type Demons = Extract<Ghosts, { demons: true }>
function areDemons(ghosts: Ghosts[]): ghosts is Demons[] {
    return ghosts.every(ghost => "demon" in ghost && ghost.demon)
}

type EctoPlasmic = Extract<Ghosts, { ectoplasmic: true }>
function areEctoPlasmic(ghosts: Ghosts[]): ghosts is EctoPlasmic[] {
    return ghosts.every(ghost => "ectoplasmic" in ghost && ghost.ectoplasmic)
}

In my solution, not only do I check for the existance of the relative property in the item, but also if that property is truthy. Since this challenge only ever uses true the truthy check is unnecessary.

Check out my challenge solution playground

Challenge 3

Try it for yourself here

Beginner/Learner - House Types

Starting off, the challenge tasks us to define three types that correspond to the three arrays:

type TreatHouse = {
  location: string,
  result: 'treat',
  treat: {
    candy: string,
    baked?: string,
  },
}

type TrickHouse = {
  location: string,
  result: 'trick',
  trick: string,
}

type NoShowHouse = {
  location: string,
  result: 'no-show',
}

The challenge then leads us to simplifying these types. Two properties location and result exist in each type, and the result type is only three literal values. So extrapolate that first:

type House = {
  location: string,
  result: 'treat' | 'trick' | 'no-show',
}

Next, intersect the House type with the previous three types and remove the location property. I kept the result property specified to the corresponding literal.

type TreatHouse = House & {
  result: 'treat',
  treat: {
    candy: string,
    baked?: string,
  },
}

type TrickHouse = House & {
  result: 'trick',
  trick: string,
}

type NoShowHouse = House & {
  result: 'no-show'
}

In a real-code examples this pattern is very useful. If you're interested check out how I used it in Fastify's type definitions for the different http/s/2 server options objects.

Check out my challenge solution playground

Intermediate/Advanced - Trunk or Treat

The Trunk or Treat challenge starts with an array of spots. It then has a type TrunkOrTreatResults that has a lot of repetitive property definitions. The challenge tells us that it is already out of sync with the list of spots... so rather than more copy-pasting, what can we do to fix this type?

Consider what the type actually is representing, a map of the spots listed in trunkOrTreatSpots. And in TypeScript there is a concept called mapped types. Often times, mapped types are used to transform one interface into another such as:

interface Person {
    name: string;
    age: number;
}

interface PersonPartial {
    [P in keyof Person]?: Person[P];
}

But the trunkOrTreatSpots is an array constant, how would one write a mapped type expression for a list of strings? Think back to the day 1 challenge; use [number] to get the index type of the list. Then, using the typeof keyword we can transform that index type returned from trunkOrTreatSpots[number] into a union of the strings. And now with this we can pass it to a mapped type expression:

type TrunkOrTreatResults = {
    [k in typeof trunkOrTreatSpots[number]]: {
        done: boolean,
        who: string,
        loot: Record<string, any>
    }
}

The new TrunkOrTreatResults type now automatically includes any value that exists in trunkOrTreatSpots. One important detail though, trunkOrTreatSpots must be a constant (denoted by as const). Since all of this needs to be static, TypeScript needs to be certain trunkOrTreatsSpots won't change in order for your code to remain type safe.

Check out my challenge solution playground

Challenge 4

Try it for yourself here

Beginner/Learner - Lock it down with Readonly

TypeScript really has a utility type for everything. This challenge tasks us with making the two given types Rectory and Room readonly so that their properties cannot be modified. One solution is to go over each property and add the readonly property, but as we learned yesterday, we can map one type into another fairly easily. Earlier in my post I showed how one may transform an interface from being all required properties, to all optional ones. This challenge requires us to transform these types into readonly versions.

type ReadonlyRectory = {
    readonly [k in keyof Rectory]: Rectory[k]
}

type ReadonlyRoom = {
    readonly [k in keyof Room]: Room[k]
}

There is alot of similarities between these two types, maybe with the help of a generic we can create a utility:

type Readonly<T> = {
    readonly [k in keyof T]: T[k]
}

😉 And look at that, we just recreated the TypeScript utility Readonly. Check it out in lib.es5.d.ts.

For my final solution I wrapped the given types with Readonly:

type Rectory = Readonly<{
  rooms: Room[]
  noises: any[]
}>

type Room = Readonly<{
  name: string
  doors: number
  windows: number
  ghost?: any
}>

Check out my challenge solution playground

Intermediate/Advanced - Cutest Halloween Competition

Alright, this one definitelly took me a little bit. It was great to use the new template literal types, but I tried to rewrite the function using some functional programming aspects and didn't get very far. The start to my solution is by defining a new type ID

type ID = `${lowercase typeof breeds[number]}-${lowercase typeof costumes[number]}`

One thing I totally missed when I first read the 4.1 post was the lowercase operator (and the corresonding uppercase, capitalize, and uncapitalize ones too). The ID type uses the same array index type thing I've used in the previous challenges. It concatenates the two resulting unions of lowercase strings together into a template string seperated by a -.

Next, I modified the const winners declaration to use a type cast.

const winners = {} as Record<ID, ReturnType<typeof decideWinner>>

I did it this way since winners at the time of declaration isn't actually filled with values, but since the immediate following lines do that, its an okay type cast. Its for this exact reason that I tried rewriting the for loops so that the interpreter infers this record type automatically.

If you can figure that out let me know! Would love to check out your algorithm.

Finally, I added one more type cast to the id declaration to complete my solution.

const id = `${breed}-${costume}`.toLowerCase() as ID

Check out my challenge solution playground

Challenge 5

Try it for yourself here

Beginner/Learner - Generic trick or treating

I really appreciated this challenge, it is a great demonstration for generics. Generics go hand-in-hand with DRY programming. In this challenge there are four houses. They all share the same object structure so we can first extrapolate the type into a shared House type.

type House = {
  doorNumber: number;
  trickOrTreat(): string;
  restock(items: string): void;
}

Now each house can be rewritten as an extension of this base type

type FirstHouse = House & {
    doorNumber: 1,
    trickOrTreat(): "book" | "candy";
  restock(items: "book" | "candy"): void;
}
// ...

But we're still repeating another part of these types, the return value of trickOrTreat and the argument of restock. Using a generic, we can make the House type even better.

type House<Items> = {
  doorNumber: number;
  trickOrTreat(): Items;
  restock(items: Items): void;
}

Now, each house type can be defined as

type FirstHouse = House<"book" | "candy"> & {
  doorNumber: 1;
};
// ...

The base House type passes its generic down to both methods.

Check out my challenge solution playground

Intermediate/Advanced - Movie Night

// You're part of a team scheduling a movie night, but someone accidentally
// considered the movie "The Nightmare Before Christmas" to be a halloween
// movie, which it really isn't.

I'd like to point out that this statement is in fact false. The Nightmare Before Chistmas is one of the best Halloween and Christmas movies since it can be enjoyed equally on both holidays. 🎃

But anyways, back to the challenge at hand, and oh is this one a doozy. I got the first part fairly easily, but the second part (kids schedule) was something else.

My solution makes use of many things previously used in these challenges, mapped types, template literals, and utility types. To kick off my solution I extracted the type from moviesToShow and defined a type for the schedule operations.

type Movies = typeof moviesToShow
type scheduleOps = 'getVHSFor' | 'makePopcornFor' | 'play'

Then using template literal types I create a movieNames type which capitalizes the keys of Movies.

type movieNames = `${capitalize keyof Movies}`

I combined this type with the scheduleOps to create another ops type. This ops type is then used in the utility type Record to define the Schedule type.

type ops = `${scheduleOps}${movieNames}`
type Schedule = Record<ops, () => void>

Inside of makeScheduler, I casted schedule to the Schedule type. Inside of the for loop, I casted the string operations to their corresponding types.

const schedule = {} as Schedule

// inside of the for loop
const capitalName = movie.charAt(0).toUpperCase() + movie.slice(1) as movieNames;

schedule[`getVHSFor${capitalName}` as ops] = () => {}
schedule[`makePopcornFor${capitalName}` as ops] = () => {}
schedule[`play${capitalName}` as ops] = () => {}

These template literal types are quite impressive!

The second part of the challenge requires doing something similar to the first, but this time filtering out any movies that is not forKids. I immediately tried to use Extract again but its not the right utility here. Instead I built a mapped-type based on the Movies type I created at the beginning.

type kidsMovies = {
  [K in keyof Movies]: Movies[K] extends { forKids: true } ? K : never
}[keyof Movies]

There is a lot of funny business going on here; lets break it down. First, the mapped expression is K in keyof Movies. This creates a union of all of the movies in Movies and assigns it to K. The value is another expression, Movies[K] extends { forKids: true } ? K : never. This expression checks if the type returned from Movies[K] satisfies the forKids truthiness. If it does, rather than returning that whole type, we just return the key K (and otherwise never). At the end of this type definition we have one last trick, we index using keyof Movies. In short, this applies every key from the original type Movies to the one just generated using mapped types. When one of these keys returns a never value (the falsy case of the mapped type expression), the interpreter leaves that value out of the resulting union. Thus, we are left with a union type of just the kids movies.

With this new list of movies in hand, I recreate the ops and schedule types and use them in the makeKidScheduler the same way I did previously

type KidsOps = `${scheduleOps}${capitalize kidsMovies}`
type KidsSchedule = Record<KidsOps, () => void>

Check out my challenge solution playground

Conclusion

First, huge thank you to the TypeScript team for putting these challenges together. Even as a seasoned TS developer, I had a blast solving each beginner and intermediate challenge. I'd like to specially promote Gabrielle Crevecoeur's finale post which has a great solution for the final challenge. Make sure you follow her on dev.to and twitter.

I understand my solutions might not be the best solutions, but thats the magic of programming. There is never just one way to do things. Furthermore, I'd like to preface that your solutions (and the ones posted by the TS team) may be quite similar (if not the same) as mine. Everything I've presented here is my own work, and any similarities are purely coincidental.

I encourage you to try them out on your own and discuss your solutions with me in the comments below.

Happy Halloween! 🎃👻🕷

Building Git with Node.js & TypeScript - Part 2

Ethan Arrowood — Sat, 23 May 2020 19:19:27 +0000

This post covers chapter 4, the concept of history between commits. Follow along with the code available here.

Read the previous posts here:

Reminders:

code highlight text references actual pieces of code such as commands, properties, variables, etc.
boldface text references file and directory names.
italic text references higher level data structures such as commit, blob, database, etc.
Most classes will referred to using italics, but may also appear as code highlights.
Imports are omitted from code examples. Assume all imports refer to other local files or Node.js core modules.
All code blocks have their respective file name commented at the top of the block.

Overview

Previously, I implemented the init and commit commands. Together they create a .git directory with a database that can track blobs of data through commits organized with trees. Additionally, it tracks the commit author, message, and timestamp. In the previous article I even demonstrated how you can get my implementation up and running! In this post I will be introducing two new structures: refs and lockfile. I'll be making some changes to the Commit and Database classes, and the commit command in jit.ts.

While working on this section I made some quick fixes to the existing code:

calls to database.store are now awaited
the slice call was removed from the database.generateTempName method as it was not necessary

History and Refs

If you have used git before, you'll already know that commits are connected in a chain-like structure. To create this chain, commits track their parent through a reference. There are more complex reference relationships which will come later in this series, but for now we are focussing on a flat, commit-to-commit chain.

Create a Refs class. We will come pack and implement the updateHead method later. The readHead method first checks if the HEAD file exists and is readable; if it is not then it returns null (this detail is important). Then, we return the contents of the HEAD file -- the latest commit.

// refs.ts
export default class Refs {
    public pathname: string

    private headPath: string

    constructor(pathname: string) {
        this.pathname = pathname
        this.headPath = path.join(pathname, 'HEAD')
    }

    public async updateHead(oid: string) {}

    public async readHead() {
        try {
            await fs.promises.access(this.headPath, fs.constants.F_OK | fs.constants.R_OK)
            return (await fs.promises.readFile(this.headPath, 'utf8')).trim()
        } catch (err) {
            return null
        }
    }
}

In jit.ts create a Refs instance alongside the Workspace and Database. And then get the latest commit using the readHead method (I do this after storing the tree in the database). Pass the parent commit object id to the new Commit constructor, and after writing the new commit to the database, update the HEAD file with refs.updateHead.

// jits.ts

// ...
const workspace = new Workspace(rootPath)
const database = new Database(dbPath)
const refs = new Refs(gitPath)
// ...
const parent = await refs.readHead()
// ...
const commit = new Commit(parent, tree.oid, author, message)
await database.store(commit)
await refs.updateHead(commit.oid)

Jump back over to refs.ts to start implementing the updateHead method. This method makes use of a new structure, lockfile.

// refs.ts

public async updateHead(oid: string) {
    const lockfile = new Lockfile(this.headPath)

    if (!(await lockfile.holdForUpdate())) {
        throw new LockDenied(`Could not acquire lock on file: ${this.headPath}`)
    }

    await lockfile.write(oid)
    await lockfile.write("\n")
    await lockfile.commit()
}

Lockfile

A lockfile, in this context, is a mechanism to protect our program from attempting to operate on the same file at the same time. If two operations were attempting to write to the HEAD file at the same time, the operations could potentially result in unexpected behavior or even a crash. By using a locking mechanism, the application can be certain that it won't accidentally be operating on a file that something else is operating on. Thus, introducing the Lockfile class.

The class contains three private properties, the most important one being the lock file handle. This file handle will not directly refer to the HEAD file, but a HEAD.lock one instead.

The holdForUpdate method first checks if the lock is null. If it is, it knows that nothing is currently being locked, so then it attempts to open HEAD.lock. The file flag constants validate a couple of conditions:

O_RDWR checks if the program has read/write access to the file
O_CREAT will create the file if it doesn't already exist
O_EXCL indicates that opening a file should fail if the O_CREAT flag is set and the file already exists

The method returns true after successfully creating the file handle. Otherwise, it handles a set of error conditions:

If the file already exists, return false.
If the parent directory does not exist, throw a custom MissingParent error
If the application does not have the right access permissions, throw a custom NoPermission error
And finally, if we don't recognize the error, throw it so we can debug and later improve the code.

The reason this method returns boolean values in certain conditions is how it will be used. Referring back to its use within refs.ts, you can see that if the lockfile update returns false, we throw an error that we couldn't lock the HEAD file.

The write method checks for the existence of the lock file and then writes the data to it.

The commit method also checks for the lock file, and then closes the file handle and renames it to the existing, non .lock path. After that it also resets the lock property to null.

// lockfile.ts

class MissingParent extends Error {}
class NoPermission extends Error {}
class StaleLock extends Error {}

export default class Lockfile {
    private filePath: string
    private lockPath: string
    private lock: fs.promises.FileHandle | null

    constructor(path: string) {
        this.filePath = path
        this.lockPath = `${path}.lock`
        this.lock = null
    }

    public async holdForUpdate () {
        try {
            if (this.lock === null) {
                const flags = fs.constants.O_RDWR | fs.constants.O_CREAT | fs.constants.O_EXCL
                this.lock = await fs.promises.open(this.lockPath, flags)
            }
            return true
        } catch (err) {
            switch (err.code) {
                case 'EEXIST':
                    return false
                case 'ENOENT':
                    throw new MissingParent(err.message)
                case 'EACCES':
                    throw new NoPermission(err.message)
                default:
                    throw err
            }
        }
    }

    public async write(data: string) {
        if (this.lock === null) {
            throw new StaleLock(`Not holding lock on file: ${this.lockPath}`)
        }
        await this.lock.write(data)
    }

    public async commit() {
        if (this.lock === null) {
            throw new StaleLock(`Not holding lock on file: ${this.lockPath}`)
        }
        await this.lock.close()
        await fs.promises.rename(this.lockPath, this.filePath)
        this.lock = null
    }
}

There is one major change I'd like to make in this class in the future; removing the use of null. I prefer to rely on undefined as null has some strange behaviors in JavaScript. This is not a hard rule for JavaScript apps, but it is my preference. For now though, using null is okay as it better aligns with the Ruby implementation this is based off of.

If you want to know more about why that is my preference, read this issue by @sindresorhus.

Now that we've completed both refs and lockfile, all thats left are some short changes to commit and database.

Commit Updates

Recall that in jit.ts we are now passing the parent commit as the first argument to the Commit class constructor. We must update the Commit method to handle these changes. The generateData method must be updated too. The parent line must only be added if it is not null. This piece maintains the current functionality for the root-commit.

// commit.ts
export default class Commit extends Entity {
    public parent: string | null
    // ...

    constructor(parent: string | null, treeOid: string, author: Author, message: string) {
        super('commit', Commit.generateData(parent, treeOid, author, message))
        this.parent = parent
        // ...
    }

    private static generateData(parent: string | null, treeOid: string, author: Author, message: string) {
        const lines = []

        lines.push(`tree ${treeOid}`)
        if (parent !== null) lines.push(`parent ${parent}`)
        lines.push(`author ${author.toString()}`)
        lines.push(`committer ${author.toString()}`)
        lines.push("")
        lines.push(message)

        const data = lines.join("\n")

        return Buffer.from(data)
    }
}

Database updates

In addition to the history feature, we can make a small edit to the database writeObject method that will prevent it from storing objects that already exist. I've added a fileExists method to simplify the logic, this can probably be written better so if you have any ideas comment them below and we can discuss them together.

// database.ts
export default class Database {
    // ...
    private async writeObject(oid: string, content: Buffer) {
        const objectPath = path.join(this.pathname, oid.substring(0, 2), oid.substring(2))
        if (await this.fileExists(objectPath)) return
        // ...
    }
    // ...
    private async fileExists(path: string) {
        try {
            await fs.promises.access(path, fs.constants.F_OK)
            return true
        } catch (err) {
            return false
        }
    }
}

Before finishing, there is one last change in jit.ts at the end of the commit command. This change improves the CLI output when creating a root vs non-root commit.

// jit.ts
const isRoot = parent === null ? "(root-commit) " : ""
console.log(`[${isRoot}${commit.oid}] ${message.substring(0, message.indexOf("\n"))}`)

Try it out

Clone the repo:

git clone git@github.com:Ethan-Arrowood/building-git-with-nodejs-and-typescript.git

Fetch and checkout the part-2 branch

git fetch origin part-2
git checkout part-2

Install dependencies, build src, and link the executable

npm i
npm run build
npm link

Set the current working directory to src and and run the commit command with the initial commit message

cd src
jit init
export GIT_AUTHOR_NAME="name" GIT_AUTHOR_EMAIL="email" && cat ../COMMIT_EDITMSG | jit commit

Write a second commit

commit ../COMMIT_EDITMSG2 | jit commit

To see if everything worked correctly use git log

git log --oneline

It should output two commits with their respective messages, mine looked like this:

a6cfc02 (HEAD) Use HEAD to set the parent of the new commit
fd5602b Initial revision of "jit", the information manager from Boston

Conclusion

That is it for the initial history feature. Thank you for reading! I encourage you to ask questions and continue the discussion in the comments; I'll do my best to respond to everyone! If you enjoyed make sure to follow me on Twitter (@ArrowoodTech). And don't forget to check out the book, Building Git.

Building Git with Node.js and TypeScript - Part 1

Ethan Arrowood — Fri, 15 May 2020 19:16:27 +0000

Read the introduction to this series here: Building Git with Node.js and TypeScript - Part 0.

In this post, I'll be sharing my work from chapter 3 section 2, implementing the commit command. Follow along with the code available here.

Throughout this article I try my best to highlight certain terms using code highlight, boldface, and italic. Any code highlight text will be referencing actual pieces of code such as commands, properties, variables, etc. Any boldface text refers to file and directory names. And any italic text references higher level data structures. Most classes will referred to using italics, but may sometimes appear as code highlights when referring to a type assignment. Keep in mind that some terms may be italicized before they are defined.

Imports are omitted from code examples. For this article, assume all imports refer to other local files or Node.js core modules. Furthermore, all code blocks will have their respective file name commented at the top of the block.

Overview

In the previous post I implemented the init command, which created a .git directory in current working directory and initialized two inner directories objects and refs. This section covers a simplified commit command. It adds all files in the current working directory to the git database as blobs, creates a tree with all of the entries, and then finally creates a commit with a message. Additionally, it tracks the commit author from data stored in environment variables, and the commit message is read from stdin rather than passing it in as a command line argument.

Adding the commit command

Inside jit.ts add a new 'commit' case to the switch statement. Then derive the database path to the objects directory created by the init command.

// jit.ts
async function jit() {
    const command = process.argv[2]

    switch (command) {
        case 'init': {
            // ...
        }
        case 'commit': {
            const rootPath = process.cwd() // get the current working directory
            const gitPath = path.join(rootPath, '.git')
            const dbPath = path.join(gitPath, 'objects')
        }
    }
}

With these paths, create Workspace and Database class instances.

// jit.ts
// inside of the `case 'commit': { }` block
const workspace = new Workspace(rootPath)
const database = new Database(dbPath)

Workspace

The workspace class contains one private property, ignore, one public property, pathname, and two public methods, listFiles and readFile. The ignore property is a list of things to ignore when scanning the current working directory. This emulates the commonly used .gitignore file. The pathname property is the absolute path to the current working directory and any files within will be included in the list. Thus, the listFiles method returns all files in the directory resolved from pathname, and filters out anything in the ignore list. Currently, this method is not recursive and will not list files contained within directories. Finally, the readFile method takes a file path, joins it with the absolute path pathname, and then reads its contents as a buffer.

It is intentional that the readFile method returns a buffer and not an encoded string version of the file contents. This method will be used when storing the entities in the Git database and we must store the binary representation of the data; not the encoded version.

// workspace.ts
import fs from 'fs'
import path from 'path'

export default class Workspace {
    private ignore = ['.', '..', '.git']

    public pathname: string

    constructor (pathname: string) {
        this.pathname = pathname
    }

    public async listFiles () {
        const dirFiles = await fs.promises.readdir(this.pathname)
        return dirFiles.filter(x => this.ignore.indexOf(x) === -1)
    }

    public async readFile (filePath: string) {
        return await fs.promises.readFile(path.join(this.pathname, filePath))
    }
}

Database

The database class is verbose, but is rightfully so as it is the basis for the entire application. It has a single public property pathname, one public method store, and two private methods writeObject and generateTempName. Start by defining the property, constructor, and methods with arguments.

// database.ts
export default class Database {
    public pathname: string

    constructor (pathname: string) {
        this.pathname = pathname
    }

    public async store(obj: Entity) {}

    private async writeObject(oid: string, content: Buffer) {}

    private generateTempName() {}
}

Starting with the store method there is already something new, the Entity class. Before continuing with the store method, let's define this class as it has some important details for the rest of the implementation.

Entity

This class is the crux for all items storable by the database. Anything that will be stored in the database (blobs, commits, trees) will need to extend from this class. It has one private property data that is a buffer of the contents of the entity, and two public properties type and oid (object id). While data and type are set by the constructor, the oid property is generated by a private method setOid. This method uses the type and data parameters, and creates a hash of a custom binary string. The code below contains comments detailing each step of this method. Lastly, the class overrides the toString method to return the underlying data buffer; this is the not the best practice as toString should generally return a string, but buffers can be implicitly turned into strings with their own toString method so this is (sorta) okay.

What is a "binary string"? This refers to a Node.js buffer, but I distinctly used this wording because I need to highlight something special about the implementation. As previously mentioned, the database needs to store the binary representation of the data and NOT the encoded string version; thus, buffers are used throughout the code samples. This detail is covered in the Building Git book, and while working on this solution, I received a wonderfully detailed stackoverflow answer on the difference between raw binary strings and encoded strings in JavaScript. If you're interested in learning more please utilize those resources.

// entity.ts

export default class Entity {
    private data: Buffer

    public type: string
    public oid: string

    constructor(type: string, data: Buffer) {
        this.type = type
        this.data = data
        this.oid = this.setOid()
    }

    private setOid () {
        // define the binary string
        const str = this.data
        // create a buffer from the type, binary string length, and a null byte
        const header = Buffer.from(`${this.type} ${str.length}\0`)
        // create the hash content by concatenating the header and the binary string
        const content = Buffer.concat([header, str], header.length + str.length)
        // create a hash generator using the 'sha1' algorithm
        const shasum = crypto.createHash('sha1')
        // update the hash generator with the content and use a hexadecimal digest to create the object id
        const oid = shasum.update(content).digest('hex')

        return oid
    }

    public toString () {
        return this.data
    }
}

Back to Database

Continuing with the database store implementation, it needs to recreate the content that was used to generate the oid property, and use that plus the oid to write the object to the database itself. Yes, the content is being generated twice (once in the Entity class and once here); I purposely did not optimize this as I didn't want to stray too far from the Ruby code. It is noted and may change in future implementations.

// database.ts

class Database {
    // ...
    async store (obj: Entity) {
        const str = obj.toString() // remember this returns the data buffer
        const header = Buffer.from(`${obj.type} ${str.length}\0`)
        const content = Buffer.concat([header, str], header.length + str.length)
        await this.writeObject(obj.oid, content)
    }
}

Next is the writeObject and generateTempName methods. Derived from the store method, writeObject has two arguments: oid and content. The binary string content will be written to a file path derived from oid. In a Git database, the objects are stored in subdirectories using the first two characters of their oid; thus, the substrings in the objectPath variable. The internal getFileDescriptor method is used to try to safely generate these directories on the fly. Unfortunately, it is not perfect and can sometimes still throw an error due to how the store method is called from jit.ts (more on this soon). Again, this is purposefully not fixed or optimized, but it is noted for future improvements. Finally, the end of the method. Another trick this method uses to prevent errors is by generating temporary names for the files, and then renaming them after. The content of the files is compressed using Zlib deflate at the Z_BEST_SPEED level.

// database.ts

class Database {
    // ...
    private async writeObject(oid: string, content: Buffer) {
        const objectPath = path.join(this.pathname, oid.substring(0, 2), oid.substring(2))
        const dirName = path.dirname(objectPath)
        const tempPath = path.join(dirName, this.generateTempName())

        const flags = fs.constants.O_RDWR | fs.constants.O_CREAT | fs.constants.O_EXCL

        const getFileDescriptor = async () => {
            try {
                return await fs.promises.open(tempPath, flags)
            } catch (err) {
                if (err.code === 'ENOENT') {
                    await fs.promises.mkdir(dirName)
                    return await fs.promises.open(tempPath, flags)
                } else if (err.code === 'EEXIST') {
                    return await fs.promises.open(tempPath, flags)
                } else {
                    throw err
                }
            }
        }

        const file = await getFileDescriptor()

        const deflate: any = util.promisify(zlib.deflate)
        const compressed = await deflate(content, { level: zlib.constants.Z_BEST_SPEED })

        await file.write(compressed)
        await file.close()

        await fs.promises.rename(tempPath, objectPath)
    }

    private generateTempName () {
        // hex ensures we only get characters 0-9 and a-f
        return `tmp_obj_${crypto.randomBytes(8).toString('hex').slice(0, 8)}`
    }
}

Back to the commit command

Continuing the commit block now that workspace and database are implemented, we list the files in the workspace, then iterating over the list, create blobs and store them in the database. Additionally, each object will be tracked as an entry which is used in the tree structure. Notice how both the blob and tree are stored in the database through the same store method. These objects are similar enough that they can both be based on the Entity class defined above.

// jit.ts
// inside of the `case 'commit': { }` block
const workspaceFiles = await workspace.listFiles()

const entries = await Promise.all(workspaceFiles.map(async path => {
    const data = await workspace.readFile(path)
    const blob = new Blob(data)

    database.store(blob)
    return new Entry(path, blob.oid)
}))

const tree = new Tree(entries)
database.store(tree)

Blob

Blobs are one of the simplest data structures in this application. They extend from Entity and set their type as 'blob'.

// blob.ts

export default class Blob extends Entity {
    constructor(data: Buffer) {
        super('blob', data)
    }
}

Entry

Another simple data structure, entry, has two public properties name and oid and both are of type string. This structure could be represented as just an object literal, but defining it as a class allows for better extensibility later on if it is needed.

// entry.ts

export default class Entry {
    public oid: string
    public name: string

    constructor (name: string, oid: string) {
        this.name = name
        this.oid = oid
    }
}

Tree

The Tree class is a bit more complicated compared to the Blob class, but it still extends from the Entity class. In the constructor, the class calls a private, static method generateData to create the data buffer passed to the parent Entity constructor. The Tree class also keeps a local, public copy of the entries list.

// tree.ts

export default class Tree extends Entity {
    public entries: Entry[]

    constructor(entries: Entry[]) {
        super('tree', Tree.generateData(entries, '100644'))
        this.entries = entries
    }

    private static generateData (input: Entry[], mode: string) {
        let totalLength = 0 // this is necessary for the final concatenation
        const entries = input
            .sort((a, b) => a.name.localeCompare(b.name)) // sort by file name
            .map(entry => {
                // encode as normal string and append a null byte
                let b1 = Buffer.from(`${mode} ${entry.name}\0`) 
                // encodes a string as hex. for example '00ce' is a string of 4 bytes; 
                // this is encoded to Buffer<00, ce>, a buffer of 2 hex bytes
                let b2 = Buffer.from(entry.oid, 'hex')
                totalLength += b1.length + b2.length
                return Buffer.concat([b1, b2], b1.length + b2.length)
            })
        // concat all of the entries into one buffer and return
        return Buffer.concat(entries, totalLength)
    }
}

The generateData function is one of my personal favorites. I think the best way to understand what it does is to first look at what it outputs. This function creates the data for the tree entry in the database. Unlike the blobs, the tree best resembles a list of all the blobs contained in the commit. Running git ls-tree <tree-hash> outputs this list:

Keep in mind this output is only possible with the rest of this sections code (i.e. commit entity), and that my commit hashes will be different from yours if you were to reproduce this yourself.

$ git ls-tree e42fafc6ea09f9b9633adc97218288b2861dd03f

100644 blob 1d15619c8d23447eac2924b07896b3be9530a42e    author.ts
100644 blob c8c1a93bf381f385bb70bcb95359ff056ee4a273    blob.ts
100644 blob fad23e45b228db3f33501691410541819e08a1e6    commit.ts
100644 blob 0355a9b19376a39700c3f44be73cb84d2398a219    database.ts
100644 blob c9a547e93c3101b3607f58469db26882645a120d    entity.ts
100644 blob c061d02df8007226fb6b4092a40f44678f533599    entry.ts
100644 blob 7a9f17b4ee76e13b062676fa74cb509aa423ee88    jit.ts
100644 blob 1adec84945be1564c70e9cdaf5b6a9c1d9326bd0    readStdin.ts
100644 blob aeafb5efdcd5e64897385341b92a33590517adae    timestamp.ts
100644 blob 377c1945ebb9aaf9f991656b7c232f7b02a55e78    tree.ts
100644 blob a331e9df15d9546f9d7dd1f28322bf1e24c2db00    workspace.ts

The ls-tree command derives this information from the contents of the tree entry itself. The entry is a hard to read as a human, but by using an inflate command and the hexdump tool we can get an output we can make sense of:

$ alias inflate="node -e 'process.stdin.pipe(zlib.createInflate()).pipe(process.stdout)'"
$ cat .git/objects/e4/2fafc6ea09f9b9633adc97218288b2861dd03f | inflate | hexdump -C

00000000  74 72 65 65 20 34 31 30  00 31 30 30 36 34 34 20  |tree 410.100644 |
00000010  61 75 74 68 6f 72 2e 74  73 00 1d 15 61 9c 8d 23  |author.ts...a..#|
00000020  44 7e ac 29 24 b0 78 96  b3 be 95 30 a4 2e 31 30  |D~.)$.x....0..10|
00000030  30 36 34 34 20 62 6c 6f  62 2e 74 73 00 c8 c1 a9  |0644 blob.ts....|
00000040  3b f3 81 f3 85 bb 70 bc  b9 53 59 ff 05 6e e4 a2  |;.....p..SY..n..|
00000050  73 31 30 30 36 34 34 20  63 6f 6d 6d 69 74 2e 74  |s100644 commit.t|
00000060  73 00 fa d2 3e 45 b2 28  db 3f 33 50 16 91 41 05  |s...>E.(.?3P..A.|
00000070  41 81 9e 08 a1 e6 31 30  30 36 34 34 20 64 61 74  |A.....100644 dat|
00000080  61 62 61 73 65 2e 74 73  00 03 55 a9 b1 93 76 a3  |abase.ts..U...v.|
00000090  97 00 c3 f4 4b e7 3c b8  4d 23 98 a2 19 31 30 30  |....K.<.M#...100|
000000a0  36 34 34 20 65 6e 74 69  74 79 2e 74 73 00 c9 a5  |644 entity.ts...|
000000b0  47 e9 3c 31 01 b3 60 7f  58 46 9d b2 68 82 64 5a  |G.<1..`.XF..h.dZ|
000000c0  12 0d 31 30 30 36 34 34  20 65 6e 74 72 79 2e 74  |..100644 entry.t|
000000d0  73 00 c0 61 d0 2d f8 00  72 26 fb 6b 40 92 a4 0f  |s..a.-..r&.k@...|
000000e0  44 67 8f 53 35 99 31 30  30 36 34 34 20 6a 69 74  |Dg.S5.100644 jit|
000000f0  2e 74 73 00 7a 9f 17 b4  ee 76 e1 3b 06 26 76 fa  |.ts.z....v.;.&v.|
00000100  74 cb 50 9a a4 23 ee 88  31 30 30 36 34 34 20 72  |t.P..#..100644 r|
00000110  65 61 64 53 74 64 69 6e  2e 74 73 00 1a de c8 49  |eadStdin.ts....I|
00000120  45 be 15 64 c7 0e 9c da  f5 b6 a9 c1 d9 32 6b d0  |E..d.........2k.|
00000130  31 30 30 36 34 34 20 74  69 6d 65 73 74 61 6d 70  |100644 timestamp|
00000140  2e 74 73 00 ae af b5 ef  dc d5 e6 48 97 38 53 41  |.ts........H.8SA|
00000150  b9 2a 33 59 05 17 ad ae  31 30 30 36 34 34 20 74  |.*3Y....100644 t|
00000160  72 65 65 2e 74 73 00 37  7c 19 45 eb b9 aa f9 f9  |ree.ts.7|.E.....|
00000170  91 65 6b 7c 23 2f 7b 02  a5 5e 78 31 30 30 36 34  |.ek|#/{..^x10064|
00000180  34 20 77 6f 72 6b 73 70  61 63 65 2e 74 73 00 a3  |4 workspace.ts..|
00000190  31 e9 df 15 d9 54 6f 9d  7d d1 f2 83 22 bf 1e 24  |1....To.}..."..$|
000001a0  c2 db 00                                          |...|
000001a3

Look closely at the table on the right of the hexdump, the mode "100644" is repeated as well as all of the file names in the tree. Following each file name is seemingly a bunch of gibberish. However, look back at the output of ls-tree and note the oid of the first entry author.ts:

1d15619c8d23447eac2924b07896b3be9530a42e

Now, take a look at the first couple lines of the hexdump, these correspond to the author.ts entry. What do you see (i've highlighted it below)?

00000000                                                    |tree 410.100644 |
00000010                                 1d 15 61 9c 8d 23  |author.ts...a..#|
00000020  44 7e ac 29 24 b0 78 96  b3 be 95 30 a4 2e        |D~.)$.x....0..10|

It is the author.ts oid in literal hex bytes! Thus, you can directly see how the generateData function transforms entries for the tree content.

Back to the commit command

Now that blob, entry, and tree have all been defined we can return to the commit code block and finally create a commit! First, read the name and email from environment variables. There are multiple ways to set these, one of the easiest is to set them in the shell profile. Then create an author instance with the name, email, and the current time. Next, read the commit message from process.stdin (the readStdin section will cover this in more detail). Create a new commit from the tree oid, the author, and the message and then write it to the database. Finally, write the commit oid to the HEAD file and the commit function is done!

// jit.ts
// inside of the `case 'commit': { }` block
const name = process.env['GIT_AUTHOR_NAME'] || ''
const email = process.env['GIT_AUTHOR_EMAIL'] || ''
const author = new Author(name, email, new Date())
const message = await readStdin()
const commit = new Commit(tree.oid, author, message)
database.store(commit)

const fd = await fs.promises.open(path.join(gitPath, 'HEAD'), fs.constants.O_WRONLY | fs.constants.O_CREAT)
await fd.write(`${commit.oid}\n`)
await fd.close()

console.log(`[(root-commit) ${commit.oid}] ${message.substring(0, message.indexOf("\n"))}`)

Author

Much like Blob and Entry, the Author class implements a unique toString method based on its properties.

// author.ts

export default class Author {
    public name: string
    public email: string
    public time: Date

    constructor(name: string, email: string, time: Date) {
        this.name = name
        this.email = email
        this.time = time
    }

    toString() {
        return `${this.name} <${this.email}> ${timestamp(this.time)}`
    }
}

This class makes use of custom timestamp method that derives the timezone offset string from a Date object:

// timestamp.ts

export default function timestamp (date: Date) {
    const seconds = Math.round(date.getTime() / 1000)
    const timezoneOffsetNum = date.getTimezoneOffset()
    const timezoneOffsetStr = timezoneOffsetNum >= 0
        ? `+${timezoneOffsetNum.toString().padStart(4, '0')}`
        : `-${(timezoneOffsetNum * -1).toString().padStart(4, '0')}`
    return `${seconds} ${timezoneOffsetStr}`
}

readStdin

The readStdin method is another utility method that helps simplify the process of reading data from process.stdin. Using async iterators, it collects chunks of the readable stream and then returns the complete string in a promise.

// readStdin.ts

export default async function () {
    let res = ''
    for await (const chunk of process.stdin) {
        res += chunk
    }
    return res
}

Commit

Finally, the last piece of the implementation is the Commit class. It extends from Entity, and thus needs to pass a type as well as data to the parent constructor. The generateData function for the Commit class joins multiple strings using the newline character and then transforms that into a buffer for the Entity data.

// commit.ts

export default class Commit extends Entity {
    public treeOid: string
    public author: Author
    public message: string

    constructor(treeOid: string, author: Author, message: string) {
        super('commit', Commit.generateData(treeOid, author, message))
        this.treeOid = treeOid
        this.author = author
        this.message = message
    }

    private static generateData(treeOid: string, author: Author, message: string) {
        const lines = [
            `tree ${treeOid}`,
            `author ${author.toString()}`,
            `committer ${author.toString()}`,
            "",
            message
        ].join("\n")

        return Buffer.from(lines)
    }
}

Running the commit command

I've posted all of this code as gist so you can clone and run it locally faster. Check it out here: Building Git with Node.js and TypeScript

Clone the sample repo:

git clone git@github.com:Ethan-Arrowood/building-git-with-nodejs-and-typescript.git

Fetch and checkout the part-1 branch

git fetch origin part-1
git checkout part-1

Install dependencies, build src, and link the executable

npm i
npm run build
npm link

Set the current working diretory to src and and run the commands

cd src
jit init
export GIT_AUTHOR_NAME="name" GIT_AUTHOR_EMAIL="email" && cat ../COMMIT_EDITMSG | jit commit

Now you should have a .git directory in the src directory that contains all of the blobs, the tree, and the commit.

To inspect the contents of the local .git directory, start by retrieving the commit hash from HEAD

cat .git/HEAD

Create an inflate command (I've added mine to my bash profile)

alias inflate="node -e 'process.stdin.pipe(zlib.createInflate()).pipe(process.stdout)'"`

Then inflate the contents of the root commit

cat .git/objects/<first two characters of HEAD>/<remaining characters of HEAD> | inflate

If everything works as expected the output should be:

commit 705tree <tree-oid>
author name <email> 1589553119 +0240
committer name <email> 1589553119 +0240

Initial revision of "jit", the information manager from Boston

This commit records a minimal set of functionality necessary for the code to store itself as a valid Git commit. This includes writing the following object types to the database:

- Blobs of ASCII text
- Trees containing a flat list of regular files
- Commits that contain a tree pointer, author info and message

These objects are written to `.git/objects`, compressed using zlib.

At this stage, there is no index and no `add` command; the `commit` command simply writes everything in the working tree to the database and commits it.

With the <tree-oid> you can then use git ls-tree to see the contents of the tree entry:

git ls-tree <tree-oid>

Conclusion

That is all for now! I intend to make following sections shorter so these posts are easier to read. I encourage you to ask questions and continue the discussion in the comments; I'll do my best to respond to everyone! If you enjoyed make sure to follow me on Twitter (@ArrowoodTech). And don't forget to check out the book, Building Git.

Happy coding 🚀

Building Git with Node.js and TypeScript - Part 0

Ethan Arrowood — Wed, 06 May 2020 13:30:10 +0000

Introduction

I recently found this amazing eBook, Building Git, by James Coglan.

Building Git is a deep dive into the internals of the Git version control system. By rebuilding it in a high-level programming language, we explore the computer science behind this widely used tool. In the process, we gain a deeper understanding of Git itself as well as covering a wide array of broadly applicable programming topics

I'm not a Ruby developer, so I'll be implementing the Git clone, jit, using Node.js and Typescript. This blog series will track that process, and I'll do my best to not spoil the book either. It is full of some incredible details not just about Git, but also about file systems, operating systems, and more!

In this first post I'm going to share my local development environment and my code from chapter 3 section 1. There won't be a remote repository for my implementation at first, but later on, I'll be sure to upload and share that too.

Also, none of this series is sponsored and the links I provide to the book are not referrals. I'm writing this series because I like to learn in public, and sharing links to the book because I want to support the author.

Development Environment

For my local environment I'm using Visual Studio Code editor on macOS Catalina. I'll be using the latest versions of Node.js and TypeScript. At the time of writing this post, I'm using Node.js v14.2 and TypeScript v3.8; I'll be keeping these up to date in the future as well.

I started my project with:

mkdir jit-ts
cd jit-ts
npm init -y
npm i -D typescript @types/node

And then added the following sections to my package.json

{
    "main": "./lib/jit.js",
    "types": "./lib/jit.d.ts",
    "bin": {
        "jit": "./lib/jit.js"
    },
    "scripts": {
        "build": "tsc -w -p ."
    }
}

Additionally, I prefer to use the built in VSCode debugger to step through and test my code so I have these two debug files in the .vscode directory:

launch.json

{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "node",
            "request": "launch",
            "name": "jit-ts init",
            "program": "${workspaceFolder}/src/jit.ts",
            "preLaunchTask": "tsc: build - jit-ts/tsconfig.json",
            "postDebugTask": "jit cleanup",
            "outFiles": ["${workspaceFolder}/lib/**/*.js"],
            "args": ["init"]
        }
    ]
}

tasks.json

{
    "version": "2.0.0",
    "tasks": [
        {
            "type": "typescript",
            "tsconfig": "tsconfig.json",
            "problemMatcher": [
                "$tsc"
            ],
            "group": "build",
            "label": "tsc: build - jit-ts/tsconfig.json"
        },
        {
            "label": "jit cleanup",
            "type": "shell",
            "command": "rm -rf ${workspaceFolder}/.git"
        }
    ]
}

These debug files will expand and change as the implementation grows, but these work good enough for the first section.

Lastly, I created a typescript configuration and the source file:

mkdir src
touch src/jit.ts
npx typescript --init

tsconfig.json

{
    "compilerOptions": {
        "target": "es5",
        "module": "commonjs",
        "declaration": true,
        "sourceMap": true,
        "outDir": "lib",
        "rootDir": "src",
        "strict": true,
        "moduleResolution": "node",
        "esModuleInterop": true,
        "forceConsistentCasingInFileNames": true
    }
}

Init Command

#!/usr/bin/env node

import fs from 'fs'
import { resolve, join } from 'path'

function jit() {
    const command = process.argv[2]

    switch (command) {
        case 'init': {
            const path = process.argv[3] || process.cwd()
            const rootPath = resolve(path)
            const gitPath = join(rootPath, ".git")

            ;["objects", "refs"].forEach(dir => {
                fs.mkdirSync(
                    join(gitPath, dir),
                    { "recursive": true }
                )
            })

            console.log(`initialized empty jit repo in ${gitPath}`)
            break
        }
        default: {
            console.log(`jit: '${command}' is not a jit command`)
        }
    }
}

jit()

Chapter 3 section 1 is all about the init command. The code is simple to start and lacks input validation. Utilizing process.argv to access the command line arguments, the function starts by setting the command. Inside the 'init' block, the code retrieves the input path from the command line arguments; this value also defaults to the user's current working directory. Next, it resolves the input path from a relative path to an absolute path and prepends .git to create the path for the git root directory.

Wondering what the difference between relative and absolute paths are? The book does a fantastic job explaining the difference and why it is necessary to use an absolute path, so I encourage you to purchase a copy of Building Git for yourself and read the explanation on page 28 😉.

After path resolution, the function iterates over a shortlist of strings and generates directories using the fs.mkdirSync command. It uses the recursive property which was introduced in Node.js v10.

Thats all the function does for now, and thus concluding the first bit of implementing Git using Node.js and TypeScript.

Conclusion

I'm very excited about this series. Learning the intricacies of Git and building a clone in my favorite language will be a great learning experience. If you have enjoyed this first post and are interested in joining me on this learning journey, make sure to follow me on Twitter (@ArrowoodTech) and here on Dev. And don't forget to check out the book, Building Git.

Until next time, happy coding!

Overriding Reach UI Styles using TailwindCSS in Parcel

Ethan Arrowood — Sun, 26 Apr 2020 20:34:34 +0000

In my React app I'm using:

In order to use tailwindcss with Parcel I'm using PostCSS. Configuration requires 3 steps:

Create tailwind.config.js and postcss.config.js files

// postcss.config.js
module.exports = {
    plugins: [
        require('tailwindcss')('./tailwind.config.js')
    ]
}

// tailwind.config.js
// This is only necessary if you want to modify tailwindcss
module.exports = {}

Create an app.pcss file

@tailwind base;
@tailwind components;
@tailwind utilities;

Link the PostCSS file to the index.html file

<head>
    <link rel="stylesheet" href="app.pcss">
</head>

In the app itself I'm using the Reach UI Tooltip element:

// import the component and its default styles
import Tooltip from '@reach/tooltip'
import "@reach/tooltip/styles.css"

return (
    <Tooltip
        label='Edit'
    >
        <button className='py-1 px-3 rounded bg-transparent border border-blue-500'>
            <span aria-hidden>✏️</span>
        </button>
    </Tooltip>
)

By default the tooltip looks like this:

To override the default styles of the tooltip element, I add a new block to the app.pcss file targetting the [data-reach-tooltip] selector and using the !important rule at the end of the @apply line.

[data-reach-tooltip] {
    @apply bg-gray-800 text-white py-2 px-4 border-none !important;
}

Now the tooltip looks like this:

And thats it! Thank you for reading. I'll do my best to answer any questions you may have.

Is Declaration Merging and Generic Inheritance at the Same Time Impossible?

Ethan Arrowood — Tue, 24 Dec 2019 18:58:05 +0000

Fastify is a fast and low overhead web framework for Node.js. It supports http, https, and http2 server types. The Fastify v3 type system provides a generic property so the user can specify which type of server they want to create; this generic property is then used to determine the type of the server request and reply objects.

The following definitions are a simplified abstraction from the actual Fastify v3 type system. You can explore the complete source code here

The start of the definition defines the list of types the server generic can be, as well as the main function fastify.

import http from 'http'
import https from 'http'
import http2 from 'http2'

type ServerTypes = http.Server | https.Server | http2.Http2Server

declare function fastify<Server extends ServerTypes>(): FastifyInstance<Server>

The FastifyInstance definition depends on two generic expressions, one for determining the base request type, and the other for reply.

type RequestExpression<Server extends ServerTypes> = (
  Server extends http.Server | https.Server
  ? http.IncomingMessage
  : http2.Http2ServerRequest 
)

type ReplyExpression<Server extends ServerTypes> = (
  Server extends http.Server | https.Server
  ? http.ServerResponse
  : http2.Http2ServerResponse
)

interface FastifyInstance<
  Server extends ServerTypes,
  Request = RequestExpression<Server>,
  Reply = ReplyExpression<Server>
> {
  request: FastifyRequest<Request>,
  reply: FastifyReply<Reply>
}

These Request and Reply generics are then passed to the FastifyRequest and FastifyReply definitions. These make use of generic inheritance to add additional properties to the base request and reply types.

type FastifyRequest<Request> = Request & {
  body: unknown,
  query: unknown
}

type FastifyReply<Reply> = Reply & {
  sent: boolean,
  code(c: number): FastifyReply<Reply>
}

Fastify supports plugins for decorating the server, request, and reply instances with additional properties from the user.

function myPlugin (inst, opts, next) {
  inst.decorateRequest('myPluginProp', 'super_secret_string')
  inst.decorareReply('myPluginProp', 5000)
}

But how can we update the type system to acknowledge these new properties? Additionally, to support module based plugins (i.e. downloadable from npm), the type override should just work by importing the plugin into a project (i.e. import myPlugin from 'myPlugin'). We can try to use declaration merging on the FastifyRequest and FastifyReply types:

declare module 'fastify' {
  type FastifyRequest = {
    myPluginProp: string
  }
  type FastifyReply = {
    myPluginProp: number
  }
}

Sadly this does not work; you cannot merge type declarations. What if we try rewriting the type declarations as interfaces?

interface FastifyRequest<Request> extends Request {
  raw: Request;
  body: unknown;
  query: unknown;
}

interface FastifyReply<Reply> extends Reply {
  raw: Reply;
  sent: boolean;
  code(c: number): FastifyReply<Reply>;
}

Sadly, this also does not work; it throws an error:

An interface can only extend an object type or intersection of object types with statically known members.

If we use types, then we can't support declaration merging. And if we use interfaces, then we can't support generic inheritance. 🤔

💡 What if we use both types and interfaces?

Encapsulate the custom properties into interfaces

interface FastifyRequestInterface<Request> {
  raw: Request;
  body: unknown;
  query: unknown;
}

interface FastifyReplyInterface<Reply> {
  raw: Reply;
  sent: boolean;
  code(c: number): FastifyReply<Reply>;
}

Replace the { ... } part of the type declaration with these new interfaces

type FastifyRequest<Request> = Request & FastifyRequestInterface<Request>

type FastifyReply<Reply> = Reply & FastifyReplyInterface<Reply>

Now, if the user wants to add custom properties too this they can use declaration merging on the FastifyRequestInterface and FastifyReplyInterface interfaces, and the type declarations can still inherit from the generic parameters!

declare module 'fastify' {
  interface FastifyRequestInterface {
    myPluginProp: string
  }
  interface FastifyReplyInterface {
    myPluginProp: number
  }
}

And in some implementation file:

import fastify from 'fastify'
import myPlugin from 'myPlugin'

const server = fastify()

server.register(myPlugin)

server.get('/', (request, reply) => {
  request.myPluginProp // -> ✅ string
  reply.myPluginProp // -> ✅ number
})

🎉 Thus, no it is not impossible to support both declaration merging and generic inheritance at the same time!

Thank you for reading! If you enjoyed this article consider following myself on Twitter @ArrowoodTech.

To learn more about Fastify checkout our GitHub repository or our website.

Observing Great Leadership in Tech

Ethan Arrowood — Tue, 09 Jul 2019 17:42:37 +0000

Amazing things are being accomplished by Tech company leaders such as Tracy Lee, Anil Dash, and Nat Friedman.

Tracy Lee is the leader behind This Dot Labs, a mentoring, training, consulting, and apprenticeship company for web developers. She is a core part of Node.js Developer Relations and is a great part of the developer community.

Anil Dash is the leader behind Glitch, a simple tool for creating web apps with a friendly, creative community. He recently posted about Glitch's 2018/2019 successes which is worth a read. Anil consistently shares his thoughts, learnings, and actions surrounding inclusive and ethical tech.

Nat Friedman has been leading GitHub since the acquisition by Microsoft. He has released many new features and is very active with the GitHub developer community. I was lucky enough to chat with him about his success and some of his leadership values. In this phone call I learned about the value of momentum and how he has attributed this to the recent success of GitHub.

What are some other leaders in our tech community we should be paying attention to?

I am especially looking for leaders from under-represented groups and leaders that are improving the community through inclusion and impactful actions.

If you aren't already I would recommend following the leaders I listed above so you can learn from them too!

Just released: tsconfig-api

Ethan Arrowood — Wed, 12 Jun 2019 01:17:41 +0000

Just released a new project called tsconfig-api. This API takes the TypeScript CLI options and makes them available via a microservice API. The data is based off of the compiler options handbook file (find the markdown version here). The API is a microservice built on Node.js. It uses now for build and deployment.

Use this API by making a GET request to https://tsconfig-api.matterhorndev.now.sh/tsconfig?option=<compiler option>

Don't forget to pass a valid compiler option to the option query parameter!

Check out the project's README on GitHub for a full list of options.

This project was created in preparation for future tools that will utilize the TypeScript compiler options. Currently planned are a web app and a command line interface for learning about and creating tsconfig files / tsc compile commands.

I hope you enjoy using it!

Learn TypeScript Linting

Ethan Arrowood — Thu, 06 Jun 2019 21:43:04 +0000

This post is an adaptation of a more detailed version found on my personal blog Mountain Top. It is recommended that you read the original Learn TypeScript Linting post series as it has much more granular details, examples, and even a follow along repository.

ESLint

1. Install ESLint & @typescript-eslint

npm i -D eslint @typescript-eslint/{eslint-plugin,parser}

2. Create eslint configuration file

touch .eslintrc.json

Copy the following text into the new eslintrc file

{
  "extends": ["eslint:recommended", "plugin:@typescript-eslint/recommended"],
  "parser": "@typescript-eslint/parser",
  "plugins": ["@typescript-eslint"],
  "env": { "node": true },
  "parserOptions": {
    "ecmaVersion": 5,
    "sourceType": "module"
  },
  "rules": {}
}

3. Add lint npm script

In package.json, add a new script:

{
  "scripts": {
    "lint": "eslint 'src/**/*.ts'"
  }
}

4. Run!

Run the new linting command:

npm run lint

If you want to automatically fix linting issues pass the --fix flag:

npm run lint -- --fix

5. Congrats 🎉

Great work! Your TypeScript is now configured with ESLint.

If you are interested in further configuring ESLint and learning about some of specific properties found in .eslintrc.json check out Part 1 of the Learn TypeScript Linting series over at Mountain Top blog. This post dives into details such as specifying environments & ecmaVersion, configuring rules like semicolon usage and indentation style, and covers one of the most common issues when converting a JavaScript project to TypeScript (no-unused-vars error).

Up next are sections about integrating popular formatters Standard and Prettier with ESLint.

Standard

This section covers how to install and configure the new ESLint set up with the popular Standard opinionated formatter.

One of my favorite things to share about the Standard library is their response to the can I configure rule x question:

No. The whole point of standard is to save you time by avoiding bikeshedding about code style. There are lots of debates online about tabs vs. spaces, etc. that will never be resolved. These debates just distract from getting stuff done. At the end of the day you have to ‘just pick something’, and that’s the whole philosophy of standard — its a bunch of sensible ‘just pick something’ opinions. Hopefully, users see the value in that over defending their own opinions.

1. Install Standard

npm i -D eslint-config-standard eslint-plugin-{standard,promise,import,node}

2. Modify eslint configuration

In .eslintrc.json add "standard" to the end of the "extends" list.

{
  "extends": ["eslint:recommended", "plugin:@typescript-eslint/recommended", "standard"],
}

3. Run and evaluate the new output

Run the lint command using npm. The output should be slightly different as new rules have now been configured incorporating Standard's opinionated formatting. Remember to use the --fix option to automatically fix certain issues.

Prettier

This section will cover how to install and set up the very popular Prettier library.

I recommend using either Standard or Prettier, but not both at the same time!

1. Install Prettier

npm i -D prettier eslint-config-prettier eslint-plugin-prettier

2. Update eslint configuration

First add "plugin:prettier/recommended" to the "extends" list, then add "prettier" to the "plugins" list.

{
- "extends": ["eslint:recommended", "plugin:@typescript-eslint/recommended"],
+ "extends": ["eslint:recommended", "plugin:@typescript-eslint/recommended", "plugin:prettier/recommended"],
  "parser": "@typescript-eslint/parser",
- "plugins": ["@typescript-eslint"],
+ "plugins": ["@typescript-eslint", "prettier"],
  "env": { "node": true },
  "parserOptions": {
    "ecmaVersion": 5,
    "sourceType": "module"
  },
  "rules": {
    "no-console": "warn",
    "@typescript-eslint/indent": ["error", 2]
  }
}

3. Run, evaluate, and fix

Just like in the previous two sections, use the npm script lint to try out the new linting rules. Use the --fix flag to automatically fix errors.

🎉 Congrats again! Now your TypeScript project is not only configured with ESLint but it also is configured with a snazzy formatter.

If you want to learn more about integrating the formatters and even configuring Prettier-specific rules check out Part 2 of the Learn TypeScript Linting series over at Mountain Top blog.

Thank you for reading this post; I hope you enjoyed it! I'd be happy to chat about linting and formatters in the comments below or over on Twitter (@ArrowoodTech).