Forem: Federico Kauffman

CRDTs and Distributed Consistency - Part 3: Building a distributed counter

Federico Kauffman — Mon, 16 Jan 2023 15:00:53 +0000

Originally published in Streaver's blog.

This is the third part of a series of posts on CRDTs and Distributed Consistency. You can find previous parts on these links:

Building a peer-to-peer game using the PNCounter, WebRTC and React

The game will be simple, we will increment a distributed counter with the keyboard. The winner will be the client that has the greatest balanced count. I call a balanced count when the number of increments are equal to the number of decrements. So if User A increments 2 units and decrements 4, the user score will be 2. If User B increments 5 and decrements 4 the user score will be 4 and it will be the winner with a total score amount of 7, because 7 increments and 8 decrements happened in total if we count across clients.

The flow of the game is the following:

User A joins, this generates a game Id.
User A shares the game link with the rest of the participants.
Each participants joins.
User A starts the game.
Each participant presses the left/right arrow keys as fast as possible to increase the score.
Fifteen seconds later the games finishes.
The winner is shown.
You take note of the global score, reload the page and play a second round.

The user that accumulates the most amount of points in N=3 rounds is the winner. Pretty basic, but it will help us cover a lot of missing parts on the usage of CRDTs within a real context.

Extending the PNCounter to be used as an external React store

In order to use the counter with React we need to be able to notify React that changes to the counter happened so React can properly rerender when things change.

The first thing to do is to emit a change event (and other events) when the PNCounter gets incremented, decremented or merged with the state from another agent. In the following example you can see that we need to extend the EventEmitter class and then emit the events:

import EventEmitter from 'events';

// Extend EventEmitter
class PNCounter extends EventEmitter {

  constructor(...) {
    // Call the EventEmitter constructor
    super();
    ...
  }

  public increment(num: number): void {
    this.positive.increment(num);

    this.emit('increment');
    // Emit a change event to notify all subscribed clients about changes
    this.emit('change');
  }

  public decrement(num: number): void {
    ...
    this.emit('decrement');
    this.emit('change');
  }

  public merge(counter: PNCounter) {
    ...
    this.emit('merge');
    this.emit('change');
  }
  ...
}

The next step is to create a hook that allows a user to create a PNCounter and use it across the application.

import PNCounter, { PNCounterId } from './PNCounter';

// Global map of counters
const counters = new Map<PNCounterId, PNCounter>();

// This function creates and cached a counter with a given Id
const createCounter = (id: PNCounterId) => {
  const newCounter = new PNCounter(id);

  counters.set(id, newCounter);

  return newCounter;
};

export default function usePNCounter(id: PNCounterId): PNCounter {
  // Find or create the counter
  const counter: PNCounter = counters.has(id) ? counters.get(id)! : createCounter(id);

  return counter;
}

Now that the counter is initialized and we have a stable reference to it we can subscribe to its changes thanks to it being an EventEmitter.

For this we will make use of the useSyncExternalStore hook that allows us to synchronize React renders with the external store. Moreover, we will use the Redux Selector pattern to fetch or calculate derived values from the counter. The combinations of these two things allows us to be very efficient in terms of renders.


import { isEqual } from 'lodash';
import { useCallback, useRef, useSyncExternalStore } from 'react';
import PNCounter, { PNCounterId } from './PNCounter';
import usePNCounter from './usePNCounter';

export default function usePNCounterSelector<T>(id: PNCounterId, selector: (counter: PNCounter) => T): T {
  const counter = usePNCounter(id);

  const subscribeToCounter = useCallback(
    (callback: () => void) => {
      counter.on('change', callback);
      return () => {
        counter.off('change', callback);
      };
    },
    [counter],
  );

  const previousValue = useRef<T>(selector(counter));

  const getSnapshot = useCallback(() => {
    const newCandidateValue = selector(counter);
    const newValue = isEqual(newCandidateValue, previousValue.current) ? previousValue.current : newCandidateValue;

    previousValue.current = newValue;

    return newValue;
  }, [counter, selector]);

  return useSyncExternalStore(subscribeToCounter, getSnapshot, getSnapshot);
}

Now that we have all the required boilerplate we can start building the actual game. For the communications between peers we are going to use PeerJS a peer-to-peer library built on top of WebRTC.

First let's clarify some things:

The code in the article is just sample code, more like pseudo-code, the actual code is much more complex.
The final result is not a mesh of clients where all clients are treated equal. For simplicity we are going to say that we have a host client.
I will provide very little details on how PeerJS works, sorry!

Sharing state between clients

First we need to create the clients and link them, if you are using PeerJS or will be something like this:


// Client A
const peer = new Peer();
const conn = peer.connect('ClientB');
// on open will be launch when you successfully connect to PeerServer
conn.on('open', function(){
  // here you have conn.id
  conn.send('hi!');
});

peer.on('connection', function(conn) {
  conn.on('data', function(data){
    // Will print 'hi!'
    console.log(data);
  });
});

// Client B
const peer = new Peer();
const conn = peer.connect('ClientA');
// on open will be launch when you successfully connect to PeerServer
conn.on('open', function(){
  // here you have conn.id
  conn.send('hi!');
});

peer.on('connection', function(conn) {
  conn.on('data', function(data){
    // Will print 'hi!'
    console.log(data);
  });
});

Each client can independently increment or decrement the counter and that change will be sent to the the other clients. In order to do that we will just subscribe to the local counter changes, something like this:

...
// On ClientA
const clients = //some how get other connected clients
const counter = usePNCounter('ClientA')

useEffect(() => {
  // This effect subscribes to all remote changes
  // it parses the messages and merges the changes into the local counter
  const clientDataHandler = (data) => {
    const receivedCounter = PNCounter.parse(data.payload);

    counter.merge(receivedCounter);
  }

  clients.forEach((conn) => {
    clientBConnection.on('data', clientDataHandler);
  });

  return () => {
    clients.forEach((conn) => {
      clientBConnection.off('data', clientDataHandler);
    });
  };
}, [counter, clients])

useEffect(() => {
  // This effect subscribes to local changes and pushes them to remote clients
  // You will need to find a way of avoiding infinite loops, because a merge
  // will send a message that will in turn result in another message from remote
  // clients. I will leave this as homework!
  const changeHandler = () => {
    clients.forEach((conn) => {
      clientBConnection.send({
        type: 'counter-state',
        payload: PNCounter.stringify(counter)
      });
    });
  }

  counter.addEventListener('change', changeHandler);

  return () => counter.removeEventListener('change', changeHandler);
}, [counter, clients])

With this ideas you should be able to iterate and build the game I described that the start of the article that uses the PNCounter.

I decided not to share the full code in the article because it would make it too long and boring, but I have a full example working here and the source code here.

Conclusions

Building a simple CRDT like a counter is not that hard, but if you are interested and keep digging into this topic you will see that other types of CRDTs have more complex challenges as I mentioned in the previous part of the series.

Also, using a CRDTs has its challenges, you need to figure out how to connect clients and share that information between them, but both problems can be thought in almost complete isolation from each other, this allows you to use them in many contexts, via the internet, HTTP, WebSocket, SMS, Letters, you name it.

These structures are really useful in distributed computing or realtime apps where network partitions will happen, so the next time you have to build a distributed system you can keep these ideas in mind (or you can send us an email 😉).

CRDTs and Distributed Consistency - Part 2: Building a distributed counter

Federico Kauffman — Mon, 16 Jan 2023 14:59:56 +0000

Originally published in Streaver's blog.

This is the second part of a series of posts on CRDTs and Distributed Consistency. If you have not read the first part I recommend you start there. You can find the first part here.

Extending a G-Counter to allow increments and decrements

The GCounter class we built in the previous post only allowed increments to be performed, now we want to allow increments and decrements. We will need to think a little bit outside the box to achieve this because our implementation depends on the Math.max function to decide whether an update coming from another agent is newer than the state we currently have.

Let's see the problem with an example, we can assume we remove the increment only restriction by commenting out the code as follows:

class GCounter {
  ...
  public increment(num: number): void {
    // if (num < 0) {
    //   throw new Error('Only positive values');
    // }

    // Increment my local counter by num
    this.state[this.id] += num;
  }
  ...
}

const NUMBER_OF_CLIENTS = 2;

// All clients start
const client1 = new GCounter(0, NUMBER_OF_CLIENTS);
const client2 = new GCounter(1, NUMBER_OF_CLIENTS);

const clients = [client1, client2];

// Each client independently increases or decreases their counter
client1.increment(1);
client2.increment(3);
client2.increment(-2);
client2.increment(-2);
client1.increment(1);
client1.increment(1);

// Before sharing any knowledge they just know about their own count
console.log({
  client1: client1.query(),
  client2: client2.query(),
});
// { client1: 3, client2: -1 }

// At some point in time they synchronize and share their state
clients.forEach((client) => {
  client1.merge(client);
  client2.merge(client);
});

// After sharing the knowledge they agree on the count, but it will not be the right count!
console.log({
  client1: client1.query(),
  client2: client2.query(),
});
// Actual: { client1: 3, client2: 3 }
// Expected: { client1: 2, client2: 2 }

This happened because when merging the states [3, 0] and [0, -1] for the client1 and client2 respectively, the max between 0 and -1 is 0 and there is no way to know which value is newer. If we have the restriction of only allowing increments the rule for deciding how to solve a conflict is trivial because we know that whatever value is bigger will be newer.

In order to solve this we need to think on the independent operations of increasing and decreasing the count. How did each client reached their independent count state?

In the case of client1 it reached its state by doing three increments of 1. For client2 the steps were one increment of 3 and two increments of -2. If we re-write the steps for client2 we can say it reached the state by doing one increment of 3 and two decrements of 2. Let's see this in a different way by placing each operation on different arrays:

// Before the re-write

client1 = {
  increments: [1, 1, 1],
}

client2 = {
  increments: [3, -2, -2],
}

// After the re-write

client1 = {
  increments: [1, 1, 1],
  decrements: [],
}

client2 = {
  increments: [3],
  decrements: [2, 2],
}

You might have noticed that now, after we changed the wording of the operations, both the increments and decrements are now positive numbers and they can be both represented as independent counts! And guess what? We already know how to represent positive counts, we do it with a G-Counter.

So, in order to remove the increment only restriction we simply need to have two counts, one for the increments and one for the decrements and to get the final result we can just subtract them from each other. This is called a PN-Counter (Positive/Negative-Counter) and the implementation is really simple because we already did most of the work with the G-Counter:

import GCounter from './GCounter';

export default class PNCounter {
  private positive: GCounter;// Counts of increments
  private negative: GCounter;// Counts of decrements

  constructor(id: number, size: number) {
    this.positive = new GCounter(id, size);
    this.negative = new GCounter(id, size);
  }

  public query(): number {
    // The result of the count is the difference of both counts
    return this.positive.query() - this.negative.query();
  }

  public increment(num: number): void {
    // Incrementing the positive count
    this.positive.increment(num);
  }

  public decrement(num: number): void {
    // Incrementing the negative count
    this.negative.increment(num);
  }

  public merge(counter: PNCounter) {
    // Merging is just done independently for each count
    this.positive.merge(counter.positive);
    this.negative.merge(counter.negative);
  }
}

We can see it working with an example, we will use the same we used to derive the idea for the counter:

import PNCounter from './PNCounter';

const NUMBER_OF_CLIENTS = 2;

// All clients start
const client1 = new PNCounter(0, NUMBER_OF_CLIENTS);
const client2 = new PNCounter(1, NUMBER_OF_CLIENTS);

const clients = [client1, client2];

// Each client independently increases or decreases their counter
client1.increment(1);
client2.increment(3);
client2.decrement(2);
client2.decrement(2);
client1.increment(1);
client1.increment(1);

// Before sharing any knowledge they just know about their own count
console.log({
  client1: client1.query(),
  client2: client2.query(),
});
// { client1: 3, client2: -1 }

// At some point in time they synchronize and share their state
clients.forEach((client) => {
  client1.merge(client);
  client2.merge(client);
});

// After sharing the knowledge they agree on the count
console.log({
  client1: client1.query(),
  client2: client2.query(),
});
// { client1: 2, client2: 2 }

Extending a G-Counter or PN-Counter to allow arbitrary clients

The second restriction that our original counter had is that it would only work if we know how many agents can be affecting the counter beforehand, that is a big restriction for any distributed system that could automatically scale up or down without notice. Let's try to remove that restriction.

If we look at the implementation of the G-Counter we will notice that we choose our data structure for the counts to be an array, that is a very efficient structure but it also is the one forcing us to use index-based IDs.

class GCounter {
  private id: number;
  public state: number[];
  ...
}

This is a problem, imagine you have two new agents "joining the counter" at the same time, if we want to keep them independent, each one of them would add one more place into the array and assign themselves the n+1 id and cause a collision, this breaks the counter.

It now becomes clear that we need to identify the agents with unique IDs and the IDs can't be index-based, we can use whatever we want but we also need to change the state structure, in this case we will use a Map to store the id-count pair.

class GCounterWithArbitraryClients {
  private id: number | string; // The ID of this client
  public state: Map<number | string, number>; // The last known count value for each client

  // We don't need the size anymore
  constructor(id: number | string, initialValue = 0) {
    this.id = id;
    this.state = new Map();

    // We don't need to initialize all clients, just our own value
    this.state.set(id, initialValue);
  }
  ...
}

If we continue adapting the code we will end up with a very similar implementation, but that will instead of assuming we have always values for all clients it will use defaults for unknown clients, see the full code below with comments:

class GCounterWithArbitraryClients {
  private id: number | string;
  public state: Map<number | string, number>;

  constructor(id: number | string, initialValue = 0) {
    this.id = id;
    this.state = new Map();

    this.state.set(id, initialValue);
  }

  public query(): number {
    // This is the same as before, but we go over the values of the Map
    return [...this.state.values()].reduce((acc, value) => acc + value, 0);
  }

  public increment(num: number): void {
    if (num < 0) {
      throw new Error('Only positive values');
    }

    // This is the same as before, but we get, increment and set the new value
    // using the Map primitives instead of the direct array assignment
    this.state.set(this.id, this.state.get(this.id)! + num);
  }

  public merge(counter: GCounterWithArbitraryClients) {
    // We create a set with all the ids known by the local agent plus
    // the ids from the remote agent that we are merging
    const ids = [...new Set([...this.state.keys(), ...counter.state.keys()])];

    // Create tuples with local count and received count,
    // but we default to zero so we can properly compare new clients
    // zipped = [[local1, remote1], [local2, remote2], ...]
    const zipped = ids.map((id) => [
      this.state.get(id) ?? 0,
      counter.state.get(id) ?? 0,
    ]);

    // Create a new map with all the ids and the new counts
    this.state = new Map(
      ids.map((id, index) => [
        id,
        Math.max(zipped[index][0], zipped[index][1]),
      ]),
    );
  }
}

We can test it with an example:

import GCounterWithArbitraryClients from './GCounterWithArbitraryClients';

// All clients start (at the same time or not, we don't care anymore)
const client1 = new GCounterWithArbitraryClients('client1');
const client2 = new GCounterWithArbitraryClients('client2');

const clients = [client1, client2];

// Each client independently increases their counter
client1.increment(1);
client2.increment(3);
client2.increment(3);
client1.increment(1);
client1.increment(1);

// Before sharing any knowledge they just know about their own count
console.log({
  client1: client1.query(),
  client2: client2.query(),
});
// { client1: 3, client2: 6 }

// At some point in time they synchronize and share their state
clients.forEach((client) => {
  client1.merge(client);
  client2.merge(client);
});

// After sharing the knowledge they agree on the count
console.log({
  client1: client1.query(),
  client2: client2.query(),
});
// { client1: 9, client2: 9 }

And that's it! We just solved both problems, our counter now supports increments and decrements and also can accommodate any number of clients as they join the count. There are some improvements that can be done to the solution in terms of performance, but for the purposes of this post the solution is enough.

Conclusions

We started with a very simple distributed counter and we have evolved it to be a more powerful counter that supports increments, decrements and an arbitrary number of clients. One problem that you might have noticed here is that there is no way of cleaning up old or dead clients. That can be achieved but it imposes many challenges and more advanced concepts as "tombstones" or "remove sets". You can read more details on the Wikipedia page for CRDTs.

Following this same approach you can build CRDTs that are increasingly more complex but at the same time they provide features that can allow you to build truly distributed apps.

In the next part of the series I will build an example that uses a peer-to-peer approach to share the state between agents.

Thanks to Martín Feldman, Martín Fitipaldo and Nicolás Wolman from our team that helped me go through the second part of this post.

CRDTs and Distributed Consistency - Part 1: Building a distributed counter

Federico Kauffman — Mon, 16 Jan 2023 14:46:00 +0000

Originally published in Streaver's blog.

You are probably reading this because you either, are really curious and asked yourself the question "How does Google Docs work?" or maybe a client you work for had a similar requirement and basically forced you to ask the question 😅. The latter is what happened to me two years ago, and this is how I met CRDTs and how Streaver has been involved with this kind of problems since 2020.

How does Google Docs (or similar apps) work?

I started by asking the question to Google directly, thinking that someone probably already asked and answered the same question but of course I only got answers from the final user's perspective, this was not enough, I wanted technical details, I wanted the internal engineering details.

I had to re-think the question, and make it more general. So I changed it for "How is collaborative document editing implemented?" This yielded better results, but I still had to do a lot of reading, the answer was not easy.

One of the results pointed me to a solution called Operational Transformation (OT) which after a lot of reading and looking at some of the outgoing requests from the Google Docs app it self I confirmed that it was indeed how it was implemented. But I also saw some mentions to the CRDT acronym, which stands for Conflict-free replicated data type, that caught my eye, because it seemed like a newer idea and with some interesting properties that OT had not, for example not needing a central server.

So, the answer is, Google Docs works by making use the Operational Transformation algorithm, which in many ways is similar to Git, but it automatically solves conflicts between editing agents.

Writing your own collaborative editor is a huge task, we have done it, but we will not do this here. I will leave this video from @martinkl which goes through some of the challenges of building this kind of apps.

From this point onwards we will explore the ideas behind CRDTs that will eventually allow you to write your own collaborative apps including rich-text editors.

What are CRDTs?

In distributed computing, a conflict-free replicated data type (CRDT) is a data structure that is replicated across multiple computers in a network, with the following features:

No Coordination: The application can update any replica independently, concurrently and without coordinating with other replicas.

Self-healing: An algorithm (itself part of the data type) automatically resolves any inconsistencies that might occur.

Convergence: Although replicas may have different state at any particular point in time, they are guaranteed to eventually converge.

An example will make it easier to understand and in this post we will do just that, build our own CRDT.

Imagine you have N instances of an app that need to count the amount of times a user clicks on a button, we don't need to store the information, just count it and eventually synchronize the information so everyone is aware of the total amount. There are many ways to solve that:

One way that comes to my mind is to have a server that holds the count and implements the transactions and APIs to ensure that the counter is updating correctly and that the clients can fetch the latest information. Just describing this was hard, it involves a DB, servers, APIs, etc.
Another way involves using a CRDT that implements a distributed counter over a P2P protocol. This type of counter is called G-Counter (Grow-only Counter) and the P2P stuff is just sugar on top, CRDTs don't care about the communication layer at all!

Building a G-Counter

Let's build this G-Counter together. A counter is just a variable that allows us to store a number. First we want to understand what a counter should do for a single user:

let counter = 0; // Creating the counter and setting initial value

counter += 1; // Increase the counter by n=1 (n could be anything);
counter += 3; // Increase the counter by n=3;

console.log(counter); // We need to query the value somehow;

Now that we know the APIs we need let's build a counter but wrapping it's functionality in a class (I am going to use TypeScript), this will prepare us for the G-Counter:

export default class GCounter {
  private count: number;

  constructor(initialValue = 0) {
    this.count = initialValue;
  }

  public query(): number {
    return this.count;
  }

  public increment(num: number): void {
    if (num < 0) {
      throw new Error('Only positive values');
    }

    this.count = this.count + num;
  }
}

The next step is to introduce a way of storing and synchronizing the counter value from other agents. We will need an internal state that allows us to compute the actual value of the counter based on what we know of the outside world. Moreover, in order to synchronize the state between agents we need a rule to decide who is right, basically we need to answer the question: Is my information of the world the latest or is the information I got from someone else newer?. This is easy to answer if we assume that our counter can only be incremented, we just have to use Math.max function and we always keep the largest count as the most recent information. Let's write the code for this:

export default class GCounter {
  private id: number; // The ID of this client
  public state: number[]; // The last known count value for each client

  constructor(id: number, size: number, initialValue = 0) {
    this.id = id;

    // Initialize an array with zeros for everyone except myself
    this.state = Array.from({ length: size }, (_, index) =>
      index === id ? initialValue : 0,
    );
  }

  public query(): number {
    // Return the sum of all counters (last known value)
    return this.state.reduce((acc, value) => acc + value, 0);
  }

  public increment(num: number): void {
    if (num < 0) {
      throw new Error('Only positive values');
    }

    // Increment my local counter by num
    this.state[this.id] += num;
  }

  public merge(counter: GCounter) {
    // Create tuples with local count and received count
    // zipped = [[local1, remote1], [local2, remote2], ...]
    const zipped = this.state.map((count, index) => [
      count,
      counter.state[index],
    ]);

    // Update the state with the max known counter for each client
    this.state = zipped.map((counts) => Math.max(counts[0], counts[1]));
  }
}

We now have a G-Counter, let's see it in action with 2 clients:

import GCounter from './GCounter';

const NUMBER_OF_CLIENTS = 2;

// All clients start
const client1 = new GCounter(0, NUMBER_OF_CLIENTS);
const client2 = new GCounter(1, NUMBER_OF_CLIENTS);

const clients = [client1, client2];

// Each client independently increases their counter
client1.increment(1);
client2.increment(3);
client2.increment(3);
client1.increment(1);
client1.increment(1);

// Before sharing any knowledge they just know about their own count
console.log({
  client1: client1.query(),
  client2: client2.query(),
});
// { client1: 3, client2: 6 }

// At some point in time they synchronize and share their state
clients.forEach((client) => {
  client1.merge(client);
  client2.merge(client);
});

// After sharing the knowledge they agree on the count
console.log({
  client1: client1.query(),
  client2: client2.query(),
});
// { client1: 9, client2: 9 }

Congratulations we have built a distributed conflict-free and replicated counter! No need to do transactions or coordinations.

In this particular case we are not actually sharing the state through the network, just on memory. But you can share the state in whatever way you prefer, HTTP, WebSockets, etc.

Conclusions

As you saw, a CRDTs can be very simple, a simple counter if written properly can be a considered a CRDT. Despite that, these data types are extremely powerful because they ensure eventual consistency without coordination between agents.

In upcoming parts of this series of posts I will build different structures and I will share how to propagate state between agents via peer-to-peer.

You can find the second part here.

Flattening a directory structure on AWS S3

Federico Kauffman — Thu, 10 Jun 2021 12:00:20 +0000

Originally published in Streaver's blog.

The problem

The other day I had to flatten a small tree of files on AWS S3, something in the order of hundreds of files. After some trial and error, I ended with a pretty straightforward series of piped commands that got the job done.

The problem was simple; we wanted to migrate from a structure that was something like s3:// <your-bucket>/posts/:postId/images/:imageId to a flat directory structure like s3:// <your-bucket>/posts/images/:imageId so we could have images as a global thing shared by all posts and not have them under a specific post folder.

You will need a bare shell and the AWS CLI setup with your credentials. I will assume you already have all of that.

Step 1: Checking for name collisions

We could have post1/image1.png and post2/image1.png, so we have to check if the flattening is possible. Doing that is pretty simple. You use some shell commands piped together, and you can get it in a matter of seconds.

We start by getting a list of all the files we want to migrate, for that we use:

$> aws s3 ls <your-bucket> --recursive
2021-06-07 12:24:28     390966 posts/1/images/image1.png
2021-06-08 13:09:16     120346 posts/1/images/image2.png
2021-06-07 12:23:37     144935 posts/2/images/image3.png
2021-06-07 12:23:37     144945 posts/3/images/image3.png
...

and we get a list with a timestamp, the size, and the path.

The second step is to filter out the information that we do not need:

$> aws s3 ls <your-bucket> --recursive | awk '{print $4}'
posts/1/images/image1.png
posts/1/images/image2.png
posts/2/images/image3.png
posts/3/images/image3.png

for that we use awk, which is a handy shell command (actually, it is a pattern-directed scanning and processing language) that lets you scan and detect patterns and extract information very quickly.

Since we are going to be flattening the directory structure, in order to check for duplicates, we need to check for the filenames only, we do not need the full path. To achieve that, the third step is:

$> aws s3 ls <your-bucket> --recursive | awk '{print $4}' | xargs -I {} basename {}
image1.png
image2.png
image3.png
image3.png

Here we use xargs which allows executing a command for each line that you pipe into it, and we combine it with basename to extract the filename of a given path.

The last step is to check for duplicated filenames. For that, we can use uniq -d, which prints the duplicated lines. If we put it all together and run it, we will know if we have duplicates:

aws s3 ls <your-bucket> --recursive | awk '{print $4}' | xargs -I {} basename {} | uniq -d
image3.png

We run it, if it is empty, we are good, the flattening can be done straight away, if not, we need to somehow modify the filename so that after the flattening of the files none of them collide. I will leave the renaming up to you since it will depend on how you are writing/reading files. Four our case, I renamed posts/3/images/image3.png to posts/3/images/image4.png.

Step 2: Moving the files

The next step is to move the files or duplicate them. If you need a zero-downtime migration, you can copy first and delete later. If you don’t need a zero-downtime migration, you can move the files from the old location to the new one, and there will be a short period where files can't be found.

We will use the xargs commands to move each file as we see them.

aws s3 ls <your-bucket> --recursive | awk '{print $4}' | xargs -I {} sh -c 'aws s3 mv s3://<your-bucket>/{} s3://<your-bucket>/posts/images/$(basename {})'

Let’s break it down! The first parts are the same as before. Then, the xargs starts similar, but instead of running the commands, it creates a subshell and passes the command into it before replacing the {} with the filename.

If you want to run it but are not sure if things will go as planned, you can add the --dryrun option to the aws s3 mv command, and that will simulate and print the result of the command without it happening.

Some personal notes

This is a pretty handy trick to have if you need to move files. It will work excellent with small amounts of files. It is also resistant to an internet connection failure; if your connection fails (or you kill the process, or it dies), you can rerun it, and it will pick up from where it left.

If you have larger amounts of files, you can launch a bunch of background processes running this command, and it should work (you probably want to paginate results or smartly split them). On the other hand, if your amount of files is too significant, maybe a copy-on-write or copy-on-read strategy works better, but you will need some programming for that.

I hope this has been useful! If you have questions, please let me know in the comments.

Animations with React: How a simple component can affect your performance

Federico Kauffman — Sun, 30 May 2021 11:50:27 +0000

Originally published in Streaver's blog.

Animations on the web

If you are working on a modern app, you will likely use some kind of animations. They might be simple transitions, for which you should probably use a CSS Transition or even if they are more complex transitions/animations, you can use CSS Keyframes. These techniques will cover most cases, but sometimes you will need customization, and JavaScript might be your only choice.

If you are going the JavaScript route (or, in our case React), you must be careful not to compromise your app's performance and always remember that JS runs a single thread for the UI.

What is the easiest way to define an animation?

Generally, the best way to define an animation is with a mathematical function. For this case, I will keep it simple and say that our function will be a function of time:

// Given a time, calculate how everything should look like
// (the something function)
const animation = (time) => {
  return something(time);
}

You can define more complex animations and functions, for example, one that depends on the previous animation state or some global state (like a game would do). But we will stay with the simplest case.

As an example we are going to animate an svg element according to a given mathematical function. Since we are going to move the svg to an x and y position it would make sense that our animation function returns what the styles of that svg should look like at a given time, something like:

const animation = (time) => {
  // X_SPEED is a constant that tells the animation
  // how many pixes per millisecond x should move.
  const x = (X_SPEED * time) % WIDTH;
  // A, B, C and D are constants that define the
  // behavior of our Sin function.
  const y = A * Math.sin(B * (x + C)) + D;

  return {
    transform: `translateX(${x}px) translateY(${y}px)`,
  };
}

This example is almost the same as you do with CSS Keyframes, the only difference is that here you need to provide a function that defines every frame, and with Keyframes, you give the essential parts, and the browser fills in the blanks.

You might be asking yourself:
Why bother writing your animations with JS if CSS Keyframes are easier?

You have to remember that our goal is to understand the performance aspects of animations. I assume you will use this for complex cases only. For everything else, pure CSS is likely the best choice.

Writing a simple animated React component

Our component will be an SVG Circle that we will move on the screen according to a provided animation function. As a first step, we simply render the SVG.

const Animation = ({ animation }) => {
  const [animatedStyle, setAnimatedStyle] = useState({});

  return (
    <svg
      viewBox="0 0 100 100"
      height="10"
      width="10"
      style={animatedStyle}
    >
      <circle cx="50" cy="50" r="50" fill="black" />
    </svg>
  );
};

Now we can use our Animation component (which is yet to be animated) as follows:

// WIDTH, HEIGHT, X_SPEED, A, B, C and D are given constants
const SlowAnimations = () => {
  return (
    <div style={{ width: WIDTH, height: HEIGHT }}>
      <Animation
        animation={(time) => {
          const x = (X_SPEED * time) % WIDTH;
          const y = A * Math.sin(B * (x + C)) + D;

          return {
            transform: `translateX(${x}px) translateY(${y}px)`,
          };
        }}
      />
    </div>
  );
};

Now that we have our component on the screen, we need to let the time run and calculate the new styles for the svg using our animation function. A simple solution could be as follows:

const Animation = ({ animation }) => {
  ...

  useEffect(() => {
    let currentTime = 0;
    let prevTime = currentTime;

    const animateFn = () => {
      // We calculate how much time has elapsed from the
      // previous run in order to know what styles we need
      // to apply at the current time.
      const now = performance.now();
      const delta = now - prevTime;
      prevTime = now;
      currentTime = currentTime + delta;

      // We set the resulting styles from the animation
      // and React renders the new state to the DOM.
      setAnimatedStyle(animation(currentTime));
    };

    /* We assume the animations start at t = 0, this means
     * that the initial style can be calculated by running
     * the animation at t = 0.
     */

    setAnimatedStyle(animation(currentTime));


    // To achieve 60 FPS you need to
    // animate every 1/60 seconds ~= 16 ms
    const intervalId = setInterval(animateFn, 16);

    return () => clearInterval(intervalId);
  }, [animation]);

  return (
    ...
  );
};

The Animation component works and animates things pretty well on the screen, but it has some big problems!

Firstly, using a setInterval that runs every 16ms is CPU intensive, and your users will notice it. Also, it does not care about anything else that is happening on your computer or mobile device. It will try to execute every 16ms even if your computer is struggling, the battery is running low, or the browser window is not visible.

Secondly, that component is going through a React render and commit cycle every ~16ms because we use the internal state of React to store the animation; when we set the state, a render and a commit happens, and that is killing the CPU even more.

You can read more about this on What are render phase and commit phase in react dom?
.

Also, if you use the React Dev Tools you can see that the component has a lot of activity. In just a few seconds of profiling, it committed and rendered hundreds of times.

But, since React is so fast and you are probably using a beefy computer, you will not feel any sluggishness on the animation.

You can also record a performance profile on your browser, which for my setup it shows that for every second we are animating, we are using our CPU/GPU ~11% of the time.

Now, let's see how to do it better.

Writing a performant animated React component

We start very similarly to the previous implementation. But you will notice we are not using React's useState hook, and that is because for this implementation after the animation gets started, we don't care about the state of the component. Our objective is to be as fast and efficient as possible.

const Animation = ({
  animation,
  style,
  ...props
}) => {
  return (
    <svg viewBox="0 0 100 100" height="10" width="10">
      <circle cx="50" cy="50" r="50" fill="black" />
    </svg>
  );
};

We are going to be writing to the DOM outside of React render and commit cycle, React is still going to be useful, because it provides the API for setting up the scene, that is mounting, unmounting the element to/from the DOM and the useEffect hook to get things started.

The next step is to use the useRef hook and get a handle to the SVG element after it is mounted so we can do the DOM updating ourselves.

const Animation = ({
  animation,
  style,
  ...props
}) => {
  const elementRef = useRef(null);
  ...

  return (
    <svg
      ref={elementRef}
      ...
    >
      ...
    </svg>
  );
};

Next, we will use the useEffect hook to synchronize our component with the DOM state. When the element is mounted, and after we have a reference, we create a animateFn which takes the time provided by the requestAnimationFrame function and calculates the next animation state. I am assuming you know what requestAnimationFrame is. If you don't, please refer to the documentation.

const Animation = ({ animation }) => {
  ...

  useEffect(() => {
    if (elementRef.current) {
      let time = 0;
      let animationFrameId, animationFramePrevTime;

      const animateFn = (currentTime: number) => {
        /* The time provided by RAF (requestAnimationFrame)
         * is a DOMHighResTimeStamp.
         *
         * But we assume our animation functions
         * start at t = 0. Because of this we need
         * to skip a frame in order to calculate the time delta
         * between each frame and use that value to get the
         * next step of our animations.
         *
         * For more details see:
         *  - https://developer.mozilla.org/en-US/docs/Web/API/window/requestAnimationFrame
         *  - https://developer.mozilla.org/en-US/docs/Web/API/DOMHighResTimeStamp
         */
        if (animationFramePrevTime !== undefined) {
          const delta = currentTime - animationFramePrevTime;
          time = time + delta;

          /* We are rendering outside the react render loop
           * so it is possible that a frame runs after the
           * element is unmounted and just before the useEffect
           * clear function is called. So we need to
           * check that the element still exists.
           */
          if (elementRef.current) {
            // Get the next position
            const { transform } = animation(time);

            elementRef.current.style.transform = transform;
          }
        }

        // Save the current RAF time as to use in the next frame
        animationFramePrevTime = currentTime;

        // This starts the requestAnimationFrame loop
        // Save the frameId for future cancellation
        animationFrameId = requestAnimationFrame(animateFn);
      };

      // First call to request animation frame
      // Save the frameId for future cancellation
      animationFrameId = requestAnimationFrame(animateFn);

      // This cancels the last requestAnimationFrame call
      return () => cancelAnimationFrame(animationFrameId);
    }
  }, [animation]);

  return (
    ...
  );
};

The previous snippet has two key differences from the first implementation. The first one is that we use requestAnimationFrame, which allows us to be conscious of the user's machine state. In other words, it lets the browser decide when to run the animation and at what FPS. That will save CPU time, battery and will likely make animations smoother.

The second important part is that instead of using useState to save the animation and let React handle the rendering, we update the DOM ourselves. And that avoids the React commit and render loop from executing at all, saving CPU time.

If you look at the React Dev Tools, you will notice that this component is only committed and rendered once even though it runs the animation.

By looking at the browser performance profile, the CPU/GPU usage is ~9% for every second of animation. It does not sound like a significant change, but this is just one small component. Imagine doing the same with a real application that has hundreds of components. You can try it yourself at the demo application

Conclusions

As with everything in life, there are tradeoffs. The biggest one for this case, in my opinion, is that the first implementation was simple and easy to read. If you know the basics of React, you could understand it. The second one not so much, you need to understand React and the browser in more depth. Sometimes this is acceptable. On the other hand, the first implementation was very inefficient, the second one is very fast, and that is the most significant tradeoff.

And finally, if you need a framework to decide when to use CSS or JS to animate things, I would start by asking the following questions:

Does my animation need some kind of state?. If no, then CSS is probably the way to go.
Do I need control of "every frame"? If the answer is no, then CSS Keyframes is worth trying.

And before you go and animate everything yourself, check out the framer-motion package. It will likely cover most of your needs.

React Adaptive Hooks and APIs for Adaptability

Federico Kauffman — Mon, 25 Nov 2019 12:33:13 +0000

Originally published in Streaver's blog.

Why should we even care about performance?

New machines and phones are more powerful than ever, internet speeds are rising, so, why should we care about delivering smaller bundles and lower load and interactivity times?

If your user-base is in countries with low internet speeds or lower-end devices the answer is simple, you should care because otherwise, you might not be able to even deliver the app to the users for them to make good use of it.

That was easy, now, what if your average user has a high-end phone and lives in a country with high-speed internet? Well, the answer is pretty simple too, just because they have powerful devices it does not mean you should kill their phones with megabytes of data and intense CPU tasks killing their batteries, you should be smart and use power just when you need it (if you can, of course).

The moral of the story is that to be a good developer you should also be mindful on all aspects, not only those that the end user see. You must care about performance independently of who is using the app, which device loads it or the time you have to wait for it to be ready.

Believe it or not, if you travel often you will experience that kind of problems more often than you think.

APIs for adaptability

Modern browsers like Chrome, Firefox, Safari, Edge, and others support some web APIs to help us adapt your apps to the user's capabilities, you can ask for Network information with the NetworkInformation API, this will even fire events when the internet speed changes to allow you to deliver very dynamic user experiences! That API also provides information about if the user has a "data saver" mode enabled, so it will allow you to be respectful of the user preferences.

The HardwareConcurrency API allows us to take advantage (or not) of multiple CPU cores, you can deliver very different experiences based on this number, for example, you can increase performance by increasing the number of Web Workers that take care of heavy loads on your app.

Talking about performance, we must mention the Performance API and the DeviceMemory API which provides important information about memory availability on the device. Just like the HardwareConcurrency API, you can use the APIs to deliver different experiences for different users, for example, you could do pagination in smaller chunks if the user will not support rendering heavy pages.

And lastly, the DateSaver API which allows you to be respectful of the user preferences and save data by loading different experiences based on this flag. I personally use this feature a lot, when I travel my internet provider is not very friendly with data caps and costs, so I love if web apps respect this preference instead of assuming I have unlimited data and just download all the things!

Some of the mentioned APIs are experimental and might not be available on all platforms. But don't worry you can check the availability with https://caniuse.com.

react-adaptive-hooks

This month Addy Osmani twitted about a new library created to take advantage of the previous mentioned APIs by using some very simple React Hooks.

// Detect dark theme var iframe = document.getElementById('tweet-1194158823948177408-499'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1194158823948177408&theme=dark" }

The hooks provided are four, each one of them allows you to target different characteristics of the user's device and/or preferences:

import { useNetworkStatus } from 'react-adaptive-hooks/network';
import { useSaveData } from 'react-adaptive-hooks/save-data';
import { useHardwareConcurrency } from 'react-adaptive-hooks/hardware-concurrency';
import { useMemoryStatus } from 'react-adaptive-hooks/memory';

If you combine them you can come up with really interesting "patterns" that allow your apps to be faster or more respectful of user's resources. The react-adaptive-hooks repository includes a demos section which demonstrates some simple (but powerful) uses of the hooks. In the next section, I will briefly describe some "advanced" uses of the APIs.

Advanced uses of adaptability

Asking the user

As simple as it sounds (and depending on the type of application) you could let the user decide which version of the app they want to use, it could be a user preference! You can have a default based on the device capabilities, and let it up to the user from there.

An example of this could be a game, the user could decide to improve different aspects of the game, as an example, the user can choose different settings based on what experience they want to achieve. A lightweight experience but very fast level loading times for when they are abroad or have data caps active, or a heavy version for when they came back from their trip or when the new billing cycle starts and the data caps are not there anymore.

Parallel processing

Imagine you have an app with image processing capabilities, like squoosh.app which allows you to process and optimize images online. You could implement a feature of bulk processing of images, you just let the user select a bunch of images and you spawn new Web Workers based on the number of CPU cores you have to process them, moreover, you can monitor the memory usage/availability and adapt your algorithms to provide the user with a nice experience allowing them to get what they need in a reasonable amount of time.

Landing pages and conversion funnels

It is a fact that many companies might not have the resources to get into this type of fine-tuning practice on every single page, but maybe you can use the adaptability APIs in very specific places to provide the average usage paths with better performance. As an example, you could optimize the landing page, with this you engage the users right away, and while they are reading your landing page details you can slowly load the rest of the app in the background.

After you have the landing you can focus on the rest of the conversion funnels of the app ensuring the users have nicer experiences throughout your site's funnel.

Conclusions

Performance is not just a whim, it affects your business, it can have an impact on sales or on user reviews, but it takes time and money to get it right.

I am confident to say with 100% certainty that it does not matter for which company or project you work, you will have limited resources, it could be 1K or 1M dollar budget, but you have a limit. So, for this kind of performance problems, which are never-ending problems that can affect every single part of the application, the important thing is choosing the right battle.

Now that you know about these APIs for adaptability you can choose the battle you want to fight with the right tool.

I know that this was not a "silver bullet" post, but the idea behind it was to tickle your brain into thinking how or where you can apply these techniques, I hope it served its purpose.

Happy performance tunning! 🚀

Translating Vue: Why internationalizing documentation actually matters?

Federico Kauffman — Wed, 03 Jul 2019 10:45:19 +0000

Originally published in Streaver's blog.

If you are reading this, then you are a pretty lucky person

Most of us (I include myself) wake up every day and just give for granted the fact that we know English, we don't think too much about it, we were lucky, you are either a native English speaker or you had enough chances to learn it in your life and thanks to that you are here, you probably have a good tech job, when you are blocked you just type a question on Google and some StackOverflow answer appears, or you can just go and read the documentation for your favorite programming language or framework.

Easy right?

How many people do you know that can't even read or write?

If your answer is zero, then you can consider yourself lucky again, according to the World Bank, the literacy rate for adults above 15 years old is 86%, but there are countries where that number is even below 50%. If you are from one of those countries then it would mean that either your mother or your father probably don't know how to read or write.

How many of those can read or write English?

Around 25% of the people in the world speak English, let's be very generous and assume that 90% of people in tech speak English (I couldn't find any data for this). Although leaving 10% of tech people out might not sound like a lot, now imagine if in that 10% percent there is a person that has a great idea, an idea that could revolutionize the tech industry, but he/she can't make it real because they don't have access to the right information or tools. For me, 10% is a big number if I think it in that way.

Language limits tech development just like hunger limits development in other general areas, this is better explained in the article 5 REASONS TO CARE ABOUT ENDING HUNGER by United Nations Foundation.

How can you help then?

Great libraries or products usually have great documentation pages, so maybe the easiest way to help is to reach out to authors or companies and let them know about this problem, ask them for a lang.library.com subdomain and start translating if you speak more than one language. If you don't, maybe just spread this issue in your circles and someone might be able to help. You can also create/attend local Meetups and show this problem up.

These are just some ways to get this started, and maybe bit by bit we reduce that hypothetical 10% to near zero.

Translating VueJS

If you go to vuejs.org you have a menu that lists all the public translations for the official documentation, that was entirely made by the community, all the people there took some time to help and get the documents translated.

Some months ago some people we know started working on the Spanish translation just because they wanted to help, after that, we joined that effort and after months of work we finally got a translation domain up, it is not 100% finished yet, we are at about 95%, but the domain is publicly accessible and you can see the results at es.vuejs.org.

You can see all the people involved at github.com/1950Labs/vuejs.org#spanish, thanks to all of them, now the Spanish speaking people can benefit from these documents ❤️!

Federico Kauffman

@fedekauffman

After months of work, the translation to Spanish of the @vuejs documentation is out there in the wild 🚀. Spanish speaking people can start taking advantage of this. Check that at es.vuejs.org. It is time to up your game, learn responsibly! Gracias a la comunidad! ❤️

20:38 PM - 10 Jun 2019

18 57

Conclusion

Helping the community can be done in many ways, you don't have to invent the greatest library ever to be a great community member, you can, for example, help on a translation effort, you can simply spread the world to find people with time to help. By helping the community we are building a better tech world and by doing it through translations we are lowering the entry barrier for those that weren't that lucky.

Spread the world 🌎 🌍 🌏!

Pull Request previews for Open Source libraries with Netlify

Federico Kauffman — Thu, 16 May 2019 18:10:36 +0000

Originally published in Streaver's blog.

What does the title even mean?

I get it, the title is hard to understand, but bear with me, I will walk you through the problem and one possible solution.

Let's say you wrote your first open source (or not) library, you wrote unit tests for it, and you have an example web app where you can test the library and check that it is working as expected, for now, you test it locally using npm link (or something similar). As a solo developer, this works just fine.

Now, people have jumped in to help you write the new features for your library, but they need a safe way to test the changes in a real app and show the reviewer a working example of it. Making the reviewers download the sample app, the new code and running npm link for every PR does not scale.

Let's fix this with pull request preview apps for libraries.

What is a pull request preview app?

If you use a "pull request based" workflow, a preview app is like having a "staging" or "pre-production" server for each of your branches or pull requests (PR). You basically have one copy of your entire application (or almost your entire app) for each PR/branch. If you had never used this and have the opportunity to try it out at work or in some side projects, trust me, it feels like heaven!

If you are still here, you might be wondering why not to write end-to-end (e2e) tests and you would be done. E2E tests are expensive and in some cases hard to write, so, this solution is for those cases where e2e is too much and you only need some basic "manual e2e" testing of your library. Moreover, you can even deploy the sample app as an example of usage of your library for end users, it is a win-win!

We need some simple things to achieve this:

Library to test, we are going to use vue-preferences
Sample application for the library, here we use vue-preferences/sample-app
A way to link the new library code to the sample application, we use npm-link
A way to deploy a preview application for PRs, for this we chose Netlify

WARNING

From now on I will provide steps assuming the items listed previously, but you can follow the steps replacing each "ingredient" with your specific needs. The general idea should apply. If you need help, let me know in the comments or @fedekauffman!

Step 1: Library with a sample app

The first step is to have the sample app in the same repository as the library you want to enable previews for, there are other ways of achieving this, but this is probably the most convenient one and "forces" you to keep the sample app up-to-date.

Here is a simplified tree view of the vue-preferences library including the sample-app:

...
├── examples
│   └── sample-app <--- This is the sample app
│       ├── package.json
│       ├── public
│       ├── src
│       └── yarn.lock
├── package.json
├── src
│   └── index.js <--- This is the library code
├── tests
│   ...
└── yarn.lock

Step 2: Linking the library

Next, you have to figure out how to create a build of your sample app that uses the latest code available in your repository in the current branch, luckily, if you are using npm that is very easy!

Enter the library repository and run npm link
Build it, in this case, we use yarn build.
Go to the sample app folder cd examples/sample-app
Run npm link vue-preferences
Built it, yarn build or for live reload (if you have it set up), yarn serve

NPM-LINK
For a great explanation on how to use npm-link check out Understanding npm-link by Franziska Hinkelmann

At this point, you should have the sample app running with the latest code in your branch!

Step 3: Deploying with Netlify

Since this example runs around a VueJS library and a web sample application it makes sense to use a service like Netlify to deploy static assets. If it's not the case and your library is a CLI (or anything not web) you could, for example, use Heroku to launch a Dyno where you can run some test commands.

From Step 2 we know how to build the sample app with the library, we now just need to put that in the netlify.toml file and we are done:

# By default all the branches/PRs build using the latest commit
# available in the branch.

[build]
  base = "."
  publish = "examples/sample-app/dist"
  command = """
    yarn build && \
    npm link && \
    cd examples/sample-app && \
    yarn install && \
    npm link vue-preferences && \
    yarn build --mode development
  """

# The master branch just builds whatever is defined in the package.json
# This is because you probably want the production sample app to be stable
# and not just build whatever is in the repository

[context.master]
  base = "examples/sample-app"
  publish = "examples/sample-app/dist"
  command = "yarn build"

Now, every time you push a new commit your sample app gets built with the latest code in the repository, moreover, if you open a pull request you will get an isolated sample app to test the library code. Cool right 🤓?

This approach might not work for every library, you might need to adapt it to your needs, but now you have a starting point!

This was actually merged to master in vue-preferences (PR 18) and you can see a test pull request (PR 19) which adds a console.log statement to the library code which is then successfully built into the preview application 🎉.

Building awesome CLIs with JavaScript and Oclif

Federico Kauffman — Tue, 23 Apr 2019 19:14:45 +0000

Originally published in Streaver's blog.

Let's first define a CLI

A quick Google search yields, of course, a Wikipedia article with the CLI definition:

A command-line interface or command language interpreter (CLI), also known as command-line user interface, console user interface and character user interface (CUI), is a means of interacting with a computer program where the user (or client) issues commands to the program in the form of successive lines of text (command lines). A program which handles the interface is called a command language interpreter or shell (computing).

So, in a nutshell, a CLI is a program that can understand requests made by a user in the form of text and then act and execute code in response to that.

This kind of programs are very useful for many different use cases, from simple CLIs like the cal Bash tool that displays the current month, to extremely complex ones like kubectl for managing Kubernetes clusters.

Even if you don't use CLIs directly every day (which is very unlikely), you probably are being indirectly affected by them, for example, git is a CLI, gcc the GNU compiler, create-react-app a scaffolding CLI for generating React apps, and many more.

How to build your own CLIs

Like many things in the tech world the answer is: "it depends". There are many ways to build them and all of them are probably valid in different contexts. In this case, I am going to explore how to build one with JavaScript and Oclif: a Node.JS Open CLI Framework (by Heroku), which includes a CLI for building CLIs 🤔.

DANGER

From now on I will assume you are comfortable with JavaScript and the NPM ecosystem in general, if you are not, you will probably get a general idea, but I recommend you read something about that before starting 😃.

Getting started with Oclif

In my opinion, building something is usually a great way to learn, so in this case, I did some brainstorming with @flarraa and decided to build a "Copa Libertadores" CLI (see Wikipedia) .

The idea is to provide a set of commands that can retrieve and display information about the already played matches and upcoming ones for the "Copa Libertadores" championship.

Let's dig in!

The Oclif CLI, has two possible ways to generate CLI projects, one is npx oclif single mynewcli and the second one is npx oclif multi mynewcli, in this case, we are going to generate a multi-command CLI.

We would like our command to look like libertadores games:all, libertadores games:past, libertadores games:upcoming and this is consistent with Oclif's multi-command CLI generation.

Initializing the project

First, we initialize the project by doing:

npx oclif multi libertadores-cli

This will ask some questions and after that, it will install everything you need to start coding!

$ npx oclif multi libertadores
npx: installed 442 in 32.454s

     _-----_     ╭──────────────────────────╮
    |       |    │      Time to build a     │
    |--(o)--|    │  multi-command CLI with  │
   `---------´   │  oclif! Version: 1.13.1  │
    ( _´U`_ )    ╰──────────────────────────╯
    /___A___\   /
     |  ~  |
   __'.___.'__
 ´   `  |° ´ Y `

? npm package name libertadores-cli
? command bin name the CLI will export libertadores
? description A simple CLI to get information about "Copa Libertadores" right in your terminal
? author Federico Kauffman
? version 0.0.0
? license MIT
? Who is the GitHub owner of repository (https://github.com/OWNER/repo) streaver
? What is the GitHub name of repository (https://github.com/owner/REPO) libertadores-cli
? Select a package manager yarn
? TypeScript No
? Use eslint (linter for JavaScript) Yes
? Use mocha (testing framework) Yes
? Add CI service config circleci (continuous integration/delivery service)

I have selected some defaults I like and now you have a bunch of files and folders which will be our main structure for the project. Next enter the directory with cd libertadores-cli.

I am going to briefly explain what Oclif has generated for us:

.
├── README.md
├── bin
│   ├── run
│   └── run.cmd
├── package.json
├── src
│   ├── commands
│   │   └── hello.js
│   └── index.js
├── test
│   ├── commands
│   │   └── hello.test.js
│   └── mocha.opts
└── yarn.lock

5 directories, 9 files

Looking at the tree of files you can see the bin directory which contains the binaries to run the command on each platform (Unix/Windows).

You see the src folder with an index.js file which simply exports an internal Oclif package which will load the available commands, and those commands are defined in the files placed in the src/commands folder. By default, Oclif generates a hello command, let's run that and see what we have:

$ ./bin/run

A simple CLI to get information about "Copa Libertadores" right in your terminal

VERSION
  libertadores-cli/0.0.0 darwin-x64 node-v11.13.0

USAGE
  $ libertadores [COMMAND]

COMMANDS
  hello  Describe the command here
  help   display help for libertadores

If you run the hello sub-command you get:

$ ./bin/run hello

hello world from ./src/commands/hello.js

Last but not least, you have the tests folder where you will place all your tests, in fact, Oclif already created some tests, and we can run them with npm run test or yarn test.

Creating the first command

First, we can delete the hello command since we are not going to use it, simply delete the src/command/hello.js and tests/commands/hello.test.js.

Now we can use the Oclif CLI generator command, let's create the games:all command with:

npx oclif command games:all

This will create all the files needed for the command (including tests) and also will update the README.md file automatically to include the new command.

We are going to get the details for "Copa Libertadores" from http://www.conmebol.com/es/copa-libertadores-2019/fixture, and we are going to use puppeteer to enter the site and get the data.

$ yarn add puppeteer --save

const puppeteer = require("puppeteer");
...
class AllCommand extends Command {
  async run() {
    ...
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.goto(
      "http://www.conmebol.com/es/copa-libertadores-2019/fixture",
      { waitUntil: "load", timeout: 0 }
    );

    // Insert some really crazy code to parse the HTML
    // you can find this at https://github.com/streaver/libertadores-cli

    this.log(results);
  }
}

Now we can execute libertadores games:all and we will get the results right there on the terminal:

As you may have noticed I also added a "loading" feature to give the user some visual feedback. In order to add that, you simply install the package cli-ux and then wrap the "slow" parts of the code in some start/stop calls:

Install it like this:

yarn add cli-ux --save

Add the spinner with something like:

const { cli } = require('cli-ux');
...

cli.action.start('Fetching data');
//Do something that takes time
cli.action.stop();
...

Now, at this point we have the CLI, we can write tests for it! Oclif comes with some nice defaults for testing this kind of CLIs. In this particular case, you just want to test that the output to the terminal is what you expect. Fortunately, that is exactly what the auto-generated test for the command does, you only need to adapt that code!

I will leave this task to you (the reader, just like Math books) 🙄...or you can check them out in the official repository for the "Copa Libertadores" CLI.

Install the CLI, stay up-to-date and don't miss games anymore ❤️⚽!

Automating your infrastructure deployments

Federico Kauffman — Mon, 11 Jun 2018 14:56:41 +0000

Originally published in WyeWorks blog.

We all know that modern applications can get very complex, very quickly. Because of that, I've been trying to add more automation to my daily life in general, as I know I’m human and will eventually mess up due to being distracted, tired, or something else.

One of the first things I did while on my automation journey was to figure out a way to automate the creation of cloud resources. In order to accomplish that, I became proficient at using Infrastructure as Code (IaC) tools such as Terraform and AWS CloudFormation, among others.

After that, the next logical step was to integrate that knowledge in infrastructure provisioning into the CI/CD pipelines, since I have (or try to have) CI/CD pipelines setup in every project I’m involved in. In this post, I will present the benefits of doing just that and will demonstrate it using a specific Terraform + CircleCI example.

Benefits of automating infrastructure

If you’re not used to the concept of automatically deploying infrastructure, it might sound crazy, but in my opinion the benefits far outweigh the risks. The main benefits are as follows:

The deployment will be automatic.
You’ll tend to be more granular when making changes.
You’ll exercise more caution when making changes to your infrastructure and this will almost always lead to less downtime.
You will never forget to update your branch before deploying.
Your development and feedback loops will be shorter; you merge your changes and everything gets applied for you. No need to wait for an "Ops" team to help you out.
If you need to perform multiple steps for a deployment, you simply merge multiple PRs, one after another.
You can automate the testing of your infrastructure by adding some automated smoke testing following deployment (or beforehand if you use blue/green deployment).
You can invest more time in developing features.
All deployments can be tracked in your pipeline.

Risks of having automated infrastructure

As with everything in life, there are also cons to this approach. They are, however, easy to overcome with some team training or by simply adding some security mechanisms.

Let's review some of the risks:

Your team must be aware of the automated process. If you merge a PR, it’s because you wanted to merge it. If you accidentally merge code, it will get applied. To solve this potential problem, you can add a manual confirmation of the deployment for the production environment.
You might forget how to manually deploy. This is easy to solve - just manually perform the deployment process every now and then.

Now that we have seen the pros and cons of automating your infrastructure, let's go through a concrete example of how to do it.

CI/CD with Terraform and CircleCI

I created an example repository that includes some ideas which you can use to kickstart your project or make your current project better. The repository includes a detailed readme explaining the reasons why I made each decision.

The main purpose is to demonstrate only one of the many ways you could manage your infrastructure using an IaC tool like Terraform and a continuous integration tool such as CircleCI.

I have followed many of the best practices described in the book Terraform: Up & Running and the CircleCI 2.0 Documentation. I will also assume you have some knowledge of these tools and Amazon Web Services in general (though no need to be an expert).

Suggested workflow

Assuming you have two environments, production and staging: when a new feature is requested, you branch from staging, commit the code and open a PR to the staging branch. After that, CircleCI will run two jobs, one for linting and one that will plan the changes to your staging infrastructure so that you can review them (see image below).

Once you merge the PR, if everything goes as planned, CircleCI will run your jobs and automatically deploy your infrastructure!

After you have tested your infrastructure in staging, you just need to open a PR from staging to master to "promote" your infrastructure to production.

In this case, we want someone to manually approve the release to master, so after you merge you need to tell CircleCI that it can proceed and it will deploy the infrastructure after receiving confirmation.

Improvements

This is a very basic workflow and there are many things that must (or can) be improved upon. Here are a couple of them:

The most important improvement that needs to be made, in my opinion, is to add a way to test the infrastructure for each PR before merging them with staging. Perhaps something like the Heroku Review Apps would be best. If you use Terraform modules and you write them in a generic way, you could very easily implement infrastructure testing.
Add tests! This is of course a must in every project. If you want quality, you need to test. You can use tools like KitchenCI or InSpec to accomplish this.

Conclusions

I always like to present some final, very general takeaways in my articles, so here are a few:

Try to automate more. This will lead your team towards efficiency and make it less error prone.
Use IaC tools. Doing so will help with maintainability and understanding of all the pieces.
Find ways around problems; don't let problems mold you.

Infrastructure as Code: A beginner's perspective

Federico Kauffman — Thu, 08 Feb 2018 17:49:47 +0000

Originally published in WyeWorks blog.

A couple of months ago, I came to realize I might actually like DevOps. I wasn’t sure what to do, so I talked to the VP of Engineering at WyeWorks. Following that conversation, we found a way for me to work in a DevOps position for several months and try it out.

I left the project I was working on at that time and was given a month to learn some basic stuff so that I would be somewhat useful in a DevOps job. And that’s exactly what I did. Beginning January 2nd of this year, I’ve been reading up on and doing DevOps work 5 days a week, 7 hours a day.

In this post, I’m going to discuss my perspective on IAC (Infrastructure as Code), what problems it solves, and how it can be implemented. I will also share some resources for further learning, as well as briefly present Terraform, an IAC tool.

What does Infrastructure as Code mean?

As the name might suggest, IAC is about writing code that describes your infrastructure, similar to writing a program in a high level language. It allows you to automate the process of provisioning infrastructure using tools that are proven to work.

How does it help you?

IAC allows you to automate, test, and reuse code pertaining to the infrastructure you are working on. You also have the option to version control your infrastructure code and check it alongside the code that will be managed. You can likewise review and test code, and generally-speaking follow all the good practices you would normally with your codebase.

You might be thinking: Ok, that all sounds great, but can you please give me a concrete use case?

I can give you two. 😁

Case 1 - Simple web app_1, app_2, app_3, ..., app_n

Imagine you are working for a company that builds basic web apps for its clients. So, what does a basic web app require? A web server and a database (I am oversimplifying) should come in handy. Every time you gain a new client for a web app, you visit AWS (or whatever cloud provider you use) and, after clicking around some, you end up with 2 VMs: one for running your app, and another for the associated database.

Oh wait, I forgot, you’ll want at least two copies of this infrastructure: one for production and one for staging.

Now that you have two copies of the same infrastructure, you can start deploying the code for that client.

Two months go by and you have another client that needs the same infrastructure, so there you go again, clicking around in AWS to get that up and running.

Instead, utilizing IAC, you can have a module that does all the heavy lifting for you. When it’s time, you run that bit of code and in a couple of minutes you are good to go. Need to update all client infrastructure at once? No problem, just update the module, run it, and you’re all set!

If you have been doing this by hand in AWS, then you can probably already see the benefits of having IAC. But if you don't believe me just yet, read the next use case.

Case 2 - Going crazy with a complex app

You came up with a great idea - a really cool web app - and it all started off simply enough, but now you have a lot of clients that rely on that app. In this case, one web server and a single database won’t be enough.

You begin by adding a load balancer and another web server, and you document the process. Some time goes by and you need to add another server, so you follow the process in the document and you're done.

The following month, you realize that running only one database server is not enough and you add a replica.

Then, you need to move some of the workload off the main servers, which means you need a message queue and a server to run workers. You do it all by hand but forget to update the documents.

Some months go by and you get to a point where you need to add more queues and workers, but since you forgot to update the document the last time around, it’s not so easy to do it in the same way. You get it done, and it works, but now the queues and workers have slightly different configurations. This is known as configuration drift.

Later, you decide to add some caching levels to your app, and since you require Redis/Memcached, you’ll have to add more servers for those services. You’ll also need to configure S3 for your static files, a CDN to serve those worldwide, and Route53 for your DNS.

By this time, the company has grown and it’s time to create IAM roles and users for the new developers, then give them the correct access rights.

You also want increased reliability, so you look to allocate more servers in different datacenters. Finally, you go nuts and close the company.

The conclusion of this last example is admittedly a bit catastrophic, but imagine trying to maintain that number of servers manually. That would not scale well.

Should you be utilizing IAC?

As with everything in tech, the answer to this question is it depends. I personally can't imagine a project that would not be bettered through the use of IAC. In my opinion, if you have a chance to incorporate it, you should give it a try and find out for yourself!

Perhaps if you are already using something like Heroku for really simple apps, implementing IAC would create too much overhead. However, if your Heroku project grows beyond a certain size, you may benefit from this kind of tool as well. Take a look at Terraform: Heroku Provider, as an example.

Terraform

Terraform is an IAC tool from Hashicorp that solves problems presented by the previous cases and more. As they say on their site,

Terraform enables you to safely and predictably create, change, and improve infrastructure. It is an open source tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.

For example, you have a load balancer with 5 VMs attached to it. Run the following simple code to return to that same state every time.

resource "aws_elb" "frontend" {
  name = "frontend-load-balancer"
  listener {
    instance_port     = 8000
    instance_protocol = "http"
    lb_port           = 80
    lb_protocol       = "http"
  }

  instances = ["${aws_instance.app.*.id}"]
}

resource "aws_instance" "app" {
  count         = 5
  ami           = "ami-408c7f28"
  instance_type = "t1.micro"
}

That’s awesome! Imagine how many times you would have had to click to do all that by hand. Definitely quite a few!

Utilizing code similar to this, I was able to recreate the second case I presented earlier in just one week, which is pretty amazing! I’m sure it would have taken me at least twice as long to do it manually.

Conclusion

Your infrastructure is as important as the code it runs, so I recommend that you treat it the way it deserves to be treated; you should complete reviews, versioning, tests, and so on.

Managing infrastructure by hand is difficult, even if you are disciplined and document everything. After all, you are only human - you could forget something someday, and that day you will regret not using IAC tools.

You can get started by managing something simple in your app with an IAC tool, then gradually migrate other things over.

If you have the opportunity to use Terraform or any other IAC tool, consider it a great way to improve how you manage your infrastructure. It can enable you to do great things quickly!

If you have questions or comments, please post them below!

Resources

Infrastructure as Code: Managing Servers in the Cloud by Kief Morris. Link
Terraform: Up & Running By Yevgeniy Brikman. Link

Is a Master's/PhD degree worth the effort/money in the Software Engineering universe?

Federico Kauffman — Thu, 18 Jan 2018 12:19:21 +0000

I recently got my Engineering degree, and immediately after that I started wondering if I should invest in a postgraduate degree.

What do you think? Is is worth the money/time?

Let's discuss this topic :)