Forem: Demangeon Julien

Programming A Space Invader In OCaml and OpenGL: Lessons Learned

Demangeon Julien — Fri, 21 Feb 2020 14:17:43 +0000

Note: This post was originally posted on marmelab.com.

During a previous hackday, I had the opportunity to try a fairly recent programming language called ReasonML. Amazed by the simplicity and power of ReasonML, I promised myself to get closer to its underlying language to give credit where it is due. Do ReasonML qualities come from OCaml?

As an experiment, I took this opportunity to test OpenGL and prove that it is possible to code a humble game in OCaml! Here's what the game looks like at the end, after a few hours of development.

The purpose of this article is to determine the magnitude of the differences between ReasonML and OCaml, and to understand why ReasonML has become so popular so quickly, and why OCaml hasn't. The aim is also to determine if it is still useful to use OCaml rather than ReasonML for future projects.

OCaml, Powerful But Discreet

OCaml is the main implementation of Caml created in 1996 by a group of researcher from the French Institute for Research in Computer Science and Automation (a.k.a. INRIA).

Even though it is known for its robustness and its large standard library, its main power lies in its strong static-typing and type inference system.

Although part of the ML (Meta-Language) family that is purely functional-oriented by nature, OCaml brings many object-oriented features that make it special. That's mainly the reason it was named Objective Caml (aka OCaml).

So, it's perfectly possible to work with Classes, Objects, Constructors (named "Initializers"), Inheritance and so on.

OpenGL, The Swiss Army Knife of Rendering

OpenGL (for Open Graphics Library) is a cross-platform and cross-language API that allows to render 2D and 3D vector graphics. It takes care of handling interactions between the source code and the system GPU to benefit from hardware rendering capabilities.

OpenGL was developed in 1992 by an old and famous commercial company called Silicon Graphics. Since 2006, the project has been managed under a BSD like license by a non-profit consortium called the Khronos Group. This consortium is also responsible of the WebGL and Vulkan standards that are very popular today.

Vulkan is usually refered to as the OpenGL next API for graphics rendering.

Practical Case: A Space Invader in OCaml

In this part, I'll give you the most objective feedback about my development experience using OCaml and OpenGL on a concrete project: the Space Invader game.

Also, I'll explain the intellectual journey that led me to design the project foundation. Then, I'll describe the different stages of project building.

How Does it Work? What Do We Need?

As for most programming languages, we need to setup a compiler which is capable of transforming our OCaml source code to a suite of machine instructions.

According to the official documentation, the OCaml core distribution provides two compilers, ocamlc and ocamlopt.

ocamlc is the base standard bytecode compiler for OCaml. It produces bytecode files (usualy *.cmi for interfaces and *.cmo files for implementations) that can be run using ocamlrun, the OCaml bytecode runner.
ocamlopt is the native code compiler for OCaml. As for ocamlc, it can take *.ml (and *.mli) files as argument and generates an executable (targeting specific architectures according to the provided parameters).

Of course, I do not pretend to give you here the full functionalities of these tools, which are in fact much more complicated to understand.

I can already imagine your faces after reading these few lines... but don't go away, I assure you that the rest will be much more exciting!

Project Setup

In practice, built-in compilers are almost never directly used on real world projects because they require a lot of work to setup and maintain. It is common to use higher level tools that also facilitate the import of dependencies (including external modules). Among these tools, we can mention Dune, Oasis and OCamlBuild.

In my case, I choose a more undisclosed tool (obuild) because of its simplicity and its brief documentation. So, the first step is to install the OCaml Package Manager (aka opam) in order to install the obuild builder.

As for every new projet, my approach consists in setting up the whole tech stack in Docker so as to avoid overloading my workstation, and to allow other people to participate easily. Therefore, I started looking for a Docker Image that includes opam on docker-hub.

Thanksfully, the ocaml/opam2 docker image exists, here is my Dockerfile configuration.

FROM ocaml/opam2:debian-10-ocaml-4.08

RUN sudo mkdir /app
WORKDIR /app
ADD . /app/

RUN sudo apt-get update

RUN opam install obuild

Since I need to work with OpenGL to draw the game, and I won't develop an OpenGL library from scratch, I'll install lablgl at the same time by adding this line at the end of the Dockerfile:

RUN opam depext lablgl

lablgl is one of the most advanced OpenGL interfaces for OCaml. Despite its lack of documentation, it remains the most talked about library after several minutes spent on search engines...

Lablgl is based on Glut (The OpenGL Utility Toolkit). Glut takes care of all the system specific mechanisms to create windows, initializing OpenGL and capture input events.

Now I have all the ingredients, I need to build the application. I create my entry point file in src/main.ml.

.
├── Dockerfile
├── ocaml-invader.obuild
└── src
    └── main.ml

Then, I create the obuild config file ocaml-invader.obuild.

name: ocaml-invader
version: 0.0.1
description: Simple OCaml Invader Game
licence: MIT
authors: Julien Demangeon <julien@marmelab.com>
obuild-ver: 1
homepage: http://github.com/marmelab/ocaml-invader

executable ocaml-invader
    main-is: main.ml
    src-dir: src
    buildDepends: lablgl, lablgl.glut

And a Makefile to facilitate development.

BIN = docker run -i -t --rm \
    -v "${PWD}:/app" \
    ocamlinvader

install: ## Install dependencies
    docker build --tag=ocamlinvader .

build: ## Build project
    $(BIN) bash -c "obuild configure && obuild build"

clean: ## Clean project
    $(BIN) bash -c "obuild clean"

start: ## Start project
    $(MAKE) build
    ./ocaml-invader
    $(MAKE) clean

OK, everything is set, I can now start developing the game!

Game Architecture

As with most video games, the whole system is based on a game state, some state mutations (through system inputs), and a rendering engine / loop.

Fortunately, I won't have to manage all these components by myself. Indeed, labgl (through its Glut module) offers most of these features out of the box:

The windowing system through Glut.createWindow
The rendering loop through Glut.mainLoop, Glut.displayFunc and Glut.idleFunc
Input management through Glut.keyboardFunc and Glut.specialFunc
The timers / tickers through Glut.timerFunc

Therefore, I only have to care about:

The rendering of the game in OpenGL (through the Glmat and Gldraw modules)
Changing the game state from the input actions

Main Program

Now that the basis of the architecture is established, let's get down to business by expanding the main function in main.ml:

(* main.ml *)

(* Declare rendering function, buffering mode, and create window *)
let initDisplay ~w ~h ~title =
  Glut.initDisplayMode ~double_buffer:true ~depth:true ~alpha:true ();
  Glut.initWindowSize ~w ~h;
  Glut.createWindow ~title;
  Glut.idleFunc ~cb:(Some Glut.postRedisplay)

(* Initialize OpenGL rendering options *)
let initView ~w ~h =
  GlDraw.viewport ~x:0 ~y:0 ~w ~h;
  GlMat.mode `projection;
  GlMat.load_identity ();
  GluMat.ortho2d ~x:(0.0, float_of_int(w)) ~y:(0.0, float_of_int(h));
  GlMat.mode `modelview

(* Create some tickers that send actions to the Game Controller *)
(* The GameController uses these actions to mutate the game state each N ms *)
let initTickers ~game =
  Glut.timerFunc ~ms:1000 ~cb:(invaderTicker game) ~value:0;
  Glut.timerFunc ~ms:10 ~cb:(bulletTicker game) ~value:0;
  Glut.timerFunc ~ms:10 ~cb:(collisionCheckTicker game) ~value:0

(* Map keyboard inputs to game actions *)
(* The GameController uses these actions to mutate the game state *)
let initInputs ~game =
  Glut.keyboardFunc ~cb:(gameController game specialKeyToAction);
  Glut.specialFunc ~cb:(gameController game commonKeyToAction)

(* Initialize each part of the game engine *)
(* Then, it returns a callable mainLoop *)
let initEngine ~game ~w ~h =
  initDisplay ~w ~h ~title: "OCaml Invader";
  initView ~w ~h;
  initTickers ~game;
  initInputs ~game;
  Glut.displayFunc (fun () -> Game.render !game);
  Glut.mainLoop

(* This is the first function executed by OCaml *)
(* We init the game object reference and pass it to the engine *)
let () =
  ignore @@ Glut.init Sys.argv;
  let game = ref (Game.init()) in
  let run = initEngine ~game ~w:450 ~h:600 in
    run()

Zoom In The Game Controller

Mutations on the game state are managed by the game controller. The game controller is a simple function that associates an action (with pattern matching on it) to a mutation on the game state (the whole game being declared mutable).

(* game.ml *)

type direction =
    | Left
    | Right

type action =
    | Move of direction
    | Shoot
    | AdvanceInvaders
    | InvaderShoot
    | AdvanceBullets
    | CheckCollisions

type game = {
    mutable over: bool;
    mutable score: int;
    mutable lifes: int;
    mutable invaders: Invader.invader list;
    mutable invadersDirection: direction;
    mutable spaceship: Spaceship.spaceship;
    mutable bullets: Bullet.bullet list;
}

let controller game = function
    | Move(direction) ->
        let op = match direction with
        | Left -> (-.)
        | Right -> (+.) in
        let coord = op game.spaceship.x 10. in
        game.spaceship.x <- min (max coord 10.) 440.;
        game
    | Shoot ->
        (* ... mutate game ... *)
        game
    | InvaderShoot ->
        (* ... mutate game ... *)
        game
    | CheckCollisions ->
        (* ... mutate game ... *)
        game
    | AdvanceInvaders ->
        (* ... mutate game ... *)
        game
    | AdvanceBullets ->
        (* ... mutate game ... *)
        game
    | _ -> game

The link between the game controller and actions (tickers and keyboard inputs) is declared in the gameController function of the Main module.

(* main.ml *)

let gameController game keyToAction = fun ~key ~x ~y ->
  match (keyToAction ~key ~x ~y) with
    | Some(action) -> game := Game.controller !game action
    | None -> ()

Rendering Logic

The entire rendering logic for the game is located in the Game module of the game.ml file. This logic is composed of a switch between two screens, a game screen, and an end screen depending on whether the game is finished or not.

(* game.ml *)

let renderHome game =
    GlClear.clear [ `color ];
    let endText = match game.score with
        | 600 -> Printf.sprintf "YOU WIN"
        | _ -> Printf.sprintf "GAME OVER" in
    Utils.drawString ~font:Glut.BITMAP_TIMES_ROMAN_24 150. 290.  endText;
    Glut.swapBuffers ()

(* Each module (Score, Life, Invader, Bullet, ...) has its own render *)
(* So we clear buffer, draw the game and swap buffers to render it to the screen *)
let renderGame game =
    GlClear.clear [ `color ];
    Score.render game.score;
    Life.render game.lifes;
    List.iter Invader.render game.invaders;
    List.iter Bullet.render game.bullets;
    Spaceship.render game.spaceship;
    Glut.swapBuffers ()

let render game =
    match game.over with
        | false -> renderGame game
        | true -> renderHome game

Each component of the game (Invader, Spaceship, ...) has its own module and its own render method, which makes it possible to separate the responsibilities in the code. For example, here are some parts of the Invader rendering.

(* invader.ml *)

type race =
    | Octopus
    | ShapeShifting
    | Crab

type invader = {
  mutable x: float;
  mutable y: float;
  race: race;
}

let renderOctopus () =
    GlDraw.color(1., 1., 1.);
    GlDraw.begins `quads;
    List.iter GlDraw.vertex2 [-10., -10.; -10., 10.; 10., 10.; 10., -10.];
    GlDraw.ends ()

let renderCrab () =
    GlDraw.color(1., 0., 1.);
    GlDraw.begins `quads;
    List.iter GlDraw.vertex2 [-10., -10.; -10., 10.; 10., 10.; 10., -10.];
    GlDraw.ends ()

let renderShapeShifting () =
    GlDraw.color(1., 0., 0.);
    GlDraw.begins `quads;
    List.iter GlDraw.vertex2 [-10., -10.; -10., 10.; 10., 10.; 10., -10.];
    GlDraw.ends ()

let render invader =
    GlMat.load_identity ();
    GlMat.translate3(invader.x, invader.y, 0.0);
    match invader.race with
        | Octopus -> renderOctopus()
        | ShapeShifting -> renderShapeShifting()
        | Crab -> renderCrab()

OpenGL In Practice

To summarize quickly, OpenGL is a large state machine on which it is possible to send instructions. Once all rendering instructions have been sent to it, it is possible to trigger the on-screen display using Buffer flush.

For each rendering, it may be necessary to:

Modify state variables (anti-aliasing enabled, ...)
Define the textures (aka Materials) to be used for the display
Send geometrical instructions (aka Vertex) to the Buffer
Trigger the Buffer flush

In the example above, for each invader I:

Reset the rendering matrix state with GlMat.load_identity()
Place cursor at the invader position with GlMat.translate3(inv.x, inv.y, 0.0);
Define a color to be used with GlDraw.color
Declare that I'll draw a quadrilateral with GlDraw.begins `quads
Send 4 vertices with GlDraw.vertex2
Close the Invader figure (a square in my case) with GlDraw.ends

In the case of the invaders, I used simple squares for lack of time. But I know that it is possible to make much more complex shapes, as in the Spaceship below:

type spaceship = {
    mutable x: float;
    mutable y: float;
}

let renderAt ~x ~y =
    GlMat.load_identity ();
    GlMat.translate3(x, y, 0.);
    GlDraw.color(0.51, 1., 0.);
    GlDraw.begins `quads;
    List.iter GlDraw.vertex2 [-20., -5.; -20., 5.; 20., 5.; 20., -5.];
    List.iter GlDraw.vertex2 [-2., 5.; -2., 14.; 2., 14.; 2., 5.];
    GlDraw.ends ()

let render spaceship =
    renderAt ~x:spaceship.x ~y:spaceship.y

The Result

As a result, the game is very fast, no lag and the compilation time is almost instantaneous, what more could you ask for? If you are interested in the code (or want to play), the game is available in open-source on Github.

Conclusion

Although confused at first, I am very proud to have achieved this project. This was an opportunity for me to measure the particularities that makes OCaml so intimidating for most developers.

Indeed, it is very complicated to install the base stack to develop in OCaml. And although some tools exist to make our life easier, they are poorly documented. This lack of documentation also applies to the language itself. The documentation is scattered among many academic and research-oriented sites, making it very intimidating for a beginner.

Clearly, it is in OCaml's weaknesses that we find the strength of ReasonML. The languages are not so different, but their communities and ways of working are clearly in contrast.

In my opinion, I would say that OCaml suffers from its original academic use, and that it deserves a little more love from mainstream developers. It is incredibly fast to compile and run, and it competes well with popular languages.

To conclude, I would still advise to prefer ReasonML over OCaml (as its community keeps growing). This way, you will adopt a functional way of coding that can be used in other derived languages such as F# or Scala.

Build Your Own Design System With StencilJS

Demangeon Julien — Fri, 06 Dec 2019 13:39:20 +0000

Note: This post was originally posted on marmelab.com.

In this post, we will define what StencilJS is and how we can use it to build a Design System. As a pre-requisite, you should know that StencilJS is not a framework. It's a compiler that produces reusable Web Components that can be embedded anywhere else.

What is a Design System?

Many publishers have a graphic charter defining a set of rules about their visual identity.

Given the technological developments in our societies, these rules increasingly and largely concern digital media, particularly the web.

In the last few years, a consensus has emerged in the world of web development. Indeed, most javascript frameworks are largely based on components. Components offer many advantages such as logic reusability, centralized style and easy testing. Their use provides a rich and uniform browsing experience far away from the early 21th century.

As a result, a new trend has emerged, merging the principles of graphical charter and standalone UI components: that's the Design System.

What Is StencilJS?

As previously said, StencilJS is not another web framework such as ReactJS or VueJS. StencilJS is a toolchain that facilitates the building of reusable and scalable Design Systems based on the Web Component standard. It was created by the Ionic Framework team especially for this purpose.

Although totally different from a frontend framework, StencilJS uses many well-known concepts and technologies from frontend development - and that's often confusing for a developer. So, StencilJS provides:

A virtual DOM
JSX (like in ReactJS)
Reactive Data (like AngularJS $watch)
Async rendering (inspired by React Fiber)
TypeScript support

By combining all these features, StencilJS is able to generate standard-compliant components. Moreover, StencilJS automatically adds the polyfill required to support older browsers. Here is the browser support grid provided by the StencilJS website.

So, to sum up, using StencilJS allows you to build a Design System that is ready for the future thanks to its compliance with upcoming standards, its automatic polyfill and its advanced API.

A Simple Case Study

In this section, I will detail the creation of a Design System from scratch using StencilJS. For the example, I will create a design system for Marmelab called mml. Here is the initialisation process below.

Create a component

Once our project is set up, we can create our first component, in our case a "Github Card". StencilJS provides a special npm script command for that purpose, generate.

julien@julien-P553UA:~/Projets/marmelab/mml$ npm run generate
> mml@0.0.1 generate /home/julien/Projets/marmelab/mml
> stencil generate

✔ Component tag name (dash-case): … github-card
✔ Which additional files do you want to generate? › Stylesheet, Spec Test, E2E Test

$ stencil generate github-card

The following files have been generated:
 - src/components/github-card/github-card.tsx
 - src/components/github-card/github-card.css
 - src/components/github-card/github-card.spec.ts
 - src/components/github-card/github-card.e2e.ts

StencilJS has generated all the required files to create our component, even the test files! Here are the generated files from that command (excluding tests).

// ./src/components/github-card/github-card.css

:host {
  display: block;
}

// ./src/components/github-card/github-card.tsx

import { Component, Host, h } from '@stencil/core';

@Component({
  tag: 'github-card',
  styleUrl: 'github-card.css',
  shadow: true
})
export class GithubCard {
  render() {
    return (
      <Host>
        <slot></slot>
      </Host>
    );
  }
}

Now you might think: We're supposed to use StencilJS, how come this command has generated Angular code? Well, although it looks like Angular code, it's actually StencilJS. And I agree, the annotations at the beginning of the module can be confusing. But don't worry, you're on the right track ;)

So, the @Component annotation allows us to declare a StencilJS component, which can be configured using several options:

tag: name of the tag on which our component will be registered.
styleUrl: relative url to the corresponding style file
shadow: enable the browser Shadow DOM encapsulation
other options...

In the render method, the Host component represents the root of the component, the tag itself. Within it, the slot allows to inject children in our custom element as described below.

<github-card>
   <div>I'm a children, and i'll replace the <!-- <slot>--></div>
</github-card>

Since we use Shadow DOM, only the styles inside our github-card.css tag will have an impact on the display of our github-card. The special :host selector refers to the tag itself (aka Host).

Then, our component exists and we're able to see it (not much at the moment) in the browser using yarn start (aka stencil build --dev --watch --serve).

If we inspects the DOM using chrome devtools, this is what we'll see.

After a few minutes in the shoes of an integrator, here is what I get from some html and css. Sadly, everything is static for the moment, and I have much less than 42 followers in real-life ^^.

In the next chapter, I'll explain how to configure which user is displayed using a special login attribute on our custom element.

<github-card login="jdemangeon"></github-card>

Passing Props To A Component

Like ReactJS, StencilJS provides state, props and lifecycle hooks. So, the first step is to declare a login prop and an user state in our component. Whereas login will receive the name of the GitHub user, user will receive the user object coming from the GitHub API.

- import { Component, Host, h } from "@stencil/core";
+ import { Component, Host, h, Prop, State } from "@stencil/core";


@Component({
  tag: "github-card",
  styleUrl: "github-card.css",
  shadow: true
})
export class GithubCard {
+  @Prop() login: string;
+  @State() user: any;

  render() {
    ...
-     <a class="avatar" href={`https://github.com/jdemangeon`}>
+     <a class="avatar" href={`https://github.com/${this.login}`}>
    ...
  }
}

Contrary to React, there's no this.props object to access props with StencilJS. Prop values are directly attached to the this instance. So we can access login using this.login.

So, our links are up to date, and point to the right profile according to the login prop. Other data such as the number of followers, repositories, etc... are not dynamic and require a call to the Github API.

External And Internal APIs

I don't really want to expose my GitHub popularity in the spotlight... But it's the lottery, honey! So, in this section we'll explore how to call the GitHub API from the component, and display real values based on the response.

First, we will declare a function that calls Github and assigns values to our user. And in a second step, we will call this function when mounting the component.


export class GithubCard {
  @Prop() login: string;
  @State() user: any;

+ async componentWillLoad() {
+   return this.fetchUser(this.login);
+ }

+ async fetchUser(login: string) {
+   const response = await fetch(`https://api.github.com/users/${login}`);
+
+   if (response.status === 200) {
+     this.user = await response.json();
+   } else {
+     this.user = null;
+   }
+ }

Then, we are able to use the user information inside the render() method. If the user doesn't exist or is not retrieved yet, we'll return null.

  render() {
+   if (!this.user) {
+     return null;
+   }

    return (
      <Host>
        <div class="card">
          <div class="header" />
          <a class="avatar" href={`https://github.com/${this.login}`}>
-           <img src="https://avatars0.githubusercontent.com/u/1064780" alt={this.login} />
+           <img src={this.user.avatar_url} alt={this.login} />
          </a>
          <div>
-           <h1>Julien Demangeon</h1>
+           <h1>{this.user.name}</h1>
            <ul>
              <li>
                <a
                  target="_blank"
                  href={`https://github.com/${this.login}?tab=repositories`}
                >
-                 <strong>42</strong>Repos
+                 <strong>{this.user.public_repos}</strong>Repos
                </a>
              </li>
              ...

Just like props, state attributes are directly attached to the instance through this. So you have to be very careful with naming to avoid conflicts between state and props.

So yeah, it works. But what happens if I update the login attribute?

- <github-card login="jdemangeon"></github-card>
+ <github-card login="marmelab"></github-card>

Well, my component doesn't change... but why? Because the componentWillLoad lifecycle method which is used is called only once, just before the component mounts to the DOM. To reflect prop (and state) changes, we must implement componentWillUpdate, too.

export class GithubCard {
  @Prop() login: string;
  @State() user: any;

  async componentWillLoad() {
    return this.fetchUser(this.login);
  }

+ async componentWillUpdate() {
+   return this.fetchUser(this.login);
+ }

componentWillLoad, componentWillUpdate and componentWillRender lifecycle methods are special. They can return a Promise that can be used to wait for the next render.

YES! Our component no shows user information, and changes when the login prop changes. What if we want to allow developers to display other information? To achieve this goal, we'll use a slot.

   return (
      <Host>
        <div class="card">
          <div class="header" />
          <a class="avatar" href={`https://github.com/${this.login}`}>
            <img src={this.user.avatar_url} alt={this.login} />
          </a>
          <div>
            <h1>{this.user.name}</h1>
+           <slot />
            <ul>
              <li>
              ...

A little change in the html...

- <github-card login="jdemangeon"></github-card>
+ <github-card login="jdemangeon">
+   <span>I like Pastis!</span>
+ </github-card>

And tada! Isn't life beautiful? In this case there's only one slot, but you need to be aware that multiple slots can be used thanks to named slots.

In reality, if you apply the example as it is, you will see the text "I like pastis" blinking before the rest of the component. This is quite normal, since we use "normal" html tags and because the browser will show the text before javascript execution.

To avoid this problem, StencilJS will apply an hydrated class on the components once they are mounted. So it's necessary to declare the style below to avoid the problem.

    <style type="text/css">
      github-card {
        display: none;
      }
      github-card.hydrated {
        display: block;
      }
    </style>

If you've already used VueJS, you probably already known the v-cloak attribute, which is based on the same principle. The same applies to AngularJS with ng-cloak.

The StencilJS API provides a lot of features that would be difficult to cover completely here. I can't recommend enough that you take a look at the official documentation, which is very extensive.

Composing Components

We have built a standalone component. How can we interact with it from other components? We're going to create a user selector, which allows us to change the user in the card.

So, we use the same generate command with a github-card-selector this time. And here is the finalized component below.

// src/components/github-card-selector/github-card-selector.tsx

import { Component, Host, h, State } from "@stencil/core";

@Component({
  tag: "github-card-selector",
  styleUrl: "github-card-selector.css",
  shadow: true
})
export class GithubSelector {
  @State() login: string;

  handleLoginChange(e: UIEvent) {
    const target = e.target as HTMLInputElement;
    this.login = target.value;
  }

  render() {
    return (
      <Host>
        <input
          onChange={this.handleLoginChange.bind(this)}
          placeholder="Github username"
        />
        {this.login && <github-card login={this.login} />}
      </Host>
    );
  }
}

As we can see, the code is simple enough to be readable. However, it presupposes that the github-card html tag is already registered on the browser (thankfully, StencilJS takes care of that when the mml library is loaded). There is no explicit import. Of course, since we use TypeScript, everything is typed, even UIEvent and HTMLInputElement browser events.

You can find the project sources at the following address: https://github.com/marmelab/mml.

Component Testing

There are 2 different test types in StencilJS: Unit Tests and End-To-End (e2e). StencilJS uses Jest for unit tests and Puppeteer for e2e tests. If you already know these two excellent tools, you won't be lost.

Thanks to Puppeteer, it is possible to fine-tune the configuration of browser tests. This includes touch management, or landscape mode, up to viewport emulation. Using unit tests, it is also possible to mock many features such as HTTP Referer, Cookies, Url, and so on.

In fact, I didn't find any significant difference between these two test modes. One is just slower because it passes through Puppeteer, but does not bring any added value. I think it's better to focus on e2e tests once the component is integrated into the final application.

For more information on testing with StencilJS, the documentation can be found here.

Framework interoperability

As already explained, StencilJS is not a framework, it's just a web-components compiler. Although dedicated to this unique task, StencilJS's underlying objective is to enable end-to-end development of web applications based on the new standards.

It's still difficult to build a complex application entirely with StencilJS. For this reason, StencilJS provides a set of functions that allow components to be injected directly into existing web applications. Here is an example with ReactJS below.

import React from 'react';
import ReactDOM from 'react-dom';
import App from './App';

import { applyPolyfills, defineCustomElements } from '<your-design-system-lib>/loader';

ReactDOM.render(<App />, document.getElementById('root'));

applyPolyfills().then(() => {
  defineCustomElements(window);
});

Not all frameworks allow easy integration of custom elements (web-components). A website lists the compatibility rate for each of them.

Design System Target

One of the greatest feature of StencilJS is that it can generate various builds at the same time to cover all expected needs and targets.

So, StencilJS can generate both the ES5 and ECMAScript Modules (esm) versions for each component. It can also generate the corresponding documentation in markdown or json.

We can configure that targets through the stencil.config.js this way.

import { Config } from '@stencil/core';

export const config: Config = {
  namespace: 'mml',
  outputTargets: [
    {
      type: 'dist',
      esmLoaderPath: '../loader'
    },
    {
      type: 'docs-readme'
    },
    {
      type: 'www',
    }
  ]
};

From the www output target folder, we can directly test our component by serving it. You can test it directly in the original blog post. Type your Github username, press Enter, and Voilà!

Here is the output target documentation.

Conclusion

StencilJS cannot replace web frameworks such as ReactJS or VueJS. Indeed, Web Components are unable to receive complex attribute data (aka prop in component terms). Like any html tag, they can only receive scalar / textual data (aka attribute). This makes their use very limited.

Some have used hacks to get around this limitation, but I'm not sure the game is worth it.

Therefore, StencilJS is a very good choice if you want to create graphical components without logic (dumb components) in an application or if you want to create widgets that can be integrated out of context everywhere.

CouchDB, The Open-Source Cloud Firestore Alternative?

Demangeon Julien — Wed, 25 Sep 2019 14:29:47 +0000

Note: This post was originally posted on marmelab.com.

During one of our last customer projects, we used Firebase, a backend-as-a-service by Google, as our backend. Althought satisfied by this "all-included" suite as a whole, we remained disappointed by its proprietary aspect.

That's why I took the initiative to look for an open-source alternative to Firebase that could cover all our needs without depending on a third party service.

The first step in this quest is to find a substitute to the Cloud Firestore realtime NoSQL database for the web.

What Do We Need?

Using Firestore rather than a more classic database is not trivial. It often results from the need to quickly develop an application with the following features:

Offline First, client writes to local database that is synchronized with remote one
Realtime, remote changes must be in sync with our local database

Some solutions exist to cover this need, most of them are based on NoSQL databases such as MongoDB, Cassandra, RethinkDB, Gun or others MongoDB based solutions like Minimongo, turtleDB or tortoiseDB.

In our case, we're going to give CouchDB (and PouchDB for the frontend) a try, because it's the more robust and best known solution from our point of view.

CouchDB & PouchDB

CouchDB is an open-source / cross-platform document oriented database software. It is developed on the basis of the concurrency-oriented Erlang language, allowing it to benefit from an high scalability. It uses JSON to store its data, and an HTTP API to expose it.

CouchDB was born in 2005. Since 2008, CouchDB became an Apache Software Foundation project, which allows it to benefit from a lot of support and a large communauty.

Here are the main features of CouchDB:

Multi-Version Concurrency Control (which lets you build offline-first solutions easily)
Distributed Architecture with Replication
Document Storage
HTTP / REST API

Since CouchDB runs on the server, many client library allows to communicate with it thanks to the HTTP interface it offers.

The most known CouchDB client library for the web is called PouchDB. PouchDB is an open-source Javascript database that is designed to run within the browser. This way, it allows to store data locally offline, and sync it with the remote CouchDB server when the user comes back online.

CouchDB & PouchDB in Practice

Enough introduction, let's get practical! In this section, I'll describe the development of a ReactJS application using CouchDB and PouchDB as database system, step by step. Meanwhile, I'll try, as much as I can, to compare the CouchDB implementation to the Firestore one.

Also, I'll present you some of my latest loves in terms on Javascript librairies: Final-Form, ElasticUI and Indicative.

In this project, I'm going to create a beer registry, which allows users to keep track of their beer stocks.

Project Setup

In order to keep this tutorial as simple as possible, I'll create a ReactJS application using create-react-app.

create-react-app reactive-beers && cd reactive-beers

npm install -S pouchdb

The application skeleton looks like the following:

julien@julien-P553UA:~/Projets/marmelab/reactive-beers$ tree -L 1
.
├── node_modules
├── package.json
├── package-lock.json
├── public
├── README.md
└── src

Then, since I don't want to install CouchDB directly on my machine, I'll use Docker. So, the first step is to configure a docker-compose.yml file and the associated Makefile to improve developer experience.

// ./docker-compose.yml

version: "2.1"

services:
  couchdb:
    image: couchdb:2.3.0
    ports:
      - "5984:5984"

  node:
    image: node:10
    command: npm start
    working_dir: "/app"
    volumes:
      - ".:/app"
    ports:
      - "4242:3000"
    depends_on:
      - couchdb

# ./Makefile

USER_ID = $(shell id -u)
GROUP_ID = $(shell id -g)

export UID = $(USER_ID)
export GID = $(GROUP_ID)

DOCKER_COMPOSE_DEV = docker-compose -p reactive-beers

help: ## Display available commands
    @fgrep -h "##" $(MAKEFILE_LIST) | fgrep -v fgrep | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'

install: ## Install docker stack
    $(DOCKER_COMPOSE_DEV) run --rm node bash -ci 'npm install'

start: ## Start all the stack
    $(DOCKER_COMPOSE_DEV) up -d

stop: ## Stop all the containers
    $(DOCKER_COMPOSE_DEV) down

log: ## Show logs
    $(DOCKER_COMPOSE_DEV) logs -f node

So, we're now ready to start our complete stack using make install start.

julien@julien-P553UA:~/Projets/marmelab/reactive-beers$ docker ps
CONTAINER ID        IMAGE            COMMAND                  CREATED       STATUS       PORTS                                        NAMES
6884f92c5341        node:10          "npm start"              3 hours ago   Up 3 hours   0.0.0.0:4242->3000/tcp                       reactive-beers_node_1
21897f166ce4        couchdb:2.3.0    "tini -- /docker-ent…"   3 hours ago   Up 3 hours   4369/tcp, 9100/tcp, 0.0.0.0:5984->5984/tcp   reactive-beers_couchdb_1

Everything is launched. You may have noticed that the 5984 port is exposed in our docker-compose.yml file, it's the CouchDB api. Then, if you open localhost:5984 in the browser, you'll see something similar to the following.

{
    "couchdb": "Welcome",
    "version": "2.3.0",
    "git_sha": "07ea0c7",
    "uuid": "49f4e7520f0e110687dcbc8fbbb5409c",
    "features": ["pluggable-storage-engines", "scheduler"],
    "vendor": {
        "name": "The Apache Software Foundation"
    }
}

Accessing The Document Store

OK, our server is up and running. But, is there an interface to visualise / supervise CouchDB just like Firestore does? The answer is YES! CouchDB already includes an administration interface called Fauxton. We can browse it at http://localhost:5984/_utils/.

Firestore Admin Interface

Fauxton Admin Interface

The Fauxton interface allows to access databases, setup nodes and clusters, configure replication, setup permissions, etc. Although practical, it is still preferable to automate these administration tasks with dedicated scripts.

React Kicks In

Now, we can start to develop our first PouchDB powered interface. Then, here are our main App.js entry point and the Home.js start screen.

// ./src/App.js

import React from 'react';
import { Home } from './screens/Home';

const App = () => <Home />;

export default App;

The App.js file has no interest for the moment. It'll certainly become useful when we need to add more routes and screens in the future.

// ./src/screens/Home.js

import React, { useState, useEffect } from 'react';
import { addBeer, getBeers, onBeersChange } from '../api/beers';

export const Home = () => {
  const [beers, setBeers] = useState([]);

  const refreshBeers = () => getBeers().then(setBeers);

  useEffect(() => {
    // We fetch beers the first time (at mounting)
    refreshBeers();

    // Each change in our beers database will call refreshBeers
    const observer = onBeersChange(refreshBeers);
    return () => {
        // Don't forget to unsubscribe our listener at unmounting
        observer.cancel();
    };
  }, []);

  return (
    <div>
      <button onClick={() => addBeer({ title: 'Beer X' })}>Add a beer</button>
      <ul>
        {/* beer._id is an unique id generated by CouchDB */}
        {beers.map(beer => <li key={beer._id}>{beer.title}</li>)}
      </ul>
    </div>
  );
};

CouchDB Requires More Requests Than Firestore

As you see, in this example we use a combination of a listener (onBeersChange) and a a query (getBeers) to get the initial beers list and refresh it when a change is issued in the database.

This operation is not optimal compared to the one Firestore offers. Indeed, while pouchdb is not able to return both changes and data to us, Firestore is able to do so thanks to a QuerySnapshot system, thereby reducing server trips back and forth. See by yourself with the Firestore example below:

  db.collection("anything")
    .onSnapshot(function(querySnapshot) {
        querySnapshot.forEach(function(doc) {
          // This forEach loop is executed at first execution
          // And executed each time the query result changes
        });
    });

So, if we had used Firestore instead, here's what it would have looked like:

  //...

  const [beers, setBeers] = useState([]);

  useEffect(() => {
    const unsubscribe =  db.collection("beers")
      .onSnapshot(function(querySnapshot) {
          const snapBeers = [];
          querySnapshot.forEach(function(doc) {
              snapBeers.push(doc.data());
          });

          setBeers(snapBeers);
      });

    return () => {
        unsubscribe();
    };
  }, []);

  // ...

It's also possible to use .map on the querySnapshot.docs attribute to retrieve all the documents in a "non-imperative" way. Unfortunately, this functionality is not sufficiently covered by the official documentation.

The Model

Just like in backend development, I like to separate the model logic from the view logic in my frontend apps. So here is the API file for our beers below:

// ./src/api/beers.js

import PouchDB from 'pouchdb';

// We declare a PouchDB instance that is "remote only"
// There's no "offline" capability for the moment, everything is sync
export const beersDatabase = new PouchDB('http://localhost:5984/beers');

// If the beers database does not already exist
// => The database is automatically created when an object is added to it
export const addBeer = beer => beersDatabase.post(beer);

// Here, we list all the documents from our beers database
// A lot of options exists. Eg: we can paginate using "startKey", "endKey" or "limit"
export const getBeers = () =>
  beersDatabase
    .allDocs({
      include_docs: true,
      descending: true,
    })
    .then(doc => doc.rows.map(row => row.doc));

// We listen all the changes that happen since now
// We can also apply a "limit" option to this method
export const onBeersChange = callback => beersDatabase
    .changes({ since: 'now', live: true })
    .on('change', callback);

Here is the result of our first CouchDB application in action. As you can see, everything is in sync between multiple windows.

Offline Sync

Sadly, our actual version only works when the internet access is up and running. In other cases, such as a bloated network or packet loss, beers will never (or slowwwwly...) be added in the beers list because of the "remote only" sync.

The right way to avoid this problem is to keep a local first approach. It means that we must achieve all our database operations on the local database, then synchronize it with the remote one when internet access comes back.

So, the first step is to declare a new PouchDB instance with a database name instead of a remote database url. This way, PouchDB automatically detects that we want to instantiate a local database.

import PouchDB from 'pouchdb';

// Declare local database
const beersDatabase = new PouchDB('beers');

// Declare remote database
const remoteBeersDatabase = new PouchDB(`http://localhost:5984/beers`);

// Keep local and remote databases in sync
PouchDB.sync(beersDatabase, remoteBeersDatabase, {
  live: true, // replicate changes in live
  timeout: false, // disable timeout
  retry: true, // retry sync if fail
});

The PouchDB.sync instruction is the equivalent of a bidirectionnal PouchDB.replicate instruction between local and remote databases.

PouchDB.replicate(beersDatabase, remoteBeersDatabase);
PouchDB.replicate(remoteBeersDatabase, beersDatabase);

By default, PouchDB uses IndexedDB as local database (just like Firestore by the way). So, now that our setup is done, we can take a look at our local database using the Chrome console.

As you can see, we find the complete list of beers we've created. Each one is uniquely identified by a key that is built from _id and _rev CouchDB attributes.

{
  "_id": "0c2738a3-d363-405f-b9bb-0ab6f5ec9655",
  "_rev": "3-b90bd9d62fbe04e36fe262a267efbd42",
  "title": "Beer X"
}

Whereas the _id represents an unique document, the _rev represents the revision identifier of it. In fact, each modification of a document implies a new version of it which then makes it possible to manage conflicts.

Unlike CouchDB, Firestore documents do not have a revision id. So, the only way not to struggle with conflicts using Firestore is to use transactions.

Moreover, since CouchDB records every submitted change, it is possible to return back or resolve conflict in a second time, which is essential in order not to risk losing data.

For more information on conflict management using PouchDB, check the PouchDB Conflict documentation.

Now that we are able to communicate with both local and remote databases, we can focus on the business logic and on the user interface. Moreover, it'll allow us to benefit from optimistic rendering while making our application more flexible in addressing network issues.

Forms & Validation

In this section, we will implement a form to be able to add new beers. To do that, I'm going to use final-form (and react-final-form, an adapter for ReactJS).

npm install -S final-form react-final-form

So, we can create a simple form to handle user input.

// ./src/components/BeerForm.js

import React from 'react';
import { Form, Field } from 'react-final-form';

export const BeerForm = ({ onSubmit }) => (
  <Form
    validate={() => ({})}
    onSubmit={onSubmit}
    render={({
      handleSubmit,
      hasValidationErrors,
      pristine,
      invalid,
      submitErrors,
      submitting,
      form,
    }) => (
        <form onSubmit={handleSubmit}>
         <div>
            <label>Title</label>
            <Field name="title" component="input" />
          </div>
          <div>
            <label>Description</label>
            <Field
              name="description"
              component="textarea"
              rows={2}
              placeholder="Tape your description here..."
            />
          <div/>
          <button type="submit" disabled={pristine || hasValidationErrors || submitting}>
            Submit
          </button>
          {submitErrors && submitErrors.global && (
            <p>{submitErrors.global}</p>
          )}
        </form>
      )
    }
  />
);

Then, we can replace our action button by the form in our home screen.

// ./src/screens/Home.js

import React, { useState, useEffect } from 'react';
import { addBeer, getBeers, onBeersChange } from '../api/beers';

export const Home = () => {
  const [beers, setBeers] = useState([]);

  /* ... */

  return (
    <div>
      <BeerForm onSubmit={beer => queries.addBeer(beer)} />
      <ul>
        {/* beer._id is an unique id generated by CouchDB */}
        {beers.map(beer => <li key={beer._id}>{beer.title}</li>)}
      </ul>
    </div>
  );
};

Data Validation With Indicative

So, we have a form, but there's no data validation for the moment. Users can send anything they want at this time. That's why we're going to set up a data validator using indicative, a library that I've just discovered and that I want to give a try.

npm install -S indicative

The Indicative API is very simple. It consists of a Validator object that uses a set of validation rules and a formatter. Here is a usage example:

import Validator from 'indicative/builds/validator';
import { Vanilla as VanillaFormatter } from 'indicative/builds/formatters';
import { required, email } from 'indicative/builds/validations';

const validator = Validator({ required, email }, VanillaFormatter);

const rules = {
  name: 'required',
  email: 'required|email',
};

const messages = {
  'required': '{{ field }} field is required', // This message works for all required rules
  'email.required': 'You must provide an email!', // This message is specific for required email
  'email.email': 'The email adress is invalid',
};

const values = {
  email: 'bad email',
};

// Validator.validate is async

validator
  .validate(values, rules, messages)
  .then(() => /* everything is ok! */)
  .catch((errors) => {
    /*
      [
        { field: 'name', message: 'name field is required!' },
        { field: 'email', message: 'The email adress is invalid' },
      ]
    */
  });

Here is our custom implementation for BeerForm.js.

// ./src/components/BeerForm.js

import React from 'react';
import { Form, Field } from 'react-final-form';
import { Vanilla } from 'indicative/builds/formatters';
import Validator from 'indicative/builds/validator';
import { required } from 'indicative/builds/validations';

const validator = Validator({ required }, Vanilla);

const rules = {
  title: 'required',
  description: 'required',
};

const messages = {
  'title.required': 'Beer title is required',
  'description.required': 'Beer description is required',
};

const validate = async values =>
  validator
    .validate(values, rules, messages)
    .then(() => ({}))
    .catch(errors => {
      return errors.reduce((acc, error) => {
        acc[error.field] = error.message;
        return acc;
      }, {});
    });

Final Form needs an object as error model, so we format errors in the catch using a reduce. Alternatively, it would have been possible to use a Custom Indicative Formatter.

So, now we have our custom validation function, we can replace our empty validate function.

export const BeerForm = ({ onSubmit }) => (
  <Form
-  validate={() => ({})}
+  validate={validate}

And tadaaa! Our validated form is up and running and we are ready to play with it.

Let's Make It Beautiful!

To summarize, we can display beers, we can add beers, everything works offline and is in sync with a remote server. But right now, it's not very aesthetic, and I wouldn't dare present it to my mother-in-law. So, how about making it a little prettier?

In this section, I'll use the Elastic UI framework (aka eui) that is in use at Elastic, the company that develops ElasticSearch.

I think we all agree that we must remove this despicable list and replace it with a nice grid. Fortunately, Eui allows it easily.

As you can see, we took the opportunity to add editing and deleting beers right from the grid. We'll also put the form in a sliding panel from the right of the page. This way, we can directly add a beer from a "+" button in the navbar, or edit a beer directly from the grid, without changing page.

Handling Image Attachments

I don't know about you, but seeing all these grey beer cans breaks my heart. So it's time to allow image upload in the form.

// ./src/components/BeerForm.js

const handleIllustration = async files => {
  if (files.length === 0) {
    form.change('_image', undefined);
    return;
  }

  const file = files[0];

  form.change('_image', {
    data: file,
    type: file.type,
  });
};

<EuiFormRow label="Beer Illustration">
  <EuiFilePicker onChange={handleIllustration} />
</EuiFormRow>

This custom _image attribute that I just added to the beer object is then handled by our beer api, and considered as a PouchDB attachement.

// ./src/api/queries.js

const saveBeer = async ({ _image, ...beer }) =>
  store
    .collection('beers')
    .post(beer)
    .then(
      ({ id, rev }) =>
        // if an "_image" attribute is present, we put an attachement to the document
        _image &&
        store
          .collection('beers')
          .putAttachment(id, 'image', rev, _image.data, _image.type)
    );

const getBeers = () =>
  store
    .collection('beers')
    .allDocs({
      include_docs: true,
      descending: true,
      attachments: true, // We include images in the output, so we can display them
    })
    .then(doc => doc.rows.map(row => row.doc));
};

In CouchDB, every file can directly be attached to its corresponding document as an attachement. This concept does not exist in Firestore. It is then preferable to use Firebase Storage (Google Cloud Storage) through its bucket system to store files and store paths in Firestore.

Conclusion

The final result of my beer registry application is available on GitHub at the following address: github.com/marmelab/reactive-beers. Feel free to comment and improve!

While I was doubtful about the power of CouchDB at first, I was quickly conquered by its stability and the ease of use of its API.

Since I have not yet deployed this type of application in production, I am not in a position to comment on the ease of maintenance of this type of database. Nevertheless, I would rather recommend using Firestore for POCs, and a third-party service like Couchbase or IBM Cloudant for critical applications in the first place.

Although this experience allowed me to balance the pros and cons of the main features of each database, it was not possible for me to go as far as I had expected.

Indeed, I didn't have time to cover many crucial points such as document access security, rights management, server-side document validation, data pagination or deployment. But no matter what, I am determined to write more articles on these topics.

So, Stay tuned!

Build Augmented Reality Applications With React-Native

Demangeon Julien — Thu, 25 Apr 2019 12:32:35 +0000

Note: This post was originally posted on marmelab.com.

Augmented Reality is one of the most important trends currently. So, after our trial using the browser over 1 year ago, I wanted to test a framework offering the possibility to create native augmented reality experiences. Read on to see how I developed a reversi game application on mobile using React-Native.

What is Augmented Reality?

As the "Artificial Intelligence" term can be mixed up with other related concepts, Augmented Reality (AR) is quite often mistaken with Virtual Reality (VR). In fact, VR and AR are not the same at all. While VR is a projection of a virtual world to our eyes, AR is a blended projection of a virtual object in the real world.

I invite you to check a more detailed description of these concepts in our previous blog post about AR in the browser.

Augmented Reality In Javascript With Native Performance

At Marmelab, we are absolute fans of React and its ecosystem. That's why we develop a lot of open-source tools and projects for our customers using this technology.

I don't pretend to be a good Java, Kotlin, CSharp or Swift developer. But I also want to have good performance on mobile, so using a web framework like React is out of the question. So I started looking for a native framework which lets me develop iOS and Android apps with both Javascript and React.

After several minutes of research, the only obvious choice was to use ViroReact. Under the hood, this framework is based on two APIs that dominate the world of Augmented and Virtual Reality for mobile phones: ARKit for iOS and ARCore for Android.

ARKit is actually the biggest existing AR platform. It allows to develop rich immersive experiences on Apple devices having at least an A9 chip and iOS 11.

ARCore is more or less the same, except that it supports a short list of devices that are considered to be powerful enough to run the API at its best. And also iOS devices, apparently?.

The rather limited support of devices is the major weakness of these APIs for the moment. Over time, phones will become more and more powerful, which will make it possible to use them more often.

Viro, The Outsider

Viro is a free AR/VR development platform that allows building cross-platform applications using React-Native, and fully native Android applications using Java. It supports multiple platforms and APIs such as ARKit, ARCore, Cardboard, Daydream or GearVR.

As previously said, Viro allows building both fully native application and React-Native ones. That's why Viro provides two distinct packages: ViroCore and ViroReact.

To use it, you're still required to sign up. The API key which is provided following registration is mandatory to be able to use the platform.

Sadly, Viro is not open-source but (only) free to use with no limits on distribution. According to the ViroMedia CEO, the API key is used for internal analytics and to guard against possible license violations.

VIRO reserves the right, at any time, to modify, suspend, or discontinue the Software, or change access requirements, with or without notice.

Regarding the license note above, it is therefore necessary to remain vigilant regarding its use since we have no guarantee on the evolution of the platform.

First Contact With ViroReact

In this section, I'll cover the major parts of the Viro Framework with a simple use case: a 3D projection of the Marmelab logo !

First, we need to create a 3D mesh to be able to include it in our project. Special thanks to @jpetitcolas who created the Marmelab logo using blender a few years ago.

Installation

Before using Viro, we need to install some npm dependencies. Viro requires react-native-cli and react-viro-cli as global packages.

npm install -g react-native-cli
npm install -g react-viro-cli

Then, we can initialize a Viro project using the special command react-viro init, followed by the project name. A folder with the same name is then created.

react-viro init marmelab_for_real

So, what can we see in this project? Well, the folder structure is quite similar to the usual ones we encounter with React-Native, no surprise on this point.

├── android
├── bin
├── ios
├── js
├── node_modules
├── App.js
├── app.json
├── index.android.js
├── index.ios.js
├── index.js
├── metro.config.js
├── package.json
├── rn-cli.config.js
├── setup-ide.sh
└── yarn.lock

Developer Experience

Once the project is initialized, we just have to launch it using the npm start command. Viro will automatically create an ngrok tunnel, which can be used by any phone connected to the internet around the globe.

julien@julien-laptop /tmp/foo $ npm start

> foo@0.0.1 prestart /tmp/foo
> ./node_modules/react-viro/bin/run_ngrok.sh

 ----------------------------------------------------------
|                                                          |
| NGrok Packager Server endpoint: http://32a5a3d7.ngrok.io |
|                                                          |
 ----------------------------------------------------------

> foo@0.0.1 start /tmp/foo
> node node_modules/react-native/local-cli/cli.js start

┌──────────────────────────────────────────────────────────────────────────────┐
│                                                                              │
│  Running Metro Bundler on port 8081.                                         │
│                                                                              │
│  Keep Metro running while developing on any JS projects. Feel free to        │
│  close this tab and run your own Metro instance if you prefer.               │
│                                                                              │
│  https://github.com/facebook/react-native                                    │
│                                                                              │
└──────────────────────────────────────────────────────────────────────────────┘

To access the application, we just have to use the special TestBed application from Viro with the corresponding tunnel or local ip (if you're connected locally). On those aspects, Viro reminds me of Expo. Then, we're able to access the test application:

In addition to these running facilities, Viro also offers hot-reloading, live-reloading, error messages & warnings directly on the device, just like any React-Native application does.

Initializing a Scene Navigator

Depending on the type of project you want, Viro provides 3 distinct SceneNavigator components which are the following:

ViroVRSceneNavigator: For VR Applications
ViroARSceneNavigator: For AR Applications
Viro3DSceneNavigator: For 3D (not AR/VR) Applications

This components are used as entry points for our application. You must choose one depending on what you want to do, in our case ViroARSceneNavigator for Augmented Reality.

Each SceneNavigator requires two distinct props which are apiKey and initialScene. The first one comes from your registration on the Viro website, the second one is an object with a scene attribute with our scene component as value.

// App.js

import React from 'react';
import { View, StyleSheet } from 'react-native';
import { ViroARSceneNavigator } from 'react-viro';
import { VIROAPIKEY } from 'react-native-dotenv';

import PlayScene from './src/PlayScene';

const styles = StyleSheet.create({
    root: {
        flex: 1,
        backgroundColor: '#fff',
    },
});

const App = () => (
    <View style={styles.root}>
        <ViroARSceneNavigator
            apiKey={VIROAPIKEY}
            initialScene={{ scene: PlayScene }}
        />
    </View>
);

export default App;

Since we want to keep our Viro apiKey private, we use the react-native-dotenv package in conjunction with a .env file at the root of our project folder.

To make it psosible, just install this package with yarn add -D react-native-dotenv and create a .env file with VIROAPIKEY=<YOUR-VIRO-API-KEY> in it.

The last step is to add the preset to babel has described below.

// .babelrc

{
  "presets": [
    "module:metro-react-native-babel-preset",
+   "module:react-native-dotenv"
  ]
}

Adding a Scene

Now that the bootstrap is done, it's time to develop our first scene!

Viro Scenes act as containers for all our UI Objects, Lights and 3D objects. There are 2 types of Scene components: ViroScene and ViroARScene.

Each Scene contains a hierarchical tree structure of nodes that are managed by a full-featured 3D scene graph engine. ViroScene children are positioned through ViroNode components that represent positions and transformations in 3D space.

So, almost every object under the tree has a position, rotation and scale prop that accept an array of coordinates/vector (x, y, z) as described below.

<ViroNode
    position={[2.0, 5.0, -2.0]}
    rotation={[0, 45, 45]}
    scale={[2.0, 2.0, 2.0]}
 />

Now that we know how it works, we can create our first ViroARScene (aka PlayScene).

// src/PlayScene.js

import React from 'react';

import {
    ViroARScene,
    Viro3DObject,
    ViroAmbientLight
} from 'react-viro';

const MarmelabLogo = () => (
    <Viro3DObject
        source={require('../assets/marmelab.obj')}
        resources={[require('../assets/marmelab.mtl')]}
        highAccuracyEvents={true}
        position={[0, 0, -1]} // we place the object in front of us (z = -1)
        scale={[0.5, 0.5, 0.5]} // we reduce the size of our Marmelab logo object
        type="OBJ"
    />
);

const PlayScene = () => (
    <ViroARScene displayPointCloud>
        <ViroAmbientLight color="#fff" />
        <MarmelabLogo />
    </ViroARScene>
);

export default PlayScene;

In the previous code, we've introduced 2 new Viro Components that are Viro3DObject and ViroAmbiantLight.

The Viro3DObject allows creating 3D objects from 3D structure / textures files that can be placed on our Viro Scene. In our case, we declare a component using our previously blended Marmelab logo object.

The ViroAmbientLight introduce some lighting in our Scene. Without that light, no object is visible.

The final result is really amazing, especially since we spent very little time on it.

Level Up: Developing A Reversi In AR

After this little exploration, it's time for us to develop a more tangible application using this technology. Since I don't want to do modeling or coding business logic this time, I'll reuse an existing codebase and blended objects (disks) from a previous projects I worked on during a hackday. It's a Reversi Game using ThreeJS.

The Reversi PlayScene

According to our previous experiment, we're going to replace our PlayScene to include a new Game component that contains a Board that itself contains Disk object components.

// src/PlayScene.js

import React from 'react';

import {
    ViroARScene,
    ViroAmbientLight,
} from 'react-viro';

import Game from './components/Game';
import { create as createGame } from './reversi/game/Game';
import { create as createPlayer } from './reversi/player/Player';
import { TYPE_BLACK, TYPE_WHITE } from './reversi/cell/Cell';

const defaultGame = createGame([
    createPlayer('John', TYPE_BLACK),
    createPlayer('Charly', TYPE_WHITE),
]);

const PlayScene = () => {
    const [game] = useState(defaultGame);

    return (
        <ViroARScene displayPointCloud>
            <ViroAmbientLight color="#fff" />
                <Game game={game} />
        </ViroARScene>
    );
};

export default PlayScene;

// src/components/Game.js

import React, { Component } from 'react';

import Board from './Board';
import { getCurrentPlayer } from '../reversi/game/Game';

class Game extends Component {
    // ...

    render() {
        const { game } = this.state;

        return (
            <Board
                board={game.board}
                currentCellType={getCurrentPlayer(game).cellType}
                onCellChange={this.handleCellChange}
            />
        );
    }
}

export default Game;

The Game relies on a Board and a Disk component:

// src/components/Board.js

import React, { Component } from 'react';
import PropTypes from 'prop-types';
import { ViroNode } from 'react-viro';

import Disk from './Disk';
import { TYPE_WHITE, TYPE_EMPTY } from '../reversi/cell/Cell';

class Board extends Component {
    // ...

    renderCellDisk = cell => (
        <Disk
            key={`${cell.x}${cell.y}`}
            position={[0.03 * cell.x, 0, -0.3 - 0.03 * cell.y]}
            rotation={[cell.type === TYPE_WHITE ? 180 : 0, 0, 0]}
            opacity={cell.type === TYPE_EMPTY ? 0.15 : 1}
            onClick={this.handleClick(cell)}
        />
    );

    render() {
        const { board } = this.props;

        return (
            <ViroNode position={[0.0, 0.0, 0.5]}>
                {board.cells
                    .reduce(
                        (agg, row, y) => [...agg, ...row.map((type, x) => createCell(x, y, type))],
                        [],
                    )
                    .map(this.renderCellDisk)}
            </ViroNode>
        );
    }
}

Board.propTypes = {
    onCellChange: PropTypes.func.isRequired,
    currentCellType: PropTypes.number.isRequired,
    board: PropTypes.shape({
        cells: PropTypes.array,
        width: PropTypes.number,
        height: PropTypes.number,
    }),
};

export default Board;

// src/Disk.js

import React from 'react';
import { Viro3DObject } from 'react-viro';

const Disk = props => (
    <Viro3DObject
        source={require('../assets/disk.obj')}
        resources={[require('../assets/disk.mtl')]}
        highAccuracyEvents={true}
        position={[0, 0, -1]}
        scale={[0.0007, 0.0007, 0.0007]}
        type="OBJ"
        {...props}
    />
);

export default Disk;

It's working! However, I think we all agree that it is not possible to play Reversi on a floating board... That's why we're going to define an Anchor on which we can place our Game / Board.

Placing Objects in Real-World

In Augmented Reality terminology, the concept of attaching virtual objects to a real-world point is called Anchoring. According to that word, Anchors are used to achieve this task.

Anchors are vertical or horizontal planes, or images (often markers) found in the real world by the AR system (ARCore or ARKit) on which we can rely to build a virtual world.

With Viro, Anchors are represented by an Anchor object which can be found through Targets using different detection methods, as described below.

ViroARPlane: This component allows to use either "manual" (though an "anchorId") or "automatic" detection of a plane in the real-world to place objects on it.
ViroARPlaneSelector: This component shows all the available planes discovered by the system and allows the user to select one.
ViroARImageMarker: This component allows to use an illustrated piece of paper as a physic anchor for our virtual objects.

In my case, I've chosen the ViroARImageMarker anchoring system because it seems more stable and performs better (at first glance).

ViroARImageMarker has a mandatory prop called target. This prop which must contain the name of a registered target which has previously been declared using ViroARTrackingTargets module.

The first thing to do is to create our target using the createTargets function. In our case, we declare an image target named marmelabAnchor (yes, I'm very corporate...) because I used the Marmelab logo as an anchor.

Then, we can use this anchor name directly as anchor prop value of our new ViroARImageMarker element around our Game.

// src/PlayScene.js

import React from 'react';

import {
    ViroARScene,
    ViroAmbientLight,
+   ViroARTrackingTargets,
+   ViroARImageMarker,
} from 'react-viro';

import Game from './components/Game';
import { create as createGame } from './reversi/game/Game';
import { create as createPlayer } from './reversi/player/Player';
import { TYPE_BLACK, TYPE_WHITE } from './reversi/cell/Cell';

const defaultGame = createGame([
    createPlayer('John', TYPE_BLACK),
    createPlayer('Charly', TYPE_WHITE),
]);

const PlayScene = () => {
    const [game] = useState(defaultGame);

    return (
        <ViroARScene displayPointCloud>
            <ViroAmbientLight color="#fff" />
+           <ViroARImageMarker target={'marmelabAnchor'}>
                <Game game={game} />
+           </ViroARImageMarker>
        </ViroARScene>
    );
};

+ ViroARTrackingTargets.createTargets({
+     marmelabAnchor: {
+         type: 'Image',
+         source: require('./assets/target.jpg'), // source of the target image
+         orientation: 'Up', // desired orientation of the image
+         physicalWidth: 0.1, // with of the target in meters (10 centimeters in our case)
+     },
+ });

export default PlayScene;

All children that are declared under the ViroARImageMarker element in the tree are placed relatively to it. In our case, the Game component is then placed over the ViroARImageMarker target.

Animating The Scene

Now the AR reversi game is working better. But it lacks a little bit of animation. So, how can we add the same disk flip effects as we made in our previous ThreeJS project?

To fill this usual need, ViroReact provides a global animation registry called ViroAnimations that can be used everywhere in conjunction with any component that accepts an animation prop.

In our case, we're gonna compose transformations together to create a complete disk flipping effect. Here is the desired scenario over time:

0 - 300ms	Move Up
300 - 600ms	Move Down
150 - 350ms	Rotate (during disk reaches the top)

First, we're gonna register an animation according to this transformation timeline.

import { ViroAnimations } from 'react-viro';

// ...

ViroAnimations.registerAnimations({
    moveUp: {
        properties: { positionY: '+=0.03' },
        duration: 300,
        easing: 'EaseInEaseOut',
    },
    moveDown: {
        properties: { positionY: '-=0.03' },
        duration: 300,
        easing: 'EaseInEaseOut',
    },
    flip: {
        properties: { rotateX: '+=180' },
        duration: 300,
        easing: 'EaseInEaseOut',
        delay: 150
    },
    flipDisk: [['moveUp', 'moveDown'], ['flip']],
});

As you see, we declare 3 distinct animations, and compose them using the fourth one, flipDisk. moveUp and moveDown are in the same array because they are executed one after the other. flip runs in parallel to these two transformations.

Secondly, we just need to use this registered animation in our Disk component using the animation prop, as follows:

    // ...

    renderCellDisk = cell => {
        const { flipping } = this.state;

        return (
            <Disk
                key={`${cell.x}${cell.y}`}
                position={[0.03 * cell.x, 0, -0.3 - 0.03 * cell.y]}
                rotation={[cell.type === TYPE_WHITE ? 180 : 0, 0, 0]}
                opacity={cell.type === TYPE_EMPTY ? 0.15 : 1}
                onClick={this.handleClick(cell)}
                animation={{
                    name: 'flipDisk',
                    run: !!flipping.find(hasSamePosition(cell)),
                    onFinish: this.handleEndFlip(cell),
                }}
            />
        );
    };

    // ...

The animation prop accepts an object of the following structure:

{
    name: string            // name of the animation
    delay: number           // number of ms before animation starts
    loop: bool              // animation can loop?
    onFinish: func          // end callback of the animation
    onStart: func           // start callback of the animation
    run: bool               // animation is active or not?
    interruptible: bool     // can we change animation when running?
}

In our case, we've just used name, run, and onFinish attributes to define which disk is currently flipping, and remove it from the flipping list when the animation ends.

Conclusion

Using ViroReact for building an Augmented Reality project was a great choice for many reasons. Whereas it was my first experience in this domain, I haven't faced any difficulties at any time. Quite the contrary, Viro has helped me to explore this world with confidence.

The developer experience is rich as it offers ReactJS binding, hot-reload and unambiguous documentation. Nevertheless, I don't recommend to use it for complex / performance-based applications because of the React-Native javascript thread which can lead to event congestion and lags. So, in case performance matters, I'd recommend full-native solutions instead.

By the way, Google is constantly adding augmented reality features within its applications, like on Google Map. Augmented Reality has never been so expanding. So, don't miss it.

Many other features remain to be explored, such as Skeletal animations, particles effects, physics, video and sounds. Don't be shy, share your experiences though comments ;)

You can find the final code on GitHub, in the marmelab/virothello repository.

Get Rid Of Toxic Bugs On Your Apps With Detox!

Demangeon Julien — Tue, 05 Jun 2018 19:56:03 +0000

Note: This post was originally posted on marmelab.com.

During the React Europe Conference last year, I discovered a promising E2E testing framework for mobile apps called Detox. Since then, I've kept in mind the will to give it a try. Recently, I developed an example app just for that purpose. It allowed me to see what's under the hood of Detox. Here is my feedback.

What Are End-to-End Tests?

In software development, we list 3 distinct kinds of automated tests:

Unit Tests: As their name suggest, they test functions individually, in isolation from the rest of the codebase. They're used to prevent unexpected code changes and to ensure that functions do what they are supposed to do.
Integration Tests (or Service Tests) are responsible for the proper connection between code parts and APIs. They test the application components altogether from a technical perspective.
End-To-End Tests (E2E): They allow to test the application as a whole, in its execution environment, as a human could do.

According to Martin Fowler, all these kinds of tests can be classified into a Test Pyramid from the slowest / most expensive to the fastest / least expensive.

At the bottom of the pyramid, Unit-Tests must be the most common tests. Utopically, each function must be tested. Some integration tests and a bit less E2E tests are needed to ensure that the entire stack is working well.

Whereas E2E tests are very important, some people sometimes go too far with an excessive E2E test coverage. Another diagram called Ice-Cream Cone represents this anti-pattern as well.

But everyone agrees that writing and debugging E2E tests is a tedious task.

Introducing Detox

Detox was first released in 2016 by Tal Kol and Rotem Mizrachi-Meidan, 2 engineers working at Wix. Wix is a cloud-based platform which allows non-technical users to create their own website.

Detox defines itself as a Gray Box End-To-End testing automation framework for mobile apps. That means that it brings the same context aware test capabilities we're already using in browser apps through Selenium. This way, Detox allows to break off from manual Quality Insurance test processes, which are time-consuming and incomplete.

Contrary to Appium, its the main competitor, Detox uses JavaScript both on the server side and on the client side. Despite this strong requirement, Detox allows using Jest, Mocha, AVA, or any other JavaScript test runner you like.

Gray Box Testing vs Black Box Testing

As a Gray Box test framework, Detox shares both White and Black Box capabilities. Let's see together what it means.

Black Box test frameworks allows to take over an execution context (a browser, a software, mobile apps, etc) and send control commands to them.

This test methodology does not allow to access to the internal state of the application though. That's why it's necessary to manually check the existence of elements to ensure test the state after a transition.

function* navigate() {
    yield driver.navigate().to(`http://localhost/#/login`);
    yield driver.wait(until.elementLocated(By.css('#loginform')));
}

Gray Box frameworks are extending White Box test frameworks capabilities. This way, they do the same thing as Black Box frameworks, except that they access the internal state of the execution context.

Accessing the internal state of the execution context permits to know when the application is idle, and to synchronize operations adequately. That's why Detox is more powerful than most classic E2E test frameworks.

Less Flakiness

If you've already used an E2E test framework before, you've certainly encountered some strange, random and unexpected errors. These errors are therefore called "flakiness errors". As you encounter them, you feel like our good old Harold and it's not very funny.

To mitigate this behavior, we usually add some sleep (or timeouts) calls into the test suite, to ensure that the application is in an idle state before resuming the test process. Even though this "hack" works, it results in slower tests, without really solving the problem because on a slow test system, the sleep delay can sometimes be not enough.

function* login() {
    yield driver.findElement(this.elements.loginButton).click();
    yield driver.sleep(5000);
}

Thankfully, as a Grey Box Framework, Detox is able to access the application state and then to determine if the application is in an idle state or not. To achieve this idle synchronization task, Detox rely on 2 natives Grey Box drivers called EarlGrey (for iOS) and Espresso (for Android).

Because Detox runs in JavaScript, it communicates with drivers using a JSON based protocol to invoke control commands on devices.

A special synchronization mechanism has also be developed for React-Native apps, so Detox supports React Native

A Concrete Use Case

As already said in the introduction, I've developed a dedicated application to give Detox a try. Since I'm a beer lover, I couldn't resist creating a simple beer registry app called beerexplorer.

Most blog posts about Detox build applications with React-Native, this one is no exception. Still, it's important to know that Detox is fully cross-platform and supports both iOS and Android.

Detox Setup

I've tried to setup Detox to run it on my own Android phone. Despite all my efforts, I've not been able to make it work. So I went back to an iOS emulator.

The Detox setup is relatively simple. It consists of installing the detox npm package, then calling 2 commands: detox build and detox test.

Then, Detox uses an existing configuration defined in package.json to determine which test runner and configuration it should use. All available devices configurations are stored under the "detox.configurations" key. Android and iOS device configurations can be mixed.

/* package.json */

{
  "name": "beerexplorer",
  "detox": {
    "test-runner": "jest",
    "runner-config": "e2e/config.json",
    "configurations": {
      "ios.sim.debug": {
        "binaryPath": "ios/build/Build/Products/Debug-iphonesimulator/beerexplorer.app",
        "build": "xcodebuild -project ios/beerexplorer.xcodeproj -scheme beerexplorer -configuration Debug -sdk iphonesimulator -derivedDataPath ios/build",
        "type": "ios.simulator",
        "name": "iPhone 7"
      }
    }
  }
}

When I call the detox test command, Detox looks in the runner-config configuration file for the setupTestFrameworkScriptFile to execute before running tests. I've called this file init.js.

// e2e/config.json

{
    "setupTestFrameworkScriptFile" : "./init.js"
}

Here is the test init file:

// e2e/init.js

const detox = require('detox');
const config = require('../package.json').detox;

beforeAll(async () => {
  await detox.init(config);
});

afterAll(async () => {
  await detox.cleanup();
});

Tests can either run on a local emulator, a hidden emulator, or even on a distant CI like Travis!

Detox Usage

Out of the box, Detox provides a small but powerful set of tools, that allows to control the device, select elements in the UI, and execute actions on these elements.

Detox tools are asynchronous. Therefore you have to use Promises, or, like in my code below, ES6 async/await.

Device

The device object allows to control the device directly, without relying on the tested application. Here are some examples of usages from the documentation.

// Launch app with specific permissions
await device.launchApp({ permissions: { calendar: 'YES' } });

// Simulate "home" button click 
await device.sendToHome();

// Simulate geolocation
await device.setLocation(32.0853, 34.7818);

Some device functions are specific to a given platform, such as device.reloadReactNative for React-Native and device.shake for iOS.

Selectors / Matchers

As with other test frameworks, Detox gives the possibility to match UI elements in different ways.

The easiest (and recommended) way to match elements is to use ids. Sadly, this technique is only available on React-Native.

// id declaration
<Touchable testID="BeerListItem">
...
</Touchable>

// element selection
await element(by.id('BeerListItem'));

It's also possible to match elements with other methods like text, label, type or traits. More informations on the corresponding matchers documentation.

Actions And Expectation

Once selected, it's possible to trigger actions and execute assertions on elements. As an example, here is a test suite from the homepage of the "beerexplorer" project.

describe('home', () => {
  beforeEach(async () => {
    await device.reloadReactNative();
  });

  it('should have a list of beers', async () => {
    await expect(element(by.id('BeerList'))).toBeVisible();
  });

  it('should go to detail on beer touch', async () => {
    await element(by.id('BeerListItem')).atIndex(0).tap();
    await expect(element(by.id('DetailBackground'))).toBeVisible();
  });

  it('should show all beers', async () => {
    await waitFor(element(by.label('Lindemans Kriek'))).toExist().whileElement(by.id('BeerList')).scroll(50, 'down');
    await expect(element(by.label('Lindemans Kriek'))).toExist();
  });
});

As you can see, tests are very expressive and easy to read. There's no need to add more test about the existence of an element between transitions, thanks to the idle state synchronization.

Conclusion

Although satisfied by Detox at the end, I'm still disappointed by the difficulty of setting up an E2E test suite on Android. Because of my poor experience on mobile application tests, I don't pretend to give you the more accurate opinion. But I still think this framework (and its documentation) still to be improved for Android.

Apart from that, the developer experience with Detox is very pleasant. I never found myself in difficulty when writing tests. Also, live preview in the emulator is very empowering.

Nevertheless, If you're testing your application on iOS only, feel free to give it a try. You won't take many risks, except to be greatly satisfied by the clarity and the stability of the tests.

If you want to read more on the subject by other authors, I recommend the following:

Enjoy Painless Typing With ReasonML!

Demangeon Julien — Mon, 09 Apr 2018 15:17:03 +0000

Note: This post was originally posted on marmelab.com.

For one of our regular hackdays at Marmelab, I wanted to learn a language I've kept in the back of my head since I heard of it at React Europe 2017.

This language, which is called Reason (shortcut for ReasonML), is in fact a syntax and a toolchain overlay for OCaml, a language that is known for its robustness, its strong static typing, and its ubiquitous functional paradigm approach.

According to the Reason website, its main goal is to provide a friendly syntax / environment to JavaScript developers looking for performance, consistency, and type safety.

Reason can almost be considered as a solidly statically typed, faster and simpler cousin of JavaScript, minus the historical crufts, plus the features of ES2030 you can use today, and with access to both the JS and the OCaml ecosystem!

By the way, I think that this simple example should suffice to illustrate its power, and whet your appetite for further reading.

type schoolPerson = Teacher | Director | Student(string);

let greeting = (stranger) =>
  switch (stranger) {
  | Teacher => "Hey professor!"
  | Director => "Hello director."
  | Student("Richard") => "Still here Ricky?"
  | Student(anyOtherName) => "Hey, " ++ anyOtherName ++ "."
  };

To explore this language in practice, I've coded a reversi game running in the browser. I'll use it to give an overview of Reason capabilities, and explain why I think that it opens a whole new horizon on the JavaScript ecosystem.

But before going into technical details, let me introduce Reason from an historical and practical point of view.

Reason Is OCaml Made Easier

As I previously said, Reason is based on OCaml. This way, it benefits from all the OCaml strengths, like polymorphic / infered typing, pattern matching, garbage collector, sophisticated module system, and so on.

OCaml is the main implementation of Caml. Caml is a safe, reliable, and expressive programming language created in 1985 by a French research institute in Computer Science called INRIA. But, what's wrong with OCaml? Why not use it directly? Indeed, the question deserves to be asked.

OCaml is based on complex principles, and uses an awkward syntax. Here is an example of OCaml code, which adds values recursively from a list:

let rec sum xs =
  match xs with
    | [] -> 0
    | x :: xs' -> x + sum xs';;

(* sum [1;2;3;4;5] => 15 *)

Tip: As a matter of fact, it's not really necessary to use recursive functions for this kind of purpose, because the Core Standard Library has many built-in functions covering most needs.

The complexity of OCaml explains why OCaml it confined to academic projects for a long time.

Until a few years ago, Jordan Walke, who works at Facebook, created the famous React library using SML (a derived OCaml language), and created ReasonML. Shortly after, he made the decision to migrate ReactJS to plain JavaScript for a wider adoption.

BuckleScript Brings OCaml to the Web

In fact, Reason does not directly compile to JavaScript. For that purpose, it maintains a strong dependency to another library called BuckleScript.

BuckleScript defines itself as a "sister" project of Reason. In fact, both share the same community (they have the same Discord), and the same purpose: bringing OCaml's capabilities to the browser. Moreover, the documentation between the two projects is very complementary.

Here is a little schema of the compilation workflow from Reason to JavaScript. Under the hood, reason files (.re) are transformed to plain OCaml AST through an OCaml preprocessor for Reason. This OCaml AST is then processed by the BuckleScript compiler called bsc, which produces plain JS Files.

The compilation process is a bit more complex than explained here. To understand it in more details, I suggest you take a look at this excellent repository owned by chenglou, the speaker who introduced me to Reason at React Europe.

Why Not TypeScript Or Flow?

JavaScript is an untyped language, but large projects often require additional programming safety throught types. That's why many tools have been created on top of JavaScript to fill this gap. Among them, the most popular are probably TypeScript and Flow.

TypeScript is an open-source programming language developed by Microsoft. It acts as a strict syntaxic superset of JavaScript that adds static typing to it. It is also considered as first class language for Angular development.
Flow is an open-source static type checker for JavaScript developed by Facebook. It acts as a specific syntax (kind of annotations) which adds types over an existing code.

How does Reason compare to those two? In fact, Reason is not a new syntax for JavaScript, or a simple typing overlay. Reason is a complete language. It introduces new control structures that you'll never find in TypeScript or Flow. Reason is built with typing at its core, and just compiles to JavaScript.

If you need strong types, I think that the only valid reason to use TypeScript or Flow is to port an existing JavaScript codebase. If you need strong types without an existing codebase, prefer a real typed language like Reason.

Reason Can Compile To Other Platforms

Reason uses OCaml as an intermediate language, and it's BuckleScript's job to translate that OCaml code into JavaScript.

But there are other toolchains for running OCaml code in other platforms. For instance, OCaml can be compiled to native code, thanks to the ocamlopt compiler. In this respect, there are some successful cross-platform projects written in Reason that are compiled to native, browser, Android and iOS.

From my point of view, this unexpected possibility opens up a whole new development horizon. Above all, this opportunity allows to move away from the "all JavaScript" trend, which I think is dangerous. We should never lock ourselves with a single technology.

OK, enough with the introduction of the language. Let's see some code!

Bindings And Expressions

Unlike JavaScript, there's only one way to declare / assign variables in Reason. Indeed, because everything is immutable by nature, the only assignation keyword is let. Therefore, the assignment action is called a "let binding".

let message = "hello world";

"Immutability" means that a value can't change over time - it doesn't means that you can't create a new binding with the same name to replace an existing one. That's why let is called a binding and not an assignment. A binding gives a name to a value, it doesn't change the value.

let message = "hello";
print_endline(message); /* Prints "hello" */
/* totally legal */
let message = "world";
print_endline(message); /* Prints "world" */

To effectively change an already bound "value", you must use a ref. This topic is discussed later in the "An Imperative Way Out" section.

Bindings can also be scoped into a "block" scope ({}). In that case, the last expression of the block is implicitly returned. There's no explicit return in Reason. As in JavaScript, bindings are only available in theirs respective scopes.

let message = {
    let part1 = "hello";
    let part2 = "world";
    part1 ++ " " ++ part2
};

/* part1 & part2 not availables here */

Basic Types

Like most other programming languages, Reason supports all basic types such as booleans, numbers, strings and chars. Since Reason is a statically typed language, types can be manually defined, or can be infered at compile time from the program AST.

let score = 10; /* type is infered */
let score: int = 10; /* type is manually defined */

With Reason, just like with Python, there's no implicit type casting. Developers must use explicit type conversion functions (like string_of_int or string_of_bool) to switch from one type to another. These functions are part of Pervasives, which is the initially opened module at the beginning of each compilation. It provides all the basic operations over the built-in types.

Custom Types

As in OCaml, it is also possible to create your own types with Reason. In this respect, here are 2 different kinds of types from the reversi "Cell" module.

The color type is called a Variant. A Variant is a kind of group of possible constants. These constants, which are called "constructors" or "tags", are separated by "|" bars. Variants are, from my point of view, the key feature of Reason. They allow us to carry values (as arguments), and enable pattern matching.

/* cell.re */
type color = White | Black;

The cell type is called a Record. In other languages, it's usually called a struct. Objects created based on a Record are immutable, fixed, and very fast. Records needs a strong type definition. That's why each field is explicitly typed.

/* cell.re */
type cell = {
    x: int,
    y: int,
    color: option(color),
};

As you see for the color field of the cell type, a type can contains another type. It allows to create complex recursives data structures (like trees) quickly and easily.

type intTree =
  | Empty
  | Node(int, intTree, intTree);

Parameterized And Special Types

In one of the previous example, you may have asked yourself about the option(color) function call. In fact, option is not a function, it's a parameterized Variant, which is directly exposed by the standard library.

Since there's no null values in Reason (and therefore no null pointer exceptions), option allows to mimic the absence of value for anything. It can either be None (null equivalent) or Some(value). It can be compared to the famous Maybe Monad.

type option('a) =
    | None
    | Some('a);

What does the 'a mean? In Reason, every type can accept parameters. The unique quote means "a value of any type". This is very useful to create generic type structures.

Lists And Arrays

In Reason, one of the most used Type is List. As its name suggests, a List is a collection of elements that are of the same type.

Lists are represented as linked lists underneath (even in the transpiled JavaScript!). Because of that, they're dynamically sized and immutable by nature, and they allow to add or remove elements very quickly.

/* board.re */
let directions = [
    (0, 1),  /* S  */
    (0, -1), /* N  */
    (1, 0),  /* E  */
    (1, 1),  /* SE */
    /* ... */
];

Lists are very fast for updates, but very slow for access. Read operation speed is proportional to the size of the List (O(n) complexity). That's why Reason also provide an Array Type.

Contrary to Lists, Arrays are fixed-size collections, which are mutable and fast for read operations (O(1) complexity).

let myArray = [|"hello", "world", "how are you"|];

In brief, Lists are better for dynamic and medium sized collections that do not require fast access. Array are better for fixed and large sized collections that require fast access.

You'll find more information about Lists and Arrays in the Exploring ReasonML online book, which is very complete.

Pattern Matching And Destructuring

I've introduced Types and Variants so that I can talk about one of the most interesting feature of Reason: pattern matching.

In brief, pattern matching allows both to check and to extract data from structures. It's a kind of mix between RegExp.test and RegExp.exec from JavaScript, but for all types of data, and anywhere (bindings and function args).

In the following example, I use the switch expression to test the color value against multiple patterns. When a pattern matches, the value just after the => is then returned and assigned to "identifier".

/* cell.re */
let identifier = switch (color) {
    | Some(Black) => "black"
    | Some(White) => "white"
    | None => ""
};

The above example is of course the simplest one. You can also match one part of the value, use it afterwards, and even match on an exception!

/* board.re */
let getCell = (x, y, cells) =>
    switch (List.find(cell => cell.x == x && cell.y == y, cells)) {
        | ({ color }) => color /* color is extracted from the cell record */
        | exception Not_found => None
    };

Naturally, matching also comes with destructuring. That's why it's even possible to extract parts of data structures easily, even from functions args!

/* board.re */
let init = (width, height) => {
    let (mw, mh) = (width / 2, height / 2); /* tuple destructuring */
    /* ... */
};

let isOutOfBound = ({ width, height }, (x, y)) => /* function args destructuring */
    x < 0 || y < 0 || x > (width - 1) || y > (height - 1);

There are a lot of things to say about pattern matching, far too much to cover everything here. I advise you to take a look at this article, which is very comprehensive.

Functions

As you may have understood, Reason is fully focused on the functional paradigm. In this regard, it highlights a lot of concepts like higher-order functions, recursivity, partial application (via currying), and so on. The Reason function syntax is very close to the ES6 one. It uses the famous "arrow / body" pair.

let add = (first, second) => first + second;
add(1,2); /* 3 */

In Reason, every function takes at least one argument, even if you don't declare / notice it! When you declare a function without any argument, under the hood, Reason adds a unit argument to it. In the example below, the pair of brackets () after locationReload is a unit. So, in reality, you effectively call locationReload with an argument without even realizing it.

let locationReload = () => {
  /* ... */
};
locationReload();

You may be disappointed by this "unit", but you must known that it's a normal value. Above all, you must not confuse "unit" with an "option". Whereas an "option" represents "a value or an empty value", a "unit" represents an absence of value (think about undefined or "nothing").

Currying

Reason offers built-in currying of all functions. That means that every function with one or more arguments is transformed into a series of functions with one argument.

let add = (first, second) => first + second;
add(1)(2); /* 3 */

You may think that it's a waste of resources to create additional function calls, but it's not. OCaml optimizes the output to avoid unnecessary function allocation if partial functions are never called in your program (see this example). This way, OCaml provides out of the box currying without any performance penalty.

Here is another example of currying, where I take advantage of partial application for my getCountForColor function from the reversi Board module:

let getCountForColor = (board, color) =>
    board.cells
        |> List.filter(c => c.color == color)
        |> List.length;

let countForColorFromMyBoard = getCountForColor(myBoard);

let countWhite = countForColorFromMyBoard(Some(White));
let countBlack = countForColorFromMyBoard(Some(Black));

The pipe operator |> (also called "reverse-application operator") allows to pass the result of an expression as the first argument of the following expression. Think of the Linux pipe. Given that functions natively allow partial application, it works like a charm!

Labeled Arguments

Reason functions also work with named arguments (called labeled arguments). They are compatible with currying as well:

let add = (~first, ~second) => first + second;
let addFirst = add(~second=1);
addFirst(~first=2);

To take full advantage of partial application, it's recommended to place args that change most often at the end of the function, or to use labeled args.

Labeled arguments can also be optional. For that purpose, you just have to add a question mark as default value, like in the example below. This way, the corresponding argument is automatically provided as an option type, described earlier.

let addOptional = (~first, ~second=?) => first + switch(second) {
  | None => 0
  | Some(nb) => nb
};

let addDefault = (~first, ~second=0) => first + second

It would be possible to write an entire book about functional paradigms and Reason. A lot of concepts were deliberately bypassed in this section.

For more informations about functors, recursivity, mutual recursivity, I suggest you to take a look at 2ality - ReasonML Functions and 2ality - ReasonML Functors. If you're interested about functional programming, I also advise you to read this Marmelab blog post by my colleague Thiery :)

Setting Up a Reason Project

During my hack day, the first step was to setup the whole Reason stack inside Docker, like we do for all our projects at Marmelab. Using Docker allows us to share projects across various environments with an easy installation.

To be honest, this setup experience was the worst I had in a long time. It took me about one hour to deal with many permission issues [1] [2]. The recommended global install from the official setup guide seems to be at the core of the problem.

Nevertheless, I'm pretty sure that the community will find solutions to make this first step easier. By the way, I'm not the only one who struggled with that. Using the "node:6" docker image seems to do the job for the moment...

Developer Experience First!

Once installed, the Reason developer experience is just amazing. The underlying BuckleScript compiler is quite fast, it builds mosts projects in less than 100ms, incrementally.

Moreover, the error reporter (based on Merlin) is just perfect. It gives a detailed explanation of all possible mistakes thanks to the 100% type coverage of OCaml. Syntax errors are a little bit less clear but still give a good feedback. Here are two little examples to give you a preview of these powers.

A React App In Seconds!

Because of its history, Reason maintains a strong relationship with React. In this regard, it is quite easy to setup a React project with Reason. I was able to confirm that during my reversi project.

create-react-app allows to focus on functionalities without worrying about toolchain configuration. So I took the decision to use it in association with reason-scripts.

create-react-app reversi-reason --scripts-version reason-scripts

After the install step, I found myself with this familiar folder structure:

reversi-reason/
  README.md
  node_modules/
  package.json
  bsconfig.json
  .gitignore
  public/
    favicon.ico
    index.html
  src/
    index.re
    index.css
    app.re
    app.css
    logo.svg

The only difference with classic JS projects are files with a .re extension (which are, as you probably guessed, Reason files), and the bsconfig.json file, which is the BuckleScript configuration.

{
  "name": "reversi-reason",
  "sources": [
    "src"
  ],
  "bs-dependencies": [
    "reason-react",
    "bs-jest"
  ],
  "reason": {
    "react-jsx": 2
  },
  "bsc-flags": [
    "-bs-super-errors"
  ],
  "refmt": 3
}

The BuckleScript configuration is a kind of mix between a linter config (like .eslintrc), and a compiler config (like .babelrc). It's quite normal because BuckleScript fulfills these 2 missions.

Domain Driven Design Made Easy

When I start a fresh new project, I always try to define the ubiquitous language and the associated domain objects before starting to code. For this project, I already knew my domain, because the reversi game is my favorite project to learn a new language, and I've already written a lot of code around it.

So, my domain objects are the following: Game, Player, Board and Cell. My first step was to create one module per object, with the associated test file. You can see them find them in the reversi-reason Github repository!

src/
├── app.re
├── app_test.re
├── board.re
├── board_test.re
├── cell.re
├── cell_test.re
├── game.re
├── game_test.re
├── index.css
├── index.re
├── player.re
└── player_test.re

In OCaml / Reason, every file maps to a module; this built-in capability empowers most projects with an out of the box Domain Driven Design architecture and a strong expressivity.

There's no need for require, use, or import to use a module in Reason. Just call the module directly. This way, all the domain objects I talked before are automatically available through their names.

It's also possible to manually create modules using the module keyword. So, you can nest and access them using the dot notation (eg: MyModuleFile.MyModule.MySubModule.myFunction).

/* MyModuleFile.re */
module MyModule = {
  module MySubModule = {
    let myFunction = () => "hello";
  };
};

/* ... in another file ... */

let message = MyModuleFile.MyModule.MySubModule.myFunction;

In fact, you don't have to use the dot notation every time you want to access a value in a module. The Module definition can be opened both locally and globally to avoid this repetition.

An example "local" opening is the applyCellClick function below. The Board.({ ... }) call exposes applyCellChange and getCountForColor from the Board module without having to repeat the module name.

/* game.re */
let applyCellClick = (game, x, y) => Board.({
    let color = Some(currentPlayer(game).color);
    let board = applyCellChange(game.board, { x, y, color });

    switch(getCountForColor(board, None)) {
        | 0 => { ...game, finished: true }
        | _ => switchPlayer({ ...game, board })
    };
});

This "local" opening could have been replaced by a "global" opening at the top of the Game module. Nevertheless, using global open is not advised and must be used sparingly because it can break reasoning.

/* game.re */
open Board;

let applyCellClick = (game, x, y) => {
    /* ... */
};

An Imperative Way Out

Reason uses a lot of concepts that are not always easy to handle (recursivity, currying, ...). Fortunately, it isn't as strict as Haskell, and it makes it possible to use some imperative and unsafe code when needed. This pragmatic approach is well highlighted in the Reason website.

Reason has great traditional imperative & mutative programming capabilities. You should use these features sparingly, but sometimes they allow your code to be more performant and written in a more familiar pattern.

Here is one of the "ugliest" code snippets from my reversi-reason project. This function collects all flipped cells in a predefined direction. It's the most suitable example because it uses a lot of "legacy" (hear "not functional") capabilities that are allowed by Reason.

/* board.re */
let flippedCellsInDirection = (board, cell, (dirx, diry)) => {
    let cells = ref([]);
    let cursor = ref((cell.x + dirx, cell.y + diry));
    let break = ref(false);

    while (! break^) {
        cells := switch(cursor^) {
            | cursor when isOutOfBound(board, cursor) => break := true; []
            | (x, y) => switch(getCell(x, y, board.cells)) {
                | None => break := true; []
                | color when (color == cell.color) => break := true; cells^
                | _ => {
                    cursor := (x + dirx, y + diry);
                    [{ x, y, color: cell.color }, ...cells^]
                }
            }
        };
    };

    cells^
};

When you look at this code, the first thing that stands out is the usage of the well-known while loop. Effectively, Reason (as well as OCaml) allows the usage of imperative loops.

Also, to be able to break a while loop, I had to use a mutable flag. In Reason, all variables are immutable by nature. To be able to mutate a value, I had to wrap it with a ref that acts like a box.

Afterward, to retrieve the underlying ref value, the postfix ^ operator is used. The truth is that ref is just a syntatic sugar for a predefined mutable record type. Test by yourself!

In the next sections, I'll try to cover how strong the link between Reason and React is. First, talking about ReasonReact, and then talking about the associated tests with Enzyme.

Built-in JSX support

Before going further, you should know that Reason natively includes JSX support. In fact, JSX only acts as a syntaxic sugar which is translated to normal function calls wrapped into expressions.

JSX translates to a make function call on the same module name as the JSX tag:

<MyComponent foo={bar} />

Becomes

([@JSX] MyComponent.make(~foo=bar, ~children=[], ()));

That's why it's necessary to properly name modules. If you wish, you can still create multiple components in the same file thanks to nested modules.

The Reason JSX syntax is not exactly the same as the JavaScript one. Indeed, there's no props spread but children spread. i.e. you can't do <Foo {...bar} /> but you can do <Foo> ...baz </Foo>.

ReasonReact As Cornerstone

How to create React components in Reson? Here is an example:

/* player.re */

let component = ReasonReact.statelessComponent("Player");

let make = (~player, _children) => {
    ...component,
    render: (_self) => 
        <div className=("player")>
            (stringToElement(player.name))
        </div>
};

I created the component template in combination with the statelessComponent function from the ReasonReact module. Spreading ...component is a bit like saying my component "inherits" from statelessComponent, except that
class components don't exist in Reason.

The "Player" string passed to statelessComponent primarily acts as a debug marker, it's the ReactJS equivalent of displayName.

Redux-Like Components!

While the statelessComponent acts as a functional component mixin, there's also another special ReasonReact function called reducerComponent. This function allows to directly include a "state machine"-like architecture into our components.

Using this component requires to define an initialState and a reducer function, which contain all the state manipulation logic. Those who have already used redux will certainly recognize this pattern.

/* game.re */

/* ... action type, state type and reducer ... */

let component = ReasonReact.reducerComponent("Game");

let make = (~game, _children) => {
  ...component,
  initialState: () => { game, message: None },
  reducer,
  render: (self) => {
    let { game, message } = self.state;

    let messageElement = switch(message) {
        | None => nullElement
        | Some(message) => stringToElement(message)
    };

    <div className="game">
        (messageElement)
        <Board
            board=game.board
            onCellClick={(x, y) => self.send(Click(x, y))}
        />
        /* ... */
    </div>
  }
};

In combination with the reducerComponent, it is usual to define 2 types:

One type for the actions (represented as a variant), and
One type for the state (represented as a record)

This way, Reason is able to infer by itself the initialState type. The action type is used to represent actions which can then be pattern-matched in the reducer function.

/* game.re */

type action = Click(int, int) | Restart;

type state = { game, message: option(string) };

let reducer = (action, state) => switch (action) {
    | Restart => ReasonReact.SideEffects(locationReload)
    | Click(x, y) => switch(applyCellClick(state.game, x, y)) {
        | game => ReasonReact.Update({
            game,
            message: None
        })
        | exception Board.InvalidMove => ReasonReact.Update({
            ...state,
            message: Some("Invalid Move")
        })
        | exception InsolubleGame(game) => ReasonReact.Update({
            game,
            message: Some("No One Can Play")
        })
        | exception CantSwitchPlayer(game) => ReasonReact.Update({
            game,
            message: Some("Opponent can't play, play again!")
        })
    }
};

/* ... react component ... */

According to the Reason philosophy, the reducer must be pure. Also using a pure function makes the code much more testable and easier to read. Only 4 distinct values can be returned:

ReasonReact.NoUpdate: don't update state
ReasonReact.Update: update state
ReasonReact.SideEffects: don't update state but trigger a side effect
ReasonReact.UpdateWithSideEffects: update state and trigger a side effect

A Rigorous Interoperability

Reason allows to communicate with other languages thanks to Externals (also known as FFI or "interop"). Using "Externals" is the way to go to write safe, predictive, and reliable typed code between Reason and any other language.

An Externals is a kind of typed contract / interface between the safe Reason world and the unsafe external world. It's this system that allows the bs-jest and bs-enzyme libraries that you'll find further to work.

Here is a simple example from the famous alert() function of JavaScript, which takes one arg and returns nothing (a.k.a. a "unit").

[@bs.val] external alert : string => unit = "alert";
alert("hello");

If you're an adventurous man, and you are using Reason with JavaScript, you can also use JS code through the JS module, or inject code directly.

/* Using the JS module */
Js.log("I'm logged via JS Module externals");

/* Direcly inject raw code */
[%bs.raw {| console.log("I'm logged via raw JS code"); |}];

A complete guide for JavaScript interop is available on Reason and Bucklescript documentations.

Unit Testing Reason Code

Under the hood, "create-react-app" uses Jest as test runner thanks to the bs-jest binding. Nothing special about it, the test architecture is almost the same as in JavaScript.

/* board_test.re */
open Jest;
open Expect;

describe("Board", () => {
    /* ... */

    describe("getCountForColor", () => {
        test("should return the number of cells of corresponding color", () => Board.({
            expect(getCountForColor(init(4, 4), Some(Cell.Black))) |> toBe(2);
            expect(getCountForColor(init(4, 4), Some(Cell.White))) |> toBe(2);
            expect(getCountForColor(init(4, 4), None)) |> toBe(12);
        }));
    });

    /* ... */
});

By the way, it's also possible to use enzyme to test components as in any other ReactJS project.

/* cell_test.re */
open Jest;
open Enzyme;
open Expect;

configureEnzyme(react_16_adapter());

describe("<Cell />", () => {
    test("should render a disk with the right identifier", () => {
        let test = (color, expectedClass) => {
            let wrapper = shallow(
                <Cell
                    color
                    onClick=((_) => ())
                />
            );

            expect(wrapper |> find(expectedClass) |> length) |> toBe(1);
        };

        test(Some(Cell.Black), ".cell.black");
        test(Some(Cell.White), ".cell.white");
        test(None, ".cell");
    });
});

A Pragmatic Community

During development, I had plenty of questions about best practices. Faced with the lack of documentation, I went to the language Discord.

Despite a notable lack of online resources, Reason benefits from a large and reactive community (~200 people always connected). My questions didn't go unanswered more than 5 minutes.

Recently, I also discovered that the community has introduced a centralized forum that appears to be very crowded and active. There are also fabulous podcasts that I've all listened to! Don't hesitate to listen to them, this initiative is rare enough to be highlighted.

Some of these quotes from other developers sound very relevant to me. They perfectly reflect the language's philosophy, which seems to be pragmatism.

My philosophy is that my time is more valuable than the computer’s time, so if I write it in a way that I understand it, it’s OK. - jdeisenberg

If this is a game that is going to be played by a human, then if evaluation of the board takes 2 milliseconds instead of 2 microseconds, will the person notice? Probably not. - jdeisenberg

A section from the "what and why?" of the language website confirms that this philosophy is not specific to the community, but to the language itself.

An eye for simplicity & pragmatism. We allow opt-in side-effect, mutation and object for familiarity & interop, while keeping the rest of the language pure, immutable and functional.

Indeed, Reason is really a pragmatic language, and I like it. Moreover, this principle is consistent with agility, which we practice on every project at marmelab.

Is This Ready To Use ?

Yes, it is ready to use in production! Reason is supported and used by Facebook everyday on big projects. This way, I think you can fairly trust it.

Moreover, Reason is a very stable programming language, it results from decades of research and compiler engineering from the OCaml language. Its interoperability capabilities give to Reason the ability to be integrated into any existing JavaScript project incrementally, so why wait?

On my side, there's no doubt that I would like to develop with Reason again. During this project, I've never felt stuck, everything is designed to produce clean code, with pragmatic alternatives in case of difficulty! The developer experience is just awesome!

To finish, its active community and cross-platform capabilities make me believe that Reason still has some beautiful days ahead. So, don't pass by it! And so, using types saves kittens, don't forget that!

You can test Reason without setting up a complete stack thanks to the online compiler which is very complete and impressively fast!

If you want to read more on the subject by other authors, I recommend the following:

Elixir GenServer Explained to Redux Developers

Demangeon Julien — Tue, 19 Dec 2017 20:30:31 +0000

Note: This post was originally posted on marmelab.com.

Although very young, the Elixir language has gained a lot of popularity since its creation. Whereas based upon an ancestral language / platform called Erlang BEAM, Elixir provides strong functional, concurrent and distributed capabilities. Its feel modern and fun. That's why I just could not resist to play with it.

In this post, I'll give the keys that helped me understand GenServer, the cornerstone of a lot of Elixir distributed applications / systems.

The GenServer Swiss Knife

Like any other language built on top of the Erlang VM, Elixir relies on lightweight and isolated threads (usually called processes), and messages passed between these threads. The key benefits of this mechanism are primarily fault-tolerance, distributivity and scalability.

The gen_server (for "Generic Server") is a standard module included in the underlying Erlang OTP (Open Telecommunications Platform). It offers abstractions to build a basic client <-> server system with a common shared resource (state). It was used in a lot of Open Source projects in the Elixir community.

GenServer usually acts as a central registry with its own state, on which we can perform (a)synchronous actions via message passing. In the web dev world, it can be roughly compared to an asynchronous Redux store.

So how does it compare with Redux? In the example below, I'll create a simple registry of quotes. I'll use both GenServer and Redux to show what they have in common.

The Elixir Version

Here is how to implement a registry of quotes with GenServer:

# quotes_list.ex

defmodule QuotesList do
    use GenServer

    # Client API

    def start_link do
        GenServer.start_link(__MODULE__, :ok, [])
    end

    def read(pid) do
        GenServer.call(pid, {:read})
    end

    def add(pid, quote) do
        GenServer.cast(pid, {:add, quote})
    end

    # Server Callbacks

    def init(:ok) do
        {:ok, []}
    end

    def handle_call({:read}, from, list) do
        {:reply, list, list}
    end

    def handle_cast({:add, quote}, list) do
        {:noreply, list ++ [quote]}
    end
end

In this snippet, I define an Elixir module called QuotesList. Since GenServer comes from an underlying Erlang library, it was provided through an Elixir Bridge with a kind of "mixin" (thanks to the "use" macro).

In the Elixir world, start_link is the common method name for starting a process which is attached to the current supervision tree. The third parameter of this method represents the initial state (an empty array) of our QuotesList module.

The Client API section can be defined as the public interface of our module. We can call read, add, ... from the outside to manipulate or access our store at our convenience.

On the other side, the Server Callbacks section is responsible for the inner working state manipulation of the GenServer. It responds respectively to the different calls / casts made in the Client API section, thanks to the strong pattern matching capabilities of Elixir. In the next chapter, I'll cover this section in a deeper way.

To conclude this part about the Elixir implementation, here is an example of GenServer usage for our QuotesList module.

# another_module.ex

# Start the quote server
{:ok, pid} = QuotesList.start_link

# Adding some items
QuotesList.add(pid, "All that glitters is not gold.")
QuotesList.add(pid, "Elementary, my dear Watson.")
QuotesList.add(pid, "Houston, we have a problem.")

# Read the quotes back
QuotesList.read(pid)

The Redux Version

On the Redux side, here is the equivalent of the preceding example in ES6.

// quoteListStore.js

import { createStore } from 'redux'

function quoteListReducer(state = [], action) {
    switch (action.type) {
    case 'add':
        return [ ...state, action.quote ];
    default:
        return state;
    }
}

return createStore(quoteListReducer)

Contrary to the Elixir implementation, there's no need to create a "read" method in Redux. We can access the state value directly with getState on the store.

Furthermore, the initial state is defined directly into the default value of the reducer and not in the createStore initializer.

Elixir relies on atom tags to bring matching capabilities to our handlers. In EcmasScript, we use a type attribute to match the right case of the reducer switch.

In Elixir, it's necessary to access data by message passing from another process. It's not required in the EcmaScript EventLoop which is mono-threaded.

And here is how to use the Redux registry in ES:

// index.js

import quoteListStore from './quoteListStore.js';

const addQuote = item => ({ type: 'add', quote });

quoteListStore.dispatch(addQuote('All that glitters is not gold.'));
quoteListStore.dispatch(addQuote('Elementary, my dear Watson.'));
quoteListStore.dispatch(addQuote('Houston, we have a problem.'));

quoteListStore.getState();

Call, Cast & Info Callbacks

If you've never encountered GenServer before, you've certainly asked yourself about the meaning of cast and call. These "verbs" corresponds to two different types of actions which can be triggered.

The cast action consists of an asynchronous action on the store, without any expectation on the result. It means that the action is queued and ready to be processed by corresponding GenServer handle_cast handler ASAP. It's similar to the famous dispatch method in Redux.

On the other side, the call action consists of a synchronous action on the store. The calling process is paused until the action is processed by corresponding GenServer handle_call handler. It's most often used to access the data.

Another info action exists, but is not usually used. This action is triggered when an incoming message comes from an unknow source / process (not the Client API section). The corresponding handle_info can be compared to some kind of "catchAll" handler (default case in a Redux reducer switch / case).

Conclusion

If I had to compare Redux and Elixir GenServer, I would say that:

GenServer != Redux because of the nature of each running platform.Erlang BEAM / Elixir is distributed, EcmaScript is not.
GenServer offers kind of RPC capabilities through calls. Redux doesn't.
When the only purpose of Redux is to manipulate the "state", it's not the case of GenServer.
Redux reducer logic is centralized, GenServer logic is split in multiple handle_* methods.

You may also be interested by the following related links to go further into Elixir and GenServer.

Let's cook some Crystal!

Demangeon Julien — Fri, 15 Dec 2017 19:15:58 +0000

Note: This post was originally posted on marmelab.com.

Crystal? Some of you may have heard about it in the best TV series of all the time, Breaking Bad! Thankfully, we're not gonna talk about drugs in this post, but about a fabulous programming language I've heard of recently.

In my constant quest to discover new languages and paradigms, I came across Crystal. Crystal is a compiled, statically typed, and object-oriented language. Its syntax is strongly inspired by Ruby, and its goals are mostly the same as Golang.

As usual when I learn a new language, I started a fresh new project that I could publish on GitHub. My use case: a CLI app intended to facilitate Caesar Cipher operations.

During this post, I'll try to give a large overview of the language capabilities, together with code snippets.

Ruby, Are You There?

As I said in the introduction, a lot of the Crystal syntax concepts were taken from Ruby. The main reason of this syntax choice comes from the language creators' background.

Ruby is an object-oriented, reflective and dynamically typed programming language. It was developed 20 years ago to provide productivity and fun to developers. It was used in many medium and large projects like GitHub, Diaspora or Redmine (my first happy Ruby experience), and made popular by RoR (Ruby on Rails).

While Ruby is an interpreted and multi-platform language (which runs on YARV VM), Crystal is not. That means Crystal needs to be compiled on the platform where it'll be executed.

As you are going to see later, Crystal is not Ruby, it mimics Ruby. If you're a Rubyist, you'll be a little disappointed by static typing and inference. Even so, many basic concepts have been taken from Ruby. Among them, we find "Blocks", "Symbols" and "Everything is an object" principle.

Crystal Blocks

Blocks are heavily used in Crystal (as much as in Ruby). They allow to capture a block of code with its own context, and execute it later. It's almost the same thing as closures or anonymous functions in other languages.

Here is a part of code from my toy project.

(1...Encoder::ALPHABET.size).each do |shift|
    decoded = Encoder.decode(encoded, shift)
    candidates << { key: decoded, count: ((decoded.split(' ') - [""]) & dictionary).size }
end

In this example, the block which is defined between do |shift| and end is called for each letter between 1 and Encoder::ALPHABET.size (thanks to Crystal Range).

The last letter (defined by Encoder::ALPHABET.size) is excluded from the range because of .... To include the last letter, it would been necessary to use .., which is inclusive.

Within the .each call, the block is "yielded" with the corresponding function from range.cr. In fact, a block is called with the yield keyword from Crystal as described in the doc example below.

def twice
  yield
  yield
end

twice do
  puts "Hello!"
end

# => prints "Hello!" twice

Symbols

The Symbol is another interesting concept also found in Ruby. A Crystal Symbol is a constant which is identified by its name. Internally, Crystal holds a registry of Symbols, which are represented as an Int32.


:hello === :hello # true
:hello === "hello" # false

Symbols are closely the same as Elixir atoms.

Everything is an Object

In Crystal, as in Ruby, everything is an object. Effectively, since reading post, you've already experienced no less than 3 objects (Blocks, Symbols and Ranges). Just like in Ruby, all objects extend Object. When you write Crystal code, most of the time, you write classes.

module Curses
  class Window
  end
end

Curses::Window.new

However, Crystal can also be used in a functional way. As a module requires a class, the solution is to make the module "self-extended". For instance, I can write an Encoder module that does not contain a class by using extend self:

module Encoder
  extend self

  ALPHABET = "abcdefghijklmnopqrstuvwxyz "

  def encode(uncoded : String, shift : (Int32 | Int64))
      dictionary = create_shift_dictionary(shift)
      normalized_uncoded = normalize(uncoded)

      normalized_uncoded.tr(dictionary.keys.join, dictionary.values.join)
  end

  [...]

end

Now I can call the encode() method directly, without a supporting class:

include Encoder
encode('o')

Crystal Functionalities

Whereas Crystal picks some concepts from Ruby, it also comes with a lot of unexpected functionalities such as Type inference, concurrency and compile time null reference checks.

Type Inference at its Core

Type inference is one of the most valuable Crystal functionnalities. It gives you the power to delegate the type attribution process to the compiler. Moreover, in association with Union Types, it allows to give different types to the same variable.

if 1 + 2 == 3
  a = 1
else
  a = "hello"
end

a # : Int32 | String

In the example above, the type of a is automaticaly infered from the AST (Abstract Syntax Tree) at compile time. It's also possible to assign the type manually, as shown below.

a : (Int32 | String) = 1

Type Inference also works with Arrays, Tuples, NamedTuples, Hashes...

[1, "hello", 'x'] # Array(Int32 | String | Char)
{1, "hello", 'x'} # Tuple(Int32, String, Char)
{name: "Crystal", year: 2017} # NamedTuple(name: String, year: Int32)
{1 => 2, 'a' => 3} # Hash(Int32 | Char, Int32)

Syntaxic sugar and shortcuts exist to declare empty typed structures:

[] of Int32 # same as Array(Int32).new
Tuple(Int32, String, Char) # Tuple(Int32, String, Char)
NamedTuple(x: Int32, y: String) # NamedTuple(x: Int32, y: String)
{} of Int32 => Int32 # same as Hash(Int32, Int32).new

Concurrency with Channels and Fibers

In addition to its fun syntax and static typing system, Crystal brings some concurrency capabilities (inspired by CSP) through Fibers. It's one of the most valuable functionalities of Crystal in comparison to other compiled languages

Concurrency consists of running several tasks in the same time interval through operation switching. This concept is often mistaken with parallelism, which consists of running multiples operations simultaneously (mathematically, in parallel).

Crystal Fibers can be compared to Golang's Goroutines. The main difference lays in the way Crystal allocate operations to CPUs. When Golang uses all availables CPUs, Crystal is only capable of using one of them for the moment.

Fibers use the same message passing system as Golang to achieve synchronisation and data passing between Fibers. It uses a kind of light-weight pipes called Channels.

Here is a little example of asynchronous operations written with Goroutines and Fibers. It'll give you a more concrete example of Fibers usage.

// GOLANG

messages := make(chan string)
defer close(messages)

go func() {
    time.Sleep(time.Second * 5)
    messages <- "ping"
}()

msg := <-messages // Program will wait for 5s
fmt.Println(msg) // => ping

# CRYSTAL

messages = Channel(String).new

spawn do
  sleep 5.seconds
  messages.send("ping")
end

msg = messages.receive # Program will wait for 5s
puts msg # => ping

In this example, I used the channel system to provide an exchange pipe between the main thread and the routine (async thread).

To declare an async thread, the spawn keyword with an associated code block is used on the Crystal side. With Golang, an anonymous function is started in a goroutine via the go keyword ahead of it.

At assignation time (in this example), the main thread is stopped until the corresponding channel sends data back.

Null Reference Checks

In you career, you've certainly heard of "NullPointerException" or the The Billion Dollar Mistake. This well known concept comes from the fact that it's deadly simple to call on a reference which is Null.

In Crystal, this kind of exception can't happen. A checks is issued at compile time to prevent it, thanks to Static Typing.

if rand(42) > 0
  hello = "hello world"
end

puts hello.upcase

The previous code will fail with a compile time exception.

$ crystal hello_world.cr
Error in hello_world.cr:5: undefined method 'e' for Nil (compile-time type is (String | Nil))

puts hello.upcase
          ^~~~~~

This exception occurs because hello can be either String or Nil, and there's no upcase method on Nil.

Crystal Tooling

Another argument in favour of Crystal is its great standard library, which covers most of the usual needs. A lot of tools are bundled with Crystal itself (HTTP, JSON, Markdown Parser and so on), making it ready to use instantly.

For example, here is a small HTTP server written in Crystal.

require "http/server"

server = HTTP::Server.new(8080) do |context|
  context.response.content_type = "text/plain"
  context.response.print "Hello world, got #{context.request.path}!"
end

puts "Listening on http://127.0.0.1:8080"
server.listen

Crystal Ecosystem

Despite its youth, Crystal has gained a lot of popularity in no time thanks to its Ruby-like syntax. It actually shares hype in the declining rubyist's world with another language, Elixir.

Most Crystal-based projects are very confidential for the moment. Nevertheless, some of them have gained popularity rapidly:

KEMAL - Fast, Effective, Simple web framework (including websockets, etc)
CRECTO - Database wrapper (inspired by Elixir's Ecto)
SIDEKIQ - Simple, efficient job processing (Including web monitoring UI)

Crystal bundles with an easy-to-use dependency management system. It consists of a simple YAML file (named shard.yml) listing dependencies and associated versions. It looks like the following:

name: MyAmbitiousProject
version: 0.1.0

dependencies:
  openssl:
    github: datanoise/openssl.cr
    branch: master

development_dependencies:
  minitest:
    git: https://github.com/ysbaddaden/minitest.cr.git
    version: ~> 0.3.1

license: MIT

This way, a simple crystal deps command is enough to retrieve needed dependencies into the project tree.

If you're curious to discover other interesting Crystal projects, a dedicated portal has been created specially for that purpose.

Conclusion

Crystal has been first released in 2014. It's a relatively young language for the moment, but it promises a bright future.

Its advanced type inference system combined with static typing and union types give Crystal the feeling of a higher-level scripting language, with performances close to the metal. It can be a good alternative to Golang for modest performance needs.

Moreover, if you're already familiar with Ruby, Crystal is the first choice language. It offers good performances without having to bear Golang's heavy syntax.

Finally, if you like this language but need distributed operations, you can take a look at the Elixir language, which is very appreciated by rubyists, too.

If you're interested in the project associated to this post, you should take a look at the repository. It'll also give you a good start to dockerize Crystal apps in the same time.