Forem: Roman Liutikov

Native Apps with ClojureScript, React and Static Hermes

Roman Liutikov — Mon, 17 Nov 2025 16:00:00 +0000

What you see in the demo below is ClojureScript UIx app driving native window with ImGui UI via custom React reconciler. Both hot-reloading and REPL-driven development are supported, as you'd expect it from a typical ClojureScript project.

The JavaScript side of the app runs in Hermes engine, which exposes ImGui to JS env. However, a release build of this exact app is a fully native 8MB executable. You can try it yourself on macOS or Linux.

While Hermes engine is a JavaScript VM, it is also an AOT compiler for JavaScript that emits either Hermes byte code or C.

Compiling this sample JavaScript program with Static Hermes: shermes -emit-c index.js

print(1 + globalThis.value);

outputs the following C program or 50KB executable, when compiled straight into a binary shermes -Os index.js

locals.t0 = _sh_ljs_get_global_object(shr);
frame[3] = _sh_ljs_try_get_by_id_rjs(shr,&locals.t0, get_symbols(shUnit)[1] /*print*/, get_read_prop_cache(shUnit) + 0);
locals.t0 = _sh_ljs_try_get_by_id_rjs(shr,&locals.t0, get_symbols(shUnit)[2] /*globalThis*/, get_read_prop_cache(shUnit) + 1);
locals.t0 = _sh_ljs_get_by_id_rjs(shr,&locals.t0,get_symbols(shUnit)[3] /*value*/, get_read_prop_cache(shUnit) + 2);
np0 = _sh_ljs_double(1);
frame[1] = _sh_ljs_add_rjs(shr, &np0, &locals.t0); // 1 + globalThis.value
np0 = _sh_ljs_undefined();
frame[4] = _sh_ljs_undefined();
frame[2] = _sh_ljs_undefined();
locals.t0 = _sh_ljs_call(shr, frame, 1);
_sh_leave(shr, &locals.head, frame);
return np0;

The above demo is 8MB executable on macOS, where 3MB is Hermes runtime, 1.8MB is React the JavaScript library compiled to C and the rest 3.2MB goes to native libraries ImGui and libwebsockets, and ClojureScript code compiled to C.

Here's a rough diagram of how things are bundled in the executable.

You may noticed there's untyped and typed JavaScript. When AOT compiling typed JavaScript, which can be a subset of either TypeScript or Flow, Hermes emits a more optimal C making it suitable for performance sensitive code, such as bindings to native libraries runnning in a hot loop.

As an example, compiling the following typed program with -typed flag: shermes -typed -emit-c index.js

function add(a: number): number {
  return a + globalThis.value;
}
print(add(1));

emits C that:

doesn’t create an add function/closure, instead the arithmetic is emitted inline
adds type assertion for unkown value globalThis.value
includes fewer symbols & caches

You can check both typed and untyped C here, here's also a diff to make it more obvious.

As I said earlier, the app runs raw JavaScript in dev to provide fast feedback loop and support interactive development. This is possible because Hermes has multiple tiers of code optimization:

During developent JavaScript VM loads the code and emits byte code (JIT) at runtime
You can also compile JavaScript to bytecode (.hbc file) ahead of time. This use case is quite popular in React Native, where application code is compiled into Hermes bytecode at build time to improve startup time of mobile apps, especially on Android.
Finally, with Static Hermes you can compile JavaScript to native object file. This way you get statically linked machine code, skipping parsing and JIT.

Now the best part about this setup is actually React itself. I didn't have to change anything in UIx to make it work with original imgui-react-runtime demo. The real power and a curse of React is that it is a platform independent abstraction. It's up to the host platform to provide specific realization of the contract. If you are Clojure developer, this might sound familiar to you.

The abstraction in question is React's reconciler. By implementing its interface you can create custom render targets, such as React Native, PixiJS React, react-three-fiber, you can even create Terminal UIs and PDFs with it. Heck, if you take a step back and think about the reconciler as a generic abstraction for turning declarative description of something into a set of imperative operations, you can even run hardware with it.

Ok, so how fast is AOT compiled JavaScript with Static Hermes? Here's a quick benchmark:

(simple-benchmark []
    (->> (range)
         (map inc)
         (filter odd?)
         (map #(zipmap (repeatedly % Math/random) (repeatedly % Math/random)))
         (take 1e2)
         (reduce #(apply + %1 (mapcat identity %2)) 0))
    1e2)

Optimized executable runs for ~6450 ms. Same code executed as JavaScript in Node and Bun runs for ~1100 ms. That's 6x slower, ouch. On the bright side, the executable is just a few MBs, while embedding JS VM will get you at least up to 60MB in the case of Bun. If you ask me, I'll always prefer 6x faster app, doesn't matter how big it is.

However, the case with React is quite specific. React being a declarative abstraction on top of imperative APIs almost always means that hot code paths are executed in underlying layers, whether it's browser's DOM, mobile views, WebGL or ImGui. And often times you have an option to use that lower level, more performant API directly, when needed.

Hermes was originally developed for React Native and mobile apps, with a focus on improving startup performance. Even generated native code can't compete with JITs in modern JavaScript engines.

Bonus point. Similarly to how shadow-cljs displays compiler warnings and errors in a browser, the demo ImGui app also captures and displays warnings and errors, as well as includes an error boundary component that catches runtime exceptions thrown during render phase.

Checkout the code of this project at roman01la/cljs-static-hermes, which is based off the work done by a member of Hermes team, at tmikov/imgui-react-runtime. If you are interested in driving native window from runtimes like Node or Bun, checkout this project showcasing use of Bun's FFI to interface with GLFW and OpenGL: roman01la/hra.

Advanced Beginner’s guide to ClojureScript

Roman Liutikov — Fri, 07 Nov 2025 20:00:00 +0000

So how does your tiny ClojureScript program transform into runnable JavaScript?

(ns app.core)

(println "Howdy!")

First of all, you need a compiler! ClojureScript is a hosted language, which means you have to compile it into JavaScript, unless you want to ship the compiler into a browser to be able to interpret ClojureScript at runtime, which is slow and in most cases doesn't make sense (but maybe you wanna build online REPL, that's ok).

This is not going to be easy. ClojureScript sits on top of JavaScript, the compiler is a library written in Clojure, which in turn hosts on JVM. This means you need Java installed, huh.

I prefer to use sdkman to manage my Java installations.

sdk install java

Next step is to install Clojure, head over to installation guide at clojure.org. If the following command returns 2, you are good!

clj -M -e "(inc 1)"

Create project directory somewhere and put deps.edn file into it with the following contents. deps.edn is kind of package.json, if you are coming from JavaScript world.

{:deps {org.clojure/clojurescript {:mvn/version "1.12.42"}}}

I hope you have Node installed? Now we can run ClojureScript REPL in Node environment. This line runs JVM process that loads Clojure, that loads ClojureScript compiler, starts Node process and runs it in REPL mode.

clj -M -m cljs.main --target node --repl

If this runs, it means you are hooked into Node process, nice!

(js/console.log "hello")

Try to list files in the current directory using Node'js fs module.

(def fs (js/require "fs"))
(.readdirSync fs "./")

It works? Great, let's move onto actually compiling a project into JavaScript files. Put our tiny program into src/app/core.cljs file. src is your project's source root directory. Also note that namespace name app.core (that's how modules are called in Clojure) in the code looks exactly like app/core.cljs. That's a convention in Clojure projects: namespace names should reflect directory structure starting from project's source root directory.

(ns app.core)

(println "Howdy!")

Now let's compile the program into Node script

clj -M -m cljs.main --target node --output-dir ./out --compile app.core

and run the script

node out/main.js

Do you see the output? Great, let's stop here and inspect compiler's output. Specified out directory includes whole bunch of files.

ls out

app cljs cljs_deps.js cljsc_opts.edn goog main.js nodejs.js nodejs.js.map nodejscli.js nodejscli.js.map

The actual compiled JavaScript program from your src/app/core.cljs file is in out/app/core.js, let's see what's inside

cat out/app/core.js

// Compiled by ClojureScript 1.12.42 {:target :nodejs, :nodejs-rt true, :optimizations :none}
goog.provide("app.core");
goog.require("cljs.core");
cljs.core.println.call(null, "Howdy!");

//# sourceMappingURL=core.js.map

cljs.core.println.call(null, "Howdy!"); is your (println "Howdy!"), where println is actually a part of implicit cljs.core namespace, which provides stdlib.

goog.provide and goog.require is a part of JavaScript module format of Google's Closure Compiler. Wait, what? Another compiler? Yeah, you see... ClojureScript compiler takes your source and outputs equivalent JavaScript code. Similarly to out/app/core.js, there's also out/cljs/core.js file which is compiled stdlib of ClojureScript. This just means that ClojureScript compiler is a source to source compiler. But how do we bundle all those JavaScript files into production ready bundle?

That's the job of Closure Compiler. Closure is an optimizing JavaScript compiler that ClojureScript is using since its initial release, in 2011. At the time JavaScript didn't have standard module format, remember AMD, UMD, RequireJS and CommonJS? Closure folks at Google invented another one, where goog.provide declares a module and goog.require imports another module.

ClojureScript emits Closure's module format so that the compiler can pick up and optimize generated JavaScript. Putting everything together, that's the stack you have to deal with:

To plug Closure into the equasion and produce optimized JavaScript bundle run the following command.

--optimizations advanced option is what tells Closure to gather all generated JavaScript files into a single bundle and optimize the program by removing unused code. Closure is quite powerful compiler, you can learn more about all kinds of optimizations it performs in this handbook that I created a while ago.

clj -M -m cljs.main --target node --output-dir ./out --optimizations advanced --compile app.core

Output out/main.js file is now a self-contained JavaScript program.

Let's make something cool now. Put the following code into src/app/core.clj

(ns app.core
  (:require [clojure.java.shell :as sh]))

(defmacro version []
  (:out (sh/sh "git" "rev-parse" "--short" "HEAD")))

and update your src/app/core.cljs to

(ns app.core
  (:require-macros [app.core :refer [version]]))

(println "Howdy!" (version))

Now init a git repo in your project directory and make the first commit

git init
git add src
git commit -m "first commit"

Compile your ClojureScript project

clj -M -m cljs.main --target node --output-dir ./out --optimizations advanced --compile app.core

and run it

node out/main.js

The script should print last git commit hash. Now inspect JavaScript generated from src/app/core.cljs

cat out/app/core.js

The hash is embedded in the code!

// Compiled by ClojureScript 1.12.42 {:static-fns true, :optimize-constants true, :target :nodejs, :nodejs-rt true, :optimizations :advanced}
goog.provide("app.core");
goog.require("cljs.core");
goog.require("cljs.core.constants");
cljs.core.println.cljs$core$IFn$_invoke$arity$variadic(
  cljs.core.prim_seq.cljs$core$IFn$_invoke$arity$2(["Howdy!", "7f7e8c3\n"], 0)
);

That's the cool part about Lisps. You can write code in your program that runs at compile time on a program itself. It's almost like a compiler inside of a compiler, except that you don't need special plugins to traverse AST, in Clojure and Lisps in general you create macros, special functions that take code, transform it and return new code.

Now, everything that you learned here is only a part of the story. We haven't touched on the language itself, how real projects are built and how ClojureScript integrates with JavaScript's ecosystem of NPM packages. ClojureScript is definitely not the easiest beast to pick up, but if you are doing front-end development and feel like digging into alternatives and learning from other languages, then there are definitely a few things you can learn from!

Integrating ClojureScript with JavaScript tooling

Roman Liutikov — Sat, 25 Apr 2020 19:28:39 +0000

Recent release of ClojureScript introduced a new way to integrate with JavaScript tooling that makes it easier to import ClojureScript into existing JavaScript projects.

In this post we will explore how to setup and use these new features and integrate with bundlers such as Webpack.

First make sure you have OpenJDK installed and then install Clojure CLI. Once that is done, it's time to create a project.

Create project folder and navigate to it

mkdir hello-bundler && cd hello-bundler

Create Clojure deps file with ClojureScript in dependencies

echo '{:deps {org.clojure/clojurescript {:mvn/version "1.10.741"}}}' > deps.edn

Install Webpack and React. We are going to use React in this guide since it is exclusively used in ClojureScript for building web apps.

yarn add webpack webpack-cli react react-dom --save-dev

Next create basic Clojure project structure and entry point namespace

mkdir -p src/hello_bundler
touch src/hello_bundler/core.cljs

Here we require react and react-dom installed in node_modules. Put the code below into core.cljs.

(ns hello-bundler.core
  (:require [react]
            [react-dom]))

(def h1 (react/createElement "h1" nil "Hello!"))

(react-dom/render h1 (.getElementById js/document "root"))

Create ClojureScript build config file build.edn with the following contents

{:main hello-bundler.core
 :output-to "out/index.js"
 :output-dir "out"
 :target :bundle
 :bundle-cmd {:none ["npx" "webpack" "--mode=development"]
              :default ["npx" "webpack"]}}

When building ClojureScript sources the compiler will output entry point file into "out/index.js" and then run Webpack build with a command provided in :bundle-cmd option.

Now create Webpack config which will consume ClojureScript's output. Make webpack.config.js with the following contents

module.exports = {
  entry: "./out/index.js",
  output: {
    path: __dirname,
    filename: "bundle.js"
  }
}

Finally create index.html file in the root of the project which will load the final build artifact produced by Webpack.

echo '<div id="root"></div><script src="bundle.js"></script>' > index.html

And run ClojureScript build command

clj -m cljs.main -co build.edn --compile --repl

This will open your browser at http://localhost:9000/ and start ClojureScript REPL in the terminal where you can evaluate ClojureScript code in browser's context, try this for example

(js/alert "Hello!")

For a better interactive development experience you may want to connect to the REPL from your editor, for example here's a plugin for VS Code.

Production build

When building for production ClojureScript compiler will run a command specified in :default field in :bundle-cmd. To create production build and start a web server afterwards execute the following command.

clj -m cljs.main -co build.edn -O advanced --compile --serve

Conclusion

With a new :bundle compiler target ClojureScript is able to produce JavaScript that can be consumed by tools like Webpack that in turn will take care of including NPM dependencies into the final bundle, so ClojureScript compiler doesn't have to reinvent the wheel here.

That also means that existing JavaScript projects can depend on ClojureScript code much easier now. In the next post we will explore how to consume ClojureScript code from JavaScript modules.

Understanding Transducers in JavaScript

Roman Liutikov — Wed, 22 Apr 2020 14:03:45 +0000

I’ve found a very good article explaining Transducers. If you are familiar with Clojure, go and read it: “Understanding Transducers”. But if you are JavaScript developer and not used to read Lisp code, I’ve translated code examples from that article into JavaScript. So you can still read the article and see code examples here.

What are Transducers?

A quick noob intro: transducers are composable and efficient data transformation functions that doesn’t create intermediate collections.

In some languages this optimization is known as loop fusion or stream fusion. However transducers offer much more than that (at a cost of being purely runtime optimization).

Here’s a visualization to show the difference between chained transformations and transduced once.

Why use them?

The above visualization means that given transformations like map, filter, or basically any other operation on sequence of values, we want to compose them together and efficiently pipe every piece of data through them step by step. But the following example is not this kind of composition:

array
  .map(fn1)
  .filter(fn2)
  .reduce(fn3);

The above example doesn’t decouple transformation from data and creates arrays on every step in the chain.

Instead we want something like this:

const transformation = compose(map(fn1), filter(fn2), reduce(fn3));
transformation(array);

This way we can reuse the transformation and compose it with others. In order to achieve such composability these functions have to be generalized. Turns out all of them can be expressed in terms of reduce.

Code examples from the article

map and filter, and how they can be combined together:

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9].map((x) => x + 1);
// [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10].filter((x) => x % 2 === 0);
// [2, 4, 6, 8, 10]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
  .map((x) => x + 1)
  .filter((x) => x % 2 === 0);
  // [2, 4, 6, 8, 10]

map and filter can be implemented using reduce. Here’s map implementation:

const mapIncReducer = (result, input) => result.concat(input + 1);
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9].reduce(mapIncReducer, []);
// [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

Let’s extract incrementing function to allow it to be passed into reducer:

const mapReducer = f => (result, input) => result.concat(f(input));
[0, 1, 2, 3, 4, 5, 6].reduce(mapReducer((x) => x + 1), []);
// [1, 2, 3, 4, 5, 6, 7]

More usage examples of map reducer:

[0, 1, 2, 3, 4, 5].reduce(mapReducer(x => x - 1), []);
// [-1, 0, 1, 2, 3, 4]
[0, 1, 2, 3, 4, 5].reduce(mapReducer(x => x * x), []);
// [0, 1, 4, 9, 16, 25]

filter implementation using reduce:

const filterEvenReducer = (result, input) =>
  input % 2 === 0 ? result.concat(input) : result;
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10].reduce(filterEvenReducer, []);
// [2, 4, 6, 8, 10]

Again, extract predicate function, so it can be passed from the outside:

const filterReducer = (predicate) => (result, input) =>
  predicate(input) ? result.concat(input) : result;
[1, 2, 3, 4, 5, 6].reduce(filterReducer(x => x % 2 === 0), []);
// [2, 4, 6]

Combine both reducers together:

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
  .reduce(mapReducer(x => x + 1), [])
  .reduce(filterReducer(x => x % 2 === 0), []);
  // [2, 4, 6, 8, 10]

Similar to what you usually do with built-in array methods:

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
  .map(x => x + 1)
  .filter(x => x % 2 === 0);
  // [2, 4, 6, 8, 10]

Here are both reducers again and both of them are using array concat as a reducing function:

const mapReducer = f => (result, input) => result.concat(f(input));
const filterReducer = (predicate) => (result, input) => 
  predicate(input) ? result.concat(input) : result;

concat and + are both reducing operations, they take initial value and input, and reduce them to a single output value:

array.concat(4); // [1, 2, 3, 4]
10 + 1; // 11

Let’s extract reducing function, so it can be also passed from the outside:

const mapping = f => reducing => (result, input) =>
  reducing(result, f(input));
const filtering = predicate => reducing => (result, input) =>
  predicate(input) ? reducing(result, input) : result;

Here’s how reducers can be used now:

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
  .reduce(mapping(x => x + 1)((xs, x) => xs.concat(x)), [])
  .reduce(filtering(x => x % 2 === 0)((xs, x) => xs.concat(x)), []);
  // [2, 4, 6, 8, 10]

Type signature of reducers is result, input -> result:

mapping(x => x + 1)((xs, x) => xs.concat(x))([], 1); // [2] 
mapping(x => x + 1)((xs, x) => xs.concat(x))([2], 2); // [2, 3]
filtering(x => x % 2 === 0)((xs, x) => xs.concat(x))([2, 4], 5);
// [2, 4]
filtering(x => x % 2 === 0)((xs, x) => xs.concat(x))([2, 4], 6);
// [2, 4, 6]

Composition of reducers has the exact same type:

mapping(x => x + 1)(filtering(x => x % 2 === 0)((xs, x) =>
  xs.concat(x)));

So it also can be used as a reducer:

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
  .reduce(mapping(x => x + 1)(filtering(x => x % 2 === 0)((xs, x) =>
    xs.concat(x))), []);
  // [2, 4, 6, 8, 10]

Let’s use R.compose from Ramda library for better readability:

const xform = R.compose(mapping(x => x + 1),
                        filtering(x => x % 2 === 0));
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
  .reduce(xform((xs, x) => xs.concat(x)), []);
  // [2, 4, 6, 8, 10]

More complex example:

const square = x => x * x;
const isEven = x => x % 2 === 0;
const inc = x => x + 1;
const xform = R.compose(filtering(isEven),
                        filtering(x => x < 10),
                        mapping(square),
                        mapping(inc));
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
  .reduce(xform((xs, x) => xs.concat(x)), []);
  // [1, 5, 17, 37, 65]

Finally let’s wrap it into transduce function:

const transduce = (xform, reducing, initial, input) =>
  input.reduce(xform(reducing), initial);

Final usage example:

const xform = R.compose(mapping((x) => x + 1),
                        filtering((x) => x % 2 === 0));  
transduce(
  xform,
  (xs, x) => xs.concat(x),
  [],
  [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]);
// [2, 4, 6, 8, 10]
transduce(
  xform,
  (sum, x) => sum + x,
  0,
  [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]);
// 30

Check out transducers-js library for a complete and performant transducers implementation in JavaScript. Read about Transducer protocol which enables safe interop between libraries (like Lodash, Underscore and Immutable.js).

Transducers are a part of standard library in Clojure. Make sure to take a look at ClojureScript.

ClojureScript REPL Workflow

Roman Liutikov — Wed, 22 Apr 2020 13:48:23 +0000

Short write up on using REPL when developing UIs in ClojureScript.

Hot-reload on save or Eval in REPL?

Everyone's standard approach to hot-reloading is to use a tool (Figwheel or shadow-cljs) that reloads changed namespaces automatically. This works really well: you change the code, the tool picks up changed files, compiles namespaces and dependants, notifies REPL client which then pulls in compiled changes, and re-runs a function that re-renders UI.

The other approach is to use ClojureScript's REPL directly and rely only on eval from the editor. This more or less matches Clojure style workflow. This approach might be useful when you don't want tools overhead or hot-reloading becomes slow for you or you just used to this style of interactions. Also changing code doesn't always mean that you want to reload all the changes. On the other hand it is very easy to change a couple of top-level forms and forget to eval one of them.

For this to work you would have to run ClojureScript REPL env from Clojure REPL. Below is a handy function to start it from the REPL.

(ns user
  (:require [cljs.repl :as repl]
            [cljs.repl.browser :as browser]))

(defn repl []
  (let [env (browser/repl-env
              :port 3000 ;; starts web server on localhost:3000
              :launch-browser false ;; do not launch browser automatically, I find this annoying
              :static-dir ["." "out" "resources/public"])] ;; serves assets from those dirs
    (repl/repl env)))

Executing (repl) will start ClojureScript REPL and print Waiting for browser to connect to http://localhost:3000 ..., when you open http://localhost:3000 in your browser, it will load REPL client and connect to REPL server. Initially nothing is executed, you have to load your entry point core namespace manually from the REPL in your editor.

Once that is done you can edit code and eval forms as usual. One thing that is annoying in UI development with this approach is re-rendering UI manually after every eval, so all those changes are reflected on the screen. Here's a useful snippet that you can put in your ClojureScript code and eval when REPL starts. This hooks into REPL client and runs provided render function after every successful evaluation.

(comment
  (let [eval-js (.. js/clojure -browser -repl -evaluate_javascript)]
    (set! (.. js/clojure -browser -repl -evaluate_javascript)
          (fn [& args]
            (let [ret (apply eval-js args)]
              (when (.includes ret ":status :success")
                (render)) ;; your render function
              ret)))))

Preserving state between reloads

This one is more related to tools like Figwheel or shadow-cljs that reload the whole namespace on every change, but still can be useful even when reloading manually.

Every UI code has stateful parts, like global state in re-frame or global event handlers or routing library that keeps its state internally or WebSocket subscription, etc. Sometimes you just want code to be executed only once, not after every reload. defonce should be used in such cases, it evaluates binding only once.

(defonce db (atom {}))

⚠️ But be aware that binding to the result of calling JavaScript function that returns undefined will break this, because defonce checks for existence of the var which involves undefined check.

(defonce global-handler
  (.addEventListener "keydown" handle-keydown)) ;; returns `undefined`

Put a random value, for example nil, as a returning value instead

(defonce global-handler
  (do (.addEventListener "keydown" handle-keydown)
      nil))

REPL & asynchrony

Asynchrony might be annoying when working in REPL, since you won't get the value back, but rather a promise or something that eventually resolves or prints. Consider this example fetching data and returning a string in a promise.

(-> (js/fetch url)
    (.then #(.text %)))

Because you don't get the value itself you'd have to perform the request every time you change the code and want to see the result. What I'm usually doing in such cases is using defonce to cache raw data and then write code that works with it.

(defn fetch-url [url]
  (-> (js/fetch url)
      (.then #(.text %))))

@state ;; this will have response string, when evaluated after the form below 

(comment
  (defonce state (atom nil)
  (-> (fetch-url url)
      (.then #(reset! state %))))

REPL and global state in re-frame

One benefit of using global state, such as in re-frame, is that it's easily accessible from REPL. Evaluating @re-frame.db/app-db will give you the whole state of your system. More to that, you can deref re-frame subscriptions in global context.

@(subscribe [:issues/by-id 1])

When a subscription is parameterized on component's arguments it becomes less trivial to run it from the REPL

(defn issue-view [id]
  @(subscribe [:issues/by-id id]))

But it's possible to capture those arguments into global vars, for example with a small macro like this one

(defmacro capture [syms & body]
  `(do
     ~@(for [sym syms]
         `(def ~sym ~sym))
     ~@body))

The macro creates defs for every provided symbol that matches argument name. When evaluating the component, after it has been re-rendered and the scope was captured you can eval id in a global context to inspect the value.

(defn issue-view [id]
  (capture [id]
    @(subscribe [:issues/by-id id])))

⚠️ Note that creating global vars pollutes namespaces and might result in unexpected results when there's a “useful” var with the same name is defined.

Conclusion

Additional tools are not strictly required to be productive
Global state is a win win for REPL workflow
Asynchrony is not ideal for REPL workflow
Macros are fun, but should be used carefully

Feel free to comment and explain your REPL workflow in ClojureScript. Tips for Node.js and other runtimes are welcome.

Recursive Data Structures and Lazy Evaluation

Roman Liutikov — Sun, 07 Jan 2018 19:25:22 +0000

In this article I’m trying to explain to myself recursive data structures and how they work together with lazy evaluation. I happen to use them everyday, but never really thought about underlying implementation and the idea in general.

First we define a problem and describe recursive solution to it. After that we are going to implement a simple data structure, recursively. And once we get the idea, I’ll cover lazy evaluation and present lazy variant of that data structure.

The Problem

A typical problem in game development is collision detection. This technique is used to detect intersection between entities in coordinate space, such as enemies and bullets in shooter games for example.

Let’s imagine we are building 2D space shooter game. The game involves interaction between enemy entities and player’s missiles. In order to say if enemy entity is destroyed the game should be able to detect when the missile intersects an entity in two-dimensional coordinate space. In the context of our game collision detection algorithm could be described as the following:

Compose a list of all missile entities and a list of all enemy entities currently on the screen
Take the first/next missile entity from the list
Check it’s position against all enemy entities
- if position matches — collision is detected, remove both entities from the list and repeat from step 2
- otherwise — repeat from step 2 until the end of the list

As you can see the algorithm is straightforward and can be easily implemented via iteration.

for (let i = 0; i < missiles.length; i++) {
  const missile = missiles[i];
  for (let j = 0; j < enemies.length; j++) {
    const enemy = enemies[j];
    if (missile.intersects(enemy)) {
      // collision is detected
    }
  }
}

The problem however is that this naive approach takes O(n*(n-1)) or simply O(n^2) time to execute, meaning that every time the number of entities doubles the number of iterations required to detect collision between all entities is going to be four times larger. That is for 3 entities it takes roughly 9 iterations, 900 iterations for 30 entities, 90000 for 300 entities and so on. It’s totally fine to use this approach if a single check is relatively cheap operation and there are not much entities on a screen, but sometimes that’s not the case.

Solution

An obvious improvement would be to perform collision check for a given entity only against entities that are close to it, discarding all distant entities. Screen area can be divided into separate areas to form a grid and now for every entity we can perform collision check only against other entities currently in the same area. This technique is called space partitioning.

More sophisticated algorithm would be to use an adaptive grid, where cells can subdivide itself based on how much entities there are currently in a given cell.

Recursive algorithm

One of the algorithms implementing adaptive grid approach is called quadtree space partitioning. A quadtree is a data structure where every node has exactly four child nodes, hence the name. A quadtree is recursive because it is defined in terms of itself (every node, except leaf nodes, has another quadtree as its subtree).

Game canvas can be divided between those nodes where each node would represent an area on the screen defined by its bounds. A node may hold entities that are currently in the area which corresponds to that node. Quadtree nodes should be set a maximum number of entities they can hold (maximum number of entities in a given area). Once capacity of the node is full, the node subdivides itself into four child nodes and spreads its entities across child nodes with respect to their position on the screen. Subdivision can be infinitely deep, but usually the maximum number of levels is set.

The benefit of this approach is that in the best case, when all entities are spread evenly across the screen, the time required to perform all collision checks is reduced by 4x with every level of subdivision. That is, for 100 entities we get 2500 checks instead of 10000 with one level of subdivision, with two levels it is 625 checks and so on. This can reduce CPU load when applied correctly, but don’t forget that maintaining a quadtree also consumes CPU cycles and memory.

What we’ve just learned is that game development is fun and algorithms are not that hard, but most importantly in this section we’ve seen how recursive data structures could be a natural fit as a solution to a certain type of problems. Now let’s see the code.

Recursive data structures

In this section we are going to build List data structure, singly linked list to be precise. A list is fundamental data structure in functional programming known from early days of Lisp programming language. Take a look at type declaration of List data type in Haskell

data List a = Nil | Cons a (List a)

You can see that a list is either Nil (empty list) or Cons (construction) of a value and another list. Having another list as the next value in a list is what makes this data structure recursive or defined in terms of itself. The last value in a list is always Nil, it is used to signal that the end of list is reached. Nil is also treated as empty list, which means that this type inherits properties of a list. To get better understanding here’s how a list of three items could be written in pseudo code:

[1 [2 [3 nil]]]

A tuple of two items here is called a cons (construction) cell. A cons cell consists of a value at the first position (head) and a pointer to another value (tail). Thus a list of three items is essentially a chain of three cons cells with Nil in the last cell to indicate the end of list. This can be also visualized as the following:

A list built on cons cells has interesting properties such as access to the head of list and its tail in constant time. On the other hand lists doesn’t provide efficient random access such as in indexed collections (arrays).

Now that we know underlying representation of a list, lets create our own implementation. First we define data types — building blocks of a list. They are: List, Cons and Nil. List type is not really necessary to build a list, because a list is just a chain of cons cells with Nil in the end. It is rather a useful abstraction which could provide us with an API for creating and manipulating lists.

First we implement type Nil as JavaScript class and create an instance of the class, a variable named nil, to be used as a marker of the end of a list.

class Nil {
  toString() {
    return 'Nil';
  }
}

const nil = new Nil();

The next type is Cons. It is a class that accepts head value and tail — a reference to the next value. We also create a helper static method .of to avoid using new keyword.

class Cons {
  constructor(head, tail = nil) {
    this.head = head;
    this.tail = tail;
  }
  toString() {
    return `Cons(${this.head}, ${this.tail})`;
  }
}

Cons.of = (head, tail) => new Cons(head, tail);

Using just cons cells we can already create lists of values and access those values:

const list = Cons.of(1, Cons.of(2, Cons.of(3, nil)));

console.log(list.head);
console.log(list.tail.toString());
console.log(list.tail.head);
console.log(list.tail.tail.tail.toString());

As I mentioned before cons cells are enough for creating lists, but list abstraction built on top of Cons could save us a little time and code by providing an API to create and manipulate lists. So lets built List type on top of Cons:

class List extends Cons {
  constructor(head, tail) {
    super(head, tail);
  }
  toString() {
    return `List(${this.first()}, ${this.rest()})`;
  }
  first() {
    return this.head;
  }
  rest() {
    return this.tail;
  }
}

List.of = (head, tail) => new List(head, tail);

List.fromValues = (head, ...tail) => {
  if (tail.length > 0) {
    return List.of(head, List.fromValues(...tail));
  } else {
    return List.of(head, nil);
  }
};

We also implement .fromValues helper for creating lists from arbitrary number of arguments. Note how it builds a list recursively.

const list = List.fromValues(1, 2, 3);

console.log(list.toString());

Another useful operation would be generating a range of values. Add a new static method to List class definition:

List.range = (start, end) => {
  if (start < end) {
    return List.of(start, List.range(start + 1, end));
  } else {
    return nil;
  }
};

take method is used to take an arbitrary number of values from a list.

take(n) {
  if (n > 0) {
    return List.of(this.first(), this.rest().take(n - 1));
  } else {
    return List.of(this.first(), nil);
  }
}

Quick test with range and take

const list = List.range(0, 10);

console.log(list.take(3).toString());

So far we’ve implemented only retrieval and list creation operations. Lets also create operations for adding values to list and mapping over its values: add and map methods.

add operation is super cheap, because list supports efficient addition to the head, just create a new list with value in head position and target list in tail position, it is as simple as that:

add(value) {
  return List.of(value, this);
}

map operation also returns a new list where a value in head position is applied recursively to mapper function:

map(fn) {    
  const first = this.first();
  const rest = this.rest();

  if (rest === nil) {
    return List.of(fn(first), nil);
  } else {
    return List.of(fn(first), rest.map(fn));
  }
}

const list = List.range(1, 4).add(0).map(n => n * n);

console.log(list.toString());

Operations such as filter and reduce could be implemented similarly to map.

In this section we’ve learnt about List data structure and its recursive implementation. Similar reasoning could be applied to build quadtree mentioned earlier and any other data structure as well. Later here we will create lazy version of List.

Lazy and eager evaluation

Before we dive into implementing lazy variant of List lets quickly remind ourselves what is lazy evaluation.

There are two evaluation strategies: eager and lazy. In short eager means now or to execute immediately and lazy — later or to delay execution until result is needed. Lazy evaluation is essentially an optimization technique. A real world analogy would be lazy loading images on a web page. When you open a website with a bunch of images the browser will start downloading them all immediately. On the other hand we could save bandwidth by loading only those images that are currently visible in browser window. Non of the images below the fold should load until the page scrolled to a position where they become visible. The same applies to code: a set of operations are executed on demand, when the result is actually needed.

Eager evaluation is more straightforward in a sense that it’s easier to reason about. It’s a standard evaluation strategy in languages like JavaScript and Ruby. Consider the following example (evaluation process is described in comments):

range(0, 10) // [0..9]
  .map(n => n + 1) // [1..10]
  .filter(n => n % 2 === 0) // [2 4 6 8 10]
  .take(3) // [2 4 6]

You can see that the code is executed in-place and result returned immediately.

Lazy evaluation strategy is different.

range(0, 10) // nothing happened
  .map(n => n + 1) // nothing happened
  .filter(n => n % 2 === 0) // nothing happened
  .take(3) // nothing happened
  .run() // [2 4 6]

Here none of the operations in the chain are executed until the call to .run method. nothing happened is of course not entirely true. When called each operation is being saved for later execution and when the result is requested the whole chain is executed and the final value is returned. Languages like Haskell and Clojure relies heavily on delayed execution. In Haskell for example everything is stored in thunks. You can think of a thunk as a container for arbitrary code that prevents its evaluation. It could be a closure that contains an expression. Eager version of 1 + 1 can be represented as lazy () => 1 + 1 in JavaScript. As an example let’s create lazy functions to add and subtract numbers:

const add = (a, b) => {
  return () => {
    a = typeof a === "function" ? a() : a;
    b = typeof b === "function" ? b() : b;
    return a + b;
  };
};

const sub = (a, b) => {
  return () => {
    a = typeof a === "function" ? a() : a;
    b = typeof b === "function" ? b() : b;
    return a - b;
  };
};

Both add and sub returns a function that is closed over the arguments, thus the actual operation is not executed yet, until we force it by calling returned thunk.

const thunk = fn => {
  fn.toString = () => '[Thunk]';
  return fn;
};



const add = (a, b) => {
  return thunk(() => {
    a = typeof a === "function" ? a() : a;
    b = typeof b === "function" ? b() : b;
    return a + b;
  });
};

const sub = (a, b) => {
  return thunk(() => {
    a = typeof a === "function" ? a() : a;
    b = typeof b === "function" ? b() : b;
    return a - b;
  });
};



const result1 = sub(20, add(10, 5));
const result2 = add(3, sub(5, 7));
const result3 = add(result1, result2);

console.log(result1);
console.log(result2);
console.log(result3);

console.log(result3());

Lazy evaluation always comes with memoization which is yet another optimization. Once a thunk was evaluated it can cache the result in closure and return cached value on subsequent calls instead of evaluating the whole thing again and again. Here’s modified version of previous example:

const memoize = (fn) => {
  let cache;
  return () => {
    if (cache) {
      console.log('Return cached');
      return cache;
    } else {
      console.log('Evaluate');
      cache = fn();
      return cache;
    }
  };
};



const add = (a, b) => {
  return memoize(() => {
    a = typeof a === "function" ? a() : a;
    b = typeof b === "function" ? b() : b;
    return a + b;
  });
};

const sub = (a, b) => {
  return memoize(() => {
    a = typeof a === "function" ? a() : a;
    b = typeof b === "function" ? b() : b;
    return a - b;
  });
};



const result1 = sub(20, add(10, 5));
const result2 = add(3, sub(5, 7));
const result3 = add(result1, result2);

console.log(result3());
console.log(result3());

Lazy and recursive data structures

Below is the initial implementation of LazyList abstraction which we are going to build on top of Cons type.

class LazyList {
  constructor(fn) {
    this._fn = fn;
  }
  toString() {
    return `LazyList(${this.next()})`;
  }
  next() {
    return this._fn();
  }
  first() {
    return this.next().head;
  }
  rest() {
    return this.next().tail;
  }
}

LazyList.of = fn => new LazyList(fn);

fn here is a thunk which should return a cons cell and next method evaluates the thunk.

See how .fromValues helper builds a list lazily and recursively. It returns an instance of LazyList which contains a thunk of Cons with head as value from arguments and tail as another lazy list.

LazyList.fromValues = (head, ...tail) => {
  return LazyList.of(() => {
    if (tail.length > 0) {
      return Cons.of(head, LazyList.fromValues(...tail));
    } else {
      return Cons.of(head, nil);
    }
  });
};

const list = LazyList.fromValues(1, 2, 3);

console.log(list.toString());

take also returns a lazy list. But in order to take actual values it evaluates thunks one by one, taking from the tail of the target list, and constructs a new one.

take(n) {
  if (n > 0) {
    return LazyList.of(() => {
      const head = this.first();
      const tail = this.rest();

      if (tail === nil) {
        return Cons.of(head, nil);
      } else {
        return Cons.of(head, tail.take(n - 1));
      }
    });
  }
}

range returns a range of values, except that now with laziness it is possible to create infinite lists.

LazyList.range = (start = 0, end = Infinity) => {
  if (start < end) {
    return LazyList.of(() => Cons.of(start, LazyList.range(start + 1, end)));
  }
};

const list = LazyList.range();

console.log(list.take(2).toString());
console.log(list.take(5).toString());

Adding a value to a lazy list is very similar to List implementation

add(value) {
  return LazyList.of(() => Cons.of(value, this));
}

map operation is also similar to its eager variant

map(fn) {    
  return LazyList.of(() => {
    const first = this.first();
    const rest = this.rest();

    return Cons.of(fn(first), rest.map(fn));
  });
}

Now we can create an infinite list of values with a set of operations defined on it.

const list = LazyList.range().map(n => n * n);

console.log(list.take(2).toString());
console.log(list.take(5).toString());

So far we had only one method for evaluating entire list: .toString. In order to be able to consume a list using common JavaScript facilities it should be converted to something meaningful, for example array. .toArray method does exactly this.

toArray() {
  const first = this.first();
  const rest = this.rest();

  if (rest === nil) {
    return [first];
  } else {
    return [first, ...rest.toArray()];
  }
}

list.take(5).toArray()

Let’s add memoization now. Every time someone calls .next method to retrieve a value from a list we either evaluate a thunk and memoize the result or just return the result if it was already cached. Here’s what modifications need to be done to LazyList class:

constructor(fn) {
  this._fn = fn;
  this._next = null; // cache
}

next() {
  // if there's a thunk
  if (typeof this._fn === 'function') {
    this._next = this._fn(); // evaluate it and cache the result
    this._fn = null; // we don't need thunk anymore
    return this._next; // return cached  value
  } else {
    // other just return cached value
    return this._next;
  }
}

See this example of pulling user records lazily from database. getUsers function produces a list of user records lazily by pulling them out of database on demand, when the thunk is evaluated.

const getUsers = (db, [id, ...ids]) => {
  return LazyList.of(() => {
    if (ids.length > 0) {
      return Cons.of(db.getUserByID(id), getUsers(db, ids));
    } else {
      return Cons.of(db.getUserByID(id), nil);
    }
  });
};

// usage
const ids = db.getUsersIDs();
const users = getUsers(db, ids); // nothing happened
const firstThreeUsers = users.take(3).map(({ id }) => id); // nothing happened

firstThreeUsers.toArray(); // [1, 2, 3]

Let’s test this. If you run the code below you will see that the first run takes ~1s but the next one takes less than 1ms, even though we execute different operations. This happens because users list was evaluated on the first run and result was memoized for subsequent calls.

// benchmark helper
const time = (fn) => {
  const start = performance.now();
  const result = fn();
  const delta = performance.now() - start;
  console.log(Math.round(delta * 100) / 100);
  return result;
};

// db mock
const db = {
  getUsersIDs() {
    return [1, 2, 3, 4, 5, 6];
  },
  getUserByID(id) {
    for (let i = 0; i < 1e8; i++) {}
    return { id };
  }
};

const getUsers = (db, [id, ...ids]) => {
  return LazyList.of(() => {
    if (ids.length > 0) {
      return Cons.of(db.getUserByID(id), getUsers(db, ids));
    } else {
      return Cons.of(db.getUserByID(id), nil);
    }
  });
};

const users = getUsers(db, [1, 2, 3, 4, 5, 6]);

time(() =>
   users
     .take(6)
     .toArray());

time(() =>
  users
    .map(id => 'user id ' + id)
    .take(3)
    .toArray());

Thats pretty much it 🎬

If you are interested in learning functional data structures, I highly recommend to read Okasaki’s book “Purely Functional Data Structures”.

Parsing DSLs with predicates

Roman Liutikov — Sun, 07 Jan 2018 16:10:38 +0000

This article illustrates how data specification can be used to validate and parse DSLs. We’ll use Clojure to create data-driven DSL for a subset of CSS, and clojure.spec library to define and parse the syntax.

clojure.spec is a specification library which provides tools (predicates and composition rules) to describe how data should look like, check if it conforms to that specification and provide an explanation of the cause in case of failure.

(s/def ::name string?)
(s/def ::age pos-int?)

(s/def ::profile (s/keys :req [::name ::age]))

(s/valid? ::profile {::name "Roman" ::age 25.9})
;; false

(s/explain ::profile {::name "Roman" ::age 25.9})

In: [:specs.profile/age]
val: 25.9 fails spec: :specs.profile/age
at: [:specs.profile/age] predicate: pos-int?

When conforming data the returning value is destructured according to the spec. This is very useful for parsing DSLs, because the output can clearly describe the structure of the syntax. Think of it as a way to produce an AST, which is much more useful for analysis and compilation. We’ll see this later.

As an example we’ll create a syntax for CSS Transforms value.

[[:translate-x 120 :px]
 [:translate-y 150 :px]]

;; -> translateX(120px), translateY(150px)

[[:translate 120 :px]]

;; -> translate(120px)

[[:translate 120 :px 150 :px]]

;; -> translate(120px, 150px)

Let’s start by looking at W3C CSS Transforms specification first. The spec clearly defines that the value of transform property is either none or <transform-list>.

This translates into the code in a straightforward way:

(s/def ::transform              ;; transform
  (s/alt                        ;; is either
    :something ::transform-list ;; <transform-list>
    :nothing   #{:none}))       ;; or `none`

The spec defines <transform-list> as one or more <transform-function> tokens.

In code this can be represented using clojure.spec’s regex operators which operate on sequences.

(s/def ::transform-list   ;; <transform-list>
  (s/cat                  ;; is a sequence
    :fns (s/+ ::transform-function)))
         ;; of one or more <transform-function>

<transform-function> is one of those CSS transform functions which you may be familiar with, e.g. translateX(x), translate(x, y).

(s/def ::transform-function     ;; <transform-function>
  (s/or                         ;; is either
    :translate-x ::translate-x  ;; translateX 
    :translate-y ::translate-y  ;; translateY
    :translate   ::translate))  ;; or translate

From the signature of these functions we can see that they accept a value of type <length-percentage>, which is either <length> (px, em, etc.) or <percentage> (%) type units.

(s/def ::length-percentage
  (s/alt
    :length     ::length
    :percentage ::percentage))

According to the syntax which we’ve specified in the beginning a value with a unit is a sequence of two elements: number and a keyword which represents the type of the unit.

(s/def ::length
  (s/cat
    :value number?
    :unit  #{:px}))

<percentage> has the same syntax, just with different type :%

(s/def ::percentage
  (s/cat
    :value number?
    :unit  #{:%}))

Let’s see how destructuring with spec works for ::length-percentage specification.

(s/conform ::length-percentage [100 :px])

;; [:length {:value 100, :unit :px}]

The returned value here is based on the spec that we’ve provided and the missing spots are filled in with values that are parsed out of the provided data. This representation is basically an AST.

Now that we have a spec for function parameters, let’s describe how the syntax of those functions look like.

(s/def ::translate-x          ;; translate-x function
  (s/cat                      ;; is a sequence
    :function #{:translate-x} ;; of :translate-x keyword
    :value    ::length-percentage))
           ;; and a nested sequence of ::length-percentage
;; translate-y is similar
(s/def ::translate-y
  (s/cat
    :function #{:translate-y}
    :value    ::length-percentage))

transform function is more interesting, because it has two parameters and the second one is optional.

(s/def ::translate          ;; translate fn
  (s/cat                    ;; is a sequence
    :function #{:translate} ;; of :translate keyword
    :value
    (s/cat                      ;; sequence
      :x ::length-percentage    ;; of ::length-percentage
      :y (s/? ::length-percentage)))))
        ;; and optional second value ::length-percentage

Finally let’s test the entire thing.

(s/conform ::transform [[:translate -190 :px 80 :px]])
[:something
 {:fns
  [[:translate
    {:function :translate,
     :value
     {:x [:length {:value -190, :unit :px}],
      :y [:length {:value 80, :unit :px}]}}]]}]

Now the compilation process goes into traversing the tree and translating tokens into a proper CSS format.

Also when conforming fails, meaning that there’s an error somewhere in the syntax, we can ask for a reason, why it is failed, in a form of data.

(s/conform ::transform [[:translate -190 :px 80 :p]])
;; :clojure.spec.alpha/invalid
(s/explain-data ::transform [[:translate -190 :px 80 :p]])
{:problems
 '({:path [:something :fns :translate :value :y :length :unit],
    :pred #{:px},
    :val  :p,
    :via
          [:specs.css/transform
           :specs.css/transform
           :specs.css/transform-list
           :specs.css/transform-function
           :specs.css/transform-function
           :specs.css/translate],
    :in   [0 4]}),
 :spec  :specs.css/transform,
 :value [[:translate -190 :px 80 :p]]}

We can see that the value :p in [[:translate -190 :px 80 :p]] failed predicate #{:px} of the specification :specs.css/translate. As you might guessed, this data can be used to report an error in a human-readable format.

For more real-world examples see the code that I’ve written for parsing and compiling syntax for CSS Media Queries.

To learn more about clojure.spec check out the rationale and overview.