Forem: Felix Terkhorn

Redis Streams API Coming to Deno

Felix Terkhorn — Mon, 29 Jun 2020 10:24:30 +0000

🦕 💾 The Redis Streams API will land in deno-redis pretty soon. I'm nearing completion with this pull request and hope for a thorough review.

🏞 🍕 There's been a lot of activity in this repository, which is encouraging!

https://github.com/denolib/deno-redis/pull/86

Redis Streams available in redis-rs

Felix Terkhorn — Tue, 23 Jun 2020 12:44:45 +0000

Woohoo! We now have high-level support for Streams commands in redis-rs:

https://github.com/mitsuhiko/redis-rs/issues/162

This is the conclusion of work described in a previous post:

https://dev.to/terkwood/adding-redis-streams-to-rust-464j

Hope someone out there finds it enjoyable! I have a couple of outstanding work items that I'm going to revisit now that there's a pleasant way to interact with this portion of the Redis API. 🦀 💾

Adding Redis Streams to Rust 💾 🦀

Felix Terkhorn — Wed, 03 Jun 2020 10:28:38 +0000

I recently worked on a pull request which adds Redis Streams capabilities to redis-rs, the most popular Redis client in Rust's ecosystem. The overwhelming majority of the effort was contributed by the community, not by me: I drafted the pull request which combines the two existing works. In addition to addressing review comments, I added a few examples of how the new API works.

I'm feeling the hype! 🔥 What is Redis Streams? Why do we need it in Rust? Does it have anything to do with Kafka Streams? And can we share any real-life examples?

Please read on...

What is Redis Streams? 🤔

To give a little bit of background, Redis Streams was released as part of Redis 5.0, all the way back in Oct 2018.

antirez posts a great explanation of the new features.

You can think of Redis Streams data as a way to communicate time-indexed data among processes. Oversimplifying this, you can imagine that each stream is a CSV with a timestamp. Digging deeper, Redis Streams exposes functionality that's more advanced than basic pub/sub, as you can have multiple consumers process a single stream: even if one consumer happens to be faster than the others, Redis will rationally distribute the data.

Unlike Redis's pub/sub mechanism, Redis Streams are somewhat durable. If you miss a message, it's still available in the stream, with limits; a stream will generally have a cap on the amount of messages allowed to accumulate.

Why Does Rust Need Redis Streams? 🔎

The Streams commands are over a year old, so there was some support expressed for including them in redis-rs.

It's great that the community came together and created a separate lib exposing the Redis Streams API. But Redis Streams is a first-class citizen of the larger Redis API, so it makes sense to include it as part of the leading Redis crate.

We've exposed the new Redis Streams commands as an opt-out feature in redis-rs. If you want to reduce your compile time 🐌 🦀, you can explicitly disable streams support. This is the same as how geospatial operators work in redis-rs, so it should be a familiar concept for developers who have experience with the lib and who want to try out Streams.

Off-the-Cuff Comparison with Kafka Streams 🌽

After looking into Redis Streams on my own, I did some initial comparisions with Kafka Streams and came to some conclusions:

Redis Streams is a subset of the Redis server API. The stream commands are exposed by the basic Redis server, not through separate lib. On the other hand, Kafka Streams is a combination of the server (Kafka) combined with a JVM-based framework/lib (Kafka Streams).
Kafka Streams apps coordinate data partitioning on the client side, while in Redis Streams, the server decides which consumer group gets which slice of data. This helps Kafka achieve extraordinary scale: clients can work together to distribute load without the server becoming a bottleneck.
Naive Kafka Streams apps consume a LOT of RAM, so if you're stingy about that sort of thing (and don't plan on scaling to infinity), Redis Streams can be a great fit.
The data types exposed by Redis Streams are somewhat deeply nested, and can take a little while to get used to.

Despite the similar naming conventions, the use cases for Kafka Streams and Redis Streams don't overlap as much as you might think. Redis works great when you have one or two boxes (potentially with an enormous amount of RAM). Kafka is built for massive, horizontal scale.

Practical Comparison 🔧

Out of sheer, nerdtastic enthusiasm, I had already written several Kafka Streams applications to process game states and various player-coordination functions for BUGOUT, our implementation of the ancient board game Go (Baduk). I went ahead and rewrote some of this functionality using rust and Redis Streams. You can see some direct comparisons of how one might structure a Redis Streams app versus a Kafka Streams app.

judging moves in Rust/Redis Streams vs Kafka Streams
assigning players to game instances in Rust/Redis Streams vs Kafka Streams

The declarative style of Kafka Streams apps really stood out as an advantage: we could just focus on the logic required by our game system, and didn't have to write quite as much boilerplate.

But our Redis Streams apps written in rust were minuscule in terms of their memory consumption: in a cold system, just after startup, the micro-judge and micro-game-lobby apps take up about 1MB of RAM, while the Kafka Streams apps ⚠️ usually initialize at 100MB+ ⚠️. Meanwhile, the Redis server itself continues to cruise along with a similarly small main-memory footprint (~3MB in our low-traffic system).

Note that these Redis examples don't actually use the nicer API that's discussed as the main focus of this article -- generally we're using the lowest-level interface which specifies Redis commands using strings. We'll work on upgrading these files once our pull request is merged!

Look Ma, I'm Learning! 🧠

As a result of working the merge, I improved my understanding of the concepts underlying Redis Streams.

Creating an example of XREADGROUP command patterns gave me an immediate insight into a shortcoming of my board game project: I was maintaining boilerplate code, tracking the time IDs processed in a given stream. This code can be destroyed once I switch from naive XREAD to Redis-controlled XREADGROUP.

Using the ">" operator in an XREADGROUP command tells Redis, "hey, give me only the newest records... and YOU keep track of where I am in the stream!" This functionality, combined with the automatic XACK provided in in the new additions to redis-rs, makes for a nice combination.

Conclusion 💛

If you're excited about seeing the Redis Streams support finally make it into the Rust ecosystem in a nice way, please dust the PR with emojis, or better yet, with critical review. 🔬

If you're using Redis Streams in your own work, we'd love to hear from you.

Attribution

Thank you to Audrey for the image used in the header. It is licensed under CC by 2.0. I cropped the image in order to make it fit a bit better as a header.

Privileged mode: Running ctop in docker under SELinux

Felix Terkhorn — Wed, 13 May 2020 13:04:52 +0000

Some programs like ctop are nice to run using docker containers, so that you don't have to manually download a binary and copy it into /usr/local/bin, where it will sit in a sad little corner, unmanaged by apt or yum.

bcicen / ctop

Top-like interface for container metrics

ctop provides a concise and condensed overview of real-time metrics for multiple containers:

as well as a single container view for inspecting a specific container.

ctop comes with built-in support for Docker and runC; connectors for other container and cluster systems are planned for future releases.

Install

Fetch the latest release for your platform:

Debian/Ubuntu

Maintained by a third party

echo "deb http://packages.azlux.fr/debian/ buster main" | sudo tee /etc/apt/sources.list.d/azlux.list
wget -qO - https://azlux.fr/repo.gpg.key | sudo apt-key add -
sudo apt update
sudo apt install docker-ctop

Arch

ctop is available for Arch in the AUR

Linux (Generic)

sudo wget https://github.com/bcicen/ctop/releases/download/v0.7.7/ctop-0.7.7-linux-amd64 -O /usr/local/bin/ctop
sudo chmod +x /usr/local/bin/ctop

OS X

brew install ctop

sudo port install ctop

sudo curl -Lo /usr/local/bin/ctop https://github.com/bcicen/ctop/releases/download/v0.7.7/ctop-0.7.7-darwin-amd64
sudo chmod +x /usr/local/bin/ctop

Docker

docker run --rm -ti \
  --name=ctop \
  --volume /var/run/docker.sock:/var/run/docker.sock:ro \
  quay.io/vektorlab/ctop:latest

Building

Build steps can be found…

View on GitHub

But if you run an SELinux-enabled distribution, you'll find that running ctop as the documentation suggests, fails:

docker run --rm -ti \
  --name=ctop \
  --volume /var/run/docker.sock:/var/run/docker.sock:ro \
  quay.io/vektorlab/ctop:latest

🐳 🐳 🐳

ctop - error ───────
  │                                                                                 │
  │  [12:54:15 UTC] attempting to reconnect...                                      │
  │                                                                                 │
  │  [12:54:16 UTC] Get http://unix.sock/info: dial unix /var/run/docker.sock: con  │
  │  nect: permission denied

What's going on here? Presumably SELinux is blocking the ctop container's access to information necessary for monitoring the other containers.

The Fix

Luckily, there's a very easy fix for this! You can just run the ctop container in privileged mode:

docker run --privileged --rm -ti  \
  --name=ctop   \
  --volume /var/run/docker.sock:/var/run/docker.sock:ro   quay.io/vektorlab/ctop:latest

Now you can see all your favorite containers, interact with their log interfaces, dip into their shells, etc:

  ctop - 12:57:42 UTC   8 containers

     NAME        CID         CPU         MEM         NET RX/TX   IO R/W      PIDS

   ◉  bugout_bot… 8f84fe3983…      1%       2M / 944M 19M / 16M   1M / 0B     6
   ◉  bugout_bug… 16f1479be4…      0%       3M / 944M 1M / 1M     6M / 0B     5
   ◉  bugout_gat… fc951914df…      0%       3M / 944M 56M / 35M   256K / 0B   21

Easy scrolling with tmux and alacritty

Felix Terkhorn — Wed, 29 Apr 2020 17:05:34 +0000

We want to have a nice experience when scrolling the terminal buffer using both alacritty and tmux on Debian linux.

When we run alacritty and tmux together and there's a lot of text on the screen (say, by invoking tree), we need to hit the following tmux key sequence to scroll back through the terminal history:

Ctrl+B 
[
PageUp

This is difficult. Others have asked about this.

We followed the workaround advice given by the author. We enabled mouse mode, and disabled faux scrolling in .tmux.conf:

set -g mouse on
set -ga terminal-overrides ',*256color*:smcup@:rmcup@'

In .alacritty.conf, the default scrolling key combination, Shift+PageUp, isn't too bad. We can just leave it commented to use it, but if we want to try our own settings, we can:

key_bindings:
#
# ...snip...
#
- { key: PageUp,   mods: Shift, action: ScrollPageUp,   mode: ~Alt       }
- { key: PageDown, mods: Shift, action: ScrollPageDown, mode: ~Alt       }
- { key: Home,     mods: Shift, action: ScrollToTop,    mode: ~Alt       }
- { key: End,      mods: Shift, action: ScrollToBottom, mode: ~Alt       }

Resolution

With the changes to .tmux.conf, we can now start an alacritty terminal, enter tmux, and scroll to our hearts' content, without using a difficult key combination.

🔥 It's FAST! 🔥

We found that scrolling through history with some basic command line apps wasn't negatively impacted.

When we entered tmux + alacritty, and then dumped a lot of text into less, or navigated man bash, we scrolled happily and without interruption.

These instructions probably apply to other terminal emulators, as well.

This is a happy time. 😄

Use Keyboard Shortcut to Focus alacritty in Gnome

Felix Terkhorn — Sun, 26 Apr 2020 17:48:24 +0000

Use Keyboard Shortcut to Focus alacritty in Gnome

alacritty is a 🔥 hot, fast terminal emulator 🔥 written in rust. We've been using it for the last 38 microseconds and really enjoy it. It pairs nicely with tmux.

We want to hit a single key and focus on alacritty while using GNOME. If alacritty isn't already open, it should start up. This would give us functionality somewhat equivalent to Guake.

As a bonus, we'll use deno to write a quick helper script and 😇 avoid learning bash. 😇

Here's what we did to accomplish this task.

Install wmctrl

Install wmctrl so that we can focus on an existing alacritty window using a program.

For debian & ubuntu-flavored linux:

sudo apt install wmctrl

For the redhats:

sudo yum install wmctrl

Start alacritty up for just a moment and take note of its handle as given by wmctrl -xl:

...snip...
0x04800002  1 Alacritty.Alacritty   mybox Alacritty
...snip...

We can then use wmctrl to focus on a running instance of alacritty:

wmctrl -xa Alacritty.Alacritty

Create script to raise or start-up terminal

Create a deno script to either focus on the existing alacritty window, or start a new one:

// saved to /home/nope/bin/raise_alacritty.ts
const p = Deno.run({
    cmd: [ "/usr/bin/pgrep", "alacritty" ]
});

const { code } = await p.status();

if (code === 0) {
    await Deno.run({ cmd: [ "wmctrl", "-xa", "Alacritty.Alacritty" ] }).status();
} else {
    await Deno.run({ cmd: [ "alacritty" ] }).status();
}

Bind keyboard shortcut

Finally, in gnome settings,
add a keyboard shortcut which runs our script.

In our case, we assigned the special MENU button on our keyboard to focus on alacritty. We use the default GNOME shortcut (Super H/WindowsKey H) to hide the window when we're done with it.

Updated for deno 1.0.0-rc3

Please note that as of deno 1.0.0-rc3, the command line invocation needed to make this work has changed. You now need to use the run subcommand:

deno run --allow-run /path/to/raise_alacritty.ts

This article was originally published under an older version, which did not require the subcommand.

deno  --allow-run /path/to/raise_alacritty.ts

If you're having trouble getting this to work under the new version of Deno, try adding run!

BUGOUT: Play Go against AI 💪 and Human Friends 👪

Felix Terkhorn — Mon, 30 Mar 2020 23:31:33 +0000

BUGOUT provides a web interface to KataGo, a leading, community-driven AI skilled in the classic board game, Go.

BUGOUT also features a multiplayer option for playing against human friends, if you're lucky enough to have them.

BUGOUT's web interface is a derivative of Sabaki, the much-loved desktop app which is already capable of hooking into KataGo, Leela Zero, and GNU Go.

Source Code for the Project

The project is open-sourced, currently under MIT license.

Terkwood / BUGOUT

AI-driven, Multiplayer Go/Weiqi/Baduk for the web 🐛🤖🦀♟

This write-up was originally posted at terkwood.farm.

Emotional Backdrop

I started playing GO sometime during high school. Several of my friends also learned to play. We would often play chess, for which I had a neat little carrying case with a simple game clock, a rollout board, and sturdy pieces. I came into a possession of a GO board of standard dimensions, and used plastic pieces that were just heavy enough to feel like they mattered.

We would sometimes meet at a Chinese dive on West 10th Street in Indianapolis, consume delicious, sugary food, and play.

GO felt different than chess. There was a sense of immense possibility. There was time to think about the future. Winning was hard. Getting your skill up with a same-stage newbie was exhilarating, because you grew together.

Eventually I got my board signed by VICTOR WOOTEN while attending a concert performance given by Béla Fleck and the Flecktones at the Indiana Roof Ballroom: we had been playing some small rounds prior to the start of the concert.

It must have been... 1999? Victor Wooten was super gracious, and inspired everyone to a wholesome life of technical mastery. I was young. I wore an undersized fedora.

The music was phenomenal, and Wooten's signature fueled my fire.

In college I dreamt that one day I would wake to the sound of an obstreperous trumpet, having been mired in games and strenuous efforts to build some sort of ultimate strategy. I dreamt that I would wake to a vision of clear dawn.

But the actual time I've spent playing GO in my life is tiny. Life goes on, and its daily demands obscure our roots.

Still, the aesthetic delight of GO has been enjoyed by millions of individuals over the centuries, and it's this very beauty which has inspired my work over the last nine months.

GO is a game which helps me combat the lassitude of experience and boredom.

GO is not a game which must be won. It's enough to face a player whose skill exceeds yours, and to learn. When I play 9x9 against KataGo running on a 5W computer, I'm happy if I manage not to lose an embarrassing number of pieces.

GO keeps me sharp.

Building the Project

BUGOUT started out as an excuse to implement something (anything) using Kafka. My previous two professional engagements had barely skirted opportunities to legitimately include a Kafka install: one was simple enough to rely on HTTP microservices, another had opted for Kinesis.

Fresh into successful semi-retirement, I didn't want to bother with finding a use case for my implementation. I wanted what I couldn't have reasonably argued for in past, commercial settings. I wanted to play with Kafka Streams!

It turns out that mapping the tiny domain model of a GO game to kafka streams is... easy. Initially BUGOUT was envisioned as a multiplayer GO board which could easily run in any modern browser and provide an enjoyable, boutique alternative to more popular servers and venues. I wanted to play again with my old friends, just like when we were kids. So the project quickly incorporated a game lobby system which allowed players to choose between joining a quick 19x19 game with the next person to visit the web page, or creating a nicely-formatted URL (https://example.com/?join=nXblGBE7erWyocXYYpRN1YOzdD) for their private game and sharing it with a friend.

Finally, both players can be fairly assigned a side, and the game can begin.

It basically worked. I spent time making sure the websocket connection between the browser and the BUGOUT gateway server was solid. I had to guarantee that Kafka and all its dependent apps started up in an orderly fashion. And I enjoyed writing functional-ish Kafka Streams code in Kotlin: from a cognitive perspective, it felt clean and tidy, even if the topology graphs quickly got out of hand.

Too Cheap for Pro Tier

It's pretty obvious to anyone that spends 30 minutes in the Kafka literature, that a production deployment of Kafka involves at least three machines. I was running a single node, because my average user load was zero. What's more, I was paying for AWS compute time using a personal credit card, so I've never been exactly fired up about running an individual host that comes anywhere near the specs recommended by the vendor.

Some additional reading turned up that 6GB of RAM is a reasonable minimum for any single host:

In most cases, Kafka can run optimally with 6 GB of RAM for heap space. For especially heavy production loads, use machines with 32 GB or more. Extra RAM will be used to bolster OS page cache and improve client throughput. While Kafka can run with less RAM, its ability to handle load is hampered when less memory is available.

Ever the miser, I settled on running a t3.medium compute host for my degenerate Kafka cluster with its single, lonely node. 4GB of RAM certainly worked well enough for my theoretically scalable backend -- at least as long as it didn't become popular.

Anyway, who cares about vendor reccomended minima or a realistic-looking deployment? I was liberated from corporate hierarchy, profit motive, etc, and was ready to experience the raw freedom that I'd always dreamed about as a junior developer. I was ready to play with toys.

The only problem was that running even a t3.medium around the clock cost about $1/day.

🤑 That's expensive! 🤑

Turning Kafka On and Off with Rust + AWS

The windmills were ready to tilt.

I spent a non-trivial amount of effort using rusoto to control the startup and shutdown of my "very expensive" t3.medium instance. Whenever the system detects that there are no users connected, a timer starts, and after a few minutes, it shuts down the t3.medium running Kafka, and all of the memory-hogging Kafka Streams apps supporting gameplay.

Whenever someone comes back to the website and wants to play a game, the system loyally boots the Kafka host.

And that's great. I can still afford to eat. Working with rusoto was painless. But the user experience of waiting for BUGOUT's Kafka backend to initialize is... abysmal. It takes about 90 seconds for the EC2 instance and its various docker containers to start up. That's 81 seconds longer than eternity.

Starting on the Micro-Stack

Enough was enough. I had no boss. I had no users. I had no chains. My self-respect was questionable.

I would re-implement the backend with redis streams, so that my game could be online 24/7, using leftover CPU and RAM on the t3.micro instance that I was already paying for.

Why not? Redis is awesome! And by choosing rust for this portion of the impl, I knew I could keep my memory and CPU footprint low.

The effort is ongoing. It will eventually be completed in the way that one completes a crocheted scarf.

Welcome KataGo

After months of relatively steady operation, I came to the sobering conclusion that most of my close friends were busy with their own lives, and playing a synchronous session of GO with them, often across international time zones, was an exercise in scheduling prowess that surpasses even the most skilled project manager.

So I began to wonder about playing against an AlphaZero-like AI through my browser.

If I can't play against my friends...

...why not BUILD MY OWN FRIEND?

Kata Meets Nano

Thus was born the effort to integrate KataGo, a community implementation of cutting edge Go AI that's gotten a nice little boost thanks to hardware sponsorship from Jane Street.

In fact, you can already play against KataGo online.

That's fine. I like building things on my own -- or in this case, integrating other people's things on my own. So I purchased an NVIDIA Jetson Nano System-on-a-Chip and set about linking it to my existing infrastructure.

🤖 Play against KataGo #67

Terkwood posted on Jul 17, 2019

Goal

Implement a system which allows playing against a KataGo instance running on an NVIDIA Jetson Nano.

Background

We picked up an NVIDIA Jetson Nano board and built KataGo. Documentation is available in the repo now

Tasks

[x] build KataGo on the nano
[x] wrap katago stdin/stdout #190
[x] make a basic docker image #189
[x] gateway: backend routing (#205)
[x] gateway: send & recv bot attachment and move-making messages
[x] use websockets correctly #210
[x] create botlink service and push game states via websocket (#193, #196, #197)
[x] connect tinybrain to Botlink service via websocket and process game states (#194)
[x] document reverse-proxy setup
[x] clean up micro-changelog init (#187)
[ ] extend Sabaki initial menu, send attach bot command, process makemoves as normal, listenForMove when human plays white
[x] fix parse error #220
[x] fix all coords #223

Architecture draft

Use KataGo analysis engine to respond to queries.

Errata

An existing service for playing against bots

View on GitHub

The effort proceeded smoothly. Soon I was suffering regular blows to my pride, delivered easily by a 128-core GPU that can subside on a mere 5W of power. NVIDIA's pre-installed CUDA libs made compilation of KataGo relatively easy. Hooking up the tiny SoC in my office to my dirt-cheap AWS t3.micro felt pleasantly conservative.

A Happy Ending

There isn't much work left before the AI subsystem is complete. It's already possible to play against KataGo through the web interface, if you're patient enough to run a read-eval-print-loop and put Kata's pieces on the board manually.

After just a few more hours of work, I'll be able to head to my website from anywhere in the world, and play a power-efficient little bot that's just right for my skill level.

And then, free from the lamentable desire for human interaction, I'll play against my own, new type of friend: a friend born of silicon of and parallel arithmetic.