Forem: Diego Resendez

Processing 1M Chess Games in 15 Seconds with Rust

Diego Resendez — Tue, 31 Mar 2026 04:24:01 +0000

I train self-supervised models on chess game data. My Python pipeline using python-chess took 25 minutes to parse and tokenize 1M games from Lichess PGN dumps. I rewrote it in Rust. It now takes 15 seconds.

This post covers the architecture, why Rust was the right choice, and what I learned.

The problem

Training a chess move predictor requires converting PGN (Portable Game Notation) files into tokenized sequences — arrays of integer IDs that a neural network can consume. A typical Lichess monthly dump has 5M+ games in a zstd-compressed PGN file.

My Python pipeline had three bottlenecks:

PGN parsing — python-chess parses SAN notation, validates moves on a board, handles edge cases. Correct, but slow. ~15 minutes for 1M games.
Tokenization — converting validated UCI moves to token IDs, tracking piece types and turns. ~10 minutes.
Memory — all games loaded into a Python list of dicts. 1M games = ~4GB RAM.

The Rust rewrite

The tool is called ailed-soulsteal (named after a Castlevania ability — the project has a theme).

Architecture

Three layers, each a clean boundary:

Input Layer → Filter Layer → Output Layer
(PGN parser)   (ELO, result)  (.somabin binary)

Everything streams — games are parsed, filtered, tokenized, and written one at a time. Memory usage stays constant regardless of input size.

PGN parsing

PGN is a messy format. Tags, comments, variations, NAGs, move numbers, results — all interleaved. I wrote a simple streaming parser that yields one RawGame at a time:

struct PgnIterator<R> {
    reader: R,
    line_buf: String,
}

impl<R: BufRead> Iterator for PgnIterator<R> {
    type Item = RawGame;
    fn next(&mut self) -> Option<RawGame> {
        // Read tags until blank line, then movetext until next blank line
        // Strip comments, NAGs, variations, move numbers
        // Return (tags, moves, result)
    }
}

No allocations per game beyond the reused line buffer. The RawGame struct holds tags as a HashMap<String, String> and moves as Vec<String>.

Move validation with shakmaty

shakmaty is a pure Rust chess library. It handles SAN parsing, move validation, and piece type lookup — the same things python-chess does, but at native speed.

let san: shakmaty::san::San = san_str.parse().ok()?;
let m = san.to_move(&pos).ok()?;
let uci = uci_string(&m);
let category = role_to_category(m.role());
pos = pos.play(&m).ok()?;

This is where most of the speedup comes from. shakmaty's play() is essentially a few bitboard operations — no Python overhead, no GC pressure.

Binary output format

Instead of writing JSON or CSV, I designed a binary format (.somabin) optimized for ML training:

Header (64 bytes): magic, version, vocab_size, num_games, ...
Index Table:       byte offset for each game (enables random access)
Data Section:      per game: [seq_len, token_ids, turn_ids, category_ids, outcome]

The index table is the key insight. A PyTorch Dataset.__getitem__(i) can seek directly to game i via mmap without scanning the file. Loading 50K games takes 20ms. Random access runs at 500K games/sec.

The Python reader is 30 lines:

class SomabinDataset:
    def __init__(self, path):
        self._file = open(path, "rb")
        self._mm = mmap.mmap(self._file.fileno(), 0, access=mmap.ACCESS_READ)
        self.header = read_header(self._mm)
        self._index = read_index(self._mm, self.header)

    def __getitem__(self, idx):
        offset = int(self._index[idx])
        return read_game(self._mm, offset)

Zstd decompression

Lichess distributes PGN files as .pgn.zst. The zstd crate handles decompression transparently:

if path.extension() == Some("zst") {
    let decoder = zstd::Decoder::new(file)?;
    Ok(Box::new(BufReader::with_capacity(1024 * 1024, decoder)))
} else {
    Ok(Box::new(BufReader::with_capacity(1024 * 1024, file)))
}

Auto-detected from the file extension. No separate decompression step needed.

Filtering

Games are filtered by metadata before tokenization — no wasted work:

pub trait GameFilter {
    fn accept(&self, game: &RawGame) -> bool;
}

// Chain of filters — all must pass
filters.add(Box::new(EloFilter::new(1000, 1800)));
filters.add(Box::new(MovesFilter::new(Some(4), None)));
filters.add(Box::new(ResultFilter::Decisive));

Filters operate on PGN tags (strings), not board positions. Checking WhiteElo >= "1000" is effectively free compared to move validation.

Benchmarks

Processing Lichess monthly dumps (zstd compressed) on an M1 MacBook:

Month	Input	Games (1000-1800)	Time	Rate
2016-01	831 MB .zst	2,060,197	45s	46K/s
2016-02	866 MB .zst	2,071,332	46s	45K/s
2016-03	994 MB .zst	2,399,234	54s	45K/s
2016-04	1.0 GB .zst	2,438,621	55s	44K/s
2016-07	1.0 GB .zst	2,598,733	59s	44K/s

11.6M games in 4.3 minutes. The equivalent Python pipeline would take roughly 5 hours.

What I learned

Streaming wins. The biggest architectural decision was making everything an iterator. Games flow through parse → filter → tokenize → write without buffering. Memory usage is constant at ~10MB regardless of input size.

Binary formats beat JSON for ML. My first version wrote JSONL. A 1M-game JSONL file was 2GB and took 30 seconds to load in Python. The .somabin binary for the same data is 550MB and loads in 20ms via mmap.

shakmaty is excellent. Chess move validation is the bottleneck in any PGN pipeline. shakmaty's bitboard implementation made this a non-issue. The crate is well-documented and the API maps cleanly to what you need for tokenization.

Rust's type system caught real bugs. The GameParser and GameTokenizer traits enforce separation between parsing (text → structured data) and tokenization (structured data → integers). When I mixed them up during development, the compiler told me immediately.

Try it

cargo install ailed-soulsteal

# Generate vocabulary
soulsteal vocab --generate -o vocab.json

# Tokenize a Lichess dump
soulsteal tokenize lichess_2016-02.pgn.zst \
  -o train.somabin \
  --vocab vocab.json \
  --elo 1000:1800

# Inspect
soulsteal info train.somabin
soulsteal stats train.somabin

Pre-tokenized datasets are available on Hugging Face.

The tool is designed to support any turn-based game — Go (SGF), Shogi (KIF), etc. Chess is the v1 implementation, but the GameParser and GameTokenizer traits are game-agnostic.

This article was drafted with assistance from Claude (Anthropic), an AI language model. The code, benchmarks, and technical decisions described are entirely my own work.

Resources:
Source: github.com/Ailed-AI/ailed-soulsteal
crates.io: ailed-soulsteal
License: MIT

Introducing Expressive Tea

Diego Resendez — Wed, 26 May 2021 22:52:46 +0000

Introduction

When I started coding Expressive Tea almost two years ago, it was just a tool to increase my laziness (as most engineers) avoid setting up a project on ExpressJS. I was thinking about combining a language that I started to learn (Typescript) and making easier a configuration for ExpressJS; by that moment, I did not know about NestJS, a very mature and fantastic Framework. I did not have a chance to research, just because I got the idea to start working in this new (at that moment) personal library.

In September 2019, that personal library turned into a microframework to help build server-side applications in NodeJS and use modern Javascript powered by Typescript. Also, one of the main goals is to make this compatible with all the express middleware and plugins available when I wrote this article; the list continues to expand with ideas and recommendations from friends and users that already started using Expressive Tea.

What the heck is Expressive Tea?

I define Expressive Tea as a flexible framework that gives freedom to build their own architectures by providing descriptive decorators, a plugin engine, shareable modules, and modern Javascript.

But what does that mean? As I mentioned in the introduction, it is just making developers lazier (for a good reason), helping them forget how to set up ExpressJS, and just working on the business logic.

Take into consideration that Expressive Tea has not come with anything out of the box; this means no additional plugins or middlewares with exceptions of certain features like Websockets, and you might need to provide the necessary plugins to achieve your unique server flavor.

So, telling that, you might still wonder where the flexibility or freedom is in something with nothing more than essential matters. Pleasantly, Expressive Tea has no attachment on any naming convention, any special plugin, or any data source, so this allows you to freely change the application in the way you prefer and introduce two critical entities in the Expressive Tea framework.

Plugins

A Plugin is an entity to implement features that we need for our applications, such as assigning a connection to MongoDB or changing the server behavior such as adding a view engine, adding authorization middleware, or just adding a body parser.

But more importantly, this can be share between Expressive Tea projects; the method depends on you (like npm package, git submodule, or copy and paste). Still, essentially you will get a path to create many plugins and combine them to flavor on your own Expressive Tea.

Modules

A Module is pretty similar to Plugin, but instead of change the behavior of the server is used to create a placeholder route to encapsulate controllers and services to respond to a user request through declared endpoints.

All the endpoints are defined in each ** controller** that have a corresponding method to respond to the client; in simple terms, a Module helps to contain routers in one place, by example, we can have a signing module that contains all the endpoints to sing in or sign up a user into our application.

As same as Plugins, Modules can be share between Expressive Tea applications if they have the following specifications:

All code must be in the same place; anyone has their own structure and scaffolding methods, ideally set all in the same root directory; the other thing is depends on you ;).
Should not contain external code, like external classes, services, or constants with certain exceptions like Expressive Tea settings and node packages.

Demo

We have enough for boring theory; it might be better with a bit of demonstration of how this is working, shall we?..

Dependencies.

We can start by installing tea CLI with the following command:

npm install -g @expressive-tea/tea

Tea CLI is under development; take patience ;), send comments or open an issue.

Create a project.

Once tea finished to install, now is time to start to create a project with the following command:

tea brew to-do

Follow all the instructions and change them if you want them. Once create project finished and create our to-do project, just go inside with:

cd to-do

and execute the project with:

tea serve

if you get the following message, congratulations; this is your first Expressive Tea application:

[INFO] 14:08:05 ts-node-dev ver. 1.1.6 (using ts-node ver. 9.1.1, typescript ver. 3.9.9)
Fri, 21 May 2021 19:08:06 GMT helmet deprecated helmet.noCache is deprecated and will be removed in helmet@4. You can use the `nocache` module instead. For more, see https://github.com/helmetjs/helmet/issues/215. at server/plugins/express.ts:59:31
Running HTTP Server on [8080]

if you open your browser and type http://localhost:8080/
you will get the following response:

time to play

Let the fun start :), it is time to move our a little bit our project; the main goal is to create a REST API for a To-Do application, and the first thing to do is remove the static HTML that shows as the index.

It is time to open the main.ts file from the application and remove the line @Static('./public') (you can use the IDE that you like) as is the following screencast.

Now the main page is gone is time to implement our first endpoint to this new REST API.

The application now will return a To-Do list that is on memory, for now, just to not overcomplicate the example.

For this example need to add one of the elements from Expressive Tea for decorating arguments in the endpoints methods; the @body() annotation is part of the package on @zerooneit/expressive-tea/decorators/annotations and helps to extract the whole body object as is showing in the screencast above.

More details you will get in the documentation.

The Extra

The main goal is to present you a little bit of Expressive Tea, but how to leave without giving you an extra; in this case, I will show you how to modify the Express plugin in the demo to just show a console log.

Check our live playground.

I know this is not extensive details in this article, and this is because I just want to inform you more than training you, but if you're going to sneak pick a little more content, there is a series of articles that you can follow, it might be a little old but contains the essential:

A Simple REST API with Expressive Tea | Part Three | by Diego Resendez | Medium

Diego Resendez ・ Oct 3, 2019 ・
Medium

A Simple REST API with Expressive Tea | Part Two | by Diego Resendez | Medium

Diego Resendez ・ Oct 8, 2019 ・
Medium

A Simple REST API with Expressive Tea | Part One. | by Diego Resendez | Medium

Diego Resendez ・ Sep 23, 2019 ・
Medium

Or you can use our sandbox in CodeSandbox, where you will be able to play a little bit or understand more about Expressive Tea.

Conclusions

As I continue improving this project, I learned and still learn many things technically. Still, the real value is discovering new things to do, challenges, and, more importantly, being active in the Open Source community. Even if this project does not grow much or maybe overextend it, the self-learning of many soft skills comes with this, for example, managing your time, executing a project, and actively checking new features.

Hence, as this work I just take it as a personal challenge it will be good to have help, so if any of you want to help, improving code, adding new issues, do a review, write an article, maybe a YouTube video, or ask for new feature down below you will get the link to the Github Repository of this project.

Expressive-Tea / expresive-tea

A Express and Typescript REST Service Template.

Without more to aggregate, let me thank you for reading my article, which is the first on this platform.