Forem: Charles Fonseca

Building a Redis Clone in Zig—Part 4

Charles Fonseca — Tue, 23 Dec 2025 01:01:58 +0000

This is the journey of a web developer learning systems programming. I’ve learned so much building Zedis, and I encourage you to try the same. If you’re coming from web development like me, building a database might seem intimidating—but it’s incredibly rewarding, and you’ll learn more about computing fundamentals than you ever thought possible.

I benchmarked Zedis with 1 million requests using 50 concurrent clients. The results showed me just how much I still have to learn.

The Bottleneck

The flame graph above immediately reveals the problem: Parse.parse consumes 75% of execution time, while Client.executeCommand takes only 13%, and Command.deinit accounts for 8%. For a Redis clone, this is backwards—parsing should be fast, and command execution should dominate.

But why is parsing so slow? The flame graph shows where time is spent, but not why. To dig deeper, I used the perf tool, which provides detailed CPU and memory statistics beyond just function call stacks:

perf stat -e cache-references,cache-misses,L1-dcache-loads,L1-dcache-load-misses,dTLB-loads,dTLB-load-misses ./zedis benchmark --clients 50 --requests 1000000

The output:

Performance counter stats for ‘zedis’:
   20,718,078,128 cache-references
    1,162,705,808 cache-misses # 5.61% of all cache refs
   81,268,003,911 L1-dcache-loads
    8,589,113,560 L1-dcache-load-misses # 10.57% of all L1-dcache accesses
      520,613,776 dTLB-loads
       78,977,433 dTLB-load-misses # 15.17% of all dTLB cache accesses
     22.936441891 seconds time elapsed
     15.761103000 seconds user
     64.451493000 seconds sys

The system spends only 15.76s in user time and 64.45s in system time over 22.93s total elapsed time. The program is clearly I/O bound—spending most of its time waiting on network operations rather than doing actual work.

First Issue: Inefficient I/O

During the migration from Zig 0.14 to 0.15, the reader API changed. Unfamiliar with the new interface, I defaulted to using readSliceShort, consuming the input stream one byte at a time. This is what I’m used to in a high-level language, where buffered readers are handled internally and you aren't concerned about sys calls:

var b_buf: [1]u8 = undefined;
const bytes_read = try reader.readSliceShort(&b_buf);

This approach is catastrophic for performance—reading a single byte per operation means excessive system calls and function overhead. For context, here’s an example Redis protocol message (line breaks added for clarity):

*3\r\n
$3\r\nSET\r\n
$4\r\nkey1\r\n
$5\r\nvalue\r\n

The *3 indicates 3 tokens follow. $3 means a 3-character string, in this case, SET. $4 represents a four-character string, key1 , and $5 represents a five-character string, value.

It’s All About Buffering

Instead of reading byte-by-byte, we allocate a buffer once and let the reader work with it:

var reader_buffer: [1024 * 8]u8 = undefined;
var sr = self.connection.stream.reader(&reader_buffer);
const reader = sr.interface();

Now we can read entire lines at once using takeDelimiterInclusive:

const line_with_crlf = reader.takeDelimiterInclusive(’\n’) catch |err| {
    if (err == error.ReadFailed) return error.EndOfStream;
    return err;
};

This change transforms parsing from thousands of tiny reads into a buffered approach that processes data in larger chunks—exactly how high-performance network servers should work.

There’s room for further optimization by navigating the buffer directly without parsing line-by-line, but I wanted to keep the code readable for now. Besides, the Redis protocol itself has several problems; you can learn more about its quirks in this lecture.

Second Issue: Know Your APIs

At first, I used std.heap.GeneralPurposeAllocator as the base allocator for all memory allocations. However, by default, numerous safety checks, stack traces, and bookkeeping features are enabled, which add tremendous overhead. As noticed in the flame graph, a significant amount of time was spent in mutex locks related to the allocator. I ended up using std.heap.page_allocator, but std.heap.smp_allocator is also an option.

Lesson Learned: Leverage Open Source

Coming from a high-level language background, it’s not expected to dig into the internals of libraries. However, in systems programming, understanding how all the pieces fit together is crucial. Zig, Redis, TigerBeetle, and PostgreSQL, one of the greatest pieces of software ever written, all have their source code available for study.

That’s the main lesson I’ve learned in this endeavor: it is hard but incredibly rewarding.

The Results

Command-line benchmarking tool used:

./redis-benchmark -t get,set -n 1000000 -c 50

Quick Comparison

Zedis is currently 5-8% slower than Redis across both operations. While not yet beating Redis in raw throughput, this represents a significant achievement for a learning project—we’re within single-digit percentage points of one of the most optimized in-memory data stores ever written.

Detailed Results

Redis Baseline

====== SET ======
  1000000 requests completed in 4.25 seconds
  50 parallel clients
  3 bytes payload

Summary:
  throughput: 235,294.12 requests per second
  latency summary (msec):
          avg min p50 p95 p99 max
        0.115 0.056 0.119 0.127 0.143 2.223

====== GET ======
  1000000 requests completed in 4.29 seconds
  50 parallel clients
  3 bytes payload

Summary:
  throughput: 233,045.92 requests per second
  latency summary (msec):
          avg min p50 p95 p99 max
        0.113 0.048 0.119 0.127 0.135 0.487

Zedis Performance

====== SET ======
  1000000 requests completed in 4.60 seconds
  50 parallel clients
  3 bytes payload

Summary:
  throughput: 217,344.06 requests per second
  latency summary (msec):
          avg min p50 p95 p99 max
        0.121 0.016 0.119 0.135 0.175 1.375

====== GET ======
  1000000 requests completed in 4.53 seconds
  50 parallel clients
  3 bytes payload

Summary:
  throughput: 220,799.30 requests per second
  latency summary (msec):
          avg min p50 p95 p99 max
        0.119 0.056 0.119 0.135 0.143 0.527

What’s Next?

I’m pretty satisfied with Zedis’ current performance. This might be the sunset of this project for now. Stay tuned for future updates as I continue to learn and explore systems programming and database internals.

If you’re a web developer curious about systems programming, I can’t recommend this enough. Start small, make mistakes (you will—I made plenty!), profile them, fix them, and watch your understanding deepen. You don’t need to be a C wizard or have a CS degree, just curiosity and persistence.

Thanks for reading! Subscribe to follow my journey—I’m learning systems programming and sharing everything I discover along the way.

Learn More

Async/Await is finally back in Zig

Charles Fonseca — Wed, 12 Nov 2025 17:48:42 +0000

I’ve been following the Zig programming language with keen interest over the past several months. I've been eagerly awaiting the new async/await API's release ever since I watched this video.

The foundation for this long-awaited feature arrived via pull request #25592, which was merged just one day before I began writing this. I was so excited to try it out that I compiled Zig from the master branch on my local machine—the easier way to access it right now.

Why This Matters

Zig removed async/await support from earlier versions while the team completely redesigned the API from the ground up. The goal is to create something entirely new that differs from what other languages like Go, JavaScript, and Rust do. The new async I/O framework is set for release in version 0.16.0. To use it, you can either compile the master branch or download the latest tarball on this page.

The Problem of Colored Functions

To understand why this redesign matters, consider JavaScript’s function model. There are regular functions and async functions—two distinct “colors” that don’t mix well. You can call regular functions from within async functions, but async functions cannot be called from regular functions without special handling.

Those who lived through JavaScript’s initial async/await rollout will remember the pain. The migration was brutal. Projects like Node.js had to maintain dual standard libraries: one callback-based and another using async/await. Codebases were effectively split in two.

The New API

The IO API is intended to work similarly to the Allocator interface; select a different allocator, and the system will continue to function normally. Similarly, you can change the Io interface without changing anything else in the codebase. For example, to declare a single-threaded Io.

const std = @import("std");

var io: std.Io.Threaded = .init_single_threaded;

Multithreaded will work like:

var io: std.Io.Threaded = .init(allocator);
// The number of threads can be configured by setting cpu_count
io.cpu_count = 4;

If you want async IO (io_uring/kqueue), it’s as simple as:

var event_io: std.Io.Evented = undefined;
try event_io.init(allocator, .{});
const io = event_io.io();

I am no expert on language design, but this is quite sophisticated.

Disclaimer : The interface std.Io.Evented does not yet work on my machine, because the Zig version I compiled (version 0.16.0-dev.1187+1d80c9540) only has io_uring implemented, while the macOS version—kqueue—is not.

Error Handling and Cancellation

One of the most important patterns in Zig’s new async I/O model is proper resource cleanup. As Andrew Kelley emphasizes in his article about the new Zig feature, directly chaining try with await can cause resource leaks when errors occur early in your code, skipping subsequent cleanup operations.

Both cancel and await are idempotent operations that share identical semantics, with one key difference: cancellation also requests task termination. This means you can safely call cancel on a task that has already been completed—the operations are designed to be safe and predictable. During my tests, if the cancel function is not used, the operation hangs.

Here’s the pattern you should follow:

var task1 = try io.async(someOperation, .{args})
defer task1.cancel(io) catch {};

var task2 = try io.async(anotherOperation, .{args});
defer task2.cancel(io) catch {};

// Await all tasks BEFORE handling errors
try task1.await(io);
try task2.await(io);

The key insight is to always use defer cancel() immediately after creating each task, then await all tasks before handling any errors. Similar to calling a defer deinit() function to clean up resources after allocating memory in the heap. This ensures that:

Resources are cleaned up even if errors occur early (before awaiting)
Cancel is safely called on completed tasks (it’s idempotent)
No resource leaks occur from early error returns

Note that we use defer because cancel should happen regardless of success or failure—it’s safe to cancel a completed task.

Async vs. Concurrent: A Critical Distinction

Asynchronous code does not automatically mean concurrent execution. This distinction is essential for understanding how Zig’s async I/O works.

Using a single-threaded pool with the async function can result in deadlock in producer-consumer patterns. The async function simply separates the call to a function from its return—it does not guarantee that multiple operations will run concurrently.

This is where the concurrent function becomes important. Unlike async, concurrent explicitly expresses concurrency needs. If you use concurrent on a truly single-threaded system, it will fail with error.ConcurrencyUnavailable rather than deadlocking, which is great, as it can be treated in the error handling flow.

The key takeaway: use async when you want asynchronous behavior (decoupled call/return), but use concurrent when you actually require operations to run simultaneously. This distinction prevents subtle bugs and makes your concurrency requirements explicit in the code.

Concurrent Execution in Practice

Here’s a complete example demonstrating true concurrent execution with multiple HTTP requests:

const std = @import(”std”);
const HostName = std.Io.net.HostName;

pub fn main() !void {
    var gpa = std.heap.GeneralPurposeAllocator(.{}){};
    defer _ = gpa.deinit();
    const allocator = gpa.allocator();

    var io_impl: std.Io.Threaded = .init(allocator);
    defer io_impl.deinit();
    const io = io_impl.io();
    const host_name: HostName = try .init(”example.com”);

    // Run 3 requests concurrently
    var results: [3]usize = undefined;

    // Start all concurrent tasks
    var task1 = try io.concurrent(request_website, .{ allocator, io, host_name, 0, &results[0] });
    defer task1.cancel(io) catch {};

    var task2 = try io.concurrent(request_website, .{ allocator, io, host_name, 1, &results[1] });
    defer task2.cancel(io) catch {};

    var task3 = try io.concurrent(request_website, .{ allocator, io, host_name, 2, &results[2] });
    defer task3.cancel(io) catch {};

    // Wait for all tasks to complete
    try task1.await(io);
    try task2.await(io);
    try task3.await(io);

    std.log.info(”All requests completed successfully!”, .{});
    std.log.info(”Results: {any}”, .{results});
}

fn request_website(allocator: std.mem.Allocator, io: std.Io, host_name: HostName, index: usize, result: *usize) !void {
    var http_client: std.http.Client = .{ .allocator = allocator, .io = io };
    defer http_client.deinit();

    var request = try http_client.request(.HEAD, .{
        .scheme = “http”,
        .host = .{ .percent_encoded = host_name.bytes },
        .port = 80,
        .path = .{ .percent_encoded = “/” },
    }, .{});
    defer request.deinit();

    try request.sendBodiless();

    var redirect_buffer: [1024]u8 = undefined;

    const response = try request.receiveHead(&redirect_buffer);
    std.log.info(”Index {d} received {d} {s}”, .{ index, response.head.status, response.head.reason });
    result.* = index;
}

Running this program produces output like:

info: Index 2 received 200 OK
info: Index 1 received 200 OK
info: Index 0 received 200 OK
info: All requests completed successfully!
info: Results: { 0, 1, 2 }

Notice how the HTTP responses arrive in non-deterministic order (2→1→0 in this run), demonstrating true concurrent execution. Despite the completion order varying, each task correctly writes to its designated position in the results array, showing proper concurrent data handling. The defer task.cancel(io) ensures cleanup happens even if errors occur.

Why I’ve Been Waiting: The Zedis Story

Since launching Zedis, I’ve been eager to benchmark it against Redis. Presently, Zedis is multithreaded—but that’s temporary. My end goal is to make it single-threaded.

The current multithreaded architecture exists solely to enable testing of pub/sub and other modules. Other high-performance databases written in Zig, like TigerBeetle, are single-threaded with async I/O based on io_uring. Redis takes a different approach: single-threaded execution with multithreaded I/O.

Knowing the Zig team was working on a proper async/await implementation, I made a conscious decision to wait. Now that the new API is available, it's time to properly rebuild Zedis' network layer.

Learn More

Thanks for reading Charles Fonseca's Substack! Subscribe for free to receive new posts and support my work.

Building a Redis Clone in Zig—Part 3

Charles Fonseca — Tue, 28 Oct 2025 01:21:32 +0000

In this series we are going through the hard parts; if you want to see the whole implementation, refer to the GitHub repository.

Why is there a software craftsmanship movement? What motivated it? What drives it now? One thing; and one thing only.
We are tired of writing crap.
— Robert C. Martin

In Part 2, we squeezed performance from our in-memory store through string interning and hash map optimizations. But what happens when the power goes out? In this article, we’ll make our Redis clone durable by implementing RDB snapshots—and learn some hard lessons about Zig’s IO interface along the way.

Implementing the RDB Writer

The RDB file format is a compact binary format used by Redis to persist its in-memory data to disk. It’s meant to be used as a snapshot of the database at a specific time, not for real-time replication. The format consists of a series of opcodes that represent different data types and commands. The specification can be found in the RDB File Format documentation.

To implement the RDB writer, we will create a new module called rdb.zig. This module will contain functions to serialize our in-memory data structures into the RDB format. We will start by defining the basic structure of our RDB writer.

pub const Writer = struct {
    allocator: std.mem.Allocator,
    buffer: *[1024]u8,
    file: std.fs.File,
    store: *Store,
    writer: std.fs.File.Writer,

    pub fn init(allocator: std.mem.Allocator, store: *Store, fileName: []const u8) !Writer {
        fs.cwd().deleteFile(fileName) catch |err| switch (err) {
            error.FileNotFound => {},
            else => return err,
        };
        const file = try std.fs.cwd().createFile(fileName, .{ .truncate = true });
        const buffer = try allocator.create([1024]u8);
        const writer = file.writer(buffer);

        return .{
            .allocator = allocator,
            .buffer = buffer,
            .file = file,
            .store = store,
            .writer = writer,
        };
    }
}

What about the so-called new IO interface?

For those following along with the latest Zig developments, you might have noticed that the IO interface has been evolving quite a bit. It introduces the std.Io.Writer and std.Io.Reader interfaces, which provide a more flexible way to handle IO operations. In our implementation, we keep the std.fs.File.Writer, why not store it as a std.Io.Writer?

pub const Writer = struct {
    allocator: std.mem.Allocator,
    buffer: *[1024]u8,
    file: std.fs.File,
    store: *Store,
    writer: *std.Io.Writer,

    pub fn init(allocator: std.mem.Allocator, store: *Store, fileName: []const u8) !Writer {
        fs.cwd().deleteFile(fileName) catch |err| switch (err) {
            error.FileNotFound => {},
            else => return err,
        };
        const file = try std.fs.cwd().createFile(fileName, .{ .truncate = true });
        const buffer = try allocator.create([1024]u8);
        var file_writer = file.writer(buffer);

        return .{
            .allocator = allocator,
            .buffer = buffer,
            .file = file,
            .store = store,
            .writer = &file_writer.interface,
        };
    }
}

This code will compile just fine, but when you run it, you might encounter unexpected behavior or crashes. The reason is that @fieldParentPtr is used internally by the new IO interface to get the parent struct from an interface pointer. Since file_writer is a local variable, once the init function returns, the memory for file_writer is no longer valid. Any attempt to use writer afterward will lead to undefined behavior. You can read more about this in Karl Seguin’s excellent post on the topic: Zig’s new Writer.

Continuing the Writer Implementation

While the RDB format specification is extensive, this article focuses on the implementation patterns for any binary file writer in Zig. For complete RDB format details, see the official spec and Zedis's RDB implementation. Let’s continue with our implementation by adding functions to write the RDB header and auxiliary fields.

fn writeAuxFields(self: *Writer) !void {
    // Magic String
    _ = try self.writer.interface.write(”REDIS”);
    // RDB Version
    _ = try self.writer.interface.write(”0012”);

    // Auxiliary fields
    // Redis version
    try self.writeMetadata(”redis-ver”, .{ .string = “255.255.255” });

    const bits = if (@sizeOf(usize) == 8) 64 else 32;
    try self.writeMetadata(”redis-bits”, .{ .int = bits });

    const now_timestamp = std.time.timestamp();
    try self.writeMetadata(”ctime”, .{ .int = now_timestamp });

    try self.writeMetadata(”used-mem”, .{ .int = 0 });

    try self.writeMetadata(”aof-base”, .{ .int = 0 });
}

fn writeHeader(self: *Writer) !void {
    const OPCODE_SELECT_DB = 0xFE;
    const OPCODE_RESIZE_DB = 0xFB;

    try self.writeAuxFields();

    try self.writer.interface.writeByte(OPCODE_SELECT_DB);
    try self.writeLength(0x00);

    try self.writer.interface.writeByte(OPCODE_RESIZE_DB);
    try self.writeLength(self.store.size());
    // TODO
    try self.writeLength(0);
}

pub fn writeFile(self: *Writer) !void {
    try self.writeHeader();
    try self.writeCache();
    try self.writeEndOfFile();

    try self.writer.interface.flush();
}

It’s a bit of a pain to have to use self.writer.interface everywhere, but it’s necessary to get things working correctly.

Writing the CRC64 Checksum

To make sure our RDB files are valid, we need to compute and write a CRC64 checksum at the end of the file. Zig has the Redis CRC implementation built-in —std.hash.crc.Crc64Redis—, so we don't need to do it ourselves, though I did for learning purposes and didn't realize Zig already had it.

The easiest way to write the checksum, yet the most inefficient, is to read back the entire file and compute the checksum. That would be outrageous! We can do better!

Computing the Checksum on the Fly

To compute the checksum on the fly, we can maintain a running CRC64 value as we write the file. This way, we don’t need to read the file back to compute the checksum.

Here’s how we can implement this:

Initialize the CRC64 value to the initial value.
Update the CRC64 value with each write operation.
Write the final CRC64 value at the end of the file.

The new IO interface shines brightly here, as we can create a custom writer, keeping the std.Io.Writer interface and updating the CRC64 value every write. This is how we can do it:

// WriterCrc.zig
const std = @import(”std”);

underlying_writer: *std.Io.Writer,
checksum: std.hash.crc.Crc64Redis,
interface: std.Io.Writer,

const WriterCrc = @This();

pub fn init(underlying: *std.Io.Writer, buffer: []u8) WriterCrc {
    return .{
        .underlying_writer = underlying,
        .checksum = .init(),
        .interface = .{
            .vtable = &.{
                .drain = drain,
                .flush = flush,
                .sendFile = std.Io.Writer.unimplementedSendFile,
                .rebase = std.Io.Writer.defaultRebase,
            },
            .buffer = buffer,
        },
    };
}

fn drain(w: *std.Io.Writer, data: []const []const u8, splat: usize) std.Io.Writer.Error!usize {
    _ = splat;
    const self: *WriterCrc = @fieldParentPtr(”interface”, w);

    // Capture the bytes and update checksum
    const bytes_to_write = data[0];
    self.checksum.update(bytes_to_write);

    // Write to underlying writer (don’t flush here - that’s done by the flush vtable function)
    try self.underlying_writer.writeAll(bytes_to_write);

    return bytes_to_write.len;
}

fn flush(w: *std.Io.Writer) std.Io.Writer.Error!void {
    const self: *WriterCrc = @fieldParentPtr(”interface”, w);

    // First, drain any buffered data in WriterCrc’s own buffer
    while (w.end > 0) {
        _ = try drain(w, &.{w.buffer[0..w.end]}, 0);
        w.end = 0;
    }

    // Then flush the underlying writer
    try self.underlying_writer.flush();
}

pub fn writer(self: *WriterCrc) *std.Io.Writer {
    return &self.interface;
}

pub fn getChecksum(self: *WriterCrc) u64 {
    return self.checksum.final();
}

Therefore, we can modify our RDB Writer struct to use this WriterCrc:

pub const Writer = struct {
    allocator: std.mem.Allocator,
    buffer: []u8,
    crc_buffer: []u8,
    file: std.fs.File,
    file_writer: *std.fs.File.Writer,
    store: *Store,
    writer_crc: WriterCrc,

    pub fn init(allocator: std.mem.Allocator, store: *Store, fileName: []const u8) !Writer {
        fs.cwd().deleteFile(fileName) catch |err| switch (err) {
            error.FileNotFound => {},
            else => return err,
        };
        const file = try std.fs.cwd().createFile(fileName, .{ .truncate = true });

        const buffer_size = 256 * 1024; // 256KB

        const buffer = try allocator.alloc(u8, buffer_size);
        errdefer allocator.free(buffer);

        const crc_buffer = try allocator.alloc(u8, buffer_size);
        errdefer allocator.free(crc_buffer);

        // Allocate file_writer on heap so its address doesn’t change
        const file_writer = try allocator.create(std.fs.File.Writer);
        errdefer allocator.destroy(file_writer);

        file_writer.* = file.writer(buffer);

        // Now initialize WriterCrc with a pointer to the heap-allocated file_writer
        const writer_crc = WriterCrc.init(&file_writer.interface, crc_buffer);

        return .{
            .allocator = allocator,
            .buffer = buffer,
            .crc_buffer = crc_buffer,
            .file = file,
            .file_writer = file_writer,
            .store = store,
            .writer_crc = writer_crc,
        };
    }
}

With this setup, every time we write data using self.writer_crc.writer(), the CRC64 checksum will be updated automatically. Why do we have two buffers, though? One for the file writer and another for the CRC writer?

The reason is that each writer maintains its buffer for efficiency. The WriterCrc needs its buffer to accumulate writes before updating the checksum, while the underlying file writer has its buffer to optimize disk writes. This two-layer buffering ensures that both the checksum calculation and file writing are efficient.

Why This Matters for Performance

Without the CRC layer’s buffer, we’d update the checksum on every single byte write, even if those bytes haven’t been flushed yet. With buffering:

Small writes accumulate in crc_buffer before checksum computation.
Checksum updates happen in larger chunks (more cache-friendly).
File writes are also batched (fewer syscalls to the kernel).

This means writing 1000 individual bytes results in far fewer checksum updates and syscalls than 1000 separate operations. The performance difference is significant for large RDB files.

Here’s a diagram to illustrate the flow of data:

When we’re done writing, we can get the final checksum and add it to the file.

fn writeEndOfFile(self: *Writer) !void {
    const OPCODE_EOF = 0xFF;
    try self.writer_crc.writer().writeByte(OPCODE_EOF);

    // Get the accumulated checksum from all writes
    const checksum = self.writer_crc.getChecksum();

    // Write checksum
    try self.writer_crc.writer().writeInt(u64, checksum, .little);
}

When to Use RDB vs. AOF

Now that we’ve implemented RDB snapshots, you might wonder, when should you use them? Understanding the trade-offs helps you choose the right persistence strategy.

RDB snapshots are great for:

Backup and restore scenarios —compact binary format transfers quickly.
Faster restarts with large datasets —loading RDB is faster than replaying millions of commands.
Point-in-time snapshots —great for disaster recovery with periodic backups.
Lower disk I/O —snapshots happen periodically, not on every write.

But they’re not ideal for:

Minimal data loss —you lose all writes since the last snapshot on crash.
Real-time durability —there's always a window of data loss between snapshots.
Frequently changing data —if data changes faster than the snapshot interval, you’re always behind.

AOF (Append-Only File) complements RDB by:

Logging every write command as it happens (minimal data loss).
Providing point-in-time recovery (replay up to any command).
Trading disk I/O for durability (every write hits disk).

That’s why Redis uses both strategies: RDB for fast restarts and backups, AOF for durability. You can configure Redis to use RDB-only, AOF-only, or both (the safest option). In production, most deployments use both: RDB for snapshots and AOF for durability between snapshots.

In our implementation, we’ve built the foundation for RDB. In future parts, we’ll add AOF logging to achieve Redis-level durability guarantees.

Wrapping Up

We’ve built a production-ready RDB writer from scratch, learning:

How to safely work with Zig’s new IO interface.
Why pointer lifetimes matter with @fieldParentPtr.
How to compute checksums on-the-fly without performance penalties.
When to choose RDB vs. AOF for persistence.

In Part 4, we’ll implement the reader side and discover how Redis handles backward compatibility with older RDB versions.

Learn More

Enjoy this post? Get new ones delivered straight to your inbox. Subscribe for free and support my work.

Building a Redis Clone in Zig—Part 2

Charles Fonseca — Tue, 14 Oct 2025 22:45:24 +0000

In the previous article, we covered the basics of building a Redis clone in Zig, focusing on the data structures and the core functionality. In this article, we will dive deeper into optimizing our in-memory store by implementing string interning and customizing our hash map for better performance.

Hash Table Context

If you noticed, this is how the OptimizedHashMap was defined in the previous article:

const OptimizedHashMap = std.ArrayHashMapUnmanaged([]const u8, ZedisObject, StringContext, true);

Let's focus on the StringContext part. The default context for HashMaps uses the standard library's std.hash.Wyhash, which is a great general-purpose hash function, and for equality, it uses std.mem.eql(u8), which is a byte-wise equality check. We can customize this by providing our own context struct.

Benchmarking

During development, I found that the default hash function and equality check were not optimal for our use case. Specifically, we want to optimize for short strings, as Redis often deals with short keys and values. I assumed SIMD (Single Instruction, Multiple Data) was the best way to speed up equality checks, and the answer was a huge it depends.

Equality Benchmark

The obvious choice for equality is to use SIMD instructions for faster comparisons. For which, again, Karl Seguin has a great article on the topic: Using SIMD to Tell if the High Bit is Set. We won't be checking the high bit, but the same principles apply.

Our baseline will be the standard library's std.mem.eql(u8) function. During the benchmark, I found that it's pretty hard to beat the standard library, my original SIMD implementation was half the speed of std.mem.eql(u8) for almost all string lengths.

You can see the benchmark code and results in this GitHub Gist. Long story short, for strings shorter than 45 bytes, std.mem.eql(u8) is the fastest option. For strings between 45 and 128 bytes, a 16-byte SIMD using XOR is the fastest; it reaches an average of 850 million operations per second on my M4 MacBook Pro.

Here's the final implementation of the SIMD equality function:

pub fn simdStringEql(a: []const u8, b: []const u8) bool {
    // Fast path
    if (a.len != b.len) return false;
    if (a.len == 0) return true;

    // Check for pointer equality first, this is a fast path for interned strings
    if (a.ptr == b.ptr) return true;

    // For strings < 45 bytes, std.mem.eql is faster due to:
    // - Highly optimized LLVM intrinsics
    // - Lower overhead for small comparisons
    // - Better branch prediction
    // Most Redis keys fall into this category
    if (a.len < 45) {
        return std.mem.eql(u8, a, b);
    }

    // For longer strings (≥45 bytes), use explicit SIMD vectorization
    // Fixed 16-byte vectors for consistent performance across platforms
    const vec_len = 16;
    const Vec = @Vector(vec_len, u8);
    var i: usize = 0;

    // Process 16-byte chunks with SIMD using XOR technique
    // XOR is slightly more efficient than equality comparison + reduce
    while (i + vec_len <= a.len) : (i += vec_len) {
        const va: Vec = a[i..][0..vec_len].*;
        const vb: Vec = b[i..][0..vec_len].*;
        const xor_result = va ^ vb;

        // If XOR result is all zeros, the vectors are equal
        // Check if any byte is non-zero (which means difference)
        if (@reduce(.Or, xor_result != @as(Vec, @splat(0)))) {
            return false;
        }
    }

    // Handle remaining bytes efficiently with std.mem.eql
    // This is faster than a scalar loop for the tail
    return std.mem.eql(u8, a[i..], b[i..]);
}

Hash Function Benchmark

I also wanted to benchmark different hash functions to see which one performs best for our use case. The standard library provides several hash functions, including Wyhash, CityHash64, Murmur3, and others. I added a few more to the benchmark, including Murmur2, XxHash (32 and 64), Adler32, and Blake3 (cryptographic). I implemented a simple FNV-1a hash function for comparison as well. You can see the benchmark code and results in this GitHub Gist. Long story short, CityHash64 outperforms Wyhash slightly, reaching an average of 490 million operations per second versus Wyhash's 460 million on my laptop.

Intern Strings

The idea is to store only one copy of each unique string value, and all references to that string point to the same memory location. This is a common technique used in many programming languages and systems to optimize memory usage and speed up string comparisons. That's why in the equality function above, we first check if the pointers are the same, if (a.ptr == b.ptr) return true;. If they are, we can immediately return true without further comparison, we get O(1) complexity for almost all string comparisons.

pub const Store = struct {
    base_allocator: std.mem.Allocator,
    allocator: std.mem.Allocator,
    map: OptimizedHashMap,

    ...

    // Map of interned strings
    interned_strings: std.StringHashMapUnmanaged([]const u8),

    fn internString(self: *Store, str: []const u8) ![]const u8 {
        assert(str.len > 0);
        const gop = try self.interned_strings.getOrPut(self.base_allocator, str);
        if (!gop.found_existing) {
            const owned_str = try self.dupeString(str);
            gop.key_ptr.* = owned_str;
            gop.value_ptr.* = owned_str;
        }
        return gop.value_ptr.*;
    }

    pub fn set(self: *Store, key: []const u8, value: []const u8) !void {
        assert(key.len > 0);
        // Check if we need to resize before adding new entries
        try self.maybeResize();

        const interned_key = try self.internString(key);
        const gop = try self.map.getOrPut(self.base_allocator, interned_key);

        // Free old value if key existed
        if (gop.found_existing) {
            switch (gop.value_ptr.value) {
                .string => |str| {
                    if (str.len > 0) self.smartFree(str);
                },
                .int => {},
                .short_string => {},
            }
        }
        // Key is already interned, no need to duplicate again
        gop.key_ptr.* = interned_key;

        // Build the new object with allocated values
        var new_object = object;
        switch (object.value) {
            .string => |str| {
                const value_copy = try self.dupeString(str);
                new_object.value = .{ .string = value_copy };
            },
            .int => {}, // No allocation needed
            .short_string => {}, // No allocation needed - stored inline
        }

        // Update the entry
        gop.value_ptr.* = new_object;
    }

By doing this, we ensure that all identical strings share the same memory, which saves space and speeds up comparisons. To summarize, the Store struct now includes an interned_strings map to keep track of all unique strings, we added the short_string variant to the ZedisObject union to store small strings inline, and we updated the hash map context to use our optimized hash function and equality check. This is a great foundation for building a high-performance in-memory cache.

Know More

Zedis GitHub Repository

Building a Redis Clone in Zig—Part 1

Charles Fonseca — Sun, 12 Oct 2025 22:38:34 +0000

I’ve been writing web applications for years, mostly high-level stuff with frameworks that abstract away the messy details. You know the type: ORMs, HTTP servers, JSON serialization—all the conveniences of modern backend development. But I’ve always been curious about what happens beneath those abstractions. How does a database actually work? What does efficient memory management look like? So a few months ago, I decided to learn Zig by rebuilding Redis from scratch.

Why Redis?

Redis is deceptively simple from the outside: it’s an in-memory key-value store with a straightforward text protocol. But underneath, it’s a masterclass in systems programming—memory management, data structures, network I/O, and concurrency all wrapped up in one project. Perfect for someone trying to bridge the gap between web development and systems programming. Also, almost every company I’ve worked for used it in some way.

The whole promise of Zig is explicit control—memory, error handling, and resources. There’s no garbage collector to save you, no exceptions to bubble up unhandled. If you want to allocate memory, you have to say exactly how, and more importantly, you have to free it yourself. This is precisely what Redis needs: tight control over memory for performance and predictability.

The Protocol

Redis uses RESP (REdis Serialization Protocol), a simple text-based format. A command like SET mykey myvalue arrives as:

*3\r\n
$3\r\n
SET\r\n
$5\r\n
mykey\r\n
$7\r\n
myvalue\r\n

The *3 means “array of 3 elements.” Each $N is a bulk string of N bytes. It’s verbose but easy to parse.

Here’s a simplified parser:

fn parseCommand(reader: anytype) ![][]const u8 {
    var args = std.ArrayList([]const u8).init(allocator);

    // Read first byte to determine type
    const type_byte = try reader.readByte();
    if (type_byte != ‘*’) return error.InvalidProtocol;

    // Read number of arguments
    const count = try readInteger(reader);

    for (0..count) |_| {
        // Each arg starts with $
        const dollar = try reader.readByte();
        if (dollar != ‘$’) return error.InvalidProtocol;

        // Length of this arg
        const len = try readInteger(reader);

        // Read the actual bytes
        const arg = try allocator.alloc(u8, len);
        _ = try reader.readAll(arg);
        try args.append(arg);

        // Consume \r\n
        _ = try reader.readByte();
        _ = try reader.readByte();
    }
    return args.toOwnedSlice();
}

Memory Management: The Hard Part

Coming from garbage-collected languages, the hardest adjustment was thinking about ownership. In Zig, when you allocate memory, you own it. You’re responsible for freeing it. There’s no runtime to clean up after you.

Here’s a simple key-value store:

const Store = struct {
    map: std.StringHashMap([]const u8),
    allocator: std.mem.Allocator,

    pub fn init(allocator: std.mem.Allocator) Store {
        return .{
            .map = std.StringHashMap([]const u8).init(allocator),
            .allocator = allocator,
        };
    }

    pub fn set(self: *Store, key: []const u8, value: []const u8) !void {
        // Duplicate the key and value so we own them
        const owned_key = try self.allocator.dupe(u8, key);
        const owned_value = try self.allocator.dupe(u8, value);

        // If key exists, free old value
        if (self.map.get(key)) |old_value| {
            self.allocator.free(old_value);
        }

        try self.map.put(owned_key, owned_value);
    }

    pub fn deinit(self: *Store) void {
        var iter = self.map.iterator();
        while (iter.next()) |entry| {
            self.allocator.free(entry.key_ptr.*);
            self.allocator.free(entry.value_ptr.*);
        }
        self.map.deinit();
    }
};

This is a great start, but there are some improvements to be made. Most of the keys and values in a typical caching system are small, so SSO (Small String Optimization) can help reduce allocations.

pub const ShortString = struct {
    data: [23]u8, // Inline storage
    len: u8, // Actual length

    pub fn fromSlice(str: []const u8) ShortString {
        var ss: ShortString = .{ .data = undefined, .len = @intCast(str.len) };
        @memcpy(ss.data[0..str.len], str);
        // Zero remaining bytes for consistent hashing
        @memset(ss.data[str.len..], 0);
        return ss;
    }

    pub fn asSlice(self: *const ShortString) []const u8 {
        return self.data[0..self.len];
    }
};

This ShortString struct can hold strings up to 23 bytes inline, avoiding heap allocations for small keys/values. Usually, a CPU cache line is 64 bytes, allowing room for multiple instances to fit in a single cache line. I still need to evaluate the performance difference between 23 bytes and 15 bytes, but this is a good starting point.

Hash Tables

Zig’s standard library provides a StringHashMap, which is a good starting point. However, we need to ensure that our keys and values are properly managed in terms of memory ownership. For that, we can use StringHashMapUnmanaged, which allows us to define custom allocators and manage memory more explicitly.

To go further, we can create an optimized hash map tailored for our use case. We must be able to store integers, short strings, larger strings, and later, more complex types like lists and hashes. Let’s take a look at how we can define this:

pub const ValueType = enum(u8) {
    string = 0,
    int,
    list,
    short_string,
};

pub const ZedisValue = union(ValueType) {
    string: []const u8,
    int: i64,
    list: ZedisList,
    // Showed before
    short_string: ShortString,
};

pub const ZedisObject = struct {
    value: ZedisValue,
    expiration_time: i64,
    // Plan to add more fields in the future
};

const OptimizedHashMap = std.ArrayHashMapUnmanaged([]const u8, ZedisObject, StringContext, true);

This will improve performance by reducing allocations for small strings and integers, which are common in Redis workloads. Additionally, we can process KEYS and SCAN commands more efficiently by iterating over the hash map directly, since the underlying structure is an array.

Memory Pool

Karl Seguin has already written a fantastic article about it; for a more in-depth understanding, see that. In summary, it keeps track of previously destroyed objects so that the same address in memory can be used again, making it an excellent choice for objects that are allocated and released frequently.

pub const Store = struct {
    base_allocator: std.mem.Allocator,
    allocator: std.mem.Allocator,
    // Cache hash map
    map: OptimizedHashMap,

    // Hybrid memory pools for different string sizes
    small_pool: std.heap.MemoryPool([32]u8), // 32 bytes - for keys
    medium_pool: std.heap.MemoryPool([128]u8), // 128 bytes - for small values
    large_pool: std.heap.MemoryPool([512]u8), // 512 bytes - for medium values

    pub fn init(allocator: std.mem.Allocator) Store {
        return .{
            .base_allocator = allocator,
            .allocator = allocator,
            .map = .{},
            .small_pool = SafeMemoryPool.init(allocator, SMALL_STRING_SIZE),
            .medium_pool = SafeMemoryPool.init(allocator, MEDIUM_STRING_SIZE),
            .large_pool = SafeMemoryPool.init(allocator, LARGE_STRING_SIZE),
        };
    }
}

It performs especially well with caching system workloads like rate limiters, session stores, user metrics, etc. With that optimization, the SET command is O(1) rather than O(n) in terms of string allocation; just beware that if the memory pool doesn’t have an address available, it will call the syscall alloc.

Know more

Zedis on GitHub.

I'm building a Redis Clone in Zig: A Deep Dive into Pub/Sub, and Memory Allocation

Charles Fonseca — Fri, 19 Sep 2025 22:16:41 +0000

It's been a while since I started a project called Zedis (Redis written in Zig). It's been a great way to learn low-level programming.

I want to take you on a tour of three core features I've implemented: Pub/Sub and the memory allocation strategy.

Why Build a Redis Clone in Zig?

I have a personal goal of mastering Zig this year; this was the perfect way to accomplish that. Zig offers a number of intriguing features for high-performance systems, such as comptime, which allows code to run at compile time, and explicit memory management. Zedis has been my playground for exploring everything from network programming to custom allocators.

Feature Deep Dive: Pub/Sub

Implementing the Publish/Subscribe mechanism was a fun challenge. At its core, it's a messaging system that decouples senders (publishers) from receivers (subscribers).

Here's how it works in Zedis:

Channels and Subscribers: When a client subscribes to a channel, they are added to a list of subscribers for that channel. I used a std.StringHashMap([]u64) to map channel names to a list of client IDs.
Publishing a Message: When a message is published to a channel, the server iterates through the list of subscribers and writes the message to each of their connections.
Entering Pub/Sub Mode: Once a client subscribes to a channel, they enter a special “Pub/Sub mode” where they can only receive messages and can't execute other commands.

Here is the subscribe function:

pub fn subscribe(client: *Client, args: []const Value) !void {
    var pubsub_context = client.pubsub_context;
    // Enter pubsub mode on first subscription
    if (!client.is_in_pubsub_mode) {
        client.enterPubSubMode();
    }

    var i: i64 = 0;
    for (args[1..]) |item| {
        const channel_name = item.asSlice();
        // Ensure channel exists
        pubsub_context.ensureChannelExists(channel_name) catch {
            try client.writeError("ERR failed to create channel");
            continue;
        };

        // Subscribe client to channel
        pubsub_context.subscribeToChannel(channel_name, client.client_id) catch |err| switch (err) {
            error.ChannelFull => {
                try client.writeError("ERR maximum subscribers per channel reached");
                continue;
            },
            else => {
                try client.writeError("ERR failed to subscribe to channel");
                continue;
            },
        };

        const subscription_count = i + 1;
        const response_tuple = .{
            "subscribe",
            channel_name,
            subscription_count,
        };
        // Use a generic writer to send the tuple as a RESP array.
        try client.writeTupleAsArray(response_tuple);
        i += 1;
    }
}

Memory Allocation Strategy

One of the most interesting aspects of this project has been designing the memory allocation strategy. I opted for a hybrid approach to balance performance and memory usage:

KeyValueAllocator: This is a custom allocator I built for the main key-value store. It uses a fixed-size memory pool and has a simple eviction policy to stay within its budget. When the allocator runs out of memory, it can evict all keys to make space for new ones.
Arena Allocator: For temporary, short-lived allocations (like parsing commands), I use an arena allocator. This is incredibly fast because it simply bumps a pointer for new allocations and frees all the memory at once when it's no longer needed.
Fixed Pools: For objects that are frequently allocated and deallocated, like client connections, I use a fixed-size pool. This avoids the overhead of dynamic allocation and deallocation.

This approach allows Zedis to be memory-efficient while still being performant.

What's Next?

I'm excited to keep building on this foundation. Here's what's on the roadmap:

Implement AOF (Append Only File) logging
Add support for more data structures like lists and sets
Implement key expiration
Add clustering support

Check it out

I'd love for you to check out Zedis on GitHub, try it out, and let me know what you think. Contributions are always welcome!

Thanks for reading!