Forem: Pierre Jambet

A toy Redis Server, in Clojure

Pierre Jambet — Sun, 31 Dec 2023 18:39:32 +0000

This is the second entry in the "Building Toy Redises in X" series". In the first article we used Go, in this one we'll use Clojure.

One of the approaches explored in this article is very similar to the one used in Go, with the use of Channels, so reading it first might help.

I've never written Clojure code professionally, what you're about to read is the result of a slow and painful process of trial and error. Take everything with a grain of salt. And if you are an experienced Clojure developer and spot something outrageous, please reach out!

What we're building

We're building something very similar to what we built in the first article, you can check out the details over there under the "What we're building" section for more information.

All the code in this post is in the content/src/clj-tcp-server/ folder, available on GitHub.

Starting small, a TCP server and nothing else

One of the great things with Clojure is that you get access to everything that Java offers. So when it comes to starting a TCP server, we can use Java's ServerSocket class, instantiate it with the port number it should listen to, and use its accept method to accept new clients.

We get back a Socket instance, which we can conveniently use clojure.java.io/writer with, to get back an instance of java.io.BufferedWriter. With that writer instance, we can now write to the socket with the write method, and we need to call flush, to make sure the data gets written, and can be read on the other end.

Finally, we close the socket with close, because we'll deal with concurrency later in this post.

(ns tcp
  (:require [clojure.java.io :as io])
  (:import (java.net ServerSocket)))

(defn handle-client
  [client]
  (let [writer (io/writer client)]
    (.write writer "Hello 👋\n")
    (.flush writer)
    (.close client)))


(defn main
  []
  (with-open [server-socket (ServerSocket. 3000)]
     (loop []
       (let [client-socket (.accept server-socket)]
         (handle-client client-socket))
       (recur))))

(main)

You can run the program with clj -M tcp.clj and connect to it with nc -v localhost 3000.

Keeping the connections open

Let's now make the server keep the connections open, wait for the clients to send something over the wire, and respond back. We will use Clojure's core/async library to help with concurrency, for the main reason that I couldn't think of any other solutions.

core/async shares a lot with Go's concurrency mechanisms with coroutines, so if you read the previous entry in this series, this should all look pretty familiar.

If you're thinking "well, since we can use anything that exists in Java, we could use Java's non blocking IO library, java.nio, it's been available since Java 7". Well, you're probably right, I'm sure we could, but I will actually dedicate a whole article to that topic, about building a Toy Redis, in Java, using the nio package!

Clojure lets us spin up new threads, which we could use to handle concurrent clients, but instead we'll use a higher level abstraction, go Blocks. If you've read the previous post, or are familiar with Go, this is very similar to coroutines created with the go keyword.

core.async provides the go macro, it asynchronously executes the body we give it. The following example starts a go block, prints immediately the first statement from the block, and then the one from the main thread, then sleeps for 5s and finally prints done:

(def sleep-time 5000)
(a/go
  (println (str "sleeping for " sleep-time "ms"))
  (Thread/sleep sleep-time)
  (println "done!"))
(println "Printing from main thread")

You can run this example from the REPL with clj -Sdeps '{:deps {org.clojure/core.async {:mvn/version "1.6.681"}}}' and then with:

Clojure 1.11.1
user=> (require '[clojure.core.async :as a])
nil
user=> (def sleep-time 5000)
#'user/sleep-time
user=> (a/go
  (println (str "sleeping for " sleep-time "ms"))
  (Thread/sleep sleep-time)
  (println "done!"))
(println "Printing from main thread")
#object[clojure.core.async.impl.channels.ManyToManyChannel 0x2b3ac26d "clojure.core.async.impl.channels.ManyToManyChannel@2b3ac26d"]
sleeping for 5000ms
user=> Printing from main thread
nil
user=> done!

We can now start a new go block for each new client, but first, let's require core.async:

(ns concurrent-server
  (:require [clojure.java.io :as io]
            [clojure.core.async :as a])
  (:import (java.net ServerSocket)))
;; ...

And we can now use a/go:

(defn handle-client
  [client]
  (a/go
    (loop []
      ;; ...
      )))

Now that each client is given its own go block, we can start reading from clients:

(loop []
  (let [request (.readLine (io/reader client))
        writer (io/writer client)]
    (do
      (.write writer "Hello 👋\n")
      (.flush writer)
      (recur))))

With io/reader, we get back an instance of java.io.BufferedReader, on which we can call the .readLine method, and get a String back. This change means that we now only write back to the client after we received something. In other words, the connection will stay open, and unused, until the client sends a line of text. After which it writes "Hello 👋\n" and waits for the next line of text.

There is an issue however, if the client disconnects before the server is stopped, an exception will be thrown, and uncaught, in the thread started by the go block. You can see this behavior by starting the server with: clj -Sdeps '{:deps {org.clojure/core.async {:mvn/version "1.6.681"}}}' -M concurrent_server.clj, connecting to it with nc localhost 3000 from another terminal, and then closing the client with Ctrl-C. The following exception will show up in the server logs:

Exception in thread "async-dispatch-1" java.net.SocketException: Broken pipe

I didn't include the full stacktrace, but if you were to look at it, despite it being kinda cryptic, we can see that the error occurs when we attempt to write to the socket. We can prevent this by checking if the result of .readLine is nil, in which case, we can close the connection on the server's end, since the client is gone.

The following is the full version of handle-client:

(defn handle-client
  [client]
  (a/go
    (loop []
      (let [request (.readLine (io/reader client))
            writer (io/writer client)]
        (if (nil? request)
          (do
            (println "Nil response, closing client")
            (.close client))
          (do
            (.write writer "Hello 👋\n")
            (.flush writer)
            (recur)))))))

Making the server stateful

The last step is to turn this whole thing stateful. We want the server to store data, so that other clients can read from it.

Clojure's collections are immutable, so we don't have that many options for our go blocks to share the same data structure to read and write on. In the previous chapter we initially tried an approach where we created a map in the main function and passed it to each coroutine, something like this:

(defn main
  []
  (with-open [server-socket (ServerSocket. 3000)]
    (loop []
      (let [client-socket (.accept server-socket)
            db {}]
        (handle-client client-socket db)
        (recur)))))

But then handle-client can't modify db in a way that would be seen by other go blocks. We can for instance add new entries to it:

(defn handle-client
  [client original-db]
  (a/go
    (loop [db original-db]
      (let [request (.readLine (io/reader client))
            writer (io/writer client)]
        (if (nil? request)
          (do
            (println "Nil response, closing client")
            (.close client))
          (let [updated-db (assoc db (.hashCode client) (System/currentTimeMillis))]
            (.write writer "Hello 👋\n")
            (.flush writer)
            (recur updated-db)))))))

In this example we store the timestamp of the most recent line we received from the client, where the key is the .hashCode value of the client. Yes, this is a very contrived example, the only purpose is to show that we have no easy way to make the updates made to the map visible to the outside. The handle-client function returns a channel, which we ignore, because we don't need it. Because we execute a go block, we cannot return data to the caller.

We will explore two approaches to address this issue:

One very similar to the Go chapter, Channels!
Atoms, clojure's builtin "[...] way to manage shared, synchronous, independent state"

Channels

Let's start with the channel version, and we'll look at atoms next.

We'll create one channel in the main function, for all the go blocks to send the requests received from clients to a go block that will handle storing the data and send it back over a different channel, so that a response can be written back to the client. Let's see what it looks like, first the main function:

(defn main
  []
  (let [command-channel (a/chan)]
    (with-open [server-socket (ServerSocket. 3000)]
      (handle-db command-channel)
      (loop []
        (let [client-socket (.accept server-socket)]
          (handle-client client-socket command-channel)
          (recur))))))

You may have noticed that on top of passing command-channel to each call of handle-client, we also pass it to handle-db, before starting the loop. Let's take a look at that function:

(defn handle-db
  [command-channel]
  (a/go
    (loop [db {}]
      (let [resp (a/<! command-channel)
            timestamp (System/currentTimeMillis)
            updated-db (assoc db (.hashCode (:client resp)) timestamp)]
        (a/>! (:channel resp) timestamp)
        (recur updated-db)))))

The function runs entirely in a go block, and starts an infinite loop with a hash map. The first thing it does is wait for a message to be sent to the channel it was given as an argument. When it receives a message, it stores the current timestamp associated with the client that sent the message. It effectively stores the timestamp of the last time it processed a message sent by a client.

Then, it extracts the :channel field from the message, and write the timestamp back to it. Finally, it calls recur with the updated hash map, so that the next iteration sees the changes made to it.

Now, let's look at handle-client, how it sends messages to command-channel, and how it reads back what handle-db sends back after receiving the message:

(defn handle-client
  [client channel]
  (a/go
    (loop []
      (let [request (.readLine (io/reader client))
            writer (io/writer client)
            message {:channel (a/chan) :client client}] ;; (1)
        (if (nil? request)
          (do (println "Nil response, closing client")
              (.close client))
          (do (a/>! channel message) ;; (2)
              (let [result (a/<! (:channel message))] ;; (3)
                (.write writer (str "OK, " result "\n"))
                (.flush writer)
                (recur))))))))

Let's look at the three main changes:

In (1), we create a hash map, called message, with two keys, :channel is a newly built channel, so that handle-db can send back a message to us.
In (2), when we have a non-nil message, we send message to the the channel created in the main function.
In (3), we wait to get a response back from the :channel field of the message hash map, and store the result in the result variable. We then write back the content to the client, sending them the timestamp stored in the internal db.

This is another contrived example, storing the last timestamp of when we received something from a client is not that useful. But we can use this architecture to build our Toy Redis!

Putting everything together.

We want to support the following commands:

GET: Accepts a string, and return the value stored for that key, if any
SET: Accepts two strings, a key and a value, and sets the value for the key, overriding any values that may have been present
DEL: Accepts a string and deletes the value that may have been there
INCR: Accepts a single argument and increments the existing value. If the value is not an integer, it's an error, if there are no values, it gets initialized to 1, resulting in an identical outcome as calling SET <key> 1.

Laying the foundations

For the final version of the server, the main function will look the same:

(defn main
  "Start a server and continuously wait for new clients to connect"
  []
  (println "About to start ...")
  (let [command-channel (a/chan)]
    (handle-db command-channel)
    (with-open [server-socket (ServerSocket. 3000)]
      (loop []
        (let [client-socket (.accept server-socket)]
          (handle-client client-socket command-channel))
        (recur)))))

We will however update the handle-db function to deal with the various commands:

(defn handle-db
  "Run a go block in which we continuously wait for clients to send commands,
  process them, and send back a response through teh channel included in the
  received hash map"
  [command-channel]
  (a/go (loop [db {}]
          (let [response (a/<! command-channel)
                command (response :command)
                key (response :key)
                value (response :value)
                chan-resp (response :resp)
                result (process-command db command key value)
                new-db (result :updated)
                response (result :response)]
            (a/>! chan-resp response)
            (recur new-db)))))

This is similar to what we looked at in the previous section, with more handling of the various elements of a command.

We start the same way, by creating a go block, in which we continuously loop over the hash map that will store all the server's data. What follows in the let call is a sequence of operations to process the command we received from the client through command-channel.

We pass the command, key, value & chan-resp variables to process-command, which we'll look at next, and we then extract two variables from whatever it returns, new-db & response. new-db is what we pass to recur to maintain the inner state of the database on the server. response is what we send back to the client through the channel that was included in the request hash map.

Let's now look at process-command:

(defn process-command
  "Perform various operations depending on the command sent by the client"
  [db command key value]
  (cond
    (= command :get)
    (if key
      (let [value (get db key "")]
        {:updated db :response value})
      {:updated db :response "ERR wrong number of arguments for 'get' command"})
    :else {:updated db :response "Unknown command"}))

To keep things short we've only included the handling of the GET command, we'll add more branches to that cond call later on.

With that cond call, we go over the handled commands, and fallback with the Unknown command error. Remember that whatever we return from the process-command function must contain an updated field with the new data, and a response field, which will be sent back to the go block running for the client that sent the command.

The GET command does not modify the database, so we return it unchanged.

Before looking at the implementation of other commands, let's now look at the handle-client function, the one we call with each newly connected client in the main function:

(defn handle-client
  "Read from a connected client, and handles the various commands accepted by the server"
  [client-socket command-channel]
  (a/go (loop [resp-channel (a/chan)]
          (let [request (.readLine (io/reader client-socket))
                writer (io/writer client-socket)]
            (if (nil? request) ;; (1)
              (do
                (println "Nil request, closing")
                (a/close! resp-channel)
                (.close client-socket))
              (let [parts (string/split request #" ") ;; (2)
                    command (get parts 0)]
                (cond
                  (contains? valid-commands command)
                  (let [request (request-for-command command parts resp-channel)]
                    (a/>! command-channel request)
                    (let [value (a/<! resp-channel)]
                      (.write writer (str value "\n"))
                      (.flush writer)
                      (recur resp-channel)))
                  (= command "QUIT")
                  (do
                    (a/close! resp-channel)
                    (.close client-socket))
                  :else (do
                          (println "Unknown request:" request)
                          (recur resp-channel)))))))))

There's a lot happening in there, so let's break it down. We first start a go block, to make sure we do not block the main thread where the main function waits for new clients to connect.

We then loop with a newly created channel, resp-channel. The purpose of this channel is for the handle-db function to send back a response after processing commands.

In the loop, we create two variables, request & writer, which is similar to what we did earlier in the chapter. After creating these variables, the first thing we do in (1) is check if what we read is nil, if it is the client disconnected, so we close everything, the channel and the socket, and don't call recur to effectively exit from the loop and the let the function terminate.

If we did receive something, in (2) we first create two new variables, parts which is the result of splitting the line we read on spaces, and command which is the first item from the parts vector. When we receive something like GET a or SET abc 123, this would respectively make command hold the "GET" & "SET" strings.

Next we use the cond function to first check if the command variable represent a valid command. For that we check if is contained in the valid-commands, which is defined as follows:

(def valid-commands
  "Valid commands"
  #{"GET"})

In case it is not a valid command, we have two more options, either command is equal to QUIT, in which case we do the same as what we do with nil results from clients, we close everything and don't recur. Finally, if the command is unknown, we do nothing and call recur to listen to any future commands from the connected client.

Back to the case where the command is a valid command, we use a helper function request-for-command to create the hash map we will send to command-channel:

(defn key-request
  "Helper to structure the basic parts of a command"
  [command key channel]
  {:command command :key key :resp channel})

(defn request-for-command
  "Return a structured representation of a client command"
  [command parts resp-channel]
  (cond
    (= command "GET")
    (key-request :get (get parts 1) resp-channel)))

This turns a string such as GET a into a hash map that looks like the following:

{:command :get :key "a" :resp {:ch #object["clojure..."]}}

Note that any arguments passed after a would be ignored, which is done for the sake of simplicity. A "real" server would instead validate that the command contains the correct number of arguments and returns an error if it doesn't.

We now have all the pieces to handle more commands!

SET

The SET commands requires two arguments, a key and a value. Let's first update the valid-commands constant:

(def valid-commands
  "Valid commands"
  #{"GET" "SET"})

Next we'll update the request-for-command function to return a hash map with all the details that handle-db needs:

(defn key-value-request
  "Helper to structure the various parts of a SET command"
  [command key value channel]
  (assoc (key-request command key channel) :value value))

(defn request-for-command
  "Return a structured representation of a client command"
  [command parts resp-channel]
  (cond
    (= command "GET")
    (key-request :get (get parts 1) resp-channel)
    (= command "SET")
    (key-value-request :set (get parts 1) (get parts 2) resp-channel)))

Note that we added a new helper function, key-value-request, to help with the creation of the hash map for SET commands where we need an additional argument for the value. The resulting hash map looks something like:

{:command :set :key "a" :value "123" :resp {:ch #object["clojure..."]}}

After making these changes, we now need to update process-command so that the cond function handles the case where command variable is :set:

(defn process-command
  "Perform various operations depending on the command sent by the client"
  [db command key value]
  (cond
    (= command :get)
    (if key
      (let [value (get db key "")]
        {:updated db :response value})
      {:updated db :response "ERR wrong number of arguments for 'get' command"})
    (= command :set)
    (if (and key value)
      {:updated (assoc db key value) :response "OK"}
      {:updated db :response "ERR wrong number of arguments for 'set' command"})
    :else {:updated db :response "Unknown command"}))

Under the (= command :set) branch, we first make sure we have values for key & value, if we don't we return an error. If we do have values, we use the assoc function to set the value in the DB, no matter what it was before.

Remember that because we return the result of (assoc) under the :updated field, the handle-db function will use that value in its recur call and effectively keep that value as the latest version of the DB.

DEL

We'll follow the same approach we did for SET here, we'll update valid-commands as well as request-for-command:

(def valid-commands
  "Valid commands"
  #{"GET" "SET" "DEL"})

(defn request-for-command
  "Return a structured representation of a client command"
  [command parts resp-channel]
  (cond
    (= command "GET")
    (key-request :get (get parts 1) resp-channel)
    (= command "SET")
    (key-value-request :set (get parts 1) (get parts 2) resp-channel)
    (= command "DEL")
    (key-request :del (get parts 1) resp-channel)))

The branch for "DEL" is very similar to "GET", with the exception that the :command field is :del and not :get.

Next, we need to update process-command:

(defn process-command
  "Perform various operations depending on the command sent by the client"
  [db command key value]
  (cond
    (= command :del)
    (if key
      (if (contains? db key)
        {:updated (dissoc db key) :response "1"}
        {:updated db :response "0"})
      {:updated db :response "ERR wrong number of arguments for 'del' command"})
    :else {:updated db :response "Unknown command"}))

Note that we omitted the branches for :get & :set.

Similarly to other commands, we check for the presence of key and return an error if it's absent. In the case where we do have a key, before deleting it from the DB we need to check if it exists or not. This is because we want to return "1" to the client if something was deleted and "0" otherwise. This is similar to what Redis does (docs), with the exception that Redis returns an integer, but we return a string, in order to keep things simpler.

INCR

Let's follow the same steps, update valid-commands & request-for-command

(def valid-commands
  "Valid commands"
  #{"GET" "SET" "DEL" "INCR"})

(defn request-for-command
  "Return a structured representation of a client command"
  [command parts resp-channel]
  (cond
    (= command "GET")
    (key-request :get (get parts 1) resp-channel)
    (= command "SET")
    (key-value-request :set (get parts 1) (get parts 2) resp-channel)
    (= command "INCR")
    (key-request :incr (get parts 1) resp-channel)
    (= command "DEL")
    (key-request :del (get parts 1) resp-channel)))

And finally, let's update process-command. Handling the INCR command requires a few more steps than the previous ones. If the key exists in the DB, we need to check that it represents an integer. If the value we found is "2" for instance, we should change it to "3", but if the value is "a", then it's an error. Once again, this behavior is copied from Redis.

In order to handle the String to Integer conversion, we'll use the valueOf class method from the Integer java class:

(defn atoi
  "Attempt to convert a string to integer, returns nil if it can't be parsed"
  [string]
  (try
    (Integer/valueOf string)
    (catch NumberFormatException _e
      nil)))

We make the function return nil when the value cannot be converted to an integer, which will more convenient that letting the exception blow up.

(defn process-command
  "Perform various operations depending on the command sent by the client"
  [db command key value]
  (cond
    (= command :incr)
    (if key
      (if (contains? db key)
        (let [current-number (atoi (get db key))
              new-number (when current-number (str (+ current-number 1)))]
          (if current-number
            {:updated (assoc db key new-number) :response new-number}
            {:updated db :response "ERR value is not an integer or out of range"}))
        {:updated (assoc db key "1") :response "1"})
      {:updated db :response "ERR wrong number of arguments for 'incr' command"})
    :else {:updated db :response "Unknown command"}))

Note that we omitted the branches for :get, :set & del.

If key is not present in the DB, then we store "1" and call it a day. Otherwise, we use our newly created helper atoi to get the integer value from the DB. If we get a non-nil value from atoi, then we override the value in the DB with the new value.

We do not need to worry about race conditions where two clients would send an INCR command at the same time thanks to the channel architecture we're using. The handle-db function will process commands sent to it one at a time, which effectively serializes the processing of commands.

This means that there can only be one command processed at a time, and therefore we know that the value we're dealing with when processing the INCR command cannot be changed by another client.

Atoms

Clojure has a built-in type which happens to be very convenient with what we need to do here, atoms (docs). With atoms we can completely remove the need for channels as all our go blocks can share the same variable and update it safely.

Before we update our server to use atoms, let's first take a quick look at how they work,

To work with atoms, you first create one, with the atom function:

(def database (atom {}))

You can then dereference it to get its content:

@database ;; {}

And update them with swap!, to which you pass the atom and a function to update its state:

(swap! database
       (fn [current-state]
         (assoc current-state "abc" "123")))
;; {"abc" "123"}

And we can confirm that the state was updated:

@database ;; {"abc" "123"}

Updating our server

The main function is now different since we don't need to create the command-channel variable:

(defn main
  "Start a server and continuously wait for new clients to connect"
  [& _args]
  (println "About to start ...")
  (let [db (atom {})]
    (with-open [server-socket (ServerSocket. 3000)]
      (loop []
        (let [client-socket (.accept server-socket)]
          (handle-client client-socket db))
        (recur)))))

Aside from not needing channels anymore, the other important difference is that we create the db variable with (atom {}). We also don't need the handle-db function anymore.

Let's then look at handle-client, now that it doesn't need a channel:

(defn handle-client
  "Read from a connected client, and handles the various commands accepted by the server"
  [client-socket db]
  (a/go (loop [] ;; (1)
          (let [request (.readLine (io/reader client-socket))
                writer (io/writer client-socket)]
            (if (nil? request)
              (do
                (println "Nil request, closing")
                (.close client-socket))
              (let [parts (string/split request #" ")
                    command (get parts 0)]
                (cond
                  (contains? valid-commands command)
                  (let [request (request-for-command command parts) ;; (2)
                        value (process-command db request)] ;; (3)
                    (.write writer (str value "\n"))
                    (.flush writer)
                    (recur))
                  (= command "QUIT")
                  (.close client-socket)
                  :else (do
                          (println "Unknown request:" request)
                          (recur)))))))))

There are three difference that are highlighted:

In (1) we don't need to have any variables in the loop anymore
In (2) we don't need to pass a channel to request-for-command anymore
In (3) we now call a new function, process-command instead of sending the request to a channel

The process-command function is where a lot of the interesting changes are, but before looking at it, let's briefly look at the request-for-command and the other helpers. They're essentially identical to the previous versions, with the absence of channels:

(defn key-request
  "Helper to structure the basic parts of a command"
  [command key]
  {:command command :key key})

(defn key-value-request
  "Helper to structure the various parts of a SET command"
  [command key value]
  (assoc (key-request command key) :value value))

(defn request-for-command
  "Return a structured representation of a client command"
  [command parts]
  (cond
    (= command "GET")
    (key-request :get (get parts 1))
    (= command "SET")
    (key-value-request :set (get parts 1) (get parts 2))
    (= command "INCR")
    (key-request :incr (get parts 1))
    (= command "DEL")
    (key-request :del (get parts 1))))

And now, let's look at process-command:

(defn process-command
  "Perform various operations depending on the command sent by the client"
  [db request]
  (let [command (request :command)
        key (request :key)
        value (request :value)]
    (cond
      (= command :get)
      (if key
        (get @db key "")
        "ERR wrong number of arguments for 'get' command")
      (= command :set)
      (if (and key value)
        (do
          (swap! db (fn [current-state]
                      (assoc current-state key value)))
          "OK")
        "ERR wrong number of arguments for 'set' command")
      (= command :del)
      (if key
        (let [[old-value _] (swap-vals! db (fn [current-state]
                                             (dissoc current-state key)))]
          (if (contains? old-value key) "1" "0"))
        "ERR wrong number of arguments for 'del' command")
      (= command :incr)
      (if key
        (let [[_ new-value] (increment-number db key)
              number (atoi (get new-value key))]
          (if number
            number
            "ERR value is not an integer or out of range"))
        "ERR wrong number of arguments for 'incr' command")
      :else "Unknown command")))

For the :get case, we use the get function and deref to read the value.

For :set we use swap! to override whatever is in the DB.

For :del things are starting to get a little trickier. This is because swap! returns the atom's new state, but in order to decide whether we need to return "0" or "1" we need to know if key was in db before the deletion.

We could have called (if (contains? @db key)) before calling swap! but we would have been subject to race conditions. This is because two go blocks could have called that, both see the value present, and therefore both deciding to return "1" whereas in reality only one of the two would have actually deleted it.

In order to prevent this issue, we use swap-vars!, which does the same as swap! with the only difference that it only returns the atom's state before the update. So we use swap-vars!, ignoring the new state, and checking if key was in the DB before the update.

Because the logic in the (=command :incr) branch was getting complicated, it was extracted to a separate function:

(defn increment-number
  "Wrap the lower level operations required to process an increment command"
  [db key]
  (swap-vals! db (fn [current-state]
                   (if (contains? current-state key)
                     (let [number (atoi (get current-state key))]
                       (if number
                         (assoc current-state key (str (+ number 1)))
                         current-state))
                     (assoc current-state key "1")))))

The logic is similar to the channel version, but with swap!. In increment-number, if the value we find in the DB cannot be converted to an integer, we left it untouched. This allows us to check the value after calling increment-number back in process-command, and check if we find an integer under key, if we don't we know that there was a value that cannot be converted an integer, and we return tha appropriate error message

Conclusion

It could be interesting to compare the performance of the two implementations, the one with channels and the one with atoms. I'd guess that the atoms one is more performant, because of the lack of overhead from channels, but there might other factors at play.

Additionally, I think we could improve the performance of the atoms-based version by making db a "regular" hash map where every value is an atom. This is because when we call swap! on an atom, clojure will retry the operation if the value was changed while the function was running, this means that with many clients, we might have many retries as db gets updated by multiple clients.
With a "per-value" atom, we'd only have to retry if two clients are operating on the same key at the same time.

A toy Redis Server, in Go

Pierre Jambet — Mon, 02 Jan 2023 14:45:28 +0000

What we're building

By the end of this post we will have a concurrent TCP server written in Go, implementing a small subset of Redis commands. We'll focus on the following features:

It will be reachable over TCP
It can handle concurrent clients. That is, it can accept connections from multiple clients and responds to them regardless of the order in which they connect and send requests. For instance, client C1 connects, client C2 connects, C2 can send a request and get a response no matter what C1 is doing, whether staying idle, disconnecting or sending requests as well.
The following will be implemented:
- GET: Accepts a string, and return the value stored for that key, if any
- SET: Accepts two strings, a key and a value, and sets the value for the key, overriding any values that may have been present
- DEL: Accepts a string and deletes the value that may have been there
- INCR: Accepts a single argument and increments the existing value. If the value is not an integer, it's an error, if there are no values, it gets initialized to 1, resulting in an identical outcome as calling SET <key> 1.

Another way to look at it is that we're building a hash map accessible over TCP.

Oh, and, finally, the last constraint, we're only using the standard library, nothing external.

I've never written Go code professionally, most of what you're seeing here is a lot of trial and error combined with finding code on the Internet.

The TCP part

We're going to build our server step by step, let's start with the "TCP server" part.

Reading command arguments

In the main function we start by getting the port number from the command arguments:


package main

import (
    "fmt"
    "os"
)

func main() {
    arguments := os.Args
    if len(arguments) == 1 {
        fmt.Println("Please provide a port number!")
        return
    }
    // [...]

We then use it to start a local server listening to that port over tcp. We use the defer keyword to make sure we close the socket that net.Listen opens.

// [...]
import (
    "fmt"
    "net"
    "os"
)
// [...]
func main() {
    // [...]
    port := ":" + arguments[1]
    server, err := net.Listen("tcp4", port)
    if err != nil {
        fmt.Println(err)
        return
    }
    defer server.Close()
    // [...]

We then start an infinite loop where we call the blocking function Accept, and for every client it returns, we run the handleConnection function in its own goroutine.

The built-in goroutine mechanism gives us a lot to achieve the concurrency goal stated earlier. The main goroutine is purposefully stuck in an infinite loop. It waits for the next client to connect and when it does, gives it its own goroutine, where it is handled without interfering with the main goroutine where new clients may or may not connect later on.

// [...]
func main() {
    // [...]
    for {
        client, err := server.Accept()
        if err != nil {
            fmt.Println(err)
            return
        }
        go handleConnection(client)
    }
}

The handleConnection function operates in a fairly similar manner. It also starts an infinite loop where it uses the bufio package to read lines of text sent by the client. The function will block until the client sends something, but because we're running this function in its goroutine, it doesn't interfere with any other clients, running in their own goroutines.

// [...]
import (
    "bufio"
    "fmt"
    "net"
    "os"
    "strings"
)

func handleConnection(client net.Conn) {
    defer client.Close()

    fmt.Printf("Serving %s\n", client.RemoteAddr().String())

    for {
        netData, err := bufio.NewReader(client).ReadString('\n')
        if err != nil {
            fmt.Println("error reading:", err)
            break
        }

        temp := strings.TrimSpace(netData)
        fmt.Println("Received:", temp)
        client.Write([]byte(temp + "\n"))
    }
}
// [...]

We're heavily leaning into the go runtime, which takes care of running the goroutines. As long as we're careful to run blocking code in ways that doesn't interfere with other parts of the server, we don't have anything else to do to handle any number of clients.

"Any number of clients" is a stretch, there is technically a limit. Handling clients is not free, it uses resources, some memory is allocated for each new client and a file descriptor is used. The specificities of the whole process vary from one OS to the next, but there is a limit of file descriptors a process can keep open, so our server cannot handle an "unlimited" number of clients.

That being said, the Go runtime is famous for being able to handle a lot of goroutines. This StackOverflow thread shows a small benchmark where 100,000 goroutines are running at the same time with a very small memory footprint.

We now have a server that accepts TCP connections, echoes back to the client what it received and keeps doing that until the client disconnects.

It's mainly copied from this blog post by Mihalis Tsoukalos, with some modifications. Here is the full version:

package main

import (
    "bufio"
    "fmt"
    "net"
    "os"
    "strings"
)

func handleConnection(client net.Conn) {
    fmt.Printf("Serving %s\n", client.RemoteAddr().String())

    for {
        netData, err := bufio.NewReader(client).ReadString('\n')
        if err != nil {
            fmt.Println("error reading:", err)
            break
        }

        temp := strings.TrimSpace(netData)
        fmt.Println("Received:", temp)
        client.Write([]byte(temp + "\n"))
    }

    fmt.Println("Closing client")
    client.Close()
}

func main() {
    arguments := os.Args
    if len(arguments) == 1 {
        fmt.Println("Please provide a port number!")
        return
    }

    port := ":" + arguments[1]
    server, err := net.Listen("tcp4", port)
    if err != nil {
        fmt.Println(err)
        return
    }
    defer server.Close()

    for {
        client, err := server.Accept()
        if err != nil {
            fmt.Println(err)
            return
        }
        go handleConnection(client)
    }
}

We can run this server with go run server.go 3000 (or any other available port), and connect to it from another terminal with nc -v localhost 3000.

In order to understand the architecture of this approach and how we'll improve it below, it's important to categorize all the goroutines started by the server in two groups. The first one has a single goroutine, started implicitly in the main function, we'll refer to it as the main coroutine, and the other group is for all the goroutines started for all the connected clients, we'll refer to them as the client-specific coroutines.

Handling commands, a first attempt

This version is subject to race conditions. Jump to the next section for a version protected from race conditions.

Let's improve the handleConnection function to do something depending on the commands we receive from clients.

So far the function reads from the client connection, prints the content to STDOUT in the server process, and waits for the next line of text.

We now create a map[string]string in the main function, to act as our main database. We pass the map to each new goroutine, so that it can either read from it for GET commands, write to it for SET & INCR commands and delete from it for DEL commands.

// [...]
func main() {
    // [...]
    db := make(map[string]string)

    for {
        // [...]
        go handleConnection(db, client)
    }
}

We now need to accept the map argument in handleConnection and use it to implement each command. The next step is to split the string we received, and treat the first element as the command. If it is a known command, we will handle it, otherwise, we will return a generic error: "ERR unknown command".

For the QUIT or STOP command, we call return, which exits the for loop and implicitly calls client.Close() through the defer mechanism, effectively disconnecting the client.

// [...]
import (
    "bufio"
    "fmt"
    "net"
    "os"
    "strconv"
    "strings"
)

func handleConnection(db map[string]string, client net.Conn) {
    defer client.Close()

    for {
        netData, err := bufio.NewReader(client).ReadString('\n')
        if err != nil {
            fmt.Println("error reading:", err)
            return
        }

        var response string
        commandString := strings.TrimSpace(netData)
        parts := strings.Split(commandString, " ")
        command := parts[0]

        switch command {
        case "STOP", "QUIT":
            return
        case "GET":
            // [...]
        case "SET":
            // [...]
        case "INCR":
            // [...]
        case "DEL":
            // [...]
        default:
            response = "ERR unknown command"
        }

        client.Write([]byte(response + "\n"))
    }
}

For each branch of the switch statement we implement the command-specific behavior through the various operations available for the map type.

GET

We return the value stored at key, with db[key]:

if len(parts) > 1 {
    key := parts[1]

    response = db[key]
} else {
    response = "ERR wrong number of arguments for 'get' command"
}

SET

We either set the value at key with value, or replace what was there before with db[key] = value

if len(parts) > 2 {
    key := parts[1]
    value := parts[2]

    db[key] = value
    response = "OK"
} else {
    response = "ERR wrong number of arguments for 'set' command"
}

INCR

We first check for the presence of the key in the map with value, ok = db[key]. If the value is present, we attempt to convert the string in the map to an int with strconv.Atoi. If that works, we increment the int and put in the back in the map as a string with strconv.Itoa. If the string cannot be converted to an int, such as "a" for instance, we do nothing and return an error string.

if len(parts) > 1 {
    key := parts[1]
    value, ok := db[key]

    if ok {
        intValue, err := strconv.Atoi(value)
        if err != nil {
            response = "ERR value is not an integer or out of range"
        } else {
            response = strconv.Itoa(intValue + 1)
            db[key] = response
        }
    } else {
        response = "1"
        db[key] = response
    }
} else {
    response = "ERR wrong number of arguments for 'incr' command"
}

DEL

We either delete the value with delete(db, key) or do nothing if the key is not present in the map

if len(parts) > 1 {
    key := parts[1]
    _, ok := db[key]

    if ok {
        delete(db, key)
        response = "1"
    } else {
        response = "0"
    }
} else {
    response = "ERR wrong number of arguments for 'del' command"
}

The full version

package main

import (
    "bufio"
    "fmt"
    "net"
    "os"
    "strconv"
    "strings"
)

func handleConnection(db map[string]string, client net.Conn) {
    defer client.Close()

    for {
        netData, err := bufio.NewReader(client).ReadString('\n')
        if err != nil {
            fmt.Println("error reading:", err)
            return
        }

        var response string
        commandString := strings.TrimSpace(netData)
        parts := strings.Split(commandString, " ")
        command := parts[0]

        switch command {
        case "STOP", "QUIT":
            return
        case "GET":
            if len(parts) > 1 {
                key := parts[1]

                response = db[key]
            } else {
                response = "ERR wrong number of arguments for 'get' command"
            }
        case "SET":
            if len(parts) > 2 {
                key := parts[1]
                value := parts[2]

                db[key] = value
                response = "OK"
            } else {
                response = "ERR wrong number of arguments for 'set' command"
            }
        case "INCR":
            if len(parts) > 1 {
                key := parts[1]
                value, ok := db[key]

                if ok {
                    intValue, err := strconv.Atoi(value)
                    if err != nil {
                        response = "ERR value is not an integer or out of range"
                    } else {
                        response = strconv.Itoa(intValue + 1)
                        db[key] = response
                    }
                } else {
                    response = "1"
                    db[key] = response
                }
            } else {
                response = "ERR wrong number of arguments for 'incr' command"
            }
        case "DEL":
            if len(parts) > 1 {
                key := parts[1]
                _, ok := db[key]

                if ok {
                    delete(db, key)
                    response = "1"
                } else {
                    response = "0"
                }
            } else {
                response = "ERR wrong number of arguments for 'del' command"
            }
        default:
            response = "ERR unknown command"
        }

        client.Write([]byte(response + "\n"))
    }
}

func main() {
    arguments := os.Args
    if len(arguments) == 1 {
        fmt.Println("Please provide a port number!")
        return
    }

    PORT := ":" + arguments[1]
    server, err := net.Listen("tcp4", PORT)
    if err != nil {
        fmt.Println(err)
        return
    }
    defer server.Close()

    db := make(map[string]string)

    for {
        client, err := server.Accept()
        if err != nil {
            fmt.Println(err)
            return
        }
        go handleConnection(db, client)
    }
}

All the client coroutines use a reference to the same map. This is fine for a command such as GET, where it doesn't really matter if multiple clients run the same command. But for a command like INCR, things can go wrong. The following diagram illustrates the issue:

        ┌───────────────────┐ ┌───────────────────┐
        │                   │ │                   │
        │   C1 coroutine    │ │   C2 coroutine    │
        │                   │ │                   │
        └───────────────────┘ └───────────────────┘
                  │                     │
                  │     ┌────────┐      │
                  │     │db = {  │      │
                  │     │  key: 1│      │
                  │     │}       │      │
┌────────────────┐│     └────────┘      │
│v = db[key] // 1││─────────────────────│
└────────────────┘│                     │┌─────────────────┐
                  │─────────────────────││v = db[key] // 1 │
     ┌───────────┐│                     │└─────────────────┘
     │v += 1 // 2││─────────────────────│
     └───────────┘│                     │┌───────────┐
                  │─────────────────────││v += 1 // 2│
     ┌───────────┐│                     │└───────────┘
     │db[key] = v││─────────────────────│
     └───────────┘│                     │┌───────────┐
                  │─────────────────────││db[key] = v│
                  │     ┌────────┐      │└───────────┘
                  │     │db = {  │      │
                  │     │  key: 2│      │
                  │     │}       │      │
                  ▼     └────────┘      ▼

We start with a map with a single key, where the value is 1. Two clients each send an INCR command for the same key, the only valid outcome for this sequence of operations if for the final value to be 3.

But with our implementation it is entirely possible for it to be 2. The likelihood of this exact problem to occur is rare in the sense that the read operation from the map is immediately followed by the conversion to an int, the increment and the update in the map. But since we're running the code inside goroutines, we have zero control over the sequence of operations. We're at the mercy of the go runtime as well as the underlying OS.

We can illustrate this issue with the following ruby script. First start the server with go server.go 3000 and separately open a ruby shell, with irb -rsocket since we're going to need what is under socket. We create an array with 100 sockets connected to our go server:

sockets = 1.upto(100).map { TCPSocket.new("localhost", 3000) }

Let's then spin up one thread per socket and make it call INCR a:

sockets.map { |s| Thread.new { s.puts("INCR a"); puts s.gets } }

We call gets to read the response from the server

For good measure, let's close all these sockets:

sockets.each(&:close)

Results will vary from one machine to the next, as well as from one run to the next, but on my macbook air, I often see the last value printed being 98 or 99, instead of the expected 100.

In some cases the server might crash with the error:

fatal error: concurrent map writes

This is an even more explicit proof that "we're doing it wrong". As it turns out, go knows that weird things can happen when there are concurrent writes on a single map, and it rejects it all together with a fatal error.

Let's fix this.

A race-condition-free version

We are going to introduce a new goroutine to handle all the operations on the map in a way that makes concurrent writes impossible. In order to achieve this we will need to use channels for our goroutines to communicate with each other. We will refer to this new coroutine as the DB coroutine.

First, we create a new type, Command, to act as en enum-like list of all the commands supported by the server.

const (
    Get  Command = iota + 1 // 1
    Set                     // 2
    Incr                    // 3
    Del                     // 4
)
// [...]

We also create a new struct type, CommandMessage. The first field will be of type Command, which identifies the command being handled. The next two string fields identify the data of the command, we always have a key, and we a have value in the case of a SET command. It was simpler to always have the field, regardless of the command, and have its value populated only if it's a SET command.

The last field, responseChannel, is a channel that will allow the DB coroutine to respond back to the client coroutine that sent the CommandMessage in the first place. Channels are bidirectional, when we create the channel in the client goroutine, no other goroutine can write to it. By including it in the message we send to commandChannel, the DB coroutine can write to it, and we can get retrieve that content from the client goroutine:

type commandMessage struct {
    commandName     Command
    key             string
    value           string
    responseChannel chan string
}

Next, we create a channel for commandMessage values in the main function, and pass it to every client goroutine as well as to the DB coroutine. The client coroutines will write to it and the DB coroutine will read from it.

// [...]
func handleDB(commandChannel chan commandMessage) {
    // [...]
}

func main() {
    // [...]
    commandChannel := make(chan commandMessage)

    go handleDB(commandChannel)

    for {
        client, err := server.Accept()
        if err != nil {
            fmt.Println(err)
            return
        }
        go handleConnection(commandChannel, client)
    }
}

The handleConnection needs to be updated to accept the channel instead of the map. It also now needs to create the commandMessage values and write them to the channel, instead of directly performing the operations for the different commands. We write to the channel with commandChannel <- commandMessage and we then wait for the DB coroutine to compute the result with response = <-commandMessage.responseChannel. In other words, we create a channel, give it to the DB goroutine, and wait for that channel to be written to.

func handleConnection(commandChannel chan commandMessage, client net.Conn) {
    // [...]
    for {
        // [...]
        switch command {
        case "STOP", "QUIT":
            return
        case "GET":
            if len(parts) > 1 {
                key := parts[1]
                commandMessage := commandMessage{
                    commandName:     Get,
                    key:             key,
                    responseChannel: make(chan string)}

                commandChannel <- commandMessage
                response = <-commandMessage.responseChannel
            } else {
                response = "ERR wrong number of arguments for 'get' command"
            }
        case "SET":
            if len(parts) > 2 {
                key := parts[1]
                value := parts[2]
                commandMessage := commandMessage{
                    commandName:     Set,
                    key:             key,
                    value:           value,
                    responseChannel: make(chan string)}

                commandChannel <- commandMessage
                response = <-commandMessage.responseChannel
            } else {
                response = "ERR wrong number of arguments for 'set' command"
            }
        case "INCR":
            if len(parts) > 1 {
                key := parts[1]
                commandMessage := commandMessage{
                    commandName:     Incr,
                    key:             key,
                    responseChannel: make(chan string)}

                commandChannel <- commandMessage
                response = <-commandMessage.responseChannel
            } else {
                response = "ERR wrong number of arguments for 'incr' command"
            }
        case "DEL":
            key := parts[1]
            commandMessage := commandMessage{
                commandName:     Del,
                key:             key,
                responseChannel: make(chan string)}

            commandChannel <- commandMessage
            response = <-commandMessage.responseChannel
        default:
            response = "ERR unknown command"
        }
        // [...]
    }
}

The final step is to move the logic that used to live in handleConnection to handleDB. The biggest change is that we need to use channels to communicate with the client coroutines. We first read from commandChannel with: command := <-commandChannel and based on the command we receive, run the appropriate branch of the switch statement.

Once we have a result, regardless of the command, we put the response string on the responseChannel field of the CommandChannel instance we received, so that the client goroutine can in turn read the value, and write it back to client.

func handleDB(commandChannel chan commandMessage) {
    db := make(map[string]string)

    for {
        select {
        case command := <-commandChannel:
            switch command.commandName {
            case Get:
                command.responseChannel <- db[command.key]
            case Set:
                db[command.key] = command.value
                command.responseChannel <- "OK"
            case Incr:
                value, ok := db[command.key]
                var response string

                if ok {
                    intValue, err := strconv.Atoi(value)
                    if err != nil {
                        response = "ERR value is not an integer or out of range"
                    } else {
                        response = strconv.Itoa(intValue + 1)
                        db[command.key] = response
                    }
                } else {
                    response = "1"
                    db[command.key] = response
                }
                command.responseChannel <- response
            case Del:
                _, ok := db[command.key]
                var response string

                if ok {
                    delete(db, command.key)
                    response = "1"
                } else {
                    response = "0"
                }
                command.responseChannel <- response
            }
        }
    }
}

With this new pattern, instead of having each client goroutine perform the sequence of operations specific to each command, they instead send a message to the DB coroutine with the data necessary to perform the operation, and wait for a response. This approach protects us from race conditions because there is a single goroutine processing the messages written to commandChannel. Sending message to commandChannel results in them being added to the channel's buffer, and in turn the DB coroutine will process these messages one by one through the select statement.

You can run the ruby script we used above and notice that the final outcome will always be 100, and the go server will never crash with fatal error: concurrent map writes 🎉

The full version

This version is inspired from this example from Go by Example:

package main

import (
    "bufio"
    "fmt"
    "net"
    "os"
    "strconv"
    "strings"
)

type Command int

const (
    Get  Command = iota + 1 // 1
    Set                     // 2
    Incr                    // 3
    Del                     // 4
)

type commandMessage struct {
    commandName     Command
    key             string
    value           string
    responseChannel chan string
}

func handleConnection(commandChannel chan commandMessage, client net.Conn) {
    defer client.Close()

    for {
        netData, err := bufio.NewReader(client).ReadString('\n')
        if err != nil {
            fmt.Println("error reading:", err)
            return
        }

        var response string
        commandString := strings.TrimSpace(netData)
        parts := strings.Split(commandString, " ")
        command := parts[0]

        switch command {
        case "STOP", "QUIT":
            return
        case "GET":
            if len(parts) > 1 {
                key := parts[1]
                commandMessage := commandMessage{
                    commandName:     Get,
                    key:             key,
                    responseChannel: make(chan string)}

                commandChannel <- commandMessage
                response = <-commandMessage.responseChannel
            } else {
                response = "ERR wrong number of arguments for 'get' command"
            }
        case "SET":
            if len(parts) > 2 {
                key := parts[1]
                value := parts[2]
                commandMessage := commandMessage{
                    commandName:     Set,
                    key:             key,
                    value:           value,
                    responseChannel: make(chan string)}

                commandChannel <- commandMessage
                response = <-commandMessage.responseChannel
            } else {
                response = "ERR wrong number of arguments for 'set' command"
            }
        case "INCR":
            if len(parts) > 1 {
                key := parts[1]
                commandMessage := commandMessage{
                    commandName:     Incr,
                    key:             key,
                    responseChannel: make(chan string)}

                commandChannel <- commandMessage
                response = <-commandMessage.responseChannel
            } else {
                response = "ERR wrong number of arguments for 'incr' command"
            }
        case "DEL":
            key := parts[1]
            commandMessage := commandMessage{
                commandName:     Del,
                key:             key,
                responseChannel: make(chan string)}

            commandChannel <- commandMessage
            response = <-commandMessage.responseChannel
        default:
            response = "ERR unknown command"
        }

        client.Write([]byte(response + "\n"))
    }
}

func handleDB(commandChannel chan commandMessage) {
    db := make(map[string]string)

    for {
        select {
        case command := <-commandChannel:
            switch command.commandName {
            case Get:
                command.responseChannel <- db[command.key]
            case Set:
                db[command.key] = command.value
                command.responseChannel <- "OK"
            case Incr:
                value, ok := db[command.key]
                var response string

                if ok {
                    intValue, err := strconv.Atoi(value)
                    if err != nil {
                        response = "ERR value is not an integer or out of range"
                    } else {
                        response = strconv.Itoa(intValue + 1)
                        db[command.key] = response
                    }
                } else {
                    response = "1"
                    db[command.key] = response
                }
                command.responseChannel <- response
            case Del:
                _, ok := db[command.key]
                var response string

                if ok {
                    delete(db, command.key)
                    response = "1"
                } else {
                    response = "0"
                }
                command.responseChannel <- response
            }
        }
    }
}

func main() {
    arguments := os.Args
    if len(arguments) == 1 {
        fmt.Println("Please provide a port number!")
        return
    }

    PORT := ":" + arguments[1]
    server, err := net.Listen("tcp4", PORT)
    if err != nil {
        fmt.Println(err)
        return
    }
    defer server.Close()

    commandChannel := make(chan commandMessage)

    go handleDB(commandChannel)

    for {
        client, err := server.Accept()
        if err != nil {
            fmt.Println(err)
            return
        }
        go handleConnection(commandChannel, client)
    }
}

Want more?

I started a repo where I'm trying to do the same thing in various languages, go check it out if for instance you're curious about how to do this with node, python, ruby, clojure, java, kotlin or rust: pjambet/tcp-servers

The code from this article is on GitHub.

Links:

Thanks to Vikram Oberoi & Brian Cobb for reviewing early versions of this article

Active Record caches generated queries for find_by queries

Pierre Jambet — Sun, 12 Dec 2021 05:15:07 +0000

tl;dr; Active Record caches generated queries, when using find_by and find. This is a different type of cache, unrelated to QueryCache and prepared statements.

What exactly does Active Record cache, and when?

When you call Post.find(1) or Post.find_by(id: 1), Active Record will cache the result of build_arel in a StatementCache instance. The build_arel method is the one that generates the actual underlying query, in this case, SELECT * FROM posts WHERE id = ?.

Why does it actually matter?

This is a small internal optimization that should be invisible in most cases. The generated queries are parameterized, so once they are cached, they can be reused regardless of the arguments. It should speed things up and all is well.

I had an issue when hooking into Active Record’s internals, using a custom module included into ActiveRecord::Relation:

module M
  def build_arel(aliases=nil)
    super.tap do |arel|
      # look into the generated query
    end
  end
end

ActiveSupport.on_load :active_record do
  ActiveRecord::Relation.include M
end

I was expecting my custom code inside #tap to always be executed, but since Active Record caches the result of build_arel, the first call would go through build_arel and subsequent calls to find would use the cached result and never hit the #tap method.

A quick workaround I found was to add arel.model.intialize_find_by_cache if Rails.env.test? at the end of my #tap block, to make sure the cache would get reset and therefore nothing would actually end up cached. In my case, this was a test only thing, and resetting the cache in this context was not an issue.

Final words

I used to be scared of digging into Rails’ code, afraid it would be “too complicated”. But it’s actually not that bad! I fully recommend checking it out, if you don’t know where to start, let me recommend the find method.

Select Syscall in Rust

Pierre Jambet — Mon, 23 Nov 2020 17:50:58 +0000

Disclaimer

I am not a Rust expert, I am just getting started, so please take everything you read here with a grain of salt.
You probably don't want to use any of this in production code. There are libraries written by actual Rust developers providing similar features, in a way that will most certainly be more efficient and more reusable, such as Tokio and nix. Additionally, select is rarely use these days, kqueue and epoll tend to be preferred.

Good? Let's get to it now.

The `select` syscall

The select syscall is really useful to write systems using an event loop. A good example is a database running as a TCP server. A database would want to maintain client connections open, and respond to them when then send queries to the server, and select can help with that.

select works by accepting four arguments, a list of file descriptors that are ready to be read from, a list of file descriptors that are ready to be written to, a list of file descriptors that have an exceptional condition pending and finally, an optional timeout. We give these file descriptors to select and it'll return as soon as one or more file descriptors are ready.

A file descriptor, often abbreviated to fd is the identifier of a file or other input/output resource, such as a pipe or network socket. It is based on the idea that, in UNIX, everything is a file. In practical terms file descriptors are non-negative integers.

In the database example, we would give file descriptors for the connected clients, based on the sockets we created when they connected, and select would wait until one of them is ready. Essentially, as soon as the TCP server receives something from one of the clients, the server would be able to respond.

If you're looking for real world example, Redis does exactly this if configured to use select, in ae_select.c. As noted earlier, it prefers other polling mechanisms, such as kqueue and epoll and select is only a last resort option.

pselect is very similar to select, with the difference that it accepts a timespec instead of a timeval and accepts a mask argument to have better control over signal handling.

Using the `libc` crate

Rust itself does not provide a way to call select or pselect directly from the standard library, but the Rust developers created the libc library that provides bindings to these, and more.

The nix crate mentioned at the beginning of this post uses the libc crate under the hood. This is how it provides bindings to the select syscall: https://github.com/nix-rust/nix/blob/master/src/sys/select.rs

Show me some code

The following is an example of using select to be notified when a TCP server sent something back to a connected client. For this example I was using a TCP server running in Ruby with the following code running in irb:

irb(main):001:0> require 'socket'
=> true
irb(main):002:0> server = TCPServer.new 'localhost', 2000
irb(main):003:0> client1 = server.accept; client2 = server.accept; client3 = server.accept

The last line does not return until three clients connect, which is what the following Rust code does:

extern crate libc;

use std::net::TcpStream;
use std::os::unix::io::{AsRawFd, RawFd};
use std::{io, mem, ptr, time};

pub struct FdSet(libc::fd_set);

impl FdSet {
    pub fn new() -> FdSet {
        unsafe {
            let mut raw_fd_set = mem::MaybeUninit::<libc::fd_set>::uninit();
            libc::FD_ZERO(raw_fd_set.as_mut_ptr());
            FdSet(raw_fd_set.assume_init())
        }
    }
    pub fn clear(&mut self, fd: RawFd) {
        unsafe { libc::FD_CLR(fd, &mut self.0) }
    }
    pub fn set(&mut self, fd: RawFd) {
        unsafe { libc::FD_SET(fd, &mut self.0) }
    }
    pub fn is_set(&mut self, fd: RawFd) -> bool {
        unsafe { libc::FD_ISSET(fd, &mut self.0) }
    }
}

fn to_fdset_ptr(opt: Option<&mut FdSet>) -> *mut libc::fd_set {
    match opt {
        None => ptr::null_mut(),
        Some(&mut FdSet(ref mut raw_fd_set)) => raw_fd_set,
    }
}
fn to_ptr<T>(opt: Option<&T>) -> *const T {
    match opt {
        None => ptr::null::<T>(),
        Some(p) => p,
    }
}

pub fn select(
    nfds: libc::c_int,
    readfds: Option<&mut FdSet>,
    writefds: Option<&mut FdSet>,
    errorfds: Option<&mut FdSet>,
    timeout: Option<&libc::timeval>,
) -> io::Result<usize> {
    match unsafe {
        libc::select(
            nfds,
            to_fdset_ptr(readfds),
            to_fdset_ptr(writefds),
            to_fdset_ptr(errorfds),
            to_ptr::<libc::timeval>(timeout) as *mut libc::timeval,
        )
    } {
        -1 => Err(io::Error::last_os_error()),
        res => Ok(res as usize),
    }
}

pub fn make_timeval(duration: time::Duration) -> libc::timeval {
    libc::timeval {
        tv_sec: duration.as_secs() as i64,
        tv_usec: duration.subsec_micros() as i32,
    }
}

pub fn connect_to_localhost_2000() -> TcpStream {
    TcpStream::connect("localhost:2000").expect("Failed to connect to localhost 2000")
}

fn main() {
    let mut fd_set = FdSet::new();

    let stream1 = connect_to_localhost_2000();
    let raw_fd1 = stream1.as_raw_fd();

    let stream2 = connect_to_localhost_2000();
    let raw_fd2 = stream2.as_raw_fd();

    let stream3 = connect_to_localhost_2000();
    let raw_fd3 = stream3.as_raw_fd();

    // let raw_fd2 = connect_to_localhost_2000().as_raw_fd(); DOES NOT WORK

    let max_fd = raw_fd1.max(raw_fd2.max(raw_fd3));

    println!("Socket 1: {}", raw_fd1);
    println!("Socket 2: {}", raw_fd2);
    println!("Socket 3: {}", raw_fd3);

    fd_set.set(raw_fd1);
    fd_set.set(raw_fd2);
    fd_set.set(raw_fd3);

    match select(
        max_fd + 1,
        Some(&mut fd_set),                               // read
        None,                                            // write
        None,                                            // error
        Some(&make_timeval(time::Duration::new(10, 0))), // timeout
    ) {
        Ok(res) => {
            println!("select result: {}", res);

            let range = std::ops::Range {
                start: 0,
                end: max_fd + 1,
            };
            for i in range {
                if (fd_set).is_set(i) {
                    println!("Socket {} received something!", i);
                }
            }
        }
        Err(err) => {
            println!("Failed to select: {:?}", err);
        }
    }
}

Let's break it down, first we declare a Rust struct, FdSet, which wraps libc::fd_set. An fd_set is not the same on every platform, but based on the libc source, we can see that it is defined as an array of integers:

pub struct fd_set {
    #[cfg(all(target_pointer_width = "64",
              any(target_os = "freebsd", target_os = "dragonfly")))]
    fds_bits: [i64; FD_SETSIZE / 64],
    #[cfg(not(all(target_pointer_width = "64",
                  any(target_os = "freebsd", target_os = "dragonfly"))))]
    fds_bits: [i32; FD_SETSIZE / 32],
}

We can use the macros that come with select to interact with it without having to worry too much about the underlying implementation details, FD_ZERO, FD_CLR, FD_SET and FD_ISSET

The next block in impl FdSet provides Rust functions for the FdSet type to use these macros. In the new function we create a new instance of FdSet, while using MaybeUninit to prevent Rust to do what it does for regular variables and that it essentially should trust us here, we know what we're doing.

FD_ZERO is used to make sure that the integers that were allocated are in a clean state, technically speaking the OS does not have to clear the bits that were allocated, so we do it, just in case.

The next two functions, to_fdset_ptr and to_ptr are helper functions to convert some Rust-y things such as Option values into C things, like a null pointer.

Next is the actual binding to libc::select, where we accept all the values we want to pass, as idiomatic Rust values, that is some Option wrapped FdSet values, instead of explicit null values.

The return value is also translated from the C tradition of returning -1 if something went wrong to a Result type, which allows us to use pattern matching when dealing with the return value of select.

The timeout passed to select is a timeval, which is a bit verbose to write, so we also create a helper function to instantiate one based on a Rust Duration value.

You can read more about the details of select in the man page, with man 2 select, which is also available online for linux, and macOS.

We will connect three clients to the server so we created a small helper function to do that for us, connect_to_localhost_2000. We use the as_raw_fd function, from the TcpStream type, which returns a RawFd, which an alias for an integer, specifically c_int, which itself is an alias for i32, a 32-bit signed integer, for most platforms.

And now, the main method, we start by creating an FdSet, this will be the one and only fd_set we'll give to select since we don't care about writable sockets neither do we care about the ones with exception pending in this example.

We then connect three sockets using connect_to_localhost_2000.

Note that while it might look tempting to write the following if you come from a different language:

let raw_fd = connect_to_localhost_2000().as_raw_fd(); DOES NOT WORK

This will not work due to how Rust automatically releases variables that are not needed anymore. In this case the TcpStream variable returned by connect_to_localhost_2000 is used to call as_raw_fd() but is not needed anymore after that, and Rust will release it. The impact is that Rust knows that the release process for this variable involves closing the socket, which we absolutely do not want here. We need the socket to stay open until the end of the main function, so that select can use it. One way of doing this is to explicitly create a variable for the stream, which will force it to stay in scope for the rest of the function.

Back to main, we create the max_fd variable and set it to the max value of the three raw sockets. This is necessary because the first argument to select must be the the value of the file descriptors given, plus one. Most of the time the file descriptors are incremented, and while running this, my machine was consistently creating these three sockets as 5, 6 and 7, but this is not something that we should rely on, and explicitly grabbing the max value is more reliable.

We then need to prepare the fd_set variable for select, and this is what FD_SET is for, which we use through the fd_set function. It will take care of setting the correct bits inside the fd_set array to store the information of the file descriptors.

We can then call select, with max_fd + 1, as mentioned above. If we have passed min fd value plus one instead, only that socket would have been monitored. If you're coming from a higher level language, such as Ruby, this might seem odd, but is essentially an "optimized" (one might see it as convoluted) way of passing an array of integers to select. select will know it will not have to look for file descriptors with a value greater than this value.

We also pass a timeout of ten seconds, which is an arbitrary value, and None values for the other arguments.

The function returns a Result, which will be an Err if something went wrong, for instance if we had used the inline version mentioned above, we would have received the following error due to the socket being closed: Failed to select: Os { code: 9, kind: Other, message: "Bad file descriptor" }

On the other hand if the result is successful, we want to know which sockets can be read from. To do that, we need to iterate through the range of all possible file descriptors that could have been described by fd_set, which is all the number between 0 and max_fd, inclusive.

For each of the file descriptors, we use the FD_ISSET macro, through the is_set function to ask if this file descriptor is set in fd_set, it will only be set if the file descriptor can be read from.

If we had been interested in which sockets could be written to, we would have created a different fd_set and use the same approach on each fd_set.

In other words, select modifies the fd_set you give it and sets the internal bits for the file descriptors that are ready, it's then up to you to look at the content of fd_set and detect which file descriptors are ready. It is also up to you to keep track of which fd_set was given to be notified for readability and which one was given for writability.

One metaphor to explain this process is that we give select a huge piece of paper with a list of file descritor ids, each followed by an empty checkbox and we tell it to not look past line n. n here is the equivalent of the max_fd argument we just discussed.

select gives it back to us with the checkbox checked for all the ones that are ready.

If you're testing this locally, you'll only have ten seconds to do something from irb, feel free to change the value, or pass None if you want an infinite timeout.

When writing to a single client in Ruby, with client2.write '123', I got the following output for my Rust program:

Socket 1: 5
Socket 2: 6
Socket 3: 7
select result: 1
Socket 6 received something!

And we can see that if multiple sockets are ready, they're all detected, which we can test by writing to two clients with client2.write '123'; client1.write '456', which gives us the following output:

Socket 1: 5
Socket 2: 6
Socket 3: 7
select result: 2
Socket 5 received something!
Socket 6 received something!

It works!

Conclusion

At the risk of repeating myself, the purpose of all this is only to learn more about Rust and select, if you're writing a real application, look into nix and Tokio instead.

Most of the code was adapted from this Gist I foound on Reddit.

You can find the code on GitHub

Liked this post? You might like my free online book about Rebuilding Redis, in Ruby.

Appendix: Same example, but with pselect:

extern crate libc;

use std::net::TcpStream;
use std::os::unix::io::AsRawFd;
use std::os::unix::io::RawFd;
use std::{io, mem, ptr, time};

pub struct FdSet(libc::fd_set);

impl FdSet {
    pub fn new() -> FdSet {
        unsafe {
            let mut raw_fd_set = mem::MaybeUninit::<libc::fd_set>::uninit();
            libc::FD_ZERO(raw_fd_set.as_mut_ptr());
            FdSet(raw_fd_set.assume_init())
        }
    }
    pub fn clear(&mut self, fd: RawFd) {
        unsafe { libc::FD_CLR(fd, &mut self.0) }
    }
    pub fn set(&mut self, fd: RawFd) {
        unsafe { libc::FD_SET(fd, &mut self.0) }
    }
    pub fn is_set(&mut self, fd: RawFd) -> bool {
        unsafe { libc::FD_ISSET(fd, &mut self.0) }
    }
}

fn to_fdset_ptr(opt: Option<&mut FdSet>) -> *mut libc::fd_set {
    match opt {
        None => ptr::null_mut(),
        Some(&mut FdSet(ref mut raw_fd_set)) => raw_fd_set,
    }
}
fn to_ptr<T>(opt: Option<&T>) -> *const T {
    match opt {
        None => ptr::null::<T>(),
        Some(p) => p,
    }
}

pub fn pselect(
    nfds: libc::c_int,
    readfds: Option<&mut FdSet>,
    writefds: Option<&mut FdSet>,
    errorfds: Option<&mut FdSet>,
    timeout: Option<&libc::timespec>,
    sigmask: Option<&libc::sigset_t>,
) -> io::Result<usize> {
    match unsafe {
        libc::pselect(
            nfds,
            to_fdset_ptr(readfds),
            to_fdset_ptr(writefds),
            to_fdset_ptr(errorfds),
            to_ptr(timeout),
            to_ptr(sigmask),
        )
    } {
        -1 => Err(io::Error::last_os_error()),
        res => Ok(res as usize),
    }
}

pub fn make_timespec(duration: time::Duration) -> libc::timespec {
    libc::timespec {
        tv_sec: duration.as_secs() as i64,
        tv_nsec: duration.subsec_nanos() as i64,
    }
}

pub fn connect_to_localhost_2000() -> TcpStream {
    TcpStream::connect("localhost:2000").expect("Failed to connect to localhost 2000")
}

fn main() {
    let ten_seconds = time::Duration::new(10, 0);
    let mut fd_set = FdSet::new();

    let stream1 = connect_to_localhost_2000();
    let raw_fd1 = stream1.as_raw_fd();

    let stream2 = connect_to_localhost_2000();
    let raw_fd2 = stream2.as_raw_fd();

    let stream3 = connect_to_localhost_2000();
    let raw_fd3 = stream3.as_raw_fd();

    let max_fd = raw_fd1.max(raw_fd2.max(raw_fd3));

    println!("Socket 1: {}", raw_fd1);
    println!("Socket 2: {}", raw_fd2);
    println!("Socket 3: {}", raw_fd3);

    fd_set.set(raw_fd1);
    fd_set.set(raw_fd2);
    fd_set.set(raw_fd3);

    match pselect(
        max_fd + 1,
        Some(&mut fd_set),                 // read
        None,                              // write
        None,                              // error
        Some(&make_timespec(ten_seconds)), // timeout
        None,                              // mask
    ) {
        Ok(res) => {
            println!("select result: {}", res);

            let range = std::ops::Range {
                start: 0,
                end: max_fd + 1,
            };
            for i in range {
                if (fd_set).is_set(i) {
                    println!("Socket {} received something!", i);
                }
            }
        }
        Err(err) => {
            println!("Failed to select: {:?}", err);
        }
    }
}

Ruby Symbol to Proc explained, the short version

Pierre Jambet — Sat, 21 Nov 2020 23:31:51 +0000

What's with this weird looking syntax with an ampersand and a symbol

When &:object_id is used as an argument to a method call, it will convert the symbol :object_id into a Proc instance with the Symbol#to_proc method.

This syntax is really useful when mixed with Enumerable methods such as map and select:

irb(main):001:0> [ 1, 2, 3, 4, 5 ].select(&:even?)
=> [2, 4]
irb(main):002:0> [ 1, 2, 3, 4, 5 ].map(&:even?)
=> [false, true, false, true, false]

The result of :even?.to_proc is a Proc instance, very similar to proc { |x| x.even? } with one major difference. They handle arguments differently. A proc is not as strict as a method, you give it too many arguments, it ignores them, you don't give enough, it fills the missing values with nil:

irb(main):001:0> a_proc = proc { |a, b, c| p a, b, c }
irb(main):002:0> a_proc.call(1)
1
nil
nil
=> [1, nil, nil]
irb(main):003:0> a_proc.call(1,2,3,4)
1
2
3
=> [1, 2, 3]

We can inspect the arity, to see the number of parameters:

irb(main):004:0> a_proc.arity
=> 3

And the Proc#parameters method confirms that all three are optional:

irb(main):005:0> a_proc.parameters
=> [[:opt, :a], [:opt, :b], [:opt, :c]]

Note that Lambdas, beside handling return and break differently, treat their arguments as required:

irb(main):006:0> a_lambda = lambda { |a, b, c| puts a, b, c }
irb(main):007:0> a_lambda.arity
=> 3
irb(main):008:0> a_lambda.parameters
=> [[:req, :a], [:req, :b], [:req, :c]]

On the other hand, the result of Symbol#to_proc is an interesting hybrid. We can confirm that it is not a lambda:

irb(main):017:0> :even?.to_proc.lambda?
=> false

But it handles arguments differently than a regular block, let's first check its arity:

irb(main):020:0> :even?.to_proc.arity
=> -1
irb(main):022:0> :even?.to_proc.parameters
=> [[:rest]]

-1 is what you'd expect from the following Proc, one that accepts any number of arguments, but we can see that the result of parameters is different. Symbol#to_proc creates a block with a nameless parameter.

irb(main):001:0> proc { |*a| p a }.arity
=> -1
irb(main):002:0> proc { |*a| p a }.parameters
=> [[:rest, :a]]

We can get close to the to_proc result with the following, but there's not much we can do with a nameless parameter, sometimes called "naked parameter". This can be useful with methods, as calling super will forward all the arguments to the parent method, but there's no parent method with a block!

irb(main):001:0> proc { |*| }.parameters
=> [[:rest]]

Back to the result of Symbol#to_proc, we get exceptions back with no arguments, or more than one:

irb(main):023:0> :even?.to_proc.call
Traceback (most recent call last):
        4: from /Users/pierre/.rbenv/versions/2.7.1/bin/irb:23:in `<main>'
        3: from /Users/pierre/.rbenv/versions/2.7.1/bin/irb:23:in `load'
        2: from /Users/pierre/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/irb-1.2.3/exe/irb:11:in `<top (required)>'
        1: from (irb):23
ArgumentError (no receiver given)
irb(main):024:0> :even?.to_proc.call(1, 2)
Traceback (most recent call last):
        6: from /Users/pierre/.rbenv/versions/2.7.1/bin/irb:23:in `<main>'
        5: from /Users/pierre/.rbenv/versions/2.7.1/bin/irb:23:in `load'
        4: from /Users/pierre/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/irb-1.2.3/exe/irb:11:in `<top (required)>'
        3: from (irb):23
        2: from (irb):24:in `rescue in irb_binding'
        1: from (irb):24:in `even?'
ArgumentError (wrong number of arguments (given 1, expected 0))

One argument is required, so that the method identified by the symbol's value can be called on it, Kernel#object_id and Integer#even?in the examples above, and remaining arguments are forwarded to the method. In this case even? does not accept any and we get an ArgumentError exception with the cause wrong number of arguments (given 1, expected 0).

It's not just for parameter-less methods

A common mistake I've seen is to think that this approach limits us to parameter-less methods, such as Integer#even?, the following is an example using the "spaceship operator" method, which is defined as :

int <=> numeric → -1, 0, +1, or nil

And the sort method has the following method signature according to the Ruby docs:

sort { |a, b| block } → array

irb(main):004:0> [ 5, 3, 1, 2, 4 ].sort(&:<=>)
=> [1, 2, 3, 4, 5]

The method <=> was called with two arguments, the two array elements it needs to compare, it is very close to the following more explicit approach:

irb(main):004:0> [ 5, 3, 1, 2, 4 ].sort { |a, b| a.<=>(b) }
=> [1, 2, 3, 4, 5]

We can pass almost anything after the ampersand

The call to to_proc is triggered in the first place because when handling a method call, Ruby needs to make sure that if it received a block argument, that this argument is actually a proc.

The ampersand character has itself nothing to do with the symbol, or whatever comes after it. The ampersand's role in an argument list is to identify the block argument, which must come last if present.

Note that while it is very common to see & used with symbols, and procs themselves when passed as block arguments to a method, in true duck-typing fashion, anything that responds to to_proc will work. For instance Hash#to_proc returns a proc that accepts one argument and returns either nil or the value at the key identified by the one argument:

irb(main):001:1* def a_method(&block)
irb(main):002:1*   block.call(1)
irb(main):003:0> end
=> :a_method
irb(main):004:0> a_method(&{ 1 => 'one' })
=> "one"
irb(main):005:0> a_method(&{ 2 => 'two' })
=> nil

And we could pass our own class, as long as it responds to to_proc:

irb(main):004:1* class A
irb(main):005:2*   def self.to_proc
irb(main):006:2*     proc { puts "Not really useful but it works" }
irb(main):007:1*   end
irb(main):008:0> end
=> :to_proc
irb(main):009:0> a_method(&A)
Not really useful but it works
=> nil

There are four classes defined with a to_proc method in the standard library: Symbol, Method, Proc and Hash. Enumerator::Yielder is not counted as a "standard library" class given its nature as an implementation detail of Enumerator.

Links:

Thank you to Étienne Barrié for pointing me to the following links. Rails used to have its own definition of Symbol#to_proc, for when it was supporting Ruby 1.8.x. The method we've talked in this chapter was added in 1.9.x. Rubinius also has its own definition.

Conclusion

A block created through Symbol#to_proc requires at least one argument, the receiver of the method, and the remaining arguments must match the arity of the method identified by the symbol itself.

even? is parameter-less, &:even?.to_proc must be called with one argument only, the receiver. <=> has an arity of one, :<=>?.to_proc must be called with two arguments, the receiver, and the one and only argument to <=>.

Wondering how Ruby actually obtains the desired result given that it's dealing with nameless parameters? Let's look at the C definition of the method:

VALUE
rb_sym_to_proc(VALUE sym)
{
}

If you think it's weird, that's because it is!

In a next article, we'll look at how exactly Ruby handles block arguments when converting a symbol to a proc, we'll explore MRI's source code, in C, and peek at the grammar definition of the Ruby language. It's gonna be fun!

Found this interesting? You'll enjoy my free online book about rebuilding Redis, in Ruby.

Appendix, Arguments vs Parameters

As of recently I used to use the terms "arguments" and "parameters" interchangeably, and I was wrong.

Parameters are the variables that you can use from withing a function/method/block, in the following example, a and b are parameters:

def a_method(a, b)
end

Arguments are the values that are given to a function/method/block, that will be be bound to the parameter variables. In the following example, 'a' and 20 are arguments:

a_method('a', 20)

Rebuilding Redis in Ruby - Chapter 10 - Adding Sorted Sets Commands

Pierre Jambet — Thu, 19 Nov 2020 13:59:14 +0000

What we'll cover

With support for Sets added in the previous chapter, our server is now only two data types short of the real Redis. In this chapter we will focus exclusively on Sorted Sets.

Sorted sets are very similar to sets, with one major difference, instead of members being only strings, members are pairs of strings and floats, where the float value is used to sort members and is called the score. As mentioned in the previous chapters, Sets do not guarantee ordering and while the IntSet structure happened to provide a sorted data structure, the Dict class doesn't guarantee any ordering and calling SMEMBERS would not always return elements in the same order. Every command that returns all the elements in a set would show this behavior, which is the case for SUNION, SDIFF and SINTER.

On the other hand, sorted sets guarantee ordering, but because the score value is not unique, members are sorted by the lexicographic of the member if scores are equal. Let's look at a few examples:

127.0.0.1:6379> ZADD z 1.1 a 2.2 b 3.3 c 4 d 5.0 e 6e2 f 0 zero
(integer) 7
127.0.0.1:6379> ZRANGE z 0 -1
1) "zero"
2) "a"
3) "b"
4) "c"
5) "d"
6) "e"
7) "f"
127.0.0.1:6379> ZRANGE z 0 -1 WITHSCORES
 1) "zero"
 2) "0"
 3) "a"
 4) "1.1000000000000001"
 5) "b"
 6) "2.2000000000000002"
 7) "c"
 8) "3.2999999999999998"
 9) "d"
10) "4"
11) "e"
12) "5"
13) "f"
14) "600"

The first thing to note is that all sorted set commands are prefixed with a Z, and a sorted set if often referred to as a "zset" throughout the Redis codebase. As a matter of fact, all the sorted set commands are implemented in the t_zset.c file.

Members are added with ZADD, which accepts an even list of arguments after the key of the zset itself, z in the example, the first element of each member pair must be a valid float, as defined in the Chapter 7 when we added the HINCRBYFLOAT command or in Chapter 6 when we added validation for the timeout values for the blocking commands on lists. This means that besides "regular" values such as 5 or 26.2, scores can also be expressed with the E notation, 7.03e1 for 70.3, and the strings inf and infinity, with or without the + and - signs are accepted, in a case insensitive way.

ZRANGE 0 -1 returns all the members, without scores, similarly to the LRANGE command for lists. We can add the WITHSCORES option to include the scores. In both examples we can see that the order of elements in the RESP array returned by ZRANGE ordered the elements by their score values.

You might have noticed that we've already observed issues with the accuracy of the float values. a, b & c all show rounding errors with the WITHSCORES option, which we can also illustrate with the ZSCORE command that returns the score of the given member:

127.0.0.1:6379> ZSCORE z a
"1.1000000000000001"
127.0.0.1:6379> ZSCORE z b
"2.2000000000000002"
127.0.0.1:6379> ZSCORE z c
"3.2999999999999998"

These issues are the exact same we discussed in Chapter 7 when implementing the HINCRBYFLOAT command. That being said, we can see that the precision of the score value in a zset seems to be worse than what HINCRBYFLOAT provided:

127.0.0.1:6379> HINCRBYFLOAT h a 1.1
"1.1"
127.0.0.1:6379> HGETALL h
1) "a"
2) "1.1"

This difference in precision is because Redis uses a double for the score field in a zset, whereas it uses a long double when performing the operation in HINCRBYFLOAT.

One reason that might justify this choice is that values are stored as strings in a hash, so the extra bytes required to allocate a long double are only temporary, while performing the operation. On the other hand, the score is always stored as a number, alongside the member value as a string, in a zset, so the extra bytes required for a long double would affect the memory usage of the server significantly for large zsets.

Let's approximate this difference. A double uses eight bytes, and a long double uses sixteen. Setting aside some of the overhead required by the data structure actually storing the sorted sets, we can infer that the size of a sorted set elements would be at least 'number of bytes in member' + 8 with a double and 'number of bytes in member' + 16 with a long double. While the difference might seem small, it means that a sorted set with 1,000,000,000 members would use an extra 8,000,000,000 bytes with a long double, 16,000,000,000 bytes compared to 8,000,000,000 bytes to store the scores, that's about 8 gigabytes (GB), or about 7.45 gibibytes (GiB) of memory!

The two units are similar but different by a small factor, a kilobyte (KB) is 1,000 bytes, and a kibibyte (KiB) is 1,024 bytes, 2^10, a megabyte (MB) is 1,000,000 bytes, and a mebibyte (MiB) is 1,048,576 bytes, 2^20, or 1,204 * 1,024, and so on, you multiply by 1,000 in one case, and by 1,024 in the other. ISO-80000, or IEC 80000 is the standard the introduced these Binary Prefix units.

And we'll see in the next section that Redis actually stores the score twice, so this difference would actually be at least two bytes per member.

It's also important to consider the trade-offs and the actual impact of these precision issues. It seems fair to expect a high level of precision for the increment commands, INCRBYFLOAT and HINCRBYFLOAT, as users are given the ability to perform operations on the values and use the results in their application. On the other hand, the score value is only used for ordering, meaning that these precision issues aren't that impactful, as long as they're consistent. In the previous example, despite the precision issues, we still observe the expected behavior, 1.1000000000000001 is lower than 2.2000000000000002 and the order of members is the same as if the scores had been precisely 1.1 and 2.2.

The consistency aspect is also important, because members are sorted by lexicographic order if members are equal, let's look at an example by adding another member with an identical score:

127.0.0.1:6379> ZADD z 1.1 a
(integer) 0
127.0.0.1:6379> ZADD z 1.1 aa
(integer) 1
127.0.0.1:6379> ZADD z 1.1 ab
(integer) 1

The first ZADD call returned 0, because the sorted set already contains the member a. The next two calls both returned 1 because aa and ab were both successfully added, with the same score as a, so let's look at the order of members now:

127.0.0.1:6379> ZRANGE z 0 -1
1) "zero"
2) "a"
3) "aa"
4) "ab"
5) "b"
6) "c"
7) "d"
8) "e"
9) "f"
127.0.0.1:6379> ZRANGE z 0 -1 WITHSCORES
 1) "zero"
 2) "0"
 3) "a"
 4) "1.1000000000000001"
 5) "aa"
 6) "1.1000000000000001"
 7) "ab"
 8) "1.1000000000000001"
 9) "b"
10) "2.2000000000000002"
11) "c"
12) "3.2999999999999998"
13) "d"
14) "4"
15) "e"
16) "5"
17) "f"
18) "600"

We can see that aa and ab were both added after a and before b. The three elements with identical scores are ordered by lexicographical order: a < aa < ab.

While we could mimic the Redis behavior and use the Ruby Float class for the score value in a zset, and keep using BigDecimal for the increment operations, we will keep using BigDecimal to keep things simpler. As mentioned previously, the Server we're building does not try to optimize every single aspects, which we would be a losing battle by using such a high level language as Ruby.

Redis supports twenty eight (!!) commands for Sorted Sets, and we'll implement all of them except ZSCAN for the same reasons outlined in Chapter 8. Because all the *SCAN commands are complicated and deserve their own chapter.

ZADD: Add one or more members (with their scores) to a sorted set, creating it if necessary
ZCARD: Return the cardinality of the set, the number of members
ZRANGE: Return all the members with an index within the given range
ZRANGEBYLEX: Return all the members with a member value within the given lexicographic range
ZRANGEBYSCORE: Return all the members with a score value within the given score range
ZSCORE: Return the score of a member
ZMSCORE: Return the scores for all the given members
ZRANK: Return the rank, its 0-based index in the set, of a member
ZREM: Remove a member from the set
ZREMRANGEBYLEX: Remove all the members within the lexicographic range
ZREMRANGEBYRANK: Remove all the members within the rank range
ZREMRANGEBYSCORE: Remove all the members within the score range
ZREVRANGE: Return all the members with an index within the given range, sorted by descending score
ZREVRANGEBYLEX: Return all the members with an index within the given lexicographic range, sorted by descending lexicographic order
ZREVRANGEBYSCORE: Return all the members with an index within the given score range, sorted by descending score
ZREVRANK: Return the rank of a member, as if it was sorted by descending score
ZINTER: Return the intersection of multiple sets
ZINTERSTORE: Store the intersection of multiple sets in another key
ZUNION: Return the union of multiple sets
ZUNIONSTORE: Store the union of multiple sets in another key
ZPOPMAX: Remove the member with the highest score
ZPOPMIN: Remove the member with the smallest score
BZPOPMAX: Blocking variant of ZPOPMAX
BZPOPMIN: Blocking variant of ZPOPMIN
ZCOUNT: Count the number of members with a score in the given range
ZLEXCOUNT: Count the number of members within the given lexicographic range
ZINCRBY: Increment the score of a member

Twenty-seven commands await us, it's gonna be a long chapter, and we have a lot of code to go through, buckle up!

How Redis does it

As we've seen in the last two chapters, Redis uses two underlying structures, depending on the size of the sorted sets to implement the sorted set API. The two criteria are similar to the ones used for the hash structure, the number of entries, configured through zset-max-ziplist-entries, with a default value of 128 and the length of the members themselves, configured with zset-max-ziplist-value, with a default value of 64.

As long as the size of the sorted set is below zset-max-ziplist-entries and as long as each member's length is below zset-max-ziplist-value, the sorted set elements will be stored in a ziplist. The choice of a ziplist as a data structure for small sorted sets is driven by the same reasons Redis uses a ziplist for small hashes.

Let's quickly confirm this with redis-cli, DEBUG OBJECT, irb and the redis gem:

127.0.0.1:6379> ZADD z 0 1
(integer) 1
127.0.0.1:6379> DEBUG OBJECT z
Value at:0x7fef01405b70 refcount:1 encoding:ziplist serializedlength:16 lru:11787108 lru_seconds_idle:3

Now let's add 127 members to the sorted set:

irb(main):001:0> require 'redis'
=> true
irb(main):002:0> red = Redis.new; 127.times { |i| red.zadd('z', i, i) }
=> 127

And let's inspect things back in redis-cli:

127.0.0.1:6379> ZCARD z
(integer) 128
127.0.0.1:6379> DEBUG OBJECT z
Value at:0x7fa078f05810 refcount:1 encoding:ziplist serializedlength:533 lru:11787457 lru_seconds_idle:2
127.0.0.1:6379> ZADD z 0 b
(integer) 1
127.0.0.1:6379> DEBUG OBJECT z
Value at:0x7fa078f05810 refcount:1 encoding:skiplist serializedlength:1292 lru:11787463 lru_seconds_idle:2
127.0.0.1:6379> ZCARD z
(integer) 129

With 128 items the set is still using a ziplist, and as long as we get past that, it becomes a skiplist.

If any of the two constraints is not met, Redis switches to a combination of a dict and a skiplist. The dict stores members as keys, which is similar to what we did in the previous chapter for sets, and the value is the score. While this is enough to guarantee uniqueness, as well as store the score values, it is problematic if we were to call the ZRANGE command to return the first element, with ZRANGE z 0 0. In order to know which member is the first one, we'd need to iterate through the whole dict, to find the member with the smallest score.

In order to make these operations faster, Redis also stores the member and score values in a skiplist. A skiplist is a data structure that maintains its element sorted and provides some of the benefits of a linked list, such as efficient adding and removal operations, while also providing an efficient way to search for elements, with an O(logn) time complexity. Using a "regular" linked list would have an O(n) time complexity for search.

Redis stores both the string and the score in the skiplist, allowing it to efficiently retrieve sorted set members based on their position according to the ordering by score. This position is called "rank".

Implementing a skiplist is fairly complicated, and Redis uses a modified version which stores extra information, so given how much work we already have ahead of us, we will not implement it in this chapter. We will instead use a similar approach by modifying our SortedArray class. It's important to note that by using a sorted array, our sorted set implementation will suffer from the same problems we described about the ziplist becoming expensive to manipulate as they grow. This drawback is a conscious decision made in order to focus on other parts of the sorted set implementations, while keeping some of the original ideas, that is the two data structures approach.

The following is an illustration of what a skiplist looks like, the arrows can be seen as "express lanes". The skiplist paper goes into more details if you're interested in learning more about this structure:

The tl;dr; is that the arrows seen above, the "express lanes", can be used to ignore big chunks of the list when searching for an element. The search process always starts from the top left, and follows arrows as needed. Say that we'd be searching for the number 7, we'd follow the first arrow, see that nothing is on the other end, so move to the second arrow, it would point us to 4, which given that we know that the list is sorted, gives us a chance to find 7 if we were to keep looping, following the arrow would take us to 6, following it again would take us to the end of the list, so we would move to the one below and find 9, which means that 7 cannot be found if we were to continue over there, so we keep going down, and land on 7. The image below highlights all the steps that we would take to find 7, where the red arrows show the paths we chose not to follow and the green ones the ones we did.

The next example shows the path we would take if we were to search for 11:

Note that Redis optimizes a few things such as storing a reference to the tail of the list, which would have sped up the process to find 11.

Updating our SortedArray class

As we mentioned earlier we will not implement a skiplist in this chapter, we will instead reuse our SortedArray class to store member/score pairs, ordered by their score, and by member if scores are equal.

The initial version of SortedArray used to accept a field argument for its constructor, and it would use this field to order elements within the array. We used this with SortedArray.new(:timeout) to order BlockedState instances, which have a timeout field.

The field was used to compare elements in blocks passed to Array#bsearch_index calls.

The main change we want is for SortedArray instances to consider multiple fields, from left to right. The use case is that we want our SortedArray to store objects with a score field, and a member field, if the scores are different, we want elements to be ordered by score, otherwise, by member. In other words, the primary ordering is through scores and the member acts as a tiebreaker.

It's worth noting that this approach, similarly to the skiplist in Redis, does not enforce member uniqueness, this is the responsibility of the caller to check for member uniqueness and is what we will use a Dict for.

There are different ways to solve the problem we're facing now, we could even create a new class, ScoreAndMemberSortedArray, and rename the current one to TimeoutSortedArray. Instead we will refactor the class to work with any number of fields. In order to do so, we will replace the argument from field, to &block, and let callers pass the block that will be fed to bsearch_index:

require 'forwardable'

module BYORedis
  class SortedArray
    extend Forwardable

    def_delegators :@underlying, :[], :delete_if, :size, :each, :delete_at, :shift,
                   :bsearch_index, :map, :each_with_index, :pop, :empty?

    def initialize(&block)
      @underlying = []
      @block = block
    end

    def push(new_element)
      if @underlying.empty?
        index = 0
      else
        index = @underlying.bsearch_index do |element|
          @block.call(element, new_element) <= 0
        end
      end

      index = @underlying.size if index.nil?
      @underlying.insert(index, new_element)
    end
    alias << push
  end
end

listing 10.1 The updated push method in the SortedArray class

Note that we added the Forwardable module to delegate a bunch of methods directly to the underlying array. The only difference between the new push method and the old one is the else branch, it used to be:

index = @underlying.bsearch_index do |element|
  element.send(@field) >= new_element.send(@field)
end

The new version expects the block to return a "comparison value", which is what we explored in the previous chapter with the "spaceship operator". A negative value indicates that the left element is lower than the right one, 0 means that both elements are equal and a positive value means that left element is greater. While technically any negative or positive values are valid, the spaceship operator always returns -1, 0 or 1.

In the previous implementation, the block passed to bsearch_index was using the find-minimum mode, and was returning a boolean. The boolean would only be true if the field of the new element, timeout in practice, was lower than or equal to the one we're comparing it with in the array.

As long as the @block given to the constructor is correctly created, it will return the same value, the following is the block that should be given for the behavior to stay the same:

SortedArray.new do |array_element, new_element|
  new_element.timeout <=> array_element.timeout
end

The block will return -1 if:

new_element.timeout < array_element.timeout

it will return 0 if:

new_element.timeout == array_element.timeout

and will return 1 if

new_element.timeout > array_element.timeout

The value returned by the block will be <= 0 if and only if new_element.timeout <= array_element.timeout, so the behavior is the same!

With this change, we can now create a SortedArray that compares multiple fields!

SortedArray.new do |array_element, new_element|
  score_comparison = new_element.score <=> array_element.score
  if score_comparison == 0
    new_element.member <=> array_element.member
  else
    score_comparison
  end
end

If score_comparison is not 0, then the scores are different, and by returning score_comparison, our push methods will end up ordering elements by score values. The difference is if the score values are equal, if score_comparison == 0, in this case we use the member values as tiebreaker, and return the result of the spaceship operators between the members values.

Let's now update the delete method:

module BYORedis
  class SortedArray

    # ...

    def delete(element)
      index = index(element)
      return if index.nil?

      element_at_index = @underlying[index]
      first_index_to_delete = nil
      number_of_items_to_delete = 0
      while element_at_index
        if element_at_index == element
          first_index_to_delete ||= index
          number_of_items_to_delete += 1
        end

        index += 1
        next_element = @underlying[index]
        if next_element && @block.call(next_element, element_at_index) == 0
          element_at_index = next_element
        else
          break
        end
      end

      @underlying.slice!(first_index_to_delete, number_of_items_to_delete)
    end

    def index(element)
      if @underlying.empty?
        nil
      else
        @underlying.bsearch_index do |existing_element|
          @block.call(existing_element, element)
        end
      end
    end
  end
end

listing 10.2 The updated delete method in the SortedArray class

There are two differences between the new delete method and the previous one. First, we extracted an index method to return the index of a member, or nil if the element is not present.

The index method uses the @block variable with bsearch_index, but this time it passes the result of the block directly, which uses the find-any mode, in which it will return the index of the element, if it exists, and nil otherwise. Note that if there are duplicates, the left-most element is returned in this mode.

The rest of the delete method is almost identical, we grab the element at index index, and as long as they are equal according to @block, which we check with @block.call(next_element, element_at_index) == 0, we keep going right.
This last step was necessary for the timeout based use case. In the timeout array we might end up with multiple values sharing the same timeout, in which case, we want to find element within these.

Final touches

Creating a new instance of SortedArray is now a bit tedious, you need to know how to craft the block argument for it to work as expected, here is what it would look like to replace the timeout based sorted array:

SortedArray.new do |array_el, new_el|
  new_el.timeout <=> array_el.timeout
end

It is even more complicated with our new use case, where we want to order items in the array by score, and fallback to member if the scores are equal:

SortedArray.new do |array_el, new_el|
  score_comparison = new_el.score <=> array_el.score
  if score_comparison == 0
    new_el.member <=> array_el.member
  else
    score_comparison
  end
end

Let's improve this by providing a class method on SortedArray that creates the correct block based on the given fields:

module BYORedis
  class SortedArray

    # ...

    def self.by_fields(*fields)
      SortedArray.new do |array_element, new_element|
        comparison = nil
        fields.each do |field|
          comparison = new_element.send(field) <=> array_element.send(field)
          # As long as the members are equal for field, we keep comparing
          if comparison == 0
            next
          else
            break
          end
        end

        comparison
      end
    end
  end
end

listing 10.3 The by_fields class method for SortedArray

We can now easily create a sorted array for BlockedState with:

SortedArray.by_fields(:timeout)

And one for sorted set members with score and member attributes:

SortedArray.by_fields(:score, :member)

Perfect!

Creating and Updating Sorted Sets

With regular sets there is no concept of "update", an element is a member of a set, or it isn't, we can either add it or remove it. Things are different with sorted sets, now that members have a score value associated with them, we can update the score of an existing member in a sorted set. All these operations are performed with the ZADD command. We only saw some of the ways in which it can be used in the introduction to this chapter. It's now time to add it to our server.

The ZADD command

In its simplest form, the ZADD command uses the format: zset-key score, member ... where score needs to be a valid float and must be followed by a string value as the member. We've already looked at some examples earlier in the chapter, so let's now look at all the possible options:

NX|XX: With NX, members can only be added and are never updated and XX never adds new members, it only updates existing ones. The two options are mutually exclusive.
LT|GT: These two options have been added in 6.2.0 and have not yet been implemented in this book. Only updates the members if the new score is respectively lower than or greater than the existing score
CH: Return the number of changed, where "changed" means members added, or members updated. By default only the number of added member is returned
INCR: Limit the number of score/member pair to one, and increment the score for the given member by the new score, defaulting to 0 if the member was not present in the set. The INCR option changes the return value to the new score and ignores the CH option.

The format is described as the following by the Redis Documentation:

ZADD key [NX|XX] [GT|LT] [CH] [INCR] score member [score member ...]

Let's look at some examples now:

127.0.0.1:6379> ZADD z NX CH INCR 10.1 a
"10.1"
127.0.0.1:6379> ZADD z NX CH 5 a 2.2 b
(integer) 1
127.0.0.1:6379> ZRANGE z 0 -1 WITHSCORES
1) "b"
2) "2.2000000000000002"
3) "a"
4) "10.1"
127.0.0.1:6379> ZADD z CH 5 a 2.2 b
(integer) 1
127.0.0.1:6379> ZRANGE z 0 -1 WITHSCORES
1) "b"
2) "2.2000000000000002"
3) "a"
4) "5"
127.0.0.1:6379> ZADD z CH 5 a
(integer) 0
127.0.0.1:6379> ZADD z CH 6 a
(integer) 1
127.0.0.1:6379> ZRANGE z 0 -1 WITHSCORES
1) "b"
2) "2.2000000000000002"
3) "a"
4) "6"

The first command, with the INCR option shows how it uses a default score value of 0 and added 10.1 to it. The CH option was overridden by the INCR option and the NX option did not do anything since the member was not already present.

In the second example, NX blocked a from being updated and only b was added, which counts as an update and is counted with the CH option.

The same command without the NX option updates both a and b, but because b has the same score, it is not updated and the count only includes the updated score of a.

The next command shows a changed count of 0 because the score is the same. Finally, changing the score to a different value, 6, returns a changed count of 1.

Let's start by creating the sorted_set_commands.rb file with the ZAddCommand class.

require_relative './redis_sorted_set'

module BYORedis

  class ZAddCommand < BaseCommand
    def call
      @options = {
        presence: nil,
        ch: false,
        incr: false,
      }
      Utils.assert_args_length_greater_than(1, @args)
      key = @args.shift
      parse_options
      raise RESPSyntaxError unless @args.length.even?

      if @options[:incr] && @args.length > 2
        raise ValidationError, 'ERR INCR option supports a single increment-element pair'
      end

      pairs = @args.each_slice(2).map do |pair|
        score = Utils.validate_float(pair[0], 'ERR value is not a valid float')
        member = pair[1]
        [ score, member ]
      end

      sorted_set = @db.lookup_sorted_set_for_write(key)
      return_count = 0
      pairs.each do |pair|
        sorted_set_add_result = sorted_set.add(pair[0], pair[1], options: @options)

        if @options[:incr]
          if sorted_set_add_result
            return_count = Utils.float_to_string(sorted_set_add_result)
          else
            return_count = nil
          end
        elsif sorted_set_add_result
          return_count += 1
        end
      end

      RESPSerializer.serialize(return_count)
    rescue FloatNaN
      RESPError.new('ERR resulting score is not a number (NaN)')
    end

    def self.describe
      Describe.new('zadd', -4, [ 'write', 'denyoom', 'fast' ], 1, 1, 1,
                   [ '@write', '@sortedset', '@fast' ])
    end

    private

    def parse_options
      @options = {}
      loop do
        # We peek at the first arg to see if it is an option
        arg = @args[0]
        case arg.downcase
        when 'nx', 'xx' then set_presence_option(arg.downcase.to_sym)
        when 'ch' then @options[:ch] = true
        when 'incr' then @options[:incr] = true
        else
          # We found none of the known options, so let's stop here
          break
        end
        # Since we didn't break, we consume the head of @args
        @args.shift
      end
    end

    def set_presence_option(option_value)
      if @options[:presence] && @options[:presence] != option_value
        raise ValidationError, 'ERR XX and NX options at the same time are not compatible'
      else
        @options[:presence] = option_value
      end
    end
  end
end

listing 10.4 The ZAddCommand class in sorted_set_commands.rb

Handling all the various options makes the method longer than most other commands, let's slowly step through it. We initially create a hash of default values for the three options, presence, which has three possible values, nil, the default, nx, or px. ch defaults to false and will be set to true only we find the ch option among the arguments. Finally, incr defaults to false and will be switched to true if we find incr among the arguments.

The validation of the length of the @args array is not as simple as it usually is, so we start by checking that we have at least one argument, and we'll perform more validations later on. The first argument is the key of the sorted set, so we extract it with Array#shift and delegate the options handling to the private method parse_options

The tricky thing about parse_options is that it operates on an array of arguments, but it doesn't know if it contains any options, since they're all optional, so if @args was set to [ '1', 'a', '2', 'b' ], it shouldn't do anything, but if the first elements of the array are valid options, it needs to extract and process them.

We use a "peek" approach, we look at the head of @args, with @args[0], and compare it with all the valid option values. We use String#downcase to make sure that we handle any case variants of the options, such as InCr or nx. If we find either nx or xx, we call set_presence_options. This method takes care of returning an error if the arguments contained both nx and xx, which is invalid, as well as setting the value in @options[:presence].

Back to parse_options, the other two cases are ch and incr, in either situation we set the corresponding value to true in the @options hash. If the head of @args does not match any of these cases, we abort the loop and exit the method, we're done parsing options and we need to treat all the remaining elements as score/member pairs. If we did not exit the loop early with break, we reach the last line @args.shift, which effectively "consumes" the head of @args so that the next iteration sees the next element when it peeks at the head again.

Back to call, all valid options have been shifted from @args, and an exception was raise if any of the options were invalid, so if we're back in call we know we have to handle all the elements in @args as member/score pairs. We start by checking that we have an even number of elements in the arguments array, to make sure that we're indeed dealing with pairs. Redis fails early in this case. It could technically process the elements one by one and abort when it fails to find a pair of element, but it instead validates the arguments eagerly.

Next up, we need to confirm that we only received a single score/member pair if @options[:incr] was set to true through the INCR option.

If all these checks pass, we iterate over all the elements, two at a time with each_slice(2) and validates that first element of each pair is a valid float string. The array returned by Array#map will be an array of pairs, where the first element of the pair is the score, as a BigDecimal and the second element is the member, as a String.

Now that all the validations are behind us, we load the sorted set with DB#lookup_sorted_set_for_write, which we need to write:

module BYORedis
  class DB

    # ...

    def lookup_sorted_set(key)
      sorted_set = @data_store[key]
      raise WrongTypeError if sorted_set && !sorted_set.is_a?(RedisSortedSet)

      sorted_set
    end

    def lookup_sorted_set_for_write(key)
      sorted_set = lookup_sorted_set(key)
      if sorted_set.nil?
        sorted_set = RedisSortedSet.new
        @data_store[key] = sorted_set

        if @blocking_keys[key]
          @ready_keys[key] = nil
        end
      end

      sorted_set
    end
  end
end

listing 10.5 The DB#lookup_sorted_set_for_write method

The @ready_keys[key] = nil line, under the if @blocking_keys[key] condition is similar to what we had to write in lookup_list_for_write when adding the BLPOP and BRPOP commands. We're dealing with a similar situation here, blocking commands such as BZPOPMIN and BZPOPMAX, which will be implemented later in this chapter, can cause clients to be blocked until a sorted set can be popped from. The same way that Redis never stores empty list, it also never stores empty sorted sets, and the same applies to hashes and sets, which means that whenever we create a new sorted set, we might be able to unblock a client blocked for that key, adding it to ready_keys will allow us to check for that. We'll explore this in more details when adding the two blocking commands for sorted sets.

Once the sorted_set variable is created, we initialize the return_count variable, its content will depend on the value of @options[:ch] or @options[:incr].

We then iterate over the pairs array, and for each pair we call RedisSortedSet#add with the score, the member and the @options hash.

If @options[:incr] was set to true we store the value returned by RedisSortedSet#add, as a BigDecimal in return_count and return it with RESPSerializer, which will either return the new cardinality as RESP integer or the new score, as string, since RESP2 does not have a dedicated float type.

We now need to dive into the RedisSortedSet#add method, which is the one that actually adds items to the sorted set:

require 'bigdecimal'

require_relative './dict'
require_relative './list'
require_relative './zset'

module BYORedis
  class RedisSortedSet

    Pair = Struct.new(:score, :member)

    attr_reader :underlying

    def initialize
      @underlying = List.new
    end

    def add(score, member, options)
      convert_to_zset if @underlying.is_a?(List) &&
                         member.length > Config.get_config(:zset_max_ziplist_value)

      case @underlying
      when List
        added = add_list(score, member, options)
        convert_to_zset if added && @cardinality >= Config.get_config(:zset_max_ziplist_entries)
        added
      when ZSet then @underlying.add(score, member, options)
      else raise "Unknown type for #{ @underlying }"
      end
    end

    private

    def convert_to_zset
      raise "#{ @underlying } is not a List" unless @underlying.is_a?(List)

      zset = ZSet.new
      @underlying.each do |pair|
        zset.dict[pair.member] = pair.score
        zset.array << pair
      end

      @underlying = zset
    end
  end
end

listing 10.6 The RedisSortedSet#add method

We declare a new Struct at the beginning of the class, Pair, which will hold the score/member pairs inside the List or within the ZSet. The ZSet class is the class that will coordinate the Dict and SortedArray instances, as described earlier in the chapter:

module BYORedis
  class ZSet

    attr_reader :dict, :array

    def initialize
      @dict = Dict.new
      @array = SortedArray.by_fields(:score, :member)
    end
  end
end

listing 10.7 The ZSet class

Back to RedisSortedSet#add, we use the tried and true pattern of a case/when against @underlying to determine which data structure we're currently dealing with. In the List case we delegate the logic to the add_list private method, and in the ZSet case we use the ZSet#add method.

Before looking at the process of adding or updating the score and member values in the List or the ZSet, let's take a look at the convert_to_zset method. We need to iterate through all the elements in the list to add them to the newly created ZSet, so instead of using the List.left_to_right_iterator, which is pretty verbose, let's add the List#each method:

module BYORedis
  class List

    # ...

    def each(&block)
      raise 'No block given' unless block

      iterator = List.left_to_right_iterator(self)
      while iterator.cursor
        block.call(iterator.cursor.value)
        iterator.next
      end
    end

    # ...
  end
end

listing 10.8 The List#each method

We won't always be able to use the each method, there are cases where we'll need to have a hold of the ListNode instance, to check the value of its prev_node for instance, and in these cases we'll still need to use List.left_to_right_iterator. We're about to see an example in the add_list method below.

Once all the members from the List instances have been added to the ZSet one, we update @underlying to point at the latter and let the Garbage Collector do its job to get rid of the list.

Let's now dive in RedisSortedSet#add_list:

module BYORedis
  class RedisSortedSet

    # ...

    private

    def add_list(score, member, options: {})
      raise "#{ @underlying } is not a List" unless @underlying.is_a?(List)

      unless [ nil, :nx, :xx ].include?(options[:presence])
        raise "Unknown presence value: #{ options[:presence] }"
      end

      iterator = List.left_to_right_iterator(@underlying)

      while iterator.cursor
        cursor = iterator.cursor
        pair = iterator.cursor.value

        if pair.member == member
          # We found a pair in the list with a matching member

          if pair.score == score && !options[:incr]
            # We found an exact match, without the INCR option, so we do nothing
            return false
          elsif options[:presence] == :nx
            # We found an element, but because of the NX option, we do nothing
            return false
          else
            # The score changed, so we might need to reinsert the element at the correct
            # location to maintain the list sorted
            new_score = options[:incr] ? Utils.add_or_raise_if_nan(pair.score, score) : score
            prev_node = cursor.prev_node
            next_node = cursor.next_node

            if (next_node.nil? ||
                next_node.value.score > new_score ||
                (next_node.value.score == score && next_node.value.member > member)) &&
               (prev_node.nil? ||
                prev_node.value.score < new_score ||
                (prev_node.value.score == score && prev_node.value.member < member))

              cursor.value.score = new_score
            else
              @underlying.remove_node(cursor)
              # We add the node back, which takes care of finding the correct index
              unless add_list(new_score, member)
                raise 'Unexpectedly failed to re-insert node after update'
              end
            end

            if options[:incr]
              return new_score
            else
              # If options[:ch] == true, then we want to count this update and return true
              return options[:ch]
            end
          end
        elsif pair.score > score || (pair.score == score && pair.member > member)
          # As soon as we find a node where its score is greater than the score of the
          # element we're attempting to insert, we store its reference in `location` so that
          # we can use insert_before_node below.
          # In case of a score equality, the right side of the || above, we use the
          # lexicographic order of the member value to sort them
          # We cannot stop here however because there might be an exact member match later in
          # the list, in which case the `if pair.member == member` check above will trigger
          # and return
          location ||= cursor
          if options[:member_does_not_exist]
            break
          else
            iterator.next
          end
        elsif pair.score < score || (pair.score == score && pair.member < member)
          # In this case we haven't found a node where the score is greater than the one we're
          # trying to insert, or the scores are equal but the lexicographic order tells us that
          # member is greater than the current node, so we keep searching for an insert location
          # to the right
          iterator.next
        else
          # We've covered all cases, this is never expected to happen
          raise "Unexpected else branch reached for #{ score }/#{ member }"
        end
      end

      return false if options[:presence] == :xx

      new_pair = Pair.new(score, member)
      if location
        @underlying.insert_before_node(location, new_pair)
      else
        @underlying.right_push(new_pair)
      end

      if options[:incr]
        score
      else
        true
      end
    end
  end
end

listing 10.9 The RedisSortedSet#add_list method

There is a lot going on in there, let's go through all of it very slowly.

The first few lines take care of performing some sanity checks, if @underlying is not a List, we raise an exception, if options[:presence] is not one of the valid values, we also raise an exception. These cases fall in the category of "bugs", they're not expected, and there's not much we could do beside reporting the error to the administrator of the server. Being so aggressive with the error handling, raising an error and letting it crash will ideally help catching these errors during the development phase.

As a quick aside, there are different philosophies with regard to how errors like this one should be handled. Redis uses the serverPanic macro in a few places such as when the encoding of what it expects to be a sorted set is neither a skiplist or ziplist. This macro ends up calling exit(3), with the argument 1, the C function that exits the current process, and 1 signifies an error, 0 is the exit code for success. In this case, we could eventually rescue the exception and return an error to the user saying something like: "Something went wrong", similar to a web server returning a 500 HTTP response.

This would let the server running and potentially serve other requests. That being said, some errors might be severe enough that we should worry about the state of the server and crashing might be safer than letting it run, potentially returning invalid values. There are also approaches where the behavior would be different in development or debug mode versus release or production mode, where the errors would be aggressively uncaught during the development phase to increase the likelihood of a developer catching them.

This is a topic that would probably deserve its own book and as a conclusion note that the approach we're taking here is one approach, among many others.

Next, we create a List iterator with List.left_to_right_iterator, and we start iterating in a while loop over each element in the List. We will need to inspect the neighbors of the current node below, and as we explained earlier, this is why we cannot use the new List#each method here. Each element of the list is a Pair instance, with score & member methods. If pair.member == member, it's a match! We found a member in the sorted set that matches the one we're trying to insert. In this case we will handle the update in the if branch and return from there, but the code in there is the most complicated part of the method, so let's skip over it for now and we'll get back to it later.

The next branches of the if are:

pair.score > score || (pair.score == score && pair.member > member)

and:

pair.score < score || (pair.score == score && pair.member < member)

The first one can be translated in plain English as "if the score of item under the cursor is greater than the score we're trying to add OR if the score of the item under the cursor is the same as the score we're trying to add but the member value, a string, of the item under the cursor is greater than the one we're trying to add".

This is the case where we found the direct successor of the new element, and we should insert the new pair before it. There's a trick though! Even though it looks like we found the right place to insert the new pair in the list, this might not be true yet, we cannot know for sure at this point. If we made it to this branch, it means we have not yet found a pair with a matching member. So there are two possibilities here. Either the member is not in the set, and this is where it should be inserted, or, the member is in the set, with a greater score, further in the list. So we need to keep iterating through the list, just in case one of the elements we have not looked at yet is a match on member. The following is an example of this situation:

127.0.0.1:6379> ZADD z 10.0 a 3.0 b
(integer) 2
127.0.0.1:6379> ZRANGE z 0 -1 WITHSCORES
1) "b"
2) "3"
3) "a"
4) "10"
127.0.0.1:6379> ZADD z INCR 1 a
"11"

If we had stopped right after location ||= cursor, then we would have inserted <1, 'a'> before <3, 'b'>, and ended up with a duplicated 'a' in the set!

We use the "or equals" operator, which will only assign a value if the left operand is truthy. We might find other members in the list that have greater scores or greater members, but we still want to keep the location of the first of those to determine which node should be the successor of the one we're trying to add.

The second elsif, pair.score < score || (pair.score == score && pair.member < member) might as well had been written as an else, but writing as such allows us to catch bugs, because this is the only condition we'd expect to happen: "if the score of the item under the cursor is lower than the score we're trying to add, OR if the score of the item under the cursor is the same as the score we're trying to add but the member value, a string, of the item under the cursor is lower than the one we're trying to add". In other words, the new member should be added the further in the list, so we need to keep looking for the correct value for location.

If we exited the loop without finding a match, and options[:presence] was set to xx, then we return false, because xx forbids the addition of new elements, and we are about to add a new element now. If options[:presence] is anything else, nil or nx, we are allowed to add new set members and we proceed to instantiating a new Pair with score and member.

Now we need to decide where to insert new_pair in the List, with the constraint that we should maintain the elements sorted by score, and member if their score values are equal. It turns out that we skipped this step, while iterating through the list, we'll store a reference to the element that should succeed new_pair, in the location variable, and we call List#insert_before_node to add the set member.

If we failed to set a location, it means that we did not find an element that should succeed new_pair, in which case we insert it last with List#right_push.

The last step of the method is to decide what to return, if options[:incr] was set, we want to return the score of the new pair, otherwise we return true, to indicate that an element was added, which counts as a change no matter what, meaning that we count it regardless of the value of options[:ch].

The List#insert_before_node is a new one, let's look at it:

module BYORedis
  class List

    # ...

    def insert_before_node(pivot_node, element)
      new_node = ListNode.new(element, pivot_node.prev_node, pivot_node)
      if @head == pivot_node
        @head = new_node
      else
        pivot_node.prev_node.next_node = new_node
      end

      pivot_node.prev_node = new_node
      @size += 1
    end

    def insert_before(pivot, element)
      generic_insert(pivot) do |node|
        insert_before_node(node, element)
      end
    end

    def insert_after(pivot, element)
      generic_insert(pivot) do |node|
        new_node = ListNode.new(element, node, node.next_node)
        if @tail == node
          @tail = new_node
        else
          node.next_node.prev_node = new_node
        end

        node.next_node = new_node
        @size += 1
      end
    end

    # ...

    private

    def generic_insert(pivot)
      cursor = @head

      while cursor
        break if cursor.value == pivot

        cursor = cursor.next_node
      end

      if cursor.nil?
        -1
      else
        yield cursor

        @size
      end
    end
  end
end

listing 10.10 The insert_before_node refactor in the List class

We use to be able to only have insert_before and insert_after, which both operate based on a pivot value, which they look for in the list. Both were initially created for the LINSERT command. But while we could still use insert_before here, it would be wasteful to start iterating from the start of the list if we already had a reference to the node we wanted to use as the insertion point.

So this is what this refactor is about, insert_before and insert_after still have the same behavior, but insert_before is now written in terms of insert_before_node, a new method containing code that used to be in insert_before. We used to increment the @size instance variable in generic_insert, but because insert_before_node does not call it, we need to move it to make sure that calling either of three insert methods correctly increment the instance variable.

Let's go back to RedisSortedSet#add, inside the while loop, inside the if pair.member == member condition. We first check if pair.score == score, in which case we found an exact match, there is a member in the set with the same score and the same member, which means that we usually have nothing to do, except if option[:incr] is set. In which case it doesn't matter what the current score is, we want to add the value in the score argument to the score that was already in the set.

Next, if options[:presence] == :nx, then updates are forbidden, we're only allowed to add new elements, and since we found a match on member, we can return early, since we're about to update the existing pair.

One more else branch, hang in there with me. We now need to perform an update, we found the node in the list that contains the member, but there are a few different possibilities depending on the values in options.

First, we compute the new score, if options[:incr] is true, then we need to add the existing score to the new one, but we need to watch out for "invalid float values". What is an "invalid float value" you ask? Let's look at an example:

irb(main):001:0> require 'bigdecimal'
=> true
irb(main):002:0> inf = BigDecimal::INFINITY
irb(main):003:0> inf + inf
=> Infinity
irb(main):004:0> inf - inf
=> NaN

We can't subtract inf from inf, it's Not a Number, let's confirm this in redis:

127.0.0.1:6379> ZADD z INCR inf a
"inf"
127.0.0.1:6379> ZADD z INCR -inf a
(error) ERR resulting score is not a number (NaN)

We handle this with the Utils.add_or_raise_if_nan method:

module BYORedis

  # ...
  IntegerOverflow = Class.new(StandardError)
  FloatNaN = Class.new(StandardError)
  InvalidIntegerString = Class.new(StandardError)
  InvalidFloatString = Class.new(StandardError)

  module Utils

    # ...

    def self.add_or_raise_if_nan(a, b)
      BigDecimal.save_exception_mode do
        BigDecimal.mode(BigDecimal::EXCEPTION_NaN, true)

        a + b
      end
    rescue FloatDomainError
      raise FloatNaN
    end
  end
end

listing 10.11 The Utils.add_or_raise_if_nan method

We use the save_exception_mode so that we don't modify the behavior of BigDecimal operations once we're done here, and if it does raise an exception, FloatDomainError, we convert it to our own exception, FloatNaN, which we added at the top of the utils.rb file.

If we're not in increment mode, then the new score is the score argument, we override what was there before.

So now, we have the new score, we need to update it, we could change the value in the list node with cursor.value.score = new_score, but how do we know if the node will still be where it should be in the list once the score is updated? Well, we can check for it. This is what this weird looking condition does:

If there is no next node, the current node is the last one, or if there is a next node, and that next node score is greater than the updated score of the current node, or if the scores are equal but the next node's member string is greater than the string of the current member

AND

If there is no previous node, the current node is the first one, or if there is a previous node, and that previous node score is lower than the updated score of the current node, or if the scores are equal but the previous node's member string is lower than the string of the current member

THEN

The list will still be in order after the score update

If this condition fails, the current node will not be in the right place after the score update. Instead of trying to find the correct location, we remove the node and call RedisSortedSet#add again, with the new score, and we know it won't find the member so we pass the member_does_not_exist option to break right after location ||= cursor, to prevent a full iteration of the list:

# ...
elsif pair.score > score || (pair.score == score && pair.member > member)
  # ...
  location ||= cursor
  if options[:member_does_not_exist]
    break
  else
    iterator.next
  end
elsif pair.score < score # ...
# ...

listing 10.12 The member_does_not_exist option handling in the RedisSortedSet#add_list method

Done! That's it, we can add members to a sorted set ... as long as it uses a List under the hood, we need to handle the other case, when the strings are either too big, or the sorted set contains too many entries.

Adding Members to a ZSet

Luckily the process is not as complicated in a ZSet, thanks the easier lookup in a Dict. Let's create the ZSet#add method:

module BYORedis
  class ZSet

    # ...

    def add(score, member, options)
      unless [ nil, :nx, :xx ].include?(options[:presence])
        raise "Unknown presence value: #{ options[:presence] }"
      end

      entry = @dict.get_entry(member)

      if entry
        return false if options[:presence] == :nx

        if entry.value != score || options[:incr]

          existing_pair = new_pair(entry.value, member)
          index = @array.index(existing_pair)
          if index.nil?
            raise "Failed to find #{ member }/#{ entry.value } in #{ @array.inspect }"
          end

          array_element = @array[index]
          while array_element.member != member && index < @array.size
            index += 1
            array_element = @array[index]
          end

          if index == @array.size
            raise "Failed to find #{ member }/#{ entry.value } in #{ @array.inspect }"
          end

          new_score = options[:incr] ? Utils.add_or_raise_if_nan(entry.value, score) : score
          next_member = @array[index + 1]
          prev_member = @array[index - 1]

          if (next_member.nil? ||
              next_member.score > new_score ||
              (next_member.score == new_score && next_member.member > member)) &&
             (prev_member.nil? ||
              prev_member.score < new_score ||
              (prev_member.score == new_score && prev_member.member < member))

            array_element.score = new_score
          else
            @array.delete_at(index)
            @array << new_pair(new_score, member)
          end
          entry.value = new_score
        end

        if options[:incr]
          new_score
        else
          options[:ch] # false by default
        end
      else
        return false if options[:presence] == :xx

        @array << new_pair(score, member)
        @dict[member] = score

        if options[:incr]
          score
        else
          true
        end
      end
    end

    private

    def new_pair(score, member)
      RedisSortedSet::Pair.new(score, member)
    end
  end
end

listing 10.13 The ZSet#add method

First we use the Dict#get_entry method to check for the existence of member in the sorted set. Things are already simpler here, we don't have to iterate over anything to determine the presence of the member we're trying to add or update.

If we found a match but options[:presence] is set to :nx then updates are forbidden and we can stop right away by returning false.

If the score value of the existing member is the same as the one we're trying to add, there's nothing to do, the update would be a no-op, except if options[:incr] is set to true, in which case we want to sum the existing score and the new one. This is what we check with if entry.value != score || options[:incr], if this condition is true, we do want to update the score of the existing member.

entry is the result of calling Dict#get_entry and is a DictEntry instance where key is the score and value is the member. We create an instance of Pair to facilitate the interaction with the SortedArray, such as calling SortedArray#index to find the position of the pair in the sorted set.

We always expect to find the pair in the sorted array, because we do the bookkeeping work necessary to maintain the consistency between the Dict and the SortedArray, but it is technically possible that SortedArray#index returns nil, and in this case we throw an exception. This is another instance of a case that falls in the category of "bugs", unexpected situations where there's nothing we can really do, and we might as well notify the administrator of the server with a crash and hope that these bugs would be caught in the development phase.

The next check is pure paranoia and could be considered useless, but we double check that the value at index index is indeed equal to existing_pair. Since returning the index for the given value is the contract of the SortedArray#index method you might wonder why we'd want to perform it. The main reason here is that such check is "cheap" in the sense of it not requiring a lot of extra instructions, and it would catch obvious bugs in the SortedArray#index method, so why not!

Back to the add method, we're using Utils.add_or_raise_if_nan similarly to how we did in the List case, to handle cases such as inf - inf. Next we perform the same check to see if the existing member's position in the sorted array will still be correct after the update. We compare it with the next and previous elements in the array, if they exist.

If the order would not be broken, then we update the Pair instance, otherwise we delete it with SortedArray#delete_at, which is delegated to Array#delete_at and we insert it again, letting the SortedArray class find the new position.

Finally, we update the value in Dict with entry.value = new_score.

The return value after an update depends on the values of options[:incr] and options[:ch]. As we've seen earlier, the INCR option takes precedence, in which case we return the new score, otherwise we return false by default, if it was an update, or true, if the CH option was used and updates should be counted.

The else branch handles the case where the member does not exist in the set, in which case we return false early if the XX option was used, forbidding adding members and only allowing updates. Otherwise, we add the Pair instance to the sorted array and add the score and member to the Dict. We use a logic similar to what we just did to determine the return value, except that regardless of the presence or not of the CH option, we always return true as additions always count as changes.

And here we are! The ZADD command works!

Counting members in a Sorted Set

Now that we added the ability to create sorted sets and to add new members to them, let's add the ZCARD command to count the number of members in a sorted set:

module BYORedis

  # ...

  class ZCardCommand < BaseCommand
    def call
      Utils.assert_args_length(1, @args)
      sorted_set = @db.lookup_sorted_set(@args[0])
      cardinality = sorted_set&.cardinality || 0

      RESPInteger.new(cardinality)
    end

    def self.describe
      Describe.new('zcard', 2, [ 'readonly', 'fast' ], 1, 1, 1,
                   [ '@read', '@sortedset', '@fast' ])
    end
  end
end

listing 10.14 The ZCardCommand class

We call the cardinality method on the RedisSortedSet instance, with the "safe navigation" operator, &., which returns nil if sorted_set itself is nil, which would then take us to the right side of the || operator and effectively default cardinality to 0. Let's add the RedisSortedSet#cardinality method:

module BYORedis
  class RedisSortedSet

    # ...

    def cardinality
      case @underlying
      when List then @underlying.size
      when ZSet then @underlying.cardinality
      else raise "Unknown type for #{ @underlying }"
      end
    end
    # ...
  end
end

listing 10.15 The RedisSortedSet#cardinality method

In the List case we return the result of calling List.size, and for a ZSet, we need to add the cardinality method to the class:

module BYORedis
  class ZSet

    # ...

    def cardinality
      @array.size
    end
  end
end

listing 10.16 The ZSet#cardinality method

We could have included Forwardable and used it to delegate #size to @array, but there's only one method we need to directly delegate, so the "cost" of manually doing the delegation is really small compared to the "complexity" of including a module, and calling def_delegators, which doesn't save us much for only one method.

This wraps up the first two commands for sorted sets, ZADD & ZCARD, next we'll look at the different range commands.

Reading from Sorted Sets

With ZADD implemented, we will now add commands to retrieve elements from sorted sets. We've already seen the ZRANGE command, but Redis provides two more similar commands, ZRANGEBYSCORE & ZRANGEBYLEX.

Re-using the sorted set z from earlier in the chapter, we can use ZRANGEBYSCORE to only select a range of members within the given score, whereas ZRANGE returns member depending on their index in the sorted set, their rank. If ZRANGE had a more explicit name it'd be called ZRANGEBYRANK.

127.0.0.1:6379> ZRANGEBYSCORE z 0 1
1) "zero"
127.0.0.1:6379> ZRANGEBYSCORE z 0 3
1) "zero"
2) "a"
3) "aa"
4) "aaa"
5) "ab"
6) "b"
127.0.0.1:6379> ZRANGEBYSCORE z 0 3 WITHSCORES
 1) "zero"
 2) "0"
 3) "a"
 4) "1.1000000000000001"
 5) "aa"
 6) "1.1000000000000001"
 7) "aaa"
 8) "1.1000000000000001"
 9) "ab"
10) "1.1000000000000001"
11) "b"
12) "2.2000000000000002"

The equivalent of ZRANGE z 0 -1, that is, "return all the members" is ZRANGE z -inf +inf. This works because all possible values, including -inf, are greater than or equal to -inf, and all possible values, including +inf are lower than or equal to +inf.

127.0.0.1:6379> ZRANGEBYSCORE z -inf +inf
 1) "zero"
 2) "a"
 3) "aa"
 4) "aaa"
 5) "ab"
 6) "b"
 7) "c"
 8) "d"
 9) "e"
10) "f"

ZRANGEBYLEX is the first command of the *BYLEX category of sorted set commands. The other ones are ZREMRANGEBYLEX and ZREVRANGEBYLEX. These three commands are meant to be used for a sorted set containing elements with identical scores. A common pattern is to set a score value of 0, but any score would work, as long as it is the same across all members. "LEX" is short for "lexicographic", which is a fancy term for "alphabetically", or what you would expects words to be sorted by in a dictionnary, the real kind, with word definitions, not the data structure. Both words are not absolutely equivalent, but they're equivalent enough for the sake of this explanation.

The reason why these three commands require an identical score is because they only operate with the lexicographic order of the member values, but if scores where different, we'd have no guarantees that all members would be sorted in lexicographical order. Let's look at an example where all scores are the same first:

127.0.0.1:6379> ZADD lex-zset 0 a 0 b 0 c 0 xylophone 0 zebra 0 something-else
(integer) 6
127.0.0.1:6379> ZRANGE lex-zset 0 -1
1) "a"
2) "b"
3) "c"
4) "something-else"
5) "xylophone"
6) "zebra"

All the scores are identical, so the lexicographic order is used as a "tiebreaker" to sort the members in the sorted set. Now let's look at the same set of members, but with different scores:

127.0.0.1:6379> ZADD lex-zset-with-scores 1 a 0 b 18 c 3.14 xylophone 1.414 zebra 0.01 something-else
(integer) 6
127.0.0.1:6379> ZRANGE lex-zset-with-scores 0 -1
1) "b"
2) "something-else"
3) "a"
4) "zebra"
5) "xylophone"
6) "c"
127.0.0.1:6379> ZRANGE lex-zset-with-scores 0 -1 WITHSCORES
 1) "b"
 2) "0"
 3) "something-else"
 4) "0.01"
 5) "a"
 6) "1"
 7) "zebra"
 8) "1.4139999999999999"
 9) "xylophone"
10) "3.1400000000000001"
11) "c"
12) "18"

The ordering by score takes precedence, and the set is not sorted alphabetically anymore.

Redis does not check that the members in the sorted set all have the same score when using a *BYLEX command, it will instead incorrect results, so it is up to the caller to make sure that the data is correctly inserted before using these commands. We can use ZRANGEBYLEX to select members within the given lexicographic range, where [ means inclusive and ( exclusive:

127.0.0.1:6379> ZRANGEBYLEX lex-zset [s [zebra
1) "something-else"
2) "xylophone"
3) "zebra"
127.0.0.1:6379> ZRANGEBYLEX lex-zset [s (zebra
1) "something-else"
2) "xylophone"

The special values - and + can be used to express values that are respectively lower than any other values and greater than any other values:

127.0.0.1:6379> ZRANGEBYLEX lex-zset [s +
1) "something-else"
2) "xylophone"
3) "zebra"
127.0.0.1:6379> ZRANGEBYLEX lex-zset - [s
1) "a"
2) "b"
3) "c"

Let's look at the same commands, but with our other sorted set, lex-zset-with-scores:

127.0.0.1:6379> ZRANGEBYLEX lex-zset-with-scores - [s
1) "b"
127.0.0.1:6379> ZRANGEBYLEX lex-zset-with-scores [s +
(empty array)
127.0.0.1:6379> ZRANGEBYLEX lex-zset-with-scores [s [zebra
(empty array)
127.0.0.1:6379> ZRANGEBYLEX lex-zset-with-scores [s (zebra
(empty array)

The behavior is undefined, Redis makes the assumption that all elements are ordered alphabetically, since they're not, the result is nonsensical.

A final word on lexicographic order, while it may look intuitive at first, the letter a comes before the letter b, it can be surprising when applied to numbers represented as strings, let's look at example in Ruby first:

irb(main):010:0> '10' < '2'
=> true

The string '2' is considered to be greater than the string '10', that's because with lexicographic order, we compare strings one character at a time, and '1' < '2' is true, so the '0' character in '10' is never even considered here.

An example that might help is to use letters instead:

irb(main):011:0> 'ba' < 'c'
=> true

a, b and c are the ASCII representation of the bytes 97, 98 and 99.

This example is very similar to the previous one, given that the byte '1' has the value 49, '0', 48 and '2', 50. We're comparing two bytes on the left, with one on the right, on the left with have 49 and 48 and on the right we have 50. 50 is greater than 49, so we know which string is greater.

By Rank

We are going to start with the ZRANGE command. We already know that we're going to implement a very similar command later on, ZREVRANGE, so let's already create a method implementing the shared logic in SortedSetUtils.

The ZRANGE command has the following format according to the Redis documentation:

ZRANGE key start stop [WITHSCORES]

module BYORedis
  module SortedSetUtils

    # ...

    def self.generic_range(db, args, reverse: false)
      Utils.assert_args_length_greater_than(2, args)
      start = Utils.validate_integer(args[1])
      stop = Utils.validate_integer(args[2])
      raise RESPSyntaxError if args.length > 4

      if args[3]
        if args[3].downcase == 'withscores'
          withscores = true
        else
          raise RESPSyntaxError
        end
      end

      sorted_set = db.lookup_sorted_set(args[0])

      if reverse
        tmp = reverse_range_index(start, sorted_set.cardinality - 1)
        start = reverse_range_index(stop, sorted_set.cardinality - 1)
        stop = tmp
      end

      if sorted_set
        range_spec =
          RedisSortedSet::GenericRangeSpec.rank_range_spec(start, stop, sorted_set.cardinality)
        SortedSetRankSerializer.new(
          sorted_set,
          range_spec,
          withscores: withscores,
          reverse: reverse,
        )
      else
        EmptyArrayInstance
      end
    end
  end

  # ...

  class ZRangeCommand < BaseCommand
    def call
      SortedSetUtils.generic_range(@db, @args)
    end

    def self.describe
      Describe.new('zrange', -4, [ 'readonly' ], 1, 1, 1, [ '@read', '@sortedset', '@slow' ])
    end
  end
end

listing 10.17 The ZRangeCommand class

The SortedSetUtils.generic_range method implements the range logic, including validating the arguments and uses the SortedSetRankSerializer class to serialize the result

Let's now look at the serializer class, as well as the "range spec" class, GenericRankRangeSpec. We'll make use of other type of range specs throughout this chapter, for score order and lexicographic order, when implementing other range related commands.

This range spec class encapsulates all the data required to define a rank spec, that is, a minimum value and a maximum value, both Integer instances. We also pass the set cardinality to the class to let it transform negative indices into actual indices. For example, -1, becomes cardinality - 1, the index of the last item in the set, -2 becomes cardinality - 2, the second to last index, and so on. The rank_range_spec also makes sure that the final values of min and max are not outside the range of valid indices, that is min cannot be lower than 0 and max cannot be greater than or equal to cardinality, because there are no elements with such ranks.

The range spec also defines a useful method, empty?, when the range cannot possibly include any elements, for instance the range 1..0 in Ruby is empty, as we can see with the result of calling .to_a on it, an empty array:

irb(main):034:0> (1..0).to_a
=> []

The comparison logic relies on "comparison values" that we've explored we discussing the uses of the bsearch_index method, -1 means that the first of the two items being compared is lower than the second one, 0 means they're equal and 1 means that the second one is greater. Using this approach will allow us to customize the actual comparison logic while still reusing the core methods of this class.

The actual comparison function used by the GenericRangeSpec is the block argument given to its constructor, in this case use a <=> b, the expected order of integers.

module BYORedis
  class RedisSortedSet

    # ...

    class GenericRangeSpec

      attr_reader :min, :max, :min_exclusive, :max_exclusive
      alias min_exclusive? min_exclusive
      alias max_exclusive? max_exclusive

      def self.rank_range_spec(min, max, cardinality)
        max = cardinality + max if max < 0
        min = cardinality + min if min < 0

        max = cardinality - 1 if max >= cardinality
        min = 0 if min < 0

        GenericRangeSpec.new(min, max, false, false) do |a, b|
          a <=> b
        end
      end

      def initialize(min, max, min_exclusive, max_exclusive, &block)
        @min = min
        @min_exclusive = min_exclusive
        @max = max
        @max_exclusive = max_exclusive
        @block = block
      end

      def empty?
        comparison = compare_with_max(min)
        comparison > 0 || (comparison == 0 && (min_exclusive? || max_exclusive?))
      end

      def compare_with_max(element)
        @block.call(element, @max)
      end
    end

    # ...
  end

  class SortedSetRankSerializer
    def initialize(sorted_set, range_spec, withscores: false, reverse: false)
      @sorted_set = sorted_set
      @range_spec = range_spec
      @withscores = withscores
      @reverse = reverse
    end

    def serialize
      return RESPArray.new([]).serialize if @range_spec.empty?

      case @sorted_set.underlying
      when List then serialize_list
      when ZSet then serialize_zset
      else raise "Unknown type for #{ @underlying }"
      end
    end

    private

    def serialize_zset
      members = []
      (@range_spec.min..@range_spec.max).each do |rank|
        pair = @sorted_set.underlying.array[rank]

        if @reverse
          members.prepend(Utils.float_to_string(pair.score)) if @withscores
          members.prepend(pair.member)
        else
          members.push(pair.member)
          members.push(Utils.float_to_string(pair.score)) if @withscores
        end
      end

      RESPArray.new(members).serialize
    end

    def serialize_list
      ltr_acc = lambda do |value, response|
        response << RESPBulkString.new(value.member).serialize
        if @withscores
          response << RESPBulkString.new(Utils.float_to_string(value.score)).serialize
        end
        @withscores ? 2 : 1
      end

      rtl_acc = lambda do |value, response|
        if @withscores
          response.prepend(RESPBulkString.new(Utils.float_to_string(value.score)).serialize)
        end
        response.prepend(RESPBulkString.new(value.member).serialize)
        @withscores ? 2 : 1
      end

      if @reverse
        tmp = ltr_acc
        ltr_acc = rtl_acc
        rtl_acc = tmp
      end

      ListSerializer.new(@sorted_set.underlying, @range_spec.min, @range_spec.max)
                    .serialize_with_accumulators(ltr_acc, rtl_acc)
    end
  end
end

listing 10.18 The SortedSetRankSerializer class

The serialize method calls serialize_list or serialize_zset depending on the type of @underlying. Let's first look look at the ZSet case, we can leverage the array structure inside the ZSet to extract the range of elements we need, based on on min and max attribute of the range spec.

The score values should only be included if the WITHSCORES option was set. Additionally, the order of the final array depends on the requested order. We're jumping ahead a little bit here, but we can already assume that the ZREVRANGE method will be very similar to the ZRANGE method. So for now reverse is always set to false, meaning that for each pair in the set we always append the member value, and conditionally add the score value afterward.

Once the members array is created, we serialize it as a RESPArray.

In the List case, we use a new method from the ListSerializer class, serialize_with_accumulators:

module BYORedis

  # ...

  class ListSerializer

    def initialize(list, start, stop)
      @list = list
      @start = start
      @stop = stop
    end

    def serialize_with_accumulators(left_to_right_accumulator, right_to_left_accumulator)
      @stop = @list.size + @stop if @stop < 0
      @start = @list.size + @start if @start < 0

      @stop = @list.size - 1 if @stop >= @list.size
      @start = 0 if @start < 0

      return EmptyArrayInstance.serialize if @start > @stop

      response = ''
      size = 0
      distance_to_head = @start
      distance_to_tail = @list.size - @stop

      if distance_to_head <= distance_to_tail
        iterator = List.left_to_right_iterator(@list)
        within_bounds = ->(index) { index >= @start }
        stop_condition = ->(index) { index > @stop }
        accumulator = left_to_right_accumulator
      else
        iterator = List.right_to_left_iterator(@list)
        within_bounds = ->(index) { index <= @stop }
        stop_condition = ->(index) { index < @start }
        accumulator = right_to_left_accumulator
      end

      until stop_condition.call(iterator.index)
        if within_bounds.call(iterator.index)
          size += accumulator.call(iterator.cursor.value, response)
        end

        iterator.next
      end

      response.prepend("*#{ size }\r\n")
    end

    def serialize
      serialize_with_accumulators(
        lambda do |value, response|
           response << RESPBulkString.new(value).serialize
           1
        end,
        lambda do |value, response|
           response.prepend(RESPBulkString.new(value).serialize)
           1
        end,
      )
    end
  end
end

listing 10.19 Updates to the ListSerializer class

We extracted most of the serialize method to the new serialize_with_accumulators method, which allows us to use the same overall logic, while being able to serialize individual list nodes differently. This is what we do in the SortedSetRankSerializer#serialize_list method.

Back to the serialize_list method in SortedSetRankSerializer, we create two different accumulators that are aware of the @withscores value and can decide whether or not to include it in the final serialized string when iterating over the list for serialization.

As a reminder, we give both a "left to right" and a "right to left" accumulator to ListSerializer so that it can use the most efficient way to serialize the list. Depending on the range it needs to serialize, it might decide to iterate from the right, if it'll be faster to find the range in the list.

Each of the accumulators now return an integer representing the number of elements added to the final response. This is necessary because the WITHSCORES option will force us to add two items in the final response for each item in the list, the member attribute, followed by the score attribute.

When iterating from the left, we'll encounter items in the order we want them to be in the final array, so we can first append the member value to response, and then, conditionally, append the score value. On the other hand, if we're iterating from right to left, we'll encounter items in the opposite order we want them to be in the final array, so we first, conditionally, prepend the score value, and then, always, prepend the member value. Let's look at an example with a small array to illustrate this.

If we have the array [ [ 10, 'a' ] , [ 20, 'b' ], [ 30, 'c' ], [ 40, 'd' ], [ 50, 'e' ] ], and we're requesting the range 1, 2, we should return the array [ 'b', 20, 'c', 30 ]. If we were requesting the range 2, 3, we should return the array [ 'c', 30, 'd', 40 ].

If we're iterating from left to right, as would be case with the range 1, 2, since it's faster to reach it from the left side, we would first encounter [ 20, 'b' ], and then [ 30, 'c' ], so we can append elements as we go, in the same order as the desired final order, member first, score, second.

On the other hand, if we're iterating from right to left, as would be the case with the range 2, 3, because it's faster to reach from the right side, we would first encounter [ 'd', 40 ] and then [ 'c', 30 ]. So when reaching the first element, we would first prepend the score and then the member, giving us the array [ 'd', 40 ], and we would do the same with the second pair, prepending 30, and then prepending 'c', giving us the desired result [ 'c', 30, 'd', 40 ].

As we did earlier, we can ignore the @reverse branch, that'll only be needed for the ZREVRANGE method.

By Score

Let's now add the ZRANGEBYSCORE command, which has the following format according to the Redis doc:

ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count]

min and max are a new type of range, a "score range", which we'll also represent with the GenericRangeSpec class.

Both values must be valid floats, that includes the values -inf and +inf, which happen to be the equivalent of 0 and -1 in the ZRANGE command, that is, a way to ask for all the elements in the sorted set:

127.0.0.1:6379> ZADD z 0.1 a 2 b 3.2 c -10 d
(integer) 4
127.0.0.1:6379> ZRANGEBYSCORE z -inf +inf
1) "d"
2) "a"
3) "b"
4) "c"

By default the values are considered inclusive, but this can be controlled with the prefix ( to mark it as exclusive, let's look at an example with the previous sorted set, z:

127.0.0.1:6379> ZRANGEBYSCORE z -inf 2
1) "d"
2) "a"
3) "b"
127.0.0.1:6379> ZRANGEBYSCORE z -inf (2
1) "d"
2) "a"

LIMIT and OFFSET are used to respectively limit the number of elements in the result and skip an arbitrary number of elements. A negative count value is the same as not passing a value and means "return all matches", negative offsets are accepted but will always result in an empty array.

Let's go ahead and add all that in sorted_set_commands.rb:

module BYORedis
  module SortedSetUtils
    def self.generic_range_by_score(db, args, reverse: false)
      # A negative count means "all of them"
      options = { offset: 0, count: -1, withscores: false }
      Utils.assert_args_length_greater_than(2, args)
      key = args.shift
      if reverse
        max = args.shift
        min = args.shift
      else
        min = args.shift
        max = args.shift
      end
      range_spec = Utils.validate_score_range_spec(min, max)
      parse_range_by_score_options(args, options) unless args.empty?

      sorted_set = db.lookup_sorted_set(key)
      if options[:offset] < 0
        EmptyArrayInstance
      elsif sorted_set
        options[:reverse] = reverse
        SortedSetSerializerBy.new(sorted_set, range_spec, **options, &:score)
      else
        EmptyArrayInstance
      end
    end

    def self.parse_limit_option(args, options)
      offset = args.shift
      count = args.shift
      raise RESPSyntaxError if offset.nil? || count.nil?

      offset = Utils.validate_integer(offset)
      count = Utils.validate_integer(count)
      options[:offset] = offset
      options[:count] = count
    end

    def self.parse_range_by_score_options(args, options)
      while arg = args.shift
        case arg.downcase
        when 'withscores' then options[:withscores] = true
        when 'limit' then SortedSetUtils.parse_limit_option(args, options)
        else raise RESPSyntaxError
        end
      end
    end

    # ...

  end

  # ...

  class ZRangeByScoreCommand < BaseCommand
    def call
      SortedSetUtils.generic_range_by_score(@db, @args, reverse: false)
    end

    def self.describe
      Describe.new('zrangebyscore', -4, [ 'readonly' ], 1, 1, 1,
                   [ '@read', '@sortedset', '@slow' ])
    end
  end
end

listing 10.20 The ZRangeByScoreCommand class

We're again anticipating the upcoming reverse command, ZREVRANGEBYSCORE in this case, and create generic_range_by_score to share the common logic, for now reverse is always false, so we can ignore the branches where it handles the true case.

We need to add the validate_score_range_spec method to the Utils module:

module BYORedis
  module Utils

    # ...

    def self.validate_score_range_spec(min, max)
      min, min_exclusive = parse_score_range_item(min)
      max, max_exclusive = parse_score_range_item(max)

      RedisSortedSet::GenericRangeSpec.score_range_spec(
        min, max, min_exclusive, max_exclusive)
    end

    def self.parse_score_range_item(str)
      if str[0] == '('
        str = str[1..-1]
        exclusive = true
      else
        exclusive = false
      end

      return validate_float(str, 'ERR min or max is not a float'), exclusive
    end
    private_class_method :parse_score_range_item
  end
end

listing 10.21 The Utils.validate_score_range_spec method

The new method in the Utils module, validate_score_range_spec, takes care of creating an instance of GenericRangeSpec, with the correct exclusive flags set depending on the presence of (. Let's add a method to create a range spec with the appropriate comparison function, passed as a block. We're also adding a new method, no_overlap_with_range? which will be used to determine when we can avoid to do any work if we can determine that no items match the requested range.

module BYORedis
  class RedisSortedSet

    # ...

    class GenericRangeSpec

      attr_reader :min, :max, :min_exclusive, :max_exclusive
      alias min_exclusive? min_exclusive
      alias max_exclusive? max_exclusive

      def self.score_range_spec(min, max, min_exclusive, max_exclusive)
        GenericRangeSpec.new(min, max, min_exclusive, max_exclusive) do |a, b|
          a <=> b
        end
      end

      # ...

    end

    # ...

    def no_overlap_with_range?(range_spec, &block)
      # Note that in that each condition the "value" is abstract and determined by the return
      # value of calling the block variable, in practice it's either score, member, or rank
      # There is no overlap under the four following conditions:
      # 1. the range spec min is greater than the max value:
      # set  : |---|
      # range:       |---| (min can be inclusive or exclusive, doesn't matter)
      # 2. the range spec min is exclusive and is equal to the max value
      # set  : |---|
      # range:     (---|   (min is exclusive)
      # 3. the min value is greater than range spec max
      # set  :       |---|
      # range: |---|       (max can be inclusive or exclusive, doesn't matter)
      # 4. the min value is equal to the range spec max which is exclusive
      # set  :     |---|
      # range: |---(       (max is exclusive)
      max_pair, max_pair_rank = max_pair_with_rank
      min_pair, min_pair_rank = min_pair_with_rank
      set_max_range_spec_min_comparison =
        range_spec.compare_with_min(block.call(max_pair, max_pair_rank))
      set_min_range_spec_max_comparison =
        range_spec.compare_with_max(block.call(min_pair, min_pair_rank))

      set_max_range_spec_min_comparison == -1 || # case 1
        (range_spec.min_exclusive? && set_max_range_spec_min_comparison == 0) || # case 2
        set_min_range_spec_max_comparison == 1 || # case 3
        (range_spec.max_exclusive? && set_min_range_spec_max_comparison == 0) # case 4
    end

    private

    # @return [Array] Two values, the first is a Pair, and the second is the rank
    def max_pair_with_rank
      case @underlying
      when List
        return @underlying.tail.value, @underlying.size
      when ZSet
        return @underlying.array[-1], @underlying.array.size - 1
      else raise "Unknown type for #{ @underlying }"
      end
    end

    # @return [Array] Two values, the first is a Pair, and the second is the rank
    def min_pair_with_rank
      case @underlying
      when List
        return @underlying.head.value, 0
      when ZSet
        return @underlying.array[0], 0
      else raise "Unknown type for #{ @underlying }"
      end
    end

    # ...

  end

  # ...
end

listing 10.22 The score_range_spec and no_overlap_with_range? methods

The new class method on GenericRangeSpec allows us to create a range specific to score ranges, which handles the exclusive boundaries.

The score_range_spec class method is almost identical to the rank_range_spec one from earlier, with the difference that it doesn't have to handle negative values, it just takes the score boundaries as in, as well as the exclusivity flags, which is something that we did not have to consider in the rank case, but the comparison of elements is the same, we're comparing numbers, Integer instances for ranks, BigDecimal instances for scores, and both can be compared with <=>, which is defined on both: Integer#<=> and BigDecimal#<=>.

If either of the boundaries is flagged as exclusive, then the range will be empty if they're equal. Looking at a Ruby example illustrates this pretty clearly:

irb(main):006:0> (0..0).to_a
=> [0]
irb(main):007:0> (0...0).to_a
=> []

The triple period notation is the exclusive range notation in Ruby, and as we can see the range "0 to 0, exclusive" is empty.

If neither of the boundaries is flagged as exclusive, then the only condition making the range empty is if min is greater than max, that is if the value of comparison is 1.

If either of the boundaries is marked as exclusive then a comparison result of 0, meaning equality, is enough to flag the range as empty.

This is the logic that is implemented in the empty? method. Let's also add the in_range? method so we can check for the inclusivity of an element in the range with awareness of the exclusivity of the min and max boundaries:

module BYORedis
  class RedisSortedSet

    # ...

    class GenericRangeSpec
      # ...

      def empty?
        comparison = compare_with_max(min)
        comparison > 0 || (comparison == 0 && (min_exclusive? || max_exclusive?))
      end

      def compare_with_min(element)
        @block.call(element, @min)
      end

      def in_range?(element)
        return false if empty?

        comparison_min = compare_with_min(element)
        comparison_max = compare_with_max(element)
        comparison_min_ok = min_exclusive? ? comparison_min == 1 : comparison_min >= 0
        comparison_max_ok = max_exclusive? ? comparison_max == -1 : comparison_max <= 0

        comparison_min_ok && comparison_max_ok
      end
    end

    # ...
  end
end

listing 10.23 The empty?, compare_with_min & in_range? methods for the GenericRangeSpec class

We can now use the same class for our different use cases, rank ranges and score ranges, let's quickly play with it in irb:

irb(main):001:0> require_relative './redis_sorted_set'
=> true
irb(main):002:0> range_spec_class = BYORedis::RedisSortedSet::GenericRangeSpec
irb(main):003:0> rank_range_spec = range_spec_class.rank_range_spec(2, 3, 10)
irb(main):004:0> rank_range_spec.in_range?(1)
=> false
irb(main):005:0> rank_range_spec.in_range?(4)
=> false
irb(main):006:0> rank_range_spec.in_range?(3)
=> true
irb(main):007:0> rank_range_spec = range_spec_class.rank_range_spec(2, -1, 10)
irb(main):008:0> rank_range_spec.in_range?(9)
=> true
irb(main):009:0> rank_range_spec.in_range?(10)
=> false
irb(main):010:0> score_range_spec = range_spec_class.score_range_spec(2, 10, true, false)
irb(main):011:0> score_range_spec.in_range?(2)
=> false
irb(main):012:0> score_range_spec.in_range?(2.1)
=> true
irb(main):013:0> score_range_spec.in_range?(10)
=> true

1 & 4 are out of range for the rank range spec 2, 3 within a set of size 10, that makes sense, only 2 and 3 are in range. We can see the transformation of negative indices when using the range 2, -1 in a set of size 10, the last rank is 9, the 0-based index of the last element, so 9 is in range, and 10 is not.

Switching to a score range, the last two boolean values determine the exclusivity of respectively the min and max boundaries, so the range we create is (2 10. We can see that 2 is out of range and 2.1 is in range, 10 is as well.

We created the range_spec_class variable only for convenience within the irb session.

Equipped with our new range spec constructor wrapper to create score range specs, the final class we need to add for this command is SortedSetSerializerBy:

module BYORedis
  # ...

  class SortedSetSerializerBy
    def initialize(sorted_set, range_spec,
                   offset: 0, count: -1, withscores: false, reverse: false, &block)
      @sorted_set = sorted_set
      @range_spec = range_spec
      @offset = offset
      @count = count
      @withscores = withscores
      @reverse = reverse
      if block.arity != 2
        @block = proc { |element, _| block.call(element) }
      else
        @block = block
      end
    end

    def serialize
      if @offset < 0 ||
         @range_spec.empty? ||
         @sorted_set.no_overlap_with_range?(@range_spec, &@block)

        return RESPArray.new([]).serialize
      end

      case @sorted_set.underlying
      when List then serialize_list
      when ZSet then serialize_zset
      else raise "Unknown type for #{ @underlying }"
      end
    end

    private

    def serialize_zset
      members = []

      if @reverse
        start_index = @sorted_set.underlying.array.last_index_in_range(@range_spec, &@block)
        if start_index.nil?
          raise "Unexpectedly failed to find last index in range for #{ self }"
        end

        indices = start_index.downto(0)
      else
        start_index = @sorted_set.underlying.array.first_index_in_range(@range_spec, &@block)
        if start_index.nil?
          raise "Unexpectedly failed to find first index in range for #{ self }"
        end

        indices = start_index.upto(@sorted_set.cardinality - 1)
      end

      indices.each do |i|
        item = @sorted_set.underlying.array[i]

        if @range_spec.in_range?(@block.call(item))
          if @offset == 0
            members << item.member
            members << Utils.float_to_string(item.score) if @withscores

            @count -= 1
            break if @count == 0
          else
            @offset -= 1
          end
        else
          break
        end
      end

      RESPArray.new(members).serialize
    end

    def serialize_list
      if @reverse
        iterator = List.right_to_left_iterator(@sorted_set.underlying)
      else
        iterator = List.left_to_right_iterator(@sorted_set.underlying)
      end
      members = []
      entered_range = false

      while iterator.cursor && @count != 0
        member = iterator.cursor.value

        if @range_spec.in_range?(@block.call(member))
          entered_range ||= true
          if @offset == 0
            members << member.member
            members << Utils.float_to_string(member.score) if @withscores

            @count -= 1
          else
            @offset -= 1
          end
        elsif entered_range == true
          break
        end

        iterator.next
      end

      RESPArray.new(members).serialize
    end
  end
end

listing 10.24 The serialize_zset & serialize_list methods for the SortedSetSerializerBy class

We start the serialize methods with three checks to return early if we can. In the first one, we test if the offset value is negative, if it is, we can return an empty array. We can also return early if the range is empty, or if the range and the set don't overlap.

Let's take a closer look at the no_overlap_with_range? method. The following illustration shows the cases in which we don't even need to go further, we can already know that there won't be any results.

There is no overlap under the four following conditions:

the range spec min is greater than the max value:

set  : |---|
range:       |---| (min can be inclusive or exclusive, doesn't matter)

the range spec min is exclusive and is equal to the max value:

set  : |---|
range:     (---|   (min is exclusive)

the min value is greater than range spec max:

set  :       |---|
range: |---|       (max can be inclusive or exclusive, doesn't matter)

the min value is equal to the range spec max which is exclusive:

set  :     |---|
range: |---(       (max is exclusive)

In other words, if the range is to the right, or to the left of the set, with special cases around exclusivity, we don't need to do any work, the result is empty.

One final word about no_overlap_with_range?, it calls the block with two arguments, the Pair instance itself, the first one in the set or the last one in the set, with their rank, which will allow us to use this method when dealing with range ranks, such as ZREMRANGEBYRANK for instance.

In practice the block.call expression will either return the score attribute, the member attribute or the rank value, for the pair. We could have written three methods, but the block based approach allows us to reuse the core logic for all three use cases.

Calling the block with two arguments might be an issue given that the block that was passed to the SortedSetSerializerBy constructor is &:score, what does passing multiple arguments to this block will do, let's take a look at that now.

A note about blocks, arity and Ruby being weird

The need for the if block.arity != 2 line in the SortedSetSerializerBy constructor deserves its own section, because it's kinda weird, Ruby does that sometimes. It's easier to start with what would happen, without this block, we would always set @block to block, which is &:score at this moment, a block that is almost the equivalent of { |x| x.score }, with a big difference, how it handles its arguments.

Both versions create a Proc, and not a Lambda, there are two differences, lambdas return or break calls only affect the lambda itself, whereas returning from a Proc returns from the outer method. You can see that by running lambda { return 1 }.call in irb, and see that it returns 1. This is because we create a lambda, call it, and get back the return value of the lambda, 1. Now, run the proc version, proc { return 1 }.call, and you'll get a LocalJumpError (unexpected return) exception back. This is because you can't return in irb. You can see that by typing return on a new line and you'll get the same error. This shows that calling return within the proc actually tried to return from the caller, the main irb session.

Blocks create procs and not lambdas, and this allows patterns such as returning early from the block, which is a pattern we use a lot in this chapter. Let's look at an example using a block, a proc and a lambda:

def a_method(proc_or_lambda)
  if block_given?
    yield
  else
    proc_or_lambda.call
  end
  'reached the end of a_method'
end


def call_with_block
  a_method(nil) { return 'nope' }
end

def call_with_lambda
  a_method(lambda { return 'nope' })
end

def call_with_proc
  a_method(proc { return 'nope' })
end

print "block: #{ call_with_block }\n"
print "lambda: #{ call_with_lambda }\n"
print "proc: #{ call_with_proc }\n"

Putting this in a file, say, proc_vs_block_vs_lambda.rb and running it with ruby proc_vs_block_vs_lambda.rb, we get the following result:

> ruby proc_vs_block_vs_lambda.rb
block: nope
lambda: reached the end of a_method
proc: nope

We only reached the end of a_method when using a lambda, because the return call only affected the lambda itself. This behavior is what allows us to do something like the following:

def first_item_or_nil(an_array)
  an_array&.each { |element| return element }
end

This method might not seem really useful, we could have used [0] to return the first item, but it will return nil for a nil array whereas calling [0] on a nil value will raise a NoMethodError exception. Beside the small safety gain with this method, the point being that we can return from within the block, and that will affect the method where the block was created, in this case, head_or_nil. We'll see a lot of actually useful cases later, such as in first_item_or_nil.

So blocks are procs, cool, now, the difference we care about here is how they handle different arguments being received:

irb(main):001:0> p = proc { |x| p x }
irb(main):002:0> p.call(1)
1
=> 1
irb(main):003:0> p.call(1, 2)
1
=> 1
irb(main):004:0> p = proc { |x, y| p x, y }
irb(main):005:0> p.call(1)
1
nil
=> [1, nil]
irb(main):006:0> p.call(1, 2)
1
2
=> [1, 2]
irb(main):007:0> l = lambda { |x| p x }
irb(main):008:0> l.call(1)
1
=> 1
irb(main):009:0> l.call(1, 2)
Traceback (most recent call last):
        5: from /Users/pierre/.rbenv/versions/2.7.1/bin/irb:23:in `<main>'
        4: from /Users/pierre/.rbenv/versions/2.7.1/bin/irb:23:in `load'
        3: from /Users/pierre/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/irb-1.2.3/exe/irb:11:in `<top (required)>'
        2: from (irb):10
        1: from (irb):8:in `block in irb_binding'
ArgumentError (wrong number of arguments (given 2, expected 1))
irb(main):010:0> l = lambda { |x, y| p x, y }
irb(main):011:0> l.call(1)
Traceback (most recent call last):
        5: from /Users/pierre/.rbenv/versions/2.7.1/bin/irb:23:in `<main>'
        4: from /Users/pierre/.rbenv/versions/2.7.1/bin/irb:23:in `load'
        3: from /Users/pierre/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/irb-1.2.3/exe/irb:11:in `<top (required)>'
        2: from (irb):12
        1: from (irb):11:in `block in irb_binding'
ArgumentError (wrong number of arguments (given 1, expected 2))
irb(main):012:0> l.call(1, 2)
1
2
=> [1, 2]

A Proc doesn't really care, if it needs more arguments that you give it, it defaults to nil, if it receives too many arguments, it just ignores them. A lambda on the other hand is like a method, it requires the exact number of arguments. Things are a bit difference when arguments have default values and become optional, but let's set that aside, it's not really what our problem is. Note that it's sometimes not explicit whether you're dealing with a lambda or a proc, but you can always use the lambda? method when in doubt.

Now back to &:member, what it does is actually a bit unclear to me, because the block that is created is defined in some internal Ruby code, in C, but what we can do is inspect the block with some of the methods available on Proc, namely #parameters & #arity, let's add this small snippet in a file called blocks.rb:

def inspect_ruby_block(&block)
  puts "parameters: #{ block.parameters }"
  puts "arity: #{ block.arity }"
  puts "lambda?: #{ block.lambda? }"
end

inspect_ruby_block { |x| x.score }
p '---'
inspect_ruby_block(&:score)

Let's run this with ruby blocks.rb:

parameters: [[:opt, :x]]
arity: 1
lambda?: false
"---"
parameters: [[:rest]]
arity: -1
lambda?: false

So neither are lambdas, cool, they should be handling arguments in a flexible way, but there's a difference, their arity is not the same. Arity is just a fancy word for "how many arguments do they expect". In the first example, the explicit block, the value is 1, which makes sense, we defined it as |x|, only one argument. But in the second example, it's -1, which happens to be kinda similar to how Redis communicates arity for its commands. -1 means that it is variadic, it accepts a variable number of arguments, but what makes this even weirder is the result of the parameters call. The one argument does not have a name.

We can mimic a similar block with the following:

irb(main):002:0> inspect_ruby_block { |*| }
parameters: [[:rest]]
arity: -1
lambda?: false
=> nil
irb(main):003:0> inspect_ruby_block { |*a| }
parameters: [[:rest, :a]]
arity: -1
lambda?: false
=> nil

How Ruby goes from a block like that, that seems to accept a variable number of arguments without naming them and is able to return the score attribute, or whatever we passed after the semicolon in &: is unclear to me, but it leads us to the problem we're trying to solve, this block, which is a proc, but of a slightly different kind, doesn't behave the way we want if we pass too many arguments.

Let's first create a method that returns the block it receives so that we can play with it. This is useful because the only way to create a block through the ampersand column approach is by passing it as an argument, so by returning it, we'll get a hold of the block in a variable:

def capture_block(&b)
  b
end

And now let's look at the differences in behavior:

irb(main):018:0> block = capture_block { |*| }
irb(main):019:0> block.call(1)
=> nil
irb(main):020:0> block.call(1, 2)
=> nil

So far, this is what we would expect, the block does nothing, and it silently accepts any number of arguments, but now let's take a look at the &:score approach:

irb(main):025:0> block = capture_block(&:score)
irb(main):026:0> block.call(Struct.new(:score).new(12))
=> 12

If we want to call block, we need an object that has a score method, so in one line we create a Struct, and instantiate it, and it works, so far so good. Now let's see what happens if we pass a second argument, which is what we want to do, call a block with a Pair and a number representing its rank:

irb(main):027:0> block.call(Struct.new(:score).new(12), 10)
Traceback (most recent call last):
        5: from /Users/pierre/.rbenv/versions/2.7.1/bin/irb:23:in `<main>'
        4: from /Users/pierre/.rbenv/versions/2.7.1/bin/irb:23:in `load'
        3: from /Users/pierre/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/irb-1.2.3/exe/irb:11:in `<top (required)>'
        2: from (irb):27
        1: from (irb):27:in `score'
ArgumentError (wrong number of arguments (given 1, expected 0))

Yup, it blows up, and honestly, I can't really tell you why. My guess? By being defined in C, it is able to use the low level APIs of the language, and can say "give me the first argument, and call the method named after the argument I was created with on it". It's also interesting to look specifically at the error we get.

It gives us a hint, wrong number of arguments (given 1, expected 0), for score, so it seems like it tried to pass 10, the second argument to score, which is a getter and only takes a single argument, so the &: approach can apparently be used for methods that require arguments, let's confirm this by creating this class in irb:

class A
  def a_variadic_method(*args)
    p args
  end
end

The method a_variadic_method is variadic, it accepts any number of arguments, and it prints them, let's now use this with a block created with the &: approach:

irb(main):012:0> b = capture_block(&:a_variadic_method)
irb(main):013:0> b.call(A.new)
[]
=> []
irb(main):014:0> b.call(A.new, 1, 2, 3)
[1, 2, 3]
=> [1, 2, 3]

We capture a block that will call a_variadic_method on its argument and first call it with an instance of A, and get back an empty array, it's because no arguments were given to a_variadic_method, so args was set to []. Now let's add arguments to the block.call line, and we see that we get them back as an array, because they were passed as arguments! So what did we learn here, that a block created with &: is variadic and requires at least one argument, to call the method it was created with on, and if it receives any other arguments, it will pass them as arguments to the method.

So anyway, what can we do? Well, we can create a block that accepts the number of arguments we need it to receive if we know it otherwise wouldn't be able to handle them. Reusing the same block variable from the previous example defined as block = capture_block(&:score):

irb(main):031:0> wrapper_block = proc { |pair, rank| block.call(pair) }
irb(main):032:0> wrapper_block.call(Struct.new(:score).new(12), 10)
=> 12
irb(main):033:0> wrapper_block.call(Struct.new(:score).new(12))
=> 12

The second argument of the block is ignored, and we can pass a single argument! Just what we wanted!

It's worth noting that we could have also changed the behavior when creating a SortedSetSerializerBy, with:

SortedSetSerializerBy.new(sorted_set, range_spec, **options) { |x, _| x.score }

But I do believe that it is convenient to be able to only pass a single argument, since that's really what this block is saying: "This block, which is used to extract a variable from a Pair returns its score". It would be a little bit annoying to have to worry about a second argument, and ignoring it.

Serializing a `ZSet`

The two methods dedicated to serializing the ZSet or the List are pretty long, because they're written in a generic way that will allow them to be reused for the *REV* commands and the *LEX* commands. Let's start with serialize_zset, once again ignoring the @reverse flag for now, and assuming it to always be false.

We start by creating the members array, which will hold all the values we need to serialize for the final result.

We then find the index of the first item that fits in the range with the SortedArray#first_index_in_range method, and use it to create the indices enumeration, which contain all the indices we want to inspect. This step is a way to skip all the elements to left of the the first element that fits in the range by directly "jumping" to the first item in the range. This approach is a big win for a large set, where for instance, jumping to an element in the middle of the array would require way less steps that iterating one by one from the beginning of the array.

We then iterate through the indices and inspect each Pair we find. Using the in_range? method, we can determine whether or not the current element is in the range. Note that we use the block variable, which was set to &:score. As discussed earlier, this notation is a shortcut for { |x| x.score }. This approach makes the SortedSetSerializerBy agnostic of which fields we're serializing by, the caller gets to decide by passing the right block.

The in_range? method will always return true for the first element, because first_index_in_range pointed us as the first element in range, but as we keep iterating to the right in the array, we might end up encountering an element that is outside the range, in which case we want to stop iterating.

We also need to handle the COUNT and OFFSET options. Because of the OFFSET option, we might need to ignore the first items we find, so if the current item is in the range, we decrement @offset until it reaches 0. If its value is 0 or less, then we start accumulating elements in members, with or without their score, depending on the WITHSCORES option. For each element we accumulate, we decrement @count, and break once it reaches 0, telling us we've accumulated enough elements. This logic relies on the fact that we use a default value of -1 for @count, meaning that by default we'll keep decrementing, and it will have no effect since its value will never reach 0.

As a quick note, the last sentence is true in Ruby, an integer that is initialized at -1 cannot possibily reach 0 if we keep decrementing, but it is false in a language where integers can overflow, like C! We've already spent a lot of time talking about integers in the previous chapter, so we'll only say that once a signed integer reaches the minimul value its encoding allows, subtracting 1 will become the maximum of the range, and we keep subtracting, we'll reach 0 again. The following program outputs: reached 0 (proof: 0) in 65535 steps:

#include <stdio.h>

int main(void) {
  short i = -1;
  int steps = 0;
  while (i != 0) {
    i--;
    steps++;
  }
  printf("reached 0 (proof: %i) in %i steps", i, steps);
  return 0;
}

listing 10.25 A C example of integer overflow and the wrapping behavior

Let's add the first_index_in_range method to SortedArray:

module BYORedis
  class SortedArray

    # ...

    def first_index_in_range(range_spec)
      return nil if empty?

      @underlying.bsearch_index do |existing_element|
        compare = range_spec.compare_with_min(yield(existing_element))
        if range_spec.min_exclusive?
          compare > 0 # existing_element.score > min
        else
          compare >= 0 # existing_element.score >= min
        end
      end
    end
  end
end

listing 10.26 The SortedArray#first_index_in_range method

This method is a generic approach to using the following block:

array.bsearch_index do |pair|
  pair.score >= range_spec.min # or pair.score > range_spec.min if exclusive
end

Using the block approach will allow us to use the same method, but with the member value instead, in the lex command below, and using the compare_with_min method handles the specifities of the comparison logic. Things are not too complicated for now, we're comparing two float values with a <=> b, but will become a bit trickier in the lex case with the + & - special values, just a heads-up!

Note that the block we're dealing with here is, in this case, the one passed to the constructor of SortedSetSerializerBy, but after being wrapped in a proc ignoring its second argument, so calling yield with a single argument will not cause any issues.

The use of the comparison instead of directly using the > or < operator adds an extra level of indirection that can make things a little bit confusing, that being said, it does allow for a greater level of reusability, so its readability cost comes with some benefit. Let's look at an example to see what the first_index_in_range really does:

irb(main):001:0> require_relative './sorted_array'
=> true
irb(main):002:0> require_relative './redis_sorted_set'
=> true
irb(main):003:0> array = BYORedis::SortedArray.by_fields(:score, :member)
irb(main):004:0> array << BYORedis::RedisSortedSet::Pair.new(1, 'a')
=> [#<struct BYORedis::RedisSortedSet::Pair score=1, member="a">]
irb(main):005:0> array << BYORedis::RedisSortedSet::Pair.new(2, 'b')
=> [#<struct BYORedis::RedisSortedSet::Pair score=1, member="a">, ...]
irb(main):006:0> array << BYORedis::RedisSortedSet::Pair.new(3, 'c')
=> [#<struct BYORedis::RedisSortedSet::Pair score=1, member="a">, ...]
irb(main):007:0> array << BYORedis::RedisSortedSet::Pair.new(4, 'd')
=> [#<struct BYORedis::RedisSortedSet::Pair score=1, member="a">, ...]
irb(main):008:0> array << BYORedis::RedisSortedSet::Pair.new(26, 'z')
=> [#<struct BYORedis::RedisSortedSet::Pair score=1, member="a">, ...]
irb(main):009:0> range_spec = BYORedis::RedisSortedSet::GenericRangeSpec.score_range_spec(2, 4, false, false)
irb(main):010:0> array.first_index_in_range(range_spec) { |x| x.score }
=> 1

This example shows that, at least in this one example, it worked, 1 is indeed the first element of the array that is in the range 2 4. The other indices of members in the range are 2 and 3. The trick relies once again on bsearch_index, which can be a really powerful once you get the hang of it. In this example we rely on its find-minimum mode. In this mode we need to give it a block that returns the following values according to the documentation:

In find-minimum mode (this is a good choice for typical use cases), the block must always return true or false, and there must be an index i (0 <= i <= ary.size) so that:

the block returns false for any element whose index is less than i, and

the block returns true for any element whose index is greater than or equal to i.

In other words, the method will return the smallest index, which can also be seen as the leftmost index, for which the block is true. Or, alternatively phrased, the index after the greatest index for which the block is false.

Equipped with this definition, we can use it to find the first element that fits in the range, with the block { |x| x > min }, the block will be false if it finds an element in the array equal to the min value of the range, and it will therefore return the index of the element after that, which is the first element of the range if the exclusive flag is set for the min value. Looking at it from the other angle, the block will be true for all values greater than min, and false for all values lower than min, including min itself.

The block { |x| x >= min } will return true as long the element being inspected by bsearch_index is greater or equal than the minimum value, so it if finds an element in the array equal to the min, it will return that index, otherwise, it will return the index of the smallest value that is still greater than the min. That's the index of the first element in the range if min is not exclusive

Now let's look at the convoluted way in which first_index_in_range ends up achieving exactly what we described. As mentioned earlier, in order to keep the method abstract, we refuse to call existing_element.score directly, and instead delegate to the caller to decide what attribute to use, which is what yield(existing_element) does. In this example, the block given is @block, in the line @sorted_set.underlying.array.first_index_in_range(@range_spec, &@block), with the ampersand to tell Ruby it's not just a regular argument, it's a block. @block is the value of the block given to the constructor, which was done in the SortedSetUtils.generic_range_by_score method and is &:score, the equivalent of { |x| x.score }. We use the value returned by yield, and feed it to compare_with_min, which calls score <=> min.

The value returned by compare_with_min will be 1 if existing_element.score > range_spec.min, 0 if they're equal and -1 if existing_element.score < range_spec.min.

So now we can look at what we return from the block in first_index_in_range, in the min_exclusive? case we return compare > 0, which is the same as returning existing_element.score > range_spec.min, and in the non exclusive case, compare >= 0 is the equivalent of existing_element.score >= range_spec.min, these are the values that will lead bsearch_index to return the index of the first element in the range, min being exclusive or not.

Why are we jumping through all these hoops? Well, it'll make writing the next similar commands a breeze!

Serializing a `List`

We've looked at the serialize_zset private method in SortedSetSerializerBy, let's now look at serialize_list. Once again, let's set the @reverse branches aside and assume it always holds its default value for now, false.

We start by creating a left to right iterator, since we have no way to know where the first element in the range is in the list, we'll start from the left and iterate from there.

Similarly to the ZSet case, we create an array, members, which we'll use to aggregate the values as we find them, we also create a boolean that will help us exiting the loop early and prevent to unnecessarily iterate through the whole list if we can avoid it.

We start iterating with the iterator, as well as with the while condition @count != 0. This is the same optimization as earlier, if a count was specified, then we'll stop iterating once we've accumulated enough elements, but if no count value is given, then the value will start at -1 and decrement from there, and this condition will never trigger.

For each element in the look, we ask the range if it is in the range, with @range_spec.in_range?(@block.call(member)). If this returns true, we first flag entered_range to true, which we only do once, there's no need to set the value again once we've done it. Next, we take the @offset value into account, the same way we did in the ZSet case, and if we've skipped enough values as indicated by @offset, then we start accumulating values, with or without scores, depending on the @withscores option. We always decrement the count value, to make sure that the while condition we just mentioned stops the loop once we've accumulated enough elements.
If the element is not in the range, then we also check if we have previously entered the range, and if we did, then it means that we just found the first element that is outside the range. This tells us that we've found the last element that could be in the range, there's no point in continuing from now on and we exit the loop.

If we have not exited the loop early, then we keep going through the list. Once the whole iteration is done, we return the serialized members array.

By Lexicographic Order

Let's add the last ZRANGE* command, ZRANGEBYLEX, but before looking at the ZRangeByLexCommand class, we need to spend some time looking at what a "lex range" is.

First things first, the same way numbers can be sorted, strings can be sorted. Well, technically, as we've seen before, string are really array of bytes under the hood, and bytes are numbers, kinda. At least bytes have a numeric value, that's the accurate way of describing it. 'a' is 97, 'b' is 98, etc ... but that's an implementation detail! We can look at a dictionary to see a real life example. A come before B, and so on, and if there is a tie, we compare the next letter, and the next, and if there's still a tie, the shorter string is always the shortest one, that makes 'aa' < 'aaa' a true comparison. Lower case letters are considered greater than their upper case equivalent, the following is true, 'A' < 'a'. And for reference, 'A'.ord == 65, 65 is indeed lower than 97.

At its core, that's what a lex range is, instead of using numbers like we've done before, such as 0 2, or 0 -1, a lex range is expressed as a z, or aa aaa. And the same way that 2 0 was nonsensical when used to return the range in a list, something like d a is nonsensical as well, these are empty ranges, there cannot be any elements that fits in them.

There are two more things we need to cover about lex ranges, exclusivity with [ & ( and special values with - & +.

Let's start with the bracket and parenthesis first. Range boundaries need to be explicitly marked as inclusive or exclusive, [ means inclusive, and ( means exclusive. Note that these characters are mandatory. Let's look at few examples:

127.0.0.1:6379> ZADD z 0 a 0 b 0 c 0 d
(integer) 4
127.0.0.1:6379> ZRANGEBYLEX z (a (d
1) "b"
2) "c"
127.0.0.1:6379> ZRANGEBYLEX z [a (d
1) "a"
2) "b"
3) "c"
127.0.0.1:6379> ZRANGEBYLEX z [d [a
(empty array)
127.0.0.1:6379> ZRANGEBYLEX z [aa (d
1) "b"
2) "c"
127.0.0.1:6379> ZRANGEBYLEX z [d [z
1) "d"
127.0.0.1:6379> ZRANGEBYLEX z (d [z
(empty array)

The set z contains the four members a, b, c & d. The first ZRANGEBYLEX calls is requesting all the elements between a and d, excluding both of them, and it returns b & c. The second example marks a as inclusive and the results includes it. The following example shows an instance of an empty range, d is greater than a, so that range does not make any sense. The next example shows how the string 'aa' is considered greater than 'a' which explains why 'a' is not returned.
In the second to last example d is marked as inclusive and is the only element returned, in the last example d is marked as exclusive and is not returned.

Finally, - and + play a role similar to -inf and +inf in the ZRANGEBYSCORE method, they can be used to specify respectively a value this lower than any other values, and a value that is greater than any other values. A different with the infinity values is what while a score can have the value -inf or inf, no strings can actually have this special value. You could set a member with the value - or +, but that's not the same, these are the characters + and -. You could request them with: ZRANGEBYLEX z [+ [- but that's different from requesting ZRANGEBYLEX z - +. As a quick aside, - is greater than +, they have the values 43 & 45. The character '*' has the value 42, so we could use it:

127.0.0.1:6379> ZADD z 0 +
(integer) 1
127.0.0.1:6379> ZADD z 0 -
(integer) 1
127.0.0.1:6379> ZRANGEBYLEX z - +
1) "+"
2) "-"
127.0.0.1:6379> zrangebylex z [+ [-
1) "+"
2) "-"
127.0.0.1:6379> zrangebylex z [* [-
1) "+"
2) "-"

I'm not sure if you'd ever want to use these values in a range, because that is pretty cryptic, but it is important to understand how these ranges work.

The ZRANGEBYLEX command has the following format according to the Redis documentation:

ZRANGEBYLEX key min max [LIMIT offset count]

module BYORedis
  module SortedSetUtils

    # ...

    def self.generic_range_by_lex(db, args, reverse: false)
      # A negative count means "all of them"
      options = { offset: 0, count: -1 }
      Utils.assert_args_length_greater_than(2, args)
      key = args.shift
      if reverse
        max = args.shift
        min = args.shift
      else
        min = args.shift
        max = args.shift
      end
      range_spec = Utils.validate_lex_range_spec(min, max)
      parse_range_by_lex_options(args, options) unless args.empty?

      sorted_set = db.lookup_sorted_set(key)
      if options[:offset] < 0
        EmptyArrayInstance
      elsif sorted_set
        options[:withscores] = false
        options[:reverse] = reverse
        SortedSetSerializerBy.new(sorted_set, range_spec, **options, &:member)
      else
        EmptyArrayInstance
      end
    end

    def self.parse_range_by_lex_options(args, options)
      raise RESPSyntaxError unless args.length == 3

      if args.shift.downcase == 'limit'
        SortedSetUtils.parse_limit_option(args, options)
      else
        raise RESPSyntaxError
      end
    end
  end

  # ...

  class ZRangeByLexCommand < BaseCommand
    def call
      SortedSetUtils.generic_range_by_lex(@db, @args, reverse: false)
    end

    def self.describe
      Describe.new('zrangebylex', -4, [ 'readonly' ], 1, 1, 1,
                   [ '@read', '@sortedset', '@slow' ])
    end
  end
end

listing 10.27 The ZRangeByLexCommand class

The generic_range_by_lex method will be useful when we add the reverse variant, ZREVRANGEBYLEX, which is why it has a reverse flag, which defaults to false. We need to validate that min and max are valid lex range items, and if they are, we create an instance of the range spec class specific to lexicographic order, with GenericRangeSpec.lex_range_spec:

module BYORedis
  module Utils

    # ...

    def self.validate_lex_range_spec(min, max)
      min_string, min_exclusive = parse_range_item(min)
      max_string, max_exclusive = parse_range_item(max)

      RedisSortedSet::GenericRangeSpec.lex_range_spec(
        min_string, max_string, min_exclusive, max_exclusive)
    end

    def self.parse_range_item(item)
      if item == '+'
        [ '+', true ]
      elsif item == '-'
        [ '-', true ]
      elsif item[0] == '['
        [ item[1..-1], false ]
      elsif item[0] == '('
        [ item[1..-1], true ]
      else
        raise ValidationError, 'ERR min or max not valid string range item'
      end
    end
    private_class_method :parse_range_item
end

listing 10.28 The Utils.validate_lex_range_spec class

validate_lex_range_spec checks the format of both range items with parse_range_item. In this method we check all the edge cases. + and - are considered exclusive since no values can actually be equal to them. Let's now create a new class method on GenericRangeSpec to create a range spec that uses lex comparison as its ordering mechanism:

module BYORedis
  class RedisSortedSet

    # ...

    class GenericRangeSpec

      # ...

      def self.lex_range_spec(min, max, min_exclusive, max_exclusive)
        GenericRangeSpec.new(min, max, min_exclusive, max_exclusive) do |a, b|
          RedisSortedSet.lex_compare(a, b)
        end
      end

      # ...

    end

    # ...

    def self.lex_compare(s1, s2)
      return 0 if s1 == s2
      return -1 if s1 == '-' || s2 == '+'
      return 1 if s1 == '+' || s2 == '-'

      s1 <=> s2
    end

    # ...

  end
  # ...
end

listing 10.29 The GenericRangeSpec.lex_range_spec class method

The comparison of lex items is a bit trickier than previous comparisons we've had to deal with previously and is why we previously created the compare_with_min and compare_with_max methods. It is implemented in the lex_compare and essentially follows these rules:

If both strings are equal, return 0
Otherwise, if the first string is -, or if the second string is +, then the first string is smaller than the second one
Otherwise, if the first string is + or the second string is -, then the first string is greater than the second string.
Otherwise, compare the strings the "regular" way, with <=>, that is b is greater than aa, which is greater than a, etc ... This is the String#<=> method.

By using this new comparison system, but still returning -1, 0, and 1 depending on the order, we can reuse all the code from GenericRangeSpec, neat!.

And now that we've looked at this new way of using GenericRangeSpec, let's look at all the pieces falling into place.

Back in SortedSetUtils.generic_range_by_lex, we now call SortedSetSerializerBy with the sorted set, the range spec, the parsed options, namely LIMIT and OFFSET, and the block &:member, the equivalent of { |x| x.member }.

We've spent a good amount of time talking about why we wrote SortedSetSerializerBy in a way that never explicitly called .score or .member, or even knew how to compare elements, and this is about to pay off.

The serialize method in SortedSetSerializerBy can now return early if the range spec is empty, and the range spec is now a GenericRangeSpec created with RedisSortedSet.lex_compare(a, b) as the comparison block, so it will know how to compare the min and max elements and determine if the lex range is empty. It will also call no_overlap_with_range? with the block we gave it, which extracts the member attribute from a pair, and will again return early, if it determines that there is no overlap between ranges.

Both specific methods, serialize_zset and serialize_list, function the exact same way as we explored in ZRANGEBYSCORE, the range spec says whether or not an item is in range, and if so they're accumulated, with or without their scores, in an array and returned to the user.

And we'll be able to reuse a lot of this code soon with the *REV* commands too!

Set Operations, Union & Intersection

Sorted sets support set operation commands similar to the ones we added in the previous chapter, the SINTER and SUNION methods in the previous chapters, as well as their *STORE variants.

As of Redis 6.0 there is no ZDIFF and ZDIFFSTORE commands, but it might be added soon as there is an active Pull Request as of November 2020: https://github.com/redis/redis/pull/7961. It should be added in 6.2.0.

The topic of the ZDIFF command has been discussed for a while, an earlier PR dates from April 2012: https://github.com/redis/redis/pull/448

The initial resistance to adding a ZDIFF commands has been discussed in the mailing list and can be summarized by this 2010 comment from Redis developer Pieter Noordhuis:

ZDIFFSTORE makes no sense, as discussed before on the ML (please search before posting). The intrinsic value of the scores gets lost when you simply start subtracting them

As we're about to see, both ZINTER and ZUNION offer mechanisms to let users decide what happens to the score values from multiple sets and how to combine them. On the other hand, things would be different with ZDIFF as values could be completely discarded.

Set Intersection

Let's look at ZINTER first. Note that ZINTER was only added in Redis 6.2.0, in earlier versions only ZINTERSTORE was available.

127.0.0.1:6379> ZADD z1 0 a 1 b 2 c
(integer) 3
127.0.0.1:6379> ZADD z2 10 z 5 x 3 c
(integer) 3
127.0.0.1:6379> ZINTER 2 z1 z2
1) "c"
127.0.0.1:6379> ZINTER 2 z1 z2 WITHSCORES
1) "c"
2) "5"

ZINTER requires the first argument to be the number of sets that will be given next. Without this explicit count argument, the command ZINTER z1 z2 WITHSCORES would be ambiguous, what if there is a set at the key WITHSCORES, did you mean the intersection of the three sets z1, z2 & WITHSCORES, or the intersection of z1 and z2 with the option WITHSCORES?

The numkeys argument prevents this and makes the number of set explicit.

We can see in the previous example that the score of the only element in the result set is 5, which is the sum of the score for member c in z1 and the score for member c in z2. The aggregation of scores default to SUM and can be controller with the AGGREGATE option. The other two possible values are MIN and MAX:

127.0.0.1:6379> ZINTER 2 z1 z2 WITHSCORES AGGREGATE MAX
1) "c"
2) "3"
127.0.0.1:6379> ZINTER 2 z1 z2 WITHSCORES AGGREGATE MIN
1) "c"
2) "2"

The last option to ZINTER is WEIGHTS, it must be followed by a list of valid float values, of length equal to numkeys. In other words, if you explicit pass the WEIGHTS option, you need to give a weight for each set we're computing the intersection of. The implicit default value of the weights is 1. When combining values in the final set, each score will be multiplied by the weight given to its set:

127.0.0.1:6379> ZINTER 2 z1 z2 WITHSCORES WEIGHTS 1 1
1) "c"
2) "5"
127.0.0.1:6379> ZINTER 2 z1 z2 WITHSCORES WEIGHTS 1 2
1) "c"
2) "8"
127.0.0.1:6379> ZINTER 2 z1 z2 WITHSCORES WEIGHTS 1 3
1) "c"
2) "11"
127.0.0.1:6379> ZINTER 2 z1 z2 WITHSCORES WEIGHTS 10 3.2
1) "c"
2) "29.600000000000001"
127.0.0.1:6379> ZINTER 2 z1 z2 WITHSCORES WEIGHTS 10 inf
1) "c"
2) "inf"

As we can see, WEIGHTS 1 1 returns the same as if we had omitted the weights, 5. With WEIGHTS 1 2, all the members from the second set, z2 will have their weight multiplied by 2 before being aggregated, so 2 * 1 + 3 * 2 == 8, and in the next example we end up at 11 with 2 * 1 + 3 * 3.

The next example show that weights can be float, so 2 * 10 + 3 * 3.2 ~= 29.6.

In the last example, we end up with inf because any values multiplied by infinity equals infinity.

The weight values are applied before the aggregation:

127.0.0.1:6379> ZINTER 2 z1 z2 WITHSCORES WEIGHTS 100 10 AGGREGATE MAX
1) "c"
2) "200"
127.0.0.1:6379> ZINTER 2 z1 z2 WITHSCORES WEIGHTS 100 10 AGGREGATE MIN
1) "c"
2) "30"
127.0.0.1:6379> ZINTER 2 z1 z2 WITHSCORES WEIGHTS 100 10 AGGREGATE SUM
1) "c"
2) "230"

200 is the result of MAX((2 * 100), (3 * 10)), 30 is the result of MIN((2 * 100), (3 * 10)) and 230 is the result of SUM((2 * 100), (3 * 10)).

Finally, ZINTER accepts regular sets, and assumes a score of 1 for each score-less set members.

127.0.0.1:6379> ZINTER 3 z1 z2 s WITHSCORES
1) "c"
2) "6"
127.0.0.1:6379> ZINTER 3 z1 z2 s WITHSCORES WEIGHTS 0 0 1
1) "c"
2) "1"

Let's now create the ZInterCommand class, but first, since we already know we have three more very similar commands coming up soon, ZINTERSTORE, ZUNION & ZUNIONSTORE, let's go ahead and add methods in SortedSetUtils in order to reuse code across these commands.

The Redis documentation describes the format of the command as:

ZINTER numkeys key [key ...] [WEIGHTS weight [weight ...]] [AGGREGATE SUM|MIN|MAX] [WITHSCORES]

module BYORedis
  module SortedSetUtils

    # ...

    def self.intersection(db, args)
      set_operation(db, args) do |sets_with_weight, aggregate|
        RedisSortedSet.intersection(sets_with_weight, aggregate: aggregate)
      end
    end

    def self.set_operation(db, args)
      options = { aggregate: :sum, withscores: false }
      sets = SortedSetUtils.validate_number_of_sets(db, args)
      options.merge!(SortedSetUtils.parse_union_or_inter_options(args, sets.size))

      sets_with_weight = sets.zip(options[:weights])
      new_set = yield sets_with_weight, options[:aggregate]

      return new_set, options[:withscores]
    end

    def self.set_operation_command(args)
      Utils.assert_args_length_greater_than(1, args)
      set_result, withscores = yield

      SortedSetRankSerializer.new(
        set_result,
        RedisSortedSet::GenericRangeSpec.rank_range_spec(0, -1, set_result.cardinality),
        withscores: withscores,
      )
    end

    def self.parse_union_or_inter_options(args, number_of_sets)
      options = { weights: Array.new(number_of_sets, 1) }
      while arg = args.shift
        case arg.downcase
        when 'weights'
          options[:weights] = validate_weights(number_of_sets, args)
        when 'aggregate'
          aggregate_mode = args.shift
          case aggregate_mode&.downcase
          when 'min' then options[:aggregate] = :min
          when 'max' then options[:aggregate] = :max
          when 'sum' then options[:aggregate] = :sum
          else raise RESPSyntaxError
          end
        when 'withscores' then options[:withscores] = true
        else raise RESPSyntaxError
        end
      end

      options
    end

    def self.validate_number_of_sets(db, args)
      number_of_sets = Utils.validate_integer(args.shift)
      if number_of_sets <= 0
        raise ValidationError, 'ERR at least 1 input key is needed for ZUNIONSTORE/ZINTERSTORE'
      else
        number_of_sets.times.map do
          set_key = args.shift
          raise RESPSyntaxError if set_key.nil?

          db.lookup_sorted_set_or_set(set_key)
        end
      end
    end

    def self.validate_weights(number_of_sets, args)
      number_of_sets.times.map do
        weight = args.shift
        raise RESPSyntaxError if weight.nil?

        Utils.validate_float(weight, 'ERR weight value is not a float')
      end
    end
  end

  # ...

  class ZInterCommand < BaseCommand
    def call
      SortedSetUtils.set_operation_command(@args) do
        SortedSetUtils.intersection(@db, @args)
      end
    end

    def self.describe
      Describe.new('zinter', -3, [ 'readonly', 'movablekeys' ], 0, 0, 0,
                   [ '@read', '@sortedset', '@slow' ])
    end
  end
end

listing 10.30 The ZInterCommand class

The ZInterCommand is surprisingly short, it essentially amounts to the two statements: "Perform a regular set operation, no store, and use the intersection operation". Using this level of abstraction will allow us to write the following commands in a very concise way as well, but let's look at what SortedSetUtils.set_operation_command and SortedSetUtils.intersection do.

set_operation_command is the method that performs the first validation, we need at least two arguments, the number of sets, numkeys and at least one set key. It then uses yield to give back the values of the actual result, as well as the options to the caller, the call method. The results from yield, which we'll look at shortly, are then fed to the SortedSetRankSerializer class, the same class we used to implement the ZRANGE command. With the range from 0 to -1 this serializer will serialize the whole set, in rank order, which is what we need to return. The value of withscores, as well as set_result, come from calling yield, which returns SortedSetUtils.intersection(@db, @args) as we can see in the ZInterCommand class. This intersection method is a wrapper around RedisSortedSet.intersection, with a few additional steps, such as parsing all the remaining options, WEIGHT, AGGREGATE and WITHSCORES.

These steps are performed in the SortedSetUtils.set_operation, which first calls the methods validate_number_of_sets, to check that we have the correct number of keys after the numkeys argument, and loads all the set with DB#lookup_sorted_set_or_set.

We need this new method because this commands and the other set union commands for sorted sets accept regular sets as arguments as well:

module BYORedis
  class DB

    # ...

    def lookup_sorted_set_or_set(key)
      set = @data_store[key]
      raise WrongTypeError if set && !set.is_a?(RedisSortedSet) && !set.is_a?(RedisSet)

      set
    end
  end
end

listing 10.31 The DB#lookup_sorted_set_or_set method

Going back to set_operation, the next step is calling parse_union_or_inter_options. At this time we've parsed the value of numkeys so we know how many sets to expect, which means that if the WEIGHTS option is present, we have to validate that the number of weights matches the numkeys value. This validation is performed in the validate_weights method. We generate the default weight values with the line Array.new(number_of_sets, 1). It initializes an array for which the size is number_of_sets, and all the elements are 1.

The other options we need to check for are aggregate, in which case the next argument must be one of min, max, or sum, and withscores, if it is present, we set the corresponding flag to true.

The parse_union_or_inter_options is a little bit simpler than the parse_options method in the ZAddCommand class because the options come at the end of the argument list here. This allows us to skip the whole "peek and only consume if the head is an option" approach we used there. Here we can shift as long as we find valid options.

The `Array#zip` method

Once the weight values have been parsed into BigDecimal instances, we combine them with the set objects with the Array#zip method. This is a method we haven't used so far so let's take a quick look at what it does, sure, it "zips" things, but what does it mean?

irb(main):001:0> a1 = [1,2,3,4]
irb(main):002:0> a2 = ['a','b','c','d']
irb(main):003:0> a1.zip(a2)
=> [[1, "a"], [2, "b"], [3, "c"], [4, "d"]]

zip is called on an array and the arguments are one or more arrays, the result is ... also an array, where each element is ... another array, which is the combination of elements from all the arrays at matching indices! Yup, that's a lot of arrays.

Let's look at what happens with three arrays:


irb(main):004:0> a3 = [true, false, true, false]
irb(main):005:0> a1.zip(a2, a3)
=> [[1, "a", true], [2, "b", false], [3, "c", true], [4, "d", false]]

We can even reuse zip an array with itself:

irb(main):006:0> a1.zip(a2, a3, a1)
=> [[1, "a", true, 1], [2, "b", false, 2], [3, "c", true, 3], [4, "d", false, 4]]

Lastly, let's look at the behavior when the lengths don't exactly match:

irb(main):007:0> a1.zip([ true ])
=> [[1, true], [2, nil], [3, nil], [4, nil]]
irb(main):008:0> [true].zip(a1)
=> [[true, 1]]

The final result is always the same length as the Array we called zip on, and nil values are thrown in there if the other arrays are too small.

Finalizing the set intersection operation

Back to the our use of the zip method in set_operation, we have an Array of RedisSortedSet or RedisSet and an Array of BigDecimal, and we zip them together in an array of pairs. This will allow us to iterate over the pairs and each element will contain a set, either sorted or not, and its weight.

The next line in set_operation is new_set = yield sets_with_weight, options[:aggregate], where we delegate the work of actually performing the operation, union, or intersection to the caller, in our case, the block SortedSetUtils.intersection method, which calls RedisSortedSet.intersection, with the sets, weights and options that set_operation took care of parsing for us.

Once the set operation is performed, the whole stack unravels and the set_operation_command ends up back in control, it finally has a value for set_result, and feeds it to SortedSetRankSerializer, which is returned by ZInterCommand#call.

That was a lot of blocks, yields, method calls, so let's summarize it:

ZInterCommand#call calls SortedSetUtils.set_operation_command with a block
SortedSetUtils.set_operation_command validates the length of the argument list and yields back to ZInterCommand#call
ZInterCommand#call calls SortedSetUtils.intersection from the block now that the argument list is confirmed to have the required number of elements
SortedSetUtils.intersection calls SortedSetUtils.set_operation with a block
SortedSetUtils.set_operation does a lot of the heavy lifting, loading the sets from memory, failing if they're of the wrong type, parsing all the arguments, and yields all that back to SortedSetUtils.intersection with the sets_with_weights and aggregate variable.
SortedSetUtils.intersection calls RedisSortedSet.intersection from the block with the sets_with_weight and aggregate variables
The result of RedisSortedSet.intersection is handled by SortedSetUtils.set_operation with the line new_set = yield sets_with_weight, options[:aggregate], and it then returns it, alongside the withscores option.
The result of SortedSetUtils.set_operation is handled by SortedSetUtils.set_operation_command with the set_result, withscores = yield line, which it uses to create an instance of SortedSetRankSerializer.
The result of SortedSetUtils.set_operation_command is the last line of ZInterCommand#call and is what is returned to the Server class.

The following is a summary of the main methods called:

call { Utils.intersection }
  set_operation_command
    validations
    Utils.intersection (through yield)
      set_operation { RedisSortedSet.intersection }
        validations
        RedisSortedSet.intersection (through yield)
    SortedSetRankSerializer.new

And with that, the ZINTER commands is done.

Well, almost, we haven't looked at the actual intersection implementation, in RedisSortedSet, let's do that now:

module BYORedis
  class RedisSortedSet

    # ...

    def self.intersection(sets_with_weight, aggregate: :sum)
      # Sort the sets smallest to largest
      sets_with_weight.sort_by! { |set, _| set.nil? ? 0 : set.cardinality }

      smallest_set = sets_with_weight[0][0]
      smallest_set_weight = sets_with_weight[0][1]
      return RedisSortedSet.new if smallest_set.nil?

      intersection_set = RedisSortedSet.new

      # Iterate over the first set, if we find a set that does not contain the member, discard
      smallest_set.each do |set_member|
        present_in_all_other_sets = true
        if set_member.is_a?(Pair)
          pair = set_member
        else
          pair = Pair.new(BigDecimal(1), set_member)
        end
        weighted_pair_score = Utils.multiply_or_zero_if_nan(smallest_set_weight, pair.score)

        # For each member of the smallest set, we loop through all the other sets and try to
        # find the member, if we don't find it, we break the loop and move on, if we do find
        # a member, then we need to apply the weight/aggregate logic to it
        sets_with_weight[1..-1].each do |set_with_weight|
          set = set_with_weight[0]
          weight = set_with_weight[1]

          if set == smallest_set
            other_pair = pair
          elsif set.is_a?(RedisSet)
            other_pair = set.member?(pair.member) ? Pair.new(BigDecimal(1), pair.member) : nil
          elsif set.is_a?(RedisSortedSet)
            other_pair = set.find_pair(pair.member)
          else
            raise "Unknown set type: #{ set }"
          end

          if other_pair
            weighted_other_pair_score = Utils.multiply_or_zero_if_nan(other_pair.score, weight)
            weighted_pair_score =
              case aggregate
              when :sum then weighted_pair_score + weighted_other_pair_score
              when :max then [ weighted_other_pair_score, weighted_pair_score ].max
              when :min then [ weighted_other_pair_score, weighted_pair_score ].min
              else raise "Unknown aggregate method: #{ aggregate }"
              end
          else
            present_in_all_other_sets = false
            break
          end
        end
        # Otherwise, keep
        if present_in_all_other_sets
          intersection_set.add(weighted_pair_score, pair.member, {})
        end
      end

      intersection_set
    end

    def self.aggregate_scores(aggregate, a, b)
      case aggregate
      when :sum then Utils.add_or_zero_if_nan(a, b)
      when :max then a < b ? b : a
      when :min then a < b ? a : b
      else raise "Unknown aggregate method: #{ aggregate }"
      end
    end
    private_class_method :aggregate_scores

    # ...

    def find_pair(member)
      case @underlying
      when List then list_find_pair(member)
      when ZSet
        dict_entry = @underlying.dict.get_entry(member)
        Pair.new(dict_entry.value, dict_entry.key) if dict_entry
      else raise "Unknown type for #{ @underlying }"
      end
    end

    def empty?
      cardinality == 0
    end

    def each(&block)
      case @underlying
      when List then @underlying.each(&block)
      when ZSet then @underlying.array.each(&block)
      else raise "Unknown type for #{ @underlying }"
      end
    end

    # ...

    private

    # ...

    def list_find_pair(member)
      @underlying.each do |pair|
        return pair if pair.member == member
      end

      nil
    end
  end
end

listing 10.32 The RedisSortedSet.intersection method

The logic is overall similar to the one we wrote in RedisSet.intersection, with the main difference being that we need to handle weights, and how to aggregate them. We start in a similar way, we sort sets from smallest to largest, because that way we will only iterate through the smallest set. If that set is nil, then we don't even have to go further, an empty set in an intersection guarantees that the result is an empty set. Like 0 in a multiplication, it doesn't matter what the other parts of the operation are.

Once sets are sorted, the smallest set is the value at sets_with_weight[0][0], the first element in the first pair of the main array, and its weight is at sets_with_weight[0][1], the second element in the first pair of the main array. Remember that sets_with_weights looks like the following where 's1', 's2' & 's3' are set instances:

[
  [ 's1', 1 ],
  [ 's2', 10 ],
  [ 's3', 0.5 ],
]

We now need to iterate through the smallest set, knowing that it might either be a RedisSet or a RedisSortedSet. RedisSet already has an each method, which we created in the previous chapter, but we're adding one on RedisSortedSet here. Both are pretty similar, they need to know how to iterate over the underlying data structure, in this case either a List or a ZSet. In the List case we use the new addition to the List class, each.

For the ZSet, we use its array attribute, which is a SortedArray, and call each on it, which is the "real" Array#each method, forwarding the block attribute to it, with the ampersand, because Ruby needs that for blocks. The array is sorted, so callers will receive the elements in the correct order.

Once we're in the iteration itself, in the block, if we're dealing with a RedisSet, then we won't get a Pair instance at each iteration, and we'll instead get a String, since RedisSet instances only store String instances as members. In this case we create a Pair, with a default score of BigDecimal(1).

Next we need to apply the weight value for that set to the score of the current member. While we might think that a multiplication with * would be enough, it's actually not! We need to handle NaN values, such as 0 * inf. We do this with the accurately named Utils.multiply_or_zero_if_nan method. If applying a weight to a score results in NaN, we default to 0.

The next step of the intersection process is to iterate over all the other sets, and as soon as we find one that does not contain the current member of the smallest set, we stop and move on to the next element in the smallest set.

We start the iteration with sets_with_weight[1..-1], and we then extract the set and weight variable from the set_with_weight pair.

It is possible that the same set is reused multiple times in a ZINTER command, such as ZINTER 3 z1 z1 z1, and in this case we do not need to look the pair up in the other set. The variable pair, from the smallest set, is the same, so we set other_pair to pair, as a shortcut, to avoid a lookup in set.

Otherwise, we need to lookup pair in set, but set might a regular set or a sorted set. We test for the presence in a RedisSet instance with set.member?(pair.member), and if we find a match, we create a new Pair instance with a default score of 1, as we did earlier with the element from the smallest set.

If set is a RedisSortedSet, then we call the RedisSortedSet#find_pair method. The find_pair method behaves differently depending on the type of @underlying. If it is a ZSet, we use the get_entry method with its dict attribute, which performs an O(1) operation to retrieve the element from the sorted set, or nil. If there is a result we instantiate a new Pair instance with the score and member values from dict.

As we've already seen many times the List case is a bit more cumbersome and we delegate it to a private method. That method, list_find_pair, uses the List#each method to iterate through the list until it either reaches the end of the list or it finds a Pair instance in it for which its member attributes matches the member argument, which is an O(n) operation. Earlier we talked about why it was great that procs allowed us to return from the outer method, and this is a great example, it allows us to short-ciruit the iteration in the each method.

Back to the intersection class method, by now we either found other_pair in set, or not. If we failed to find it, we break from the loop and move to the next element in the smallest set while setting present_in_all_sets to false. This variable is necessary because when we exit the loop over all the other sets, we need to know whether the current element in the smallest set, pair should be added to the result set, and it should only be added if we did find it in all the other sets.

On the other hand, if other_pair is not nil, then we need to aggregate the scores, but first we need to apply the weight of the other set to other_pair, which we again use multiply_or_zero_if_nan for. The aggregation is delegated to the private class method aggregate_scores.

This method takes an aggregate symbol, either :min, :max or :sum and two numeric values. It uses a case/when to apply the correct function, using Utils.add_or_zero_if_nan as a safety measure because some additions might return NaN, such as inf - inf. Let's add this method as well as multiply_or_zero_if_nan:

module BYORedis
  module Utils

    # ...

    def self.multiply_or_zero_if_nan(a, b)
      BigDecimal.save_exception_mode do
        BigDecimal.mode(BigDecimal::EXCEPTION_NaN, true)

        a * b
      end
    rescue FloatDomainError
      BigDecimal(0)
    end

    def self.add_or_zero_if_nan(a, b)
      add_or_raise_if_nan(a, b)
    rescue FloatNaN
      BigDecimal(0)
    end
  end
end

listing 10.33 The add_or_zero_if_nan & multiply_or_zero_if_nan methods

If the aggregate value is neither of the expected ones we follow the "this is not supposed to happen so throw a generic exception so that bugs are caught in the development cycle" approach.

As previously mentioned, once we exit the loop over all the other sets, we need to check why we exited it. Did we check all the sets and found pair in all of them? If so we need to add it, with the aggregated score, to the result set, intersection_set, otherwise, we can move on to the next member of the smallest set.

And with that, we're now done, for real, with the ZINTER command.

Sorted Set Intersection, but store the result this time

ZINTERSTORE behaves almost exactly as ZINTER, with the only difference being that the first argument is the key where the result set will be stored and the return value is the cardinality of the new set.

The format is described as the following by the Redis documentation:

ZINTERSTORE destination numkeys key [key ...]
  [WEIGHTS weight [weight ...]]
  [AGGREGATE SUM|MIN|MAX]

Let's create the ZInterStoreCommand class:

module BYORedis
  module SortedSetUtils

    # ...

    def self.set_operation_store_command(db, args)
      Utils.assert_args_length_greater_than(2, args)
      destination_key = args.shift

      result_set, _ = yield

      if result_set.empty?
        db.data_store.delete(destination_key)
      else
        db.data_store[destination_key] = result_set
      end

      RESPInteger.new(result_set.cardinality)
    end

    # ...
  end

  # ...

  class ZInterStoreCommand < BaseCommand
    def call
      SortedSetUtils.set_operation_store_command(@db, @args) do
        SortedSetUtils.intersection(@db, @args)
      end
    end

    def self.describe
      Describe.new('zinterstore', -4, [ 'write', 'denyoom', 'movablekeys' ], 0, 0, 0,
                   [ '@write', '@sortedset', '@slow' ])
    end
  end

  # ...
end

listing 10.34 The ZInterStoreCommand class

Earlier we created the SortedSetUtils.set_operation_command method so we could share some of the logic between ZINTER and ZUNION, because they accept the same arguments, the only difference is that one performs an intersection and one a union. We're doing the same here with SortedSetUtils.set_operation_store_command, so that we share some logic between ZINTERSTORE and ZUNIONSTORE, which are both similar to the non-store commands, but different because they accept an extra key, destination.

The set_operation_store_command checks that we have more than two arguments, the smallest list of arguments we can accept is destination, numkeys set to 1 and one key. We then call Array#shift, to extract the destination argument, and we end up with an argument list identical to what ZINTER accepts, so we use the exact same method in the SortedSetUtils module we did earlier, intersection.

The other difference is in how we handle the result of the intersection operation, instead of returning it to the client, we store it at destination_key, and we return its cardinality. This is what set_operation_store_command does once it received the result from yield. Note that we discard the withscores value because we're not serializing the set. This command, like other *STORE commands deletes whatever was stored at destination if the result of the operation is empty. Redis never stores empty collection, but it clears the destination, to make the following sequence of operation sensible:

127.0.0.1:6379> SET a-string whatever
OK
127.0.0.1:6379> TYPE z
none
127.0.0.1:6379> ZINTERSTORE a-string 2 z z
(integer) 0
127.0.0.1:6379> ZCARD a-string
(integer) 0

The result of ZINTERSTORE is 0, telling us that the result set is empty, and it seems reasonable that calling ZCARD on it would also return 0, to keep things consistent. If Redis did not delete whatever might have already existed, the a-string key would still have been a string, and this is what we would have observed, which would have been very surprising for users:

127.0.0.1:6379> SET a-string whatever
OK
127.0.0.1:6379> TYPE z
none
127.0.0.1:6379> ZINTERSTORE a-string 2 z z
(integer) 0
127.0.0.1:6379> ZCARD a-string
(error) WRONGTYPE Operation against a key holding the wrong kind of value

Set Union

Like ZINTER, ZUNION is a recent addition to Redis and was added in 6.2.0.

ZUNION accepts the exact same options than ZINTER, the only difference being that it returns the set union instead of the set intersection.

module BYORedis
  module SortedSetUtils

    # ...

    def self.union(db, args)
      set_operation(db, args) do |sets_with_weight, aggregate|
        RedisSortedSet.union(sets_with_weight, aggregate: aggregate)
      end
    end

    # ...

  end

  # ...

  class ZUnionCommand < BaseCommand
    def call
      SortedSetUtils.set_operation_command(@args) do
        SortedSetUtils.union(@db, @args)
      end
    end

    def self.describe
      Describe.new('zunion', -3, [ 'readonly', 'movablekeys' ], 0, 0, 0,
                   [ '@read', '@sortedset', '@slow' ])
    end
  end
end

listing 10.35 The ZUnionCommand class

We can already see some of the decisions we made earlier when implementing ZINTER paying off. The content of ZUnionCommand#call is really concise and very similar to ZInterCommand#call, the only difference is the content of the block passed to SortedSetUtils.set_operation_command, which is SortedSetUtils.union this time.

This method, SortedSetUtils.union is very similar to SortedSetUtils.intersection, it also calls SortedSetUtils.set_operation, which does the heavy lifting with regards to all the possible options, and calls RedisSortedSet.union instead:

module BYORedis
  class RedisSortedSet

    # ...

    def self.union(sets_with_weight, aggregate: :sum)
      return RedisSortedSet.new({}) if sets_with_weight.empty?

      accumulator = Dict.new

      sets_with_weight[0..-1].each do |set_with_weight|
        set = set_with_weight[0]
        weight = set_with_weight[1]
        next if set.nil?

        set.each do |set_member|
          if set.is_a?(RedisSet)
            pair = Pair.new(BigDecimal(1), set_member)
          elsif set.is_a?(RedisSortedSet)
            pair = set_member
          else
            raise "Unknown set type: #{ set }"
          end

          weighted_score = Utils.multiply_or_zero_if_nan(pair.score, weight)
          existing_entry = accumulator.get_entry(pair.member)
          if existing_entry
            new_score = aggregate_scores(aggregate, existing_entry.value, weighted_score)
            existing_entry.value = new_score
          else
            accumulator[pair.member] = weighted_score
          end
        end
      end

      union_set = RedisSortedSet.new
      accumulator.each do |key, value|
        union_set.add(value, key)
      end
      union_set
    end

    # ...
  end
end

listing 10.36 The RedisSortedSet.union method

Set intersection was the complicated one, and we've kept the simpler version of the two for last. RedisSet.union was also the simplest of the set operations in the previous chapter, and it is the same here. We need to iterate through all the sets no matter what, there's no shortcut here.

For each set member, we need to check what type of set it is, and give it the default score of 1 if it's a "regular" set. We then weigh the scores, aggregate them, and throw the result in accumulator, a Dict. The use of a Dict here might be a bit surprising, because we need to return a RedisSortedSet after all. This is a small optimization because inserting in a Dict is cheaper on average than inserting in a RedisSortedSet. There might be a lot of insertions and updates required as we go through all the set members. It's entirely possible that members change positions through the iteration, but we don't care about maintaining the set sorted while we iterate, we only care about it at the end.

So we use a Dict, which is an efficient way to keep track of all the members, updating their scores is "cheap" as we can find the dict entry, update its score and move on, no re-ordering or anything.

Once the whole iteration is over, for all the sets, we convert the Dict to a RedisSortedSet, and we're done!

Sorted Set Union stored instead of returned

Finally, ZUNIONSTORE is the union variant, which behaves similarly to ZINTERSTORE, let's create the ZUnionStoreCommand class:

module BYORedis

  # ...

  class ZUnionStoreCommand < BaseCommand
    def call
      SortedSetUtils.set_operation_store_command(@db, @args) do
        SortedSetUtils.union(@db, @args)
      end
    end

    def self.describe
      Describe.new('zunionstore', -4, [ 'write', 'denyoom', 'movablekeys' ], 0, 0, 0,
                   [ '@write', '@sortedset', '@slow' ])
    end
  end
end

listing 10.37 The ZUnionStoreCommand class

By now we're already implemented all these methods! We've seen SortedSetUtils.set_operation_store_command when adding ZINTERSTORE and we've just seen SortedSetUtils.union when adding ZUNION. We can reuse the existing, and like Lego bricks, assemble them in a slightly different way and get what we need, sweet!

Done with set operations, next we're going to move to a few utility commands returning information about sorted set members.

Member data retrieval commands

Redis supports some commands to retrieve information for one or more members in a sorted set. ZRANK returns the "rank", that is the position as a zero-based index of the member in the sorted set, or a nil string if it is not present. ZSCORE & ZMSCORE are similar, with the difference that ZMSCORE accepts multiple members and returns an array, similar so SMISMEMBER in the previous chapter. As their name implies, they return the score of the given member(s).

Note that ZMSCORE is a recent addition in Redis 6.2.0.

Let's start with the ZRankCommand class:

module BYORedis

  # ...

  class ZRankCommand < BaseCommand
    def call
      Utils.assert_args_length(2, @args)
      sorted_set = @db.lookup_sorted_set(@args[0])

      if sorted_set
        RESPSerializer.serialize(sorted_set.rank(@args[1]))
      else
        NullBulkStringInstance
      end
    end

    def self.describe
      Describe.new('zrank', 3, [ 'readonly', 'fast' ], 1, 1, 1,
                   [ '@read', '@sortedset', '@fast' ])
    end
  end
end

listing 10.38 The ZRankCommand class

The ZRANK command accepts two arguments, key and member, and nothing else, if we find a RedisSortedSet for key, we call the new RedisSortedSet#rank method:

module BYORedis
  class RedisSortedSet

    # ...

    def rank(member)
      case @underlying
      when List then find_member_in_list(member) { |_, rank| rank }
      when ZSet
        entry = @underlying.dict.get_entry(member)
        return nil unless entry

        @underlying.array.index(Pair.new(entry.value, member))
      else raise "Unknown type for #{ @underlying }"
      end
    end

    private

    # ...

    def find_member_in_list(member)
      index = 0
      @underlying.each do |pair|
        return yield pair, index if pair.member == member

        index += 1
      end

      nil
    end
  end
end

listing 10.39 The RedisSortedSet#rank method

To retrieve the rank of a member in List, we start iterating from the left with each, and count, until we find it. The rank of an element is its 0-based index after all.

In a ZSet things are bit different. We could iterate through the sorted array, until we find the element we're looking for, but the combination of the Dict and the SortedArray allow us to be more efficient!

We first call Dict#get_entry, to retrieve the Pair instance. This is already a big win, because if there is no entry, we can return nil right away, we know that there is no rank value to return. On the other hand, if we did find a member, the Dict itself cannot tell us its rank, because the Pair instances it stores are not ordered. We use the SortedArray for that, and because we have both the score and member values, we can leverage the sorted property to find the element faster than by having to iterate through the whole array, in O(logn) time. By knowing the score it's looking for, the SortedArray#index method will use binary search, with the bsearch_index method, to get to the element in less steps.

The index returned by SortedArray#index happens to be the rank value, we found what we were looking for.

Next is the ZScoreCommand class:

module BYORedis

  # ...

  class ZScoreCommand < BaseCommand
    def call
      Utils.assert_args_length(2, @args)
      sorted_set = @db.lookup_sorted_set(@args[0])

      if sorted_set
        RESPSerializer.serialize(sorted_set.score(@args[1]))
      else
        NullBulkStringInstance
      end
    end

    def self.describe
      Describe.new('zscore', 3, [ 'readonly', 'fast' ], 1, 1, 1,
                   [ '@read', '@sortedset', '@fast' ])
    end
  end
end

listing 10.40 The ZScoreCommand class

The ZSCORE command has a similar structure to ZRANK, it takes two arguments, key and member. We need to add the RedisSortedSet#score method:

module BYORedis
  class RedisSortedSet

    # ...

    def score(member)
      case @underlying
      when List then find_member_in_list(member) { |pair, _| pair.score }
      when ZSet then @underlying.dict[member]
      else raise "Unknown type for #{ @underlying }"
      end
    end

    # ...

  end
end

listing 10.41 The RedisSortedSet#score method

In the List case we use the same method we used earlier, find_member_in_list, and we return the score value of the element, if we find it.

Things are easier in the Dict case, compared to ZRANK, the result of Dict#[] will return the value we store in each DictEntry, which happens to be the score, remember the keys are the member values, because these are the ones we need to maintain uniqueness on.

So we have the score, we can return it.

And finally, let's add the ZMScoreCommand class:

module BYORedis

  # ...

  class ZMScoreCommand < BaseCommand
    def call
      Utils.assert_args_length_greater_than(1, @args)
      sorted_set = @db.lookup_sorted_set(@args[0])

      scores = @args[1..-1].map do |member|
        sorted_set.score(member) if sorted_set
      end

      RESPArray.new(scores)
    end

    def self.describe
      Describe.new('zmscore', -3, [ 'readonly', 'fast' ], 1, 1, 1,
                   [ '@read', '@sortedset', '@fast' ])
    end
  end
end

listing 10.42 The ZMScoreCommand class

ZMSCORE can use mostly the same approach we used in ZSCORE, with the difference being that we iterate over each member we received in the argument list, and call RedisSortedSet#score for each of them, within a block given to Array#map so that the return value is a map of scores, potentially containing nil values.

Remove commands

Redis provides different ways to remove members from a sorted set. In this section we'll implement the ZREM, ZREMRANGEBYLEX, ZREMRANGEBYRANK & ZREMRANGEBYSCORE commands. Four other commands also remove members from sorted sets, ZPOPMIN & ZPOPMAX as well as their blocking variants, we'll explore these later in the chapter.

We'll add ZREM first:

module BYORedis

  # ...

  class ZRemCommand < BaseCommand
    def call
      Utils.assert_args_length_greater_than(1, @args)
      sorted_set = @db.lookup_sorted_set(@args.shift)
      removed_count = 0

      if sorted_set
        @args.each do |member|
          removed_count += 1 if sorted_set.remove(member)
        end
      end

      RESPInteger.new(removed_count)
    end

    def self.describe
      Describe.new('zrem', -3, [ 'write', 'fast' ], 1, 1, 1,
                   [ '@write', '@sortedset', '@fast' ])
    end
  end
end

listing 10.43 The ZRemCommand class

ZREM accepts one or more members, so we iterate over all of them, after shifting the key argument from the argument array. For each member, we call RedisSortedSet#remove and increment a counter if it returned true. Let's add this method:

module BYORedis
  class RedisSortedSet

    # ...

    def remove(member)
      case @underlying
      when List then remove_list(member)
      when ZSet then @underlying.remove_member(member)
      else raise "Unknown type for #{ @underlying }"
      end
    end

    # ...

    private

    # ...

    def remove_list(member)
      iterator = List.left_to_right_iterator(@underlying)
      while iterator.cursor
        if iterator.cursor.value.member == member
          @underlying.remove_node(iterator.cursor)
          return true
        end

        iterator.next
      end

      false
    end
  end
end

listing 10.44 The RedisSortedSet#remove method

In the List case, as has been the case a few times already, we delegate the work to a private method remove_list. In this method we iterate from left to right, until we find the member we're looking for. If we don't find it, we return false, to notify the caller that nothing was deleted.

If we do find the member, we call List#remove_node with the node we get from the iterator variable to remove the member from the list.

This is a rare case where we don't use the List#each method because we need a direct reference to the node value to pass to List#remove_node, and the each method skips over it to return the value the node contains, so we need a little bit more control here and use the lower level approach.

In the Dict case, we call the Dict#remove_member method. This method needs to make sure that both internal collections, the Dict and the SortedArray are updated and no longer contain the member, if it was present.

module BYORedis
  class ZSet

    # ...

    def remove_member(member)
      entry = @dict.delete_entry(member)
      return false unless entry

      index = @array.index(new_pair(entry.value, member))
      @array.delete_at(index)

      true
    end

    # ...
  end
end

listing 10.45 The ZSet#remove_member method

We start by calling Dict#delete_entry, which will return false if it fails to find the key we're looking for, or the entry that was removed if it contained it.

With the value attribute of the entry variable, which is the score of the member, we can call SortedArray#index method, and it will efficiently find the index, that is, without a full scan of the array. Now that we have the index value, we can call delete_at to remove this Pair from the array.

And with that the ZSet is now updated and the member is completely removed.

Rank Ranges

We are now going to add the ZREMRANGEBYRANK command, which has the following format according to the Redis documentation:

ZREMRANGEBYRANK key start stop

start and stop must be valid integers and negative values count as starting from the last member, the one with the highest score. The semantics are equivalent the the ones used in the ZRANGE command. The return value is the number of deleted members.

module BYORedis

  # ...

  class ZRemRangeByRankCommand < BaseCommand
    def call
      Utils.assert_args_length(3, @args)
      start = Utils.validate_integer(@args[1])
      stop = Utils.validate_integer(@args[2])
      sorted_set = @db.lookup_sorted_set(@args[0])
      removed_count = 0

      if sorted_set
        range_spec =
          RedisSortedSet::GenericRangeSpec.rank_range_spec(start, stop, sorted_set.cardinality)
        removed_count = sorted_set.remove_rank_range(range_spec)
      end

      RESPInteger.new(removed_count)
    end

    def self.describe
      Describe.new('zremrangebyrank', 4, [ 'write' ], 1, 1, 1,
                   [ '@write', '@sortedset', '@slow' ])
    end
  end
end

listing 10.46 The ZRemRangeByRankCommand class

Validating the number of arguments, checking that start and stop are both valid integers and looking up the set, once all these steps are successfully completed, we call remove_rank_range on the sorted set, which is a new method:

module BYORedis
  class RedisSortedSet

    # ...

    def remove_rank_range(range_spec)
      return 0 if range_spec.empty? || no_overlap_with_range?(range_spec) { |_, rank| rank }

      case @underlying
      when List then remove_rank_range_list(range_spec)
      when ZSet then @underlying.remove_rank_range(range_spec.min, range_spec.max)
      else raise "Unknown type for #{ @underlying }"
      end
    end

    # ...

    private

    # ...

    def remove_rank_range_list(range_spec)
      generic_remove_range_list(range_spec) { |_, rank| rank }
    end

    def generic_remove_range_list(range_spec)
      removed_count = 0
      iterator = List.left_to_right_iterator(@underlying)
      entered_range = false
      rank = 0

      while iterator.cursor
        pair = iterator.cursor.value
        in_range = range_spec.in_range?(yield(pair, rank))

        if in_range
          entered_range ||= true
          removed_count += 1
          next_node = iterator.cursor.next_node
          @underlying.remove_node(iterator.cursor)
          iterator.cursor = next_node
        elsif entered_range
          break
        else
          iterator.next
        end
        rank += 1
      end

      removed_count
    end
  end
end

listing 10.47 The RedisSortedSet#remove_rank_range method

We already used no_overlap_with_range? earlier, when serializing sub ranges of sorted sets in SortedSetSerializerBy, which is used for both ZRANGEBYSCORE and ZRANGEBYLEX. We can also use it here, since there won't be anything to remove for a rank range that is completely outside the set. For instance, the range 5 10 would not have any overlap with the sorted set { < 1, 'a' >, < 2, 'b' > }. This sorted set only contains two elements, with ranks 0 and 1. This is why we made no_overlap_with_range? calls its block argument with two arguments, the Pair, and the rank, this lets the block we pass from remove_rank_range tell the range to use the rank values to compare elements: return 0 if range_spec.empty? || no_overlap_with_range?(range_spec) { |_, rank| rank }.

The List branch delegates to the remove_rank_range_list private method, and the ZSetone to a method in that class, let's stay in RedisSortedSet for now and look at the list case.

The process to delete a range, whether it's a rank range, a score range or a lex range is very similar, so we create the generic_remove_range_list method to encapsulate the logic. The only thing the method needs, beside a range spec, is a block that tells it what to consider when sorting elements, the score, the member, or the rank. In this case, we pass a block that tells it to use the rank: { |_, rank| rank }. The range_spec we pass is the one we received from the ZRemRangeByRankCommand#call.

In generic_remove_range_list we iterate from left to right, with a few additional variables to keep track of where we are in the set. removed_count is necessary since this is what we need to return, entered_range is a helper like we've seen earlier that we'll use to abort before the end of the set if we can, and rank is necessary to yield it to the given block.

For each element we encounter, we ask the range_spec variable if it is in the range, using the result from yield(pair, rank), which in this case always returns the rank value. Note how we once again use a method that does not specifically know what to call, .score, .member or something else, and instead relies on a block to parameterize its behavior.

If the member is in range, we flag that we've entered the range with entered_range ||= true, increment removed_count, delete the node from the list, and tell the iterator to move to the next node.

If the member is not in the range, but we've entered the range, it means we just exited it, and we can exit the loop, otherwise, we haven't entered the range yet so we need to keep iterating. Finally, we increment the rank before jumping to the next loop iteration.

Once we exit the while loop, we return removed_count. This is another instance where we need a direct reference to the ListNode object and therefore need to use an iterator instead of List#each.

Let's now take a look at hew ZSet#remove_rank_range method. It accepts two arguments, the start and stop values, which happen to map directly to the index of the elements in the SortedArray. This lets us use the Array#slice! method to delete the whole range, which returns the deleted elements. We then iterate over the deleted elements to also remove the entries in the Dict.

module BYORedis
  class ZSet

    # ...

    def remove_rank_range(start, stop)
      removed = @array.slice!(start..stop)
      return 0 if removed.nil?

      removed.each do |pair|
        @dict.delete(pair.member)
      end
      removed.size
    end

    # ...

  end
end

listing 10.48 The ZSet#remove_rank_range method

Lex Ranges

Next is ZREMRANGEBYLEX, which has the following format according to the Redis documentation:

ZREMRANGEBYLEX key min max [LIMIT offset count]

We'll need to handle a lex range in the same way we did with ZRANGEBYLEX earlier, let's create the ZRemRangeByLexCommand class:

module BYORedis

  # ...

  class ZRemRangeByLexCommand < BaseCommand
    def call
      Utils.assert_args_length(3, @args)
      range_spec = Utils.validate_lex_range_spec(@args[1], @args[2])
      sorted_set = @db.lookup_sorted_set(@args[0])
      removed_count = 0

      if sorted_set
        removed_count = sorted_set.remove_lex_range(range_spec)
      end

      RESPInteger.new(removed_count)
    end

    def self.describe
      Describe.new('zremrangebylex', 4, [ 'write' ], 1, 1, 1,
                   [ '@write', '@sortedset', '@slow' ])
    end
  end
end

listing 10.49 The ZRemRangeByLexCommand class

The min and max values use the same format used in ZRANGEBYLEX, so we reuse the validate_lex_range_spec method from the Utils module. We pass the range spec it returns, which is a range spec aware of the specificities of lex comparisons, created with GenericRangeSpec.lex_range_spec. Let's add the remove_lex_range to RedisSortedSet:

module BYORedis
  class RedisSortedSet

    # ...

    def remove_lex_range(range_spec)
      return 0 if range_spec.empty? ||
                  no_overlap_with_range?(range_spec) { |pair, _| pair.member }

      case @underlying
      when List then remove_lex_range_list(range_spec)
      when ZSet then @underlying.remove_lex_range(range_spec)
      else raise "Unknown type for #{ @underlying }"
      end
    end

    # ...

    private

    # ...

    def remove_lex_range_list(range_spec)
      generic_remove_range_list(range_spec) { |pair, _| pair.member }
    end
  end
end

listing 10.50 The RedisSortedSet#remove_lex_range method

The method starts almost identically to remove_rank_range, with the exception that this time we tell no_overlap_with_range? to use the member attribute from the pair variable, instead of the rank variable.

The List case delegates to the remove_lex_range_list private method, which also uses the generic_remove_range_list private method, like remove_rank_range_list did, but tells it to use the pair.member value, instead of the rank, to decide what to remove from the list. This is another example of the benefits of generic_remove_range_list being so abstract, we can now express remove_lex_range_list very succinctly, and let generic_remove_range_list do the heavy lifting, removing elements from the list, counting the number of elements deleted, and so on.

For the ZSet case, things are not as simple because this time we do not have the index values of the elements we need to delete like we had for ZREMRANGEBYRANK, let's create the remove_lex_range method:

module BYORedis
  class ZSet

    # ...

    def remove_lex_range(range_spec)
      generic_remove(range_spec, &:member)
    end

    private

    # ...

    def generic_remove(range_spec, &block)
      first_in_range_index = @array.first_index_in_range(range_spec, &block)
      last_in_range_index = first_in_range_index
      (first_in_range_index.upto(@array.size - 1)).each do |rank|
        pair = @array[rank]
        in_range = range_spec.in_range?(yield(pair))

        if in_range
          last_in_range_index = rank
        else
          break
        end
      end
      remove_rank_range(first_in_range_index, last_in_range_index)
    end
  end
end

listing 10.51 The ZSet#remove_lex_range method

remove_lex_range uses the generic_remove method, which we'll be able to use when removing score ranges. We use first_index_in_range, from SortedArray, which we've added earlier when serializing ranges in ZRANGEBYLEX and ZRANGEBYSCORE. By giving the block that returns the member attribute of a Pair instance, first_index_in_range will be able to find the first element in the array that is in the range.

Once we found this element, we iterate from there, and as long as elements are in the range, we mark the current index as the last index of elements in range with last_in_range_index. As soon as we find an element that is not in range, we exit the loop.

At this point, we have the boundaries of the range we need to delete, in terms of indices, the index of the first element, which first_index_in_range gave us, and the last index, last_in_range_index, which we found by iterating through the array. These values are also ranks, since the rank is the index of the element in the set, meaning that we can use the remove_rank_range we added earlier when adding the ZREMRANGEBYRANK method. This method items from @array and @dict, and return the number of elements deleted, so we can return what it returns.

Score Ranges

And finally we are adding ZRemRangeByScoreCommand, for the ZREMRANGEBYSCORE command, which has the following format according to the Redis Documentation:

ZREMRANGEBYSCORE key min max

min and max must be valid scores, with the same semantics as in ZRANGEBYSCORE where a score can be marked as exclusive with the ( prefix.

module BYORedis

  # ...

  class ZRemRangeByScoreCommand < BaseCommand
    def call
      Utils.assert_args_length(3, @args)
      range_spec = Utils.validate_score_range_spec(@args[1], @args[2])
      sorted_set = @db.lookup_sorted_set(@args[0])
      removed_count = 0

      removed_count = sorted_set.remove_score_range(range_spec) if sorted_set

      RESPInteger.new(removed_count)
    end

    def self.describe
      Describe.new('zremrangebyscore', 4, [ 'write' ], 1, 1, 1,
                   [ '@write', '@sortedset', '@slow' ])
    end
  end
end

listing 10.52 The ZRemRangeByScoreCommand class

We reuse the validate_score_range_spec method from the Utils module, to create the right type of range, a GenericRangeSpec created with score_range_spec, we then pass this range spec to RedisSortedSet#remove_score_range:

module BYORedis
  class RedisSortedSet

    # ...

    def remove_score_range(range_spec)
      return 0 if range_spec.empty? ||
                  no_overlap_with_range?(range_spec) { |pair, _| pair.score }

      case @underlying
      when List then remove_score_range_list(range_spec)
      when ZSet then @underlying.remove_score_range(range_spec)
      else raise "Unknown type for #{ @underlying }"
      end
    end

    # ...

    private

    # ...

    def remove_score_range_list(range_spec)
      generic_remove_range_list(range_spec) { |pair, _| pair.score }
    end
  end
end

listing 10.53 The RedisSortedSet#remove_score_range method

The structure of the method might look pretty familiar by now, this is almost identical to remove_lex_range, with the exception of the block given to no_overlap_with_range? which returns the score attribute this time.

In the List case we call remove_score_range_list, which itself uses the generic_remove_range_list, with a block telling it to extract the score attribute from the Pair instances it handles.

For ZSet instances, we call remove_score_range, which can be expressed in terms of the method we just wrote earlier for ZREMRANGEBYLEX, generic_remove:

module BYORedis
  class ZSet

    # ...

    def remove_score_range(range_spec)
      generic_remove(range_spec, &:score)
    end

    # ...

  end
end

listing 10.54 The ZSet#remove_score_range method

With ZREMRANGEBYSCORE we have now implemented all the *REM* methods, next up, the *REV* variants. Did you notice how we did not have to write any real logic this time, we just called a few existing methods with the right block telling them what values to extract!

Reverse commands

All the commands we've implemented so far used an implicit sorting order by ascending score. Redis provides four commands that let users reverse the order and use the scores in descending order: ZREVRANGE, ZREVRANGEBYLEX, ZREVRANGEBYSCORE, & ZREVRANK. These commands behave almost identically to ZRANGE, ZRANGEBYLEX, ZRANGEBYSCORE & ZRANK, all of which were implemented earlier in the chapter, but use the reverse score ordering instead.

We've spent some time earlier explaining why we were building things in a way that may have look convoluted, well, now we're about to see why. These commands will all be really short to write, because most of the logic already exists, we just have to reverse things in a few places, here and there.

Let's start with the ZREVRANGE command, which has the following format according to the Redis Documentation:

ZREVRANGE key start stop [WITHSCORES]

Let's create the ZRevRangeCommand class:

module BYORedis

  # ...

  class ZRevRangeCommand < BaseCommand
    def call
      SortedSetUtils.generic_range(@db, @args, reverse: true)
    end

    def self.describe
      Describe.new('zrevrange', -4, [ 'readonly' ], 1, 1, 1, [ '@read', '@sortedset', '@slow' ])
    end
  end
end

listing 10.55 The ZRevRangeCommand class

We call the generic_range method we created earlier, but this time we set the reverse flag to true. The flag is used to decide in which order to handle the start and stop argument. We use a trick to convert start and stop in a way that will make it easy to reuse, before looking at the code, which is convoluted, let's first look at an example with the following sorted set:

127.0.0.1:6379> ZADD z 10 a 15 c 20 f 50 m 100 z
(integer) 5
127.0.0.1:6379> ZRANGE z 0 -1
1) "a"
2) "c"
3) "f"
4) "m"
5) "z"
127.0.0.1:6379> ZRANGE z 2 3
1) "f"
2) "m"
127.0.0.1:6379> ZREVRANGE z 0 -1
1) "z"
2) "m"
3) "f"
4) "c"
5) "a"
127.0.0.1:6379> ZREVRANGE z 2 3
1) "f"
2) "c"

With ZRANGE, the members a, c, f, m, & z have the ranks 0, 1, 2, 3 & 4, with ZREVRANGE, a has now the rank 4, c 3, f, 2, m 1 and z 0.

It turns out that we can always convert a "regular index" to a reversed index with the following logic:

if index is greater than or equal to 0, convert it to the maximum index minus the index value
if index is negative, convert it to the maximum index value, minus the sum of the index, the maximum index value, and 1

Yup, that works, let's look at the values we used in the example above and start with 0, its "reverse equivalent" is 4, that is when we pass 0 to ZREVRANGE, we can get the desired value by fetching the element at rank 4 in the sorted set, which is where z is, at index 4 in the set, and ZREVRANGE z 0 0 does return z.

We apply the first rule, where max index is 4, the max rank in the set, so 4 - 0 = 0, 4 is indeed the index in the set where we would find z.

Let's keep going, with a negative value this time, -1, in the set, this is the last index, the member z, when passed to ZREVRANGE, -1 maps to a, which is the member at index 0 in the sorted set, so -1 should give us 0 back. Let's check, it applies to the second rule, 4 - (-1 + 4 + 1) = 0, it worked again!

Let's check 2 and 3 now, in the set, they return the members f and m, but in ZREVRANGE, they respectively map to f and c, which are the members at index 2 and 1 in the set, so 2 should stay 2 and 3 should become 1. They both trigger the first rule: 4 - 2 = 2 and 4 - 3 = 1.

These examples show us that we can apply these transformations to know where to look in the sorted set. For instance when we receive the command ZREVRANGE z 2 3, we'll transform start from 2 to 2 and stop from 3 to 1, we can then swap start and stop, so that we maintain the condition start <= stop and we're almost done. The last thing is that we now need to tell the serializer to serialize the items from right to left.

In ZREVRANGE 0 -1, both boundaries will be converted to 4 and 0, and then swapped so that start is 0 and stop is 4.

Let's make these changes:

module BYORedis
  module SortedSetUtils

    # ...

    def self.generic_range(db, args, reverse: false)
      Utils.assert_args_length_greater_than(2, args)
      start = Utils.validate_integer(args[1])
      stop = Utils.validate_integer(args[2])
      raise RESPSyntaxError if args.length > 4

      if args[3]
        if args[3].downcase == 'withscores'
          withscores = true
        else
          raise RESPSyntaxError
        end
      end

      sorted_set = db.lookup_sorted_set(args[0])

      if reverse
        tmp = reverse_range_index(start, sorted_set.cardinality - 1)
        start = reverse_range_index(stop, sorted_set.cardinality - 1)
        stop = tmp
      end

      if sorted_set
        range_spec =
          RedisSortedSet::GenericRangeSpec.rank_range_spec(start, stop, sorted_set.cardinality)
        SortedSetRankSerializer.new(
          sorted_set,
          range_spec,
          withscores: withscores,
          reverse: reverse,
        )
      else
        EmptyArrayInstance
      end
    end

    def self.reverse_range_index(index, max)
      if index >= 0
        max - index
      elsif index < 0
        max - (index + max + 1)
      end
    end

    # ...
  end

  # ...

end

listing 10.56 The generic_range class method for the SortedSetUtils module

If the reverse flag is true we convert the indices as described earlier and we swap the values to maintain the order of start and stop. We propagate the reverse flag to SortedSetRankSerializer:

module BYORedis

  # ...

  class SortedSetRankSerializer

    # ...

    private

    def serialize_zset
      sub_array = @sorted_set.underlying.array[@range_spec.min..@range_spec.max]
      members = []
      sub_array.each do |pair|
        if @reverse
          members.prepend(Utils.float_to_string(pair.score)) if @withscores
          members.prepend(pair.member)
        else
          members.push(pair.member)
          members.push(Utils.float_to_string(pair.score)) if @withscores
        end
      end

      RESPArray.new(members).serialize
    end

    def serialize_list
      ltr_acc = lambda do |value, response|
        response << RESPBulkString.new(value.member).serialize
        if @withscores
          response << RESPBulkString.new(Utils.float_to_string(value.score)).serialize
        end
        @withscores ? 2 : 1
      end

      rtl_acc = lambda do |value, response|
        if @withscores
          response.prepend(RESPBulkString.new(Utils.float_to_string(value.score)).serialize)
        end
        response.prepend(RESPBulkString.new(value.member).serialize)
        @withscores ? 2 : 1
      end

      if @reverse
        tmp = ltr_acc
        ltr_acc = rtl_acc
        rtl_acc = tmp
      end

      ListSerializer.new(@sorted_set.underlying, @range_spec.min, @range_spec.max)
                    .serialize_with_accumulators(ltr_acc, rtl_acc)
    end

      # ...

    end
  end
end

listing 10.57 The serialize_zset & serialize_list methods for the SortedSetRankSerializer class

In the serialize_list method we look at the @reverse flag and if it set to true we swap the values or ltr_acc and rtl_acc. Swapping these values has the effect of reverting the order in which elements get serialized. Looking again at the ZREVRANGE z 0 -1 example from earlier, where the desired result is z, m, f, c, a, the ListSerializer will iterate from 0 to 4, but instead of appending elements as it finds them, it'll prepend them. It will first find a, at index 0, and will add it to the string serializing the resp array, and it will then find c, prepending it and so on, until it prepends z. If scores are to be included, it knows how to do that as well, the same way we did earlier for ZRANGE.

The process is similar in the ZSet case, if the @reverse flag is set to true, we use members.prepend instead of members.<< to accumulate elements.

Reverse Range by Score and Lex

ZREVRANGEBYLEX

Next is the ZREVRANGEBYLEX command, which has the following format according to the Redis Documentation:

ZREVRANGEBYLEX key max min [LIMIT offset count]

min and max have the same format we used in the ZRANGEBYLEX and ZREMRANGEBYLEX commands, and note how their order is reversed, the max boundary is specified first here.

module BYORedis

  # ...

  class ZRevRangeByLexCommand < BaseCommand
    def call
      SortedSetUtils.generic_range_by_lex(@db, @args, reverse: true)
    end

    def self.describe
      Describe.new('zrevrangebylex', -4, [ 'readonly' ], 1, 1, 1,
                   [ '@read', '@sortedset', '@slow' ])
    end
  end
end

listing 10.58 The ZRevRangeByLexCommand class

We call generic_range_by_lex, but this time we set the reverse flag to true, which changes how the arguments are read:

module BYORedis
  module SortedSetUtils

    # ...

    def self.generic_range_by_lex(db, args, reverse: false)
      # A negative count means "all of them"
      options = { offset: 0, count: -1 }
      Utils.assert_args_length_greater_than(2, args)
      key = args.shift
      if reverse
        max = args.shift
        min = args.shift
      else
        min = args.shift
        max = args.shift
      end
      range_spec = Utils.validate_lex_range_spec(min, max)
      parse_range_by_lex_options(args, options) unless args.empty?

      sorted_set = db.lookup_sorted_set(key)
      if options[:offset] < 0
        EmptyArrayInstance
      elsif sorted_set
        options[:withscores] = false
        options[:reverse] = reverse
        SortedSetSerializerBy.new(sorted_set, range_spec, **options, &:member)
      else
        EmptyArrayInstance
      end
    end

    # ...
  end

  # ...

end

listing 10.59 The generic_range_by_lex class method for the SortedSetUtils module

We read min and max in the reverse order, max first if reverse is true, and this is the only difference. The reverse flag is forwarded to SortedSetSerializerBy when we call it: SortedSetSerializerBy.new(sorted_set, range_spec, **options, &:member)

We ignored it earlier, but looking at the @reverse branch of the serialize_zset method in SortedSetSerializerBy, we now need a new method in SortedArray, last_index_in_range:

module BYORedis

  # ...

  class SortedSetSerializerBy

    # ...

    def serialize_zset
      members = []

      if @reverse
        start_index = @sorted_set.underlying.array.last_index_in_range(@range_spec, &@block)
        if start_index.nil?
          raise "Unexpectedly failed to find last index in range for #{ self }"
        end

        indices = start_index.downto(0)
      else
        start_index = @sorted_set.underlying.array.first_index_in_range(@range_spec, &@block)
        if start_index.nil?
          raise "Unexpectedly failed to find first index in range for #{ self }"
        end

        indices = start_index.upto(@sorted_set.cardinality - 1)
      end

      # ...

    end

    def serialize_list
      if @reverse
        iterator = List.right_to_left_iterator(@sorted_set.underlying)
      else
        iterator = List.left_to_right_iterator(@sorted_set.underlying)
      end
      members = []
      entered_range = false

      while iterator.cursor && @count != 0
        member = iterator.cursor.value

        if @range_spec.in_range?(@block.call(member))
          entered_range ||= true
          if @offset == 0
            members << member.member
            members << Utils.float_to_string(member.score) if @withscores

            @count -= 1
          else
            @offset -= 1
          end
        elsif entered_range == true
          break
        end

        iterator.next
      end

      RESPArray.new(members).serialize
    end
  end
end

listing 10.60 The serialize_zset & serialize_list methods in the SortedSetSerializerBy class

Let's look at serialize_zset first. The order of the range items is still what we'd expect, that is @range_spec.max is expected to be greater than or equal to @range_spec.min, using lexicographic order, but the "first" item in what we want to return is not the first item in the set to match the range, it's the last one. Let's look at an example:

127.0.0.1:6379> ZADD z 0 a 0 b 0 c 0 d 0 e 0 f
(integer) 6
127.0.0.1:6379> ZRANGEBYLEX z - +
1) "a"
2) "b"
3) "c"
4) "d"
5) "e"
6) "f"
127.0.0.1:6379> ZREVRANGEBYLEX z + -
1) "f"
2) "e"
3) "d"
4) "c"
5) "b"
6) "a"
127.0.0.1:6379> ZREVRANGEBYLEX z [e [b
1) "e"
2) "d"
3) "c"
4) "b"

The set z has the elements a, b, c, d, e and f. When we request the range + -, we get the whole set, starting from the member with the greater value in lexicographic order, z, down to the smallest one, a. The next example is what interests us here, the range is [e [b, that is, from e, to b, inclusive. But remember that the range spec always needs to maintain the condition min <= max, so the range spec we created is actually one where min is b, and max is e, because we read the arguments in the reversed order, max first.
The layout of the SortedSet is roughly the following, ignoring the score since they're all 0:

["a", "b", "c", "d", "e", "f"]

If we were to call first_index_in_range with the range spec for [b [e, we'd get 1, the index of b, but since we need to return e, d, c, b, the first index we want is 4, the index of e. This is why we call last_index_in_range, it'll give us just what we need. And once we have the index of the last index, which is where we want to start iterating from, we create an enumerator that goes the other way, down to 0, with start_index.downto(0).

The rest of the method is the same, we iterate over all the indices in the range, we check if the item is in the range, and as soon as we find an item that is not, we exit the loop, while taking into account the values of @count and @offset.

Let's take a look at the last_index_in_range method:

module BYORedis
  class SortedArray

    # ...

    def last_index_in_range(range_spec)
      return nil if empty?

      first_index_outside = @underlying.bsearch_index do |existing_element|
        compare = range_spec.compare_with_max(yield(existing_element))
        if range_spec.max_exclusive?
          compare >= 0 # existing_element.score > max
        else
          compare > 0 # existing_element.score >= max
        end
      end

      case first_index_outside
      when nil then @underlying.size - 1 # last
      when 0 then nil # the max of the range is smaller than the smallest item
      else first_index_outside - 1
      end
    end
  end
end

listing 10.61 The SortedArray#last_index_in_range method

We use bsearch_index in a similar way we used it in first_index_in_range with a few differences. We compare each value with compare_with_max instead of compare_with_min, and the condition related to the exclusivity of the max boundary are reversed. Once again, it's easier to look at an example in irb:

irb(main):001:0> set = [ 'a', 'b', 'c', 'd', 'e', 'f' ]
irb(main):002:0> set.bsearch_index { |x| x >= 'e' }
=> 4
irb(main):003:0> set.bsearch_index { |x| x > 'e' }
=> 5
irb(main):004:0> set.bsearch_index { |x| x >= 'f' }
=> 5
irb(main):005:0> set.bsearch_index { |x| x > 'f' }
=> nil

In all the examples we get the leftmost index, if it exists, for which the condition is true, in the first case, the smallest index where the value is greater than or equal to e is 4, because set[4] == 'e', and if we swap the condition to be strict, then it return 5, because the element at index 5, f, is the first one to be strictly greater than e. If we translate that to the inclusive/exclusive cases we're trying to handle, we can see that the result are almost what we want, just off by one to the right. If the upper boundary is exclusive, (e, then the last index would be 3, and if it is inclusive, [e, then it would be 4. We can obtain these values by using the >= operator in the exclusive case and the > operator in the inclusive case, and subtracting one to the result.

There's a small edge case we need to handle, what if the set does not contain an element that is outside the set, then the last element in the set would be the last in the range. This is what the last example above demonstrates, the set array does not contain a value greater than f, and returns nil, and in this case, because we have nothing we can subtract 1 to to obtain the last index, we need to handle this specific case.

This logic is what the case/when conditions at the end of last_index_in_range handle. There is also a special case where the value returned would be 0, in which case we don't want to return -1 and want to return 0 instead. The following is an example that could potentially trigger this path but that will be prevented in practice by the no_overlap_with_range? check.

irb(main):011:0> '0' < 'a' # this is included as a reminder that in lex order '0' < 'a'
=> true
irb(main):012:0> set.bsearch_index { |x| x >= '0' }
=> 0
irb(main):013:0> set.bsearch_index { |x| x > '0' }
=> 0

In both cases the range does not overlap with the set, the max value of the range, '0' is lower than the mininum value of the set, 'a', and we return nil, none of the set members is within the range.

ZREVRANGEBYSCORE

Next is the ZREVRANGEBYSCORE command, which has the following format according to the Redis Documentation:

ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count]

min and max have the same format we used in the ZRANGEBYSCORE and ZREMRANGEBYSCORE commands, and note how their order is reversed, the max boundary is specified first.

We'll first create the ZRevRangeByScoreCommand class:

module BYORedis

  # ...

  class ZRevRangeByScoreCommand < BaseCommand
    def call
      SortedSetUtils.generic_range_by_score(@db, @args, reverse: true)
    end

    def self.describe
      Describe.new('zrevrangebyscore', -4, [ 'readonly' ], 1, 1, 1,
                   [ '@read', '@sortedset', '@slow' ])
    end
  end
end

listing 10.62 The ZRevRangeByScoreCommand class

We call generic_range_by_score with the reverse flag set to true, let's look at how the flag is handled in generic_range_by_score:

module BYORedis
  module SortedSetUtils

    # ...

    def self.generic_range_by_score(db, args, reverse: false)
      # A negative count means "all of them"
      options = { offset: 0, count: -1, withscores: false }
      Utils.assert_args_length_greater_than(2, args)
      key = args.shift
      if reverse
        max = args.shift
        min = args.shift
      else
        min = args.shift
        max = args.shift
      end

      # ...
    end

    # ...
  end

  # ...

end

listing 10.63 Handling of the reverse flag in the SortedSetUtils.generic_range_by_score class method

Things are very similar to generic_range_by_lex, if reverse is true, we read max first, and we then create the range spec with these values.

And that's it ... yes, everything else "just works". The code path is essentially the same as in ZRANGEBYSCORE, but the reverse flag will make sure that either the List or the ZSet are serialized in the reverse order, returning the desired result to the client.

ZREVRANK

ZREVRANK is simpler since it only deals with a single set member and returns its rank as if the set was sorted from highest scores to lowest:

127.0.0.1:6379> ZADD z 0 a 0 b 0 c 0 d 0 e 0 f
(integer) 6
127.0.0.1:6379> ZRANGE z 0 -1
1) "a"
2) "b"
3) "c"
4) "d"
5) "e"
6) "f"
127.0.0.1:6379> ZRANK z a
(integer) 0
127.0.0.1:6379> ZRANK z e
(integer) 4
127.0.0.1:6379> ZREVRANGE z 0 -1
1) "f"
2) "e"
3) "d"
4) "c"
5) "b"
6) "a"
127.0.0.1:6379> ZREVRANK z a
(integer) 5
127.0.0.1:6379> ZREVRANK z e
(integer) 1

Let's create the ZRevRankCommand class:

module BYORedis

  # ...

  class ZRevRankCommand < BaseCommand
    def call
      Utils.assert_args_length(2, @args)
      sorted_set = @db.lookup_sorted_set(@args[0])

      if sorted_set
        RESPSerializer.serialize(sorted_set.rev_rank(@args[1]))
      else
        NullBulkStringInstance
      end
    end

    def self.describe
      Describe.new('zrevrank', 3, [ 'readonly', 'fast' ], 1, 1, 1,
                   [ '@read', '@sortedset', '@fast' ])
    end
  end
end

listing 10.64 The ZRevRankCommand class

We nee to add the rev_rank method in RedisSortedSet:

module BYORedis
  class RedisSortedSet

    # ...

    def rev_rank(member)
      member_rank = rank(member)
      cardinality - 1 - member_rank if member_rank
    end

    # ...

  end
end

listing 10.65 The RedisSortedSet#rev_rank method

We don't need to change the order of the set to get the reversed rank of a member, we can use its "regular" rank, that we get with rank(member) and subtract it to the the cardinality of the set, and subtract one to that.

With the example from above, the rank of a in the set is 0, and the cardinality of z is 6, 6 - 0 - 1 == 5, the correct result. It also works for e, which has a rank of 4 and a "revrank" of 1: 6 - 4 - 1 == 1.

Pop Commands

We mentioned earlier the pop commands as ways to remove members from a sorted set. ZPOPMIN removes the element with the lowest score, and returns it and ZPOPMAX does the same with the member with the highest score.

Similarly to how we implemented BLPOP, BRPOP and BRPOPLPUSH in Chapter 7, we are going to add the two blocking variants of ZPOPMIN & ZPOPMAX: BZPOPMIN & BZPOPMAX. The semantics are similar to the blocking list commands, a timeout is given, where 0 means infinite, and the server will block the client until a value can be returned or until the timeout expires.

The format of ZPOPMAX is the following according to the Redis Documentation:

ZPOPMAX key [count]

count must be an integer and has a default value of 1. Negative values are accepted but turn the command in a no-op since an array cannot have a negative number of items, so Redis always returns an empty array.

Let's start with the ZPopMaxCommand class:

module BYORedis
 module SortedSetUtils

    # ...

    def self.generic_zpop(db, args)
      Utils.assert_args_length_greater_than(0, args)
      key = args.shift
      count = args.shift
      raise RESPSyntaxError unless args.empty?

      count = if count.nil?
                1
              else
                Utils.validate_integer(count)
              end

      sorted_set = db.lookup_sorted_set(key)
      popped = []

      if sorted_set
        popped = db.generic_pop(key, sorted_set, count) do
          yield sorted_set, count
        end
      end

      RESPArray.new(popped)
    end
  end

  # ...

  class ZPopMaxCommand < BaseCommand
    def call
      SortedSetUtils.generic_zpop(@db, @args) do |sorted_set, count|
        sorted_set.pop_max(count)
      end
    end

    def self.describe
      Describe.new('zpopmax', -2, [ 'write', 'fast' ], 1, 1, 1,
                   [ '@write', '@sortedset', '@fast' ])
    end
  end
end

listing 10.66 The ZPopMaxCommand class

We already know we have to implement two very similar commands, ZPOPMAX and ZPOPMIN, and both accept the same options, so we use the generic_zpop method to encapsulate all the logic that can be shared. The command handles the argument validation, as well as using a default value for count and calls yield with the sorted_set loaded from memory and the count value parsed an Integer. Elements are always returned with their scores when popped, so we'll need to make sure to include when returning it. The popped variable is expected to be an array from the result of yield and is returned back to the client. Back in ZPopMaxCommand#call, we call RedisSortedSet#pop_max from the block with the variables given to us by generic_zpop.

And we now need to add the RedisSortedSet#pop_max method:

module BYORedis
  class RedisSortedSet

    # ...

    def pop_max(count)
      case @underlying
      when List then generic_pop(count) { @underlying.right_pop&.value }
      when ZSet
        generic_pop(count) do
          max = @underlying.array.pop
          @underlying.dict.delete(max.member) if max
          max
        end
      else raise "Unknown type for #{ @underlying }"
      end
    end

    # ...

    private

    # ...

    def generic_pop(count)
      popped = []
      return popped if count < 0

      while count > 0
        min = yield

        if min
          popped.push(min.member, min.score)
          count -= 1
        else
          break
        end
      end

      popped
    end
  end
end

listing 10.67 The RedisSortedSet#pop_max method

Once again trying to always be one step ahead, we use a generic method that will be useful when implementing ZPOPMIN, generic_pop, to which we pass the count value.

This method handles the negative count scenario and returns an empty array right away in this case. Otherwise, it keeps iterating until count reaches 0, at each step we yield back to the caller, which in this case is either a call to @underlying.right_pop if we're dealing with a List or @underlying.array.pop followed by a call to Dict#delete if we're dealing with a ZSet. In either case, the Pair instance is returned and generic_pop aggregates it in popped, and decrements count. If the previous iteration deleted the last element in the set, we'll receive nil from yield, and in this case we must stop iterating right away, there's nothing else to pop.

Next up is ZPopMinCommand:

module BYORedis

  # ...

  class ZPopMinCommand < BaseCommand
    def call
      SortedSetUtils.generic_zpop(@db, @args) do |sorted_set, count|
        sorted_set.pop_min(count)
      end
    end

    def self.describe
      Describe.new('zpopmin', -2, [ 'write', 'fast' ], 1, 1, 1,
                   [ '@write', '@sortedset', '@fast' ])
    end
  end
end

listing 10.68 The ZPopMinCommand class

The only different with ZPopMaxCommand is that the block given to generic_zpop calls pop_min instead of pop_max, let's add the command to RedisSortedSet:

module BYORedis
  class RedisSortedSet

    # ...

    def pop_min(count)
      case @underlying
      when List then generic_pop(count) { @underlying.left_pop&.value }
      when ZSet
        generic_pop(count) do
          min = @underlying.array.shift
          @underlying.dict.delete(min.member) if min
          min
        end
      else raise "Unknown type for #{ @underlying }"
      end
    end

    # ...

  end
end

listing 10.69 The RedisSortedSet#pop_min method

The block we give to generic_pop is the one that actually performs the pop operation and in this case we use List#left_pop to remove the element with the smallest rank or Array#shift for a ZSet. We return the Pair instance in either case to let generic_pop handle everything for us and return the list of deleted pairs.

Blocking Commands

Both blocking commands have identical formats according to the Redis Documentation for BZPOPMIN:

BZPOPMIN key [key ...] timeout

and the Redis Documentation for BZPOPMAX

BZPOPMAX key [key ...] timeout

Let's go ahead and add both classes, BZPopMaxCommand & BZPopMinCommand:

module BYORedis
 module SortedSetUtils

    # ...

    def self.generic_bzpop(db, args, operation)
      Utils.assert_args_length_greater_than(1, args)
      timeout = Utils.validate_timeout(args.pop)
      args.each do |set_name|
        sorted_set = db.lookup_sorted_set(set_name)
        next if sorted_set.nil?

        popped = db.generic_pop(set_name, sorted_set, 1) do
          yield sorted_set
        end
        return RESPArray.new([ set_name ] + popped)
      end

      Server::BlockedState.new(
        BlockedClientHandler.timeout_timestamp_or_nil(timeout), args, operation)
    end
  end

  # ...

  class BZPopMaxCommand < BaseCommand
    def call
      SortedSetUtils.generic_bzpop(@db, @args, :zpopmax) do |sorted_set|
        sorted_set.pop_max(1)
      end
    end

    def self.describe
      Describe.new('bzpopmax', -3, [ 'write', 'noscript', 'fast' ], 1, -2, 1,
                   [ '@write', '@sortedset', '@fast', '@blocking' ])
    end
  end

  class BZPopMinCommand < BaseCommand
    def call
      SortedSetUtils.generic_bzpop(@db, @args, :zpopmin) do |sorted_set|
        sorted_set.pop_min(1)
      end
    end

    def self.describe
      Describe.new('bzpopmin', -3, [ 'write', 'noscript', 'fast' ], 1, -2, 1,
                   [ '@write', '@sortedset', '@fast', '@blocking' ])
    end
  end
end

listing 10.70 The BZPopMinCommand & BZPopMaxCommand classes

Let's ignore the blocking behavior for now, if any of sets for the given keys exist, we want to pop from it, either with pop_min or pop_max. We only need to pop one element so we set the count argument to 1 with .pop_min(1) and .pop_max(1). This is what we call from the block we give to generic_bzpop, but let's look at what this method does in details. Once the timeout value is validated, we iterate over all the arguments one by one, each time we load the set, and jump to the next one if it's nil. If it is not nil, we yield it back to the caller, which as we've just seen will call the appropriate pop method, and we exit the loop, returning the value that was popped. There's no blocking behavior in this case.

In the case where we exhausted the list of all arguments, we want to block until one of these sets receives an element with ZADD, which is very similar to the BLPOP and BRPOP behavior, but with sorted sets this time.

We are going to reuse a lot of the BlockedClientHandler logic, but we need to make it aware of sorted sets now, back then it only knew about lists, but first we return an instance of BlockedState, with the result of timeout_timestamp_or_nil, a method we introduced in Chapter 7, that either converts the float timeout value by adding the value to the current time or return nil, indicating an infinite timeout. The last two arguments are args and operation. args has the all the sorted set keys, and operation is either the symbol :zpopmin or the symbol :zpopmax.

The server class already knows how to handle BlockedState instances with the block_client method, and will add it to its internal client_timeouts sorted array. Let's take a a look at all the changes in the BlockedClientHandler class. As a reminder, the handle_client method is called from the Server class, from the handle_clients_blocked_on_keys method, which is called after each command is processed.

module BYORedis
  class BlockedClientHandler

    def initialize(server, db)
      # ...
    end

    def self.timeout_timestamp_or_nil(timeout)
      if timeout == 0
        nil
      else
        Time.now + timeout
      end
    end

    def handle(key)
      clients = @db.blocking_keys[key]

      list_or_set = @db.data_store[key]
      raise "Unexpected empty list/sorted set for #{ key }" if list_or_set&.empty?

      unblocked_clients = serve_client_blocked_on(key, list_or_set, clients)

      @db.blocking_keys.delete(key) if clients.empty?

      unblocked_clients
    end

    private

    def pop_operation(key, list, operation, target)
      case operation
      when :lpop
        RESPArray.new([ key, @db.left_pop_from(key, list) ])
      when :rpop
        RESPArray.new([ key, @db.right_pop_from(key, list) ])
      when :rpoplpush
        raise "Expected a target value for a brpoplpush handling: #{ key }" if target.nil?

        ListUtils.common_rpoplpush(@db, key, target, list)
      when :zpopmax
        RESPArray.new([ key ] + @db.pop_max_from(key, list))
      when :zpopmin
        RESPArray.new([ key ] + @db.pop_min_from(key, list))
      else
        raise "Unknown pop operation #{ operation }"
      end
    end

    def rollback_operation(key, response, operation, target_key)
      case operation
      when :lpop
        element = response.underlying_array[1]
        list = @db.lookup_list_for_write(key)
        list.left_push(element)
      when :rpop
        element = response.underlying_array[1]
        list = @db.lookup_list_for_write(key)
        list.right_push(element)
      when :rpoplpush
        list = @db.lookup_list_for_write(key)
        target_list = @db.lookup_list(target_key)
        element = target_list.left_pop
        @db.data_store.delete(target_key) if target_list.empty?
        list.right_push(element.value)
      when :zpopmax, :zpopmin
        sorted_set = @db.lookup_sorted_set_for_write(key)
        member = response.underlying_array[1]
        score = response.underlying_array[2]
        sorted_set.add(score, member)
      else
        raise "Unknown pop operation #{ operation }"
      end
    end

    def handle_client(client, key, list_or_set)
      blocked_state = client.blocked_state

      # The client is expected to be blocked on a set of keys, we unblock it based on the key
      # arg, which itself comes from @db.ready_keys, which is populated when a key that is
      # blocked on receives a push
      # So we pop (left or right for list, min or max for a set) at key, and send the response
      # to the client
      if client.blocked_state

        response =
          pop_operation(key, list_or_set, blocked_state.operation, blocked_state.target)

        serialized_response = response.serialize
        @logger.debug "Writing '#{ serialized_response.inspect } to #{ client }"

        unless Utils.safe_write(client.socket, serialized_response)
          # If we failed to write the value back, we put the element back in the list or set
          rollback_operation(key, response, blocked_state.operation, blocked_state.target)
          @server.disconnect_client(client)
          return
        end
      else
        @logger.warn "Client was not blocked, weird!: #{ client }"
        return
      end

      true
    end

    def serve_clients_blocked_on_lists(key, list_or_set, clients)
      generic_serve_clients(clients, list_or_set) do |client, clients_waiting_on_different_type|
        if is_client_blocked_on_list?(client)
          handle_client(client, key, list_or_set)
        else
          clients_waiting_on_different_type << client
          nil
        end
      end
    end

    def serve_clients_blocked_on_sorted_sets(key, list_or_set, clients)
      generic_serve_clients(clients, list_or_set) do |client, clients_waiting_on_different_type|
        if is_client_blocked_on_sorted_set?(client)
          handle_client(client, key, list_or_set)
        else
          clients_waiting_on_different_type << client
          nil
        end
      end
    end

    def generic_serve_clients(clients, list_or_set)
      unblocked_clients = []
      clients_waiting_on_different_type = []
      cursor = clients.left_pop

      while cursor
        client = cursor.value

        unblocked_clients << client if yield(client, clients_waiting_on_different_type)

        if list_or_set.empty?
          break
        else
          cursor = clients.left_pop
        end
      end

      return unblocked_clients, clients_waiting_on_different_type
    end

    def serve_client_blocked_on(key, list_or_set, clients)
      unblocked_clients = []
      clients_waiting_on_different_type = List.new

      case list_or_set
      when List then
        unblocked_clients, clients_waiting_on_different_type =
          serve_clients_blocked_on_lists(key, list_or_set, clients)
      when RedisSortedSet
        unblocked_clients, clients_waiting_on_different_type =
          serve_clients_blocked_on_sorted_sets(key, list_or_set, clients)
      else
        @logger.warn "Found neither a list or sorted set: #{ key } / #{ list_or_set }"
        raise "Found nil or neither a list or sorted set: #{ key } / #{ list_or_set }"
      end

      # Take all the clients we set aside and add them back
      clients_waiting_on_different_type.each do |client|
        clients.right_push(client)
      end

      unblocked_clients
    end

    def is_client_blocked_on_list?(client)
      return false unless client.blocked_state

      [ :lpop, :rpop, :rpoplpush ].include?(client.blocked_state.operation)
    end

    def is_client_blocked_on_sorted_set?(client)
      return false unless client.blocked_state

      [ :zpopmax, :zpopmin ].include?(client.blocked_state.operation)
    end
  end
end

listing 10.71 The updates to the BlockedClientHandler class for sorted sets

handle is called with the key of a collection, either a sorted set or a list, that was just added. The same way we used to, we get the list of all clients blocked for that key, through the @db.blocking_keys dictionary and we also get the actual collection, from @db.data_store, which is still of an unknown type at this point. It could either be a List or a SortedSet. We call serve_client_blocked_on with all the variable we just created.

In serve_client_blocked_on we check for the type list_or_set and call the appropriate method depending on what we find, serve_clients_blocked_on_lists or serve_clients_blocked_on_sorted_sets. Each of these methods make use of the generic generic_serve_clients method, in which we left pop from the clients list, to get the client who is first in line.

The clients list acts a queue where elements are right pushed and left popped to use a FIFO mechanism. We also create a new list, clients_waiting_on_different_type which will become handy shortly.

Because we might be able to unblock more than one client, we iterate over all the nodes in client, and for each of them we yield back to either serve_clients_blocked_on_sorted_sets or serve_clients_blocked_on_lists. We need to check which type of keys the client was blocked for. If they ended up in the blocked queue because of BLPOP, BRPOP or BRPOPLPUSH, they need any of the keys they're blocked on to become a new list, and if they were blocked because of a BZPOPMIN or BZPOPMAX command, they need it to be a sorted. An example will illustrate this clearly:

127.0.0.1:6379> BLPOP something something-else 0

While this client is blocked, if I open another redis-cli and run the following:

127.0.0.1:6379> SET something 123
OK
127.0.0.1:6379> ZADD something-else 0 a 0 b
(integer) 2

Then the first client is still blocked, because neither something or something-else were created as a List. The same goes if a client is blocked on a sorted set, it will only unblock if one the keys is created as a sorted set.

We check for this with the methods is_client_blocked_on_list? and is_client_blocked_on_sorted_set? which both look at the operation attribute of the BlockedState object to determine which type of object the client is waiting for. If they match, that is if the client is waiting for a List and list_or_set is a List, or the other way around for sorted sets, then we proceed and call handle_client, otherwise we accumulate the clients in a different array, to make sure they are not deleted from the queue and so they are eventually unblocked in the future.

handle_client returns a boolean, because it might actually fail to unblock the client in some uncommon edge cases. generic_serve_clients accumulates the unblocked clients in the unblocked_clients array, and it returns it, alongside all the clients that were waiting for the the other type.

These two arrays are handled in serve_client_blocked_on, where we iterate through clients_waiting_on_different_type to put them back in the queue, and unblocked_clients is returned to handle.

handle_client has been updated as well, mainly to rename the variable list to list_or_set, but the actual changes are in the pop_operation and rollback_operation methods. These methods use a case/when against the operation attribute to determine what to do. We need to add branches for :zpopmax and :zpopmin in order to be able to perform the right pop operation when a sorted set is created and we're in the process of unblocking a client.

For :zpopmax we call RESPArray.new([ key ] + @db.pop_max_from(key, list_or_set)) and for :zpopmin we call RESPArray.new([ key ] + @db.pop_min_from(key, list_or_set)). The array returned in each case is a three-element array with the name of the sorted array, the key, and the member/score pair. The key is required because a client might pass more than a single key to block on and without returning it, they wouldn't know which of the keys triggered the unblocking.

We need to add these two methods to the DB class:

module BYORedis
  class DB

    # ...

    def pop_max_from(key, sorted_set)
      generic_pop(key, sorted_set) do
        sorted_set.pop_max(1)
      end
    end

    def pop_min_from(key, sorted_set)
      generic_pop(key, sorted_set) do
        sorted_set.pop_min(1)
      end
    end

    # ...
  end
end

listing 10.72 The pop_min_from and pop_max_from methods in the DB class

Both methods wrap the specific pop method from RedisSet and use generic_pop to make sure that if the popped element was the last one the sorted set is deleted from the database

In the rollback_operation we can handle both operations the same way since calling RedisSortedSet#add will insert the element at the correct location.

This wraps up the blocking commands. We now have three more small commands to add to complete this chapter.

Misc commands

There are three more commands left to add to complete all the sorted set commands. Let's start with ZCOUNT which counts the number of elements in the given score range. Its format is the following according to the Redis Documentation:

ZCOUNT key min max

We kept the simplest commands for last, this one takes three arguments, the key of the sorted set and the min and max values using the same semantic as we've seen in ZRANGEBYSCORE, that is we can use the exclusive prefix (.

module BYORedis
  module SortedSetUtils

    # ...

    def self.generic_count(db, args)
      Utils.assert_args_length(3, args)
      key = args[0]
      min = args[1]
      max = args[2]
      sorted_set = db.lookup_sorted_set(key)

      count = yield(sorted_set, min, max) || 0

      RESPInteger.new(count)
    end
  end

  # ...

  class ZCountCommand < BaseCommand
    def call
      SortedSetUtils.generic_count(@db, @args) do |sorted_set, min, max|
        range_spec = Utils.validate_score_range_spec(min, max)
        sorted_set&.count_in_score_range(range_spec)
      end
    end

    def self.describe
      Describe.new('zcount', 4, [ 'readonly', 'fast' ], 1, 1, 1,
                   [ '@read', '@sortedset', '@fast' ])
    end
  end
end

listing 10.73 The ZCountCommand class

We use the generic_count method from ZCountCommand, because we'll be able to use later with ZLEXCOUNT. The generic method validates the argument array length, looks up the sorted set and yields the value back to the call method, and return what the block returns, as an Integer, which is the count value.

The min and max value are sent back to the caller with yield because their handling will depend on the type of range we're expecting. With this command we need to treat them as range score items and in ZLEXCOUNT we will treat them as lex range items.

We use validate_score_range_spec to create a new instance of GenericRangeSpec, specific to scores, and pass it to count_in_score_range_spec on RedisSortedSet:

module BYORedis
  class RedisSortedSet

    # ...

    def count_in_score_range(range_spec)
      return 0 if range_spec.empty? ||
                  no_overlap_with_range?(range_spec) { |pair, _| pair.score }

      case @underlying
      when List then count_in_score_range_list(range_spec)
      when ZSet then @underlying.count_in_score_range(range_spec)
      else raise "Unknown type for #{ @underlying }"
      end
    end

    # ...

    private

    def generic_count_list(range_spec)
      count = 0
      entered_range = false

      @underlying.each do |pair|
        in_range = range_spec.in_range?(yield(pair))

        if in_range
          entered_range ||= true
          count += 1
        elsif entered_range
          break
        end
      end

      count
    end

    def count_in_score_range_list(range_spec)
      generic_count_list(range_spec, &:score)
    end
  end
end

listing 10.74 The RedisSortedSet#count_in_score_range method

The first two lines try to return early if the range spec is empty or if there's no overlap with the set, otherwise we call count_in_score_range_list or ZSet#count_in_score_range. Let's in RedisSortedSet and look at the list method.

Most of the work is performed by the generic_count_list method, which iterates from the left to right with each, and keeps track of when it enters the range with the entered_range boolean. Determining if an element is in range is done by calling in_range? with the range spec, with the value returned by yield(pair), which in this case returns the score attribute of pair.

We keep incrementing the count variable as long as we find elements in the range, and as soon as we find an element outside of the range, we exit early, since there's no point in continuing iterating at this point.

The ZSet case can be optimized thanks to the SortedArray class:

module BYORedis
  class ZSet

    # ...

    def count_in_score_range(range_spec)
      generic_count(range_spec, &:score)
    end

    private

    # It is more than recommended to check that there is some overlap between the range_spec and
    # this set RedisSortedSet provides that with the no_overlap_with_range? method
    def generic_count(range_spec, &block)
      first_in_range_index = @array.first_index_in_range(range_spec, &block)
      last_in_range_index = @array.last_index_in_range(range_spec, &block)

      # We need to add 1 because the last index - the first index is off by one:
      # < 1, 2, 3, 4, 5>, with the range 2, 4, has the indices 1 & 3, 3 - 1 + 1 == 3
      # If 4 had been specified as exclusive with (4, then the result should be 2, because
      # only the scores 2 & 3 (at index 1 & 2) fit in that range, and 2 - 1 + 1 == 2
      last_in_range_index - first_in_range_index + 1
    end
  end
end

listing 10.75 The ZSet#count_in_score_range method

ZSet#count_in_score_range acts very similarly to count_in_rank_range_list, it is a wrapper around the method that actually does the work, generic_count in this case, with a block that returns the score attribute of a Pair instance.

generic_count uses methods we've created before on SortedArray, first_index_in_range and last_index_in_range, which will both be "fast" by relying on bsearch_index under the hood. What's important to note is that it won't require a full iteration of the array, and each call to bsearch_index will have a O(logn) time complexity, making the time complexity of generic_count O(2logn), which happens to be on the same ballpark than O(logn). Put simply, the number of steps it takes to complete this method will grow as the set grows, but the growth will not be near to a linear growth, essentially, it will slow down "slowly".

Once we have the first and last index, we can subtract them and add one to get the number of items in the range.

We could have chosen a different approach, to only call first_index_in_range, to "jump" to the first element in the range, and to keep counting from there until we find an element outside the range. This alternative approach might actually be more efficient for very small ranges, within a very large sorted sets. In this case the binary search approach, while always being more efficient than a full scan, will require a lot of iterations to reach the element it's trying to find, but if the range is really small, it might actually be really fast to advance by that many items in the range and return that small count.

Being able to find an acceptable threshold is complicated, and would require a lot of manual testing to find appropriate values.

There would be a potential problem by always using this alternative approach, we could trigger a very slow count if the range was really big, say, more than a million items, we'd be looping a million times. The approach we picked might not always be the best but it will never be terrible thanks to the O(logn) time complexity of the binary search, which is at least a good starting point.

An approximation of the number of steps required to find an element with bsearch_index can be done by using the log2 n function, the binary logarithm. log2n(1,000,000) ~= 19. This is because at first the binary search operates on the full array, 1,000,000 items, it then slices it in half and looks at 500,000, and so on, and after doing that 19 times, there should only be one or two items left. This tells us that in the worst case scenario, calling first_index_in_range and then last_index_in_range would require about ~38 iterations to retrieve both items, which is way less than iterating a million times to count items one by one!

The ZLEXCOUNT command is very similar, except that it works with a lexicographic range instead of a score range, its format is the following according to the Redis Documentation:

ZLEXCOUNT key min max

Let's create the ZLexCountCommand class:

module BYORedis

  # ...

  class ZLexCountCommand < BaseCommand
    def call
      SortedSetUtils.generic_count(@db, @args) do |sorted_set, min, max|
        range_spec = Utils.validate_lex_range_spec(min, max)
        sorted_set&.count_in_lex_range(range_spec)
      end
    end

    def self.describe
      Describe.new('zlexcount', 4, [ 'readonly', 'fast' ], 1, 1, 1,
                   [ '@read', '@sortedset', '@fast' ])
    end
  end
end

listing 10.76 The ZLexCountCommand class

We call generic_count the same way we did with ZCOUNT, but this time we use validate_lex_range_spec with the min and max variable, to create an instance of GenericRangeSpec specific to lexicographic order. We then call count_in_lex_range from RedisSortedSet:

module BYORedis
  class RedisSortedSet

    # ...

    def count_in_lex_range(range_spec)
      return 0 if range_spec.empty? ||
                  no_overlap_with_range?(range_spec) { |pair, _| pair.member }

      case @underlying
      when List then count_in_lex_range_list(range_spec)
      when ZSet then @underlying.count_in_lex_range(range_spec)
      else raise "Unknown type for #{ @underlying }"
      end
    end

    # ...

    private

    # ...

    def count_in_lex_range_list(range_spec)
      generic_count_list(range_spec, &:member)
    end
  end
end

listing 10.77 The RedisSortedSet#count_in_lex_range method

count_in_lex_range is very similar to count_in_rank_range, with the same early return check, and also delegates to two methods, count_in_lex_range_list and ZSet#count_in_lex_range. Both have these methods make use of the generic methods we created earlier and are very concise. count_in_lex_range_list can use generic_count_list, which does the iteration and counting work and only need a block to tell it what attribute to use from Pair instances.

The same goes for ZSet#count_in_lex_range, which uses generic_count giving it the range spec to use, and telling it to use the pair attribute from the Pair instances:

module BYORedis
  class ZSet

    # ...

    def count_in_lex_range(range_spec)
      generic_count(range_spec, &:member)
    end

    # ...

  end
end

listing 10.78 The ZSet#count_in_range method

This wraps up the ZLEXCOUNT method.

Finally, the ZINCRBY command increments the score of a member by the given increment value, its format is the following according to the Redis Documentation:

ZINCRBY key increment member

The command is very similar to HINCRBYFLOAT, and performs a floating point increment, with all the infinity and NaN handling it requires.

Let's create the ZIncrByCommand class:

module BYORedis

  # ...

  class ZIncrByCommand < BaseCommand
    def call
      Utils.assert_args_length(3, @args)
      incr = Utils.validate_float(@args[1], 'ERR value is not a valid float')

      key = @args[0]
      member = @args[2]

      sorted_set = @db.lookup_sorted_set_for_write(key)
      new_score = sorted_set.increment_score_by(member, incr)

      RESPBulkString.new(Utils.float_to_string(new_score))
    rescue InvalidFloatString
      RESPError.new('ERR hash value is not a float')
    rescue FloatNaN
      RESPError.new('ERR resulting score is not a number (NaN)')
    end

    def self.describe
      Describe.new('zincrby', 4, [ 'write', 'denyoom', 'fast' ], 1, 1, 1,
                   [ '@write', '@sortedset', '@fast' ])
    end
  end
end

listing 10.79 The ZIncrByCommand class

Once the sorted set is loaded, we call RedisSortedSet#increment_score_by:

module BYORedis
  class RedisSortedSet

    # ...

    def increment_score_by(member, increment)
      current_score = score(member) || BigDecimal(0)

      new_score = Utils.add_or_raise_if_nan(current_score, increment)
      add(new_score, member)

      new_score
    end

    # ...

  end
end

listing 10.80 The RedisSortedSet#increment_score_by method

If the result of score(member) is nil, which would happen if member is not already in the set, then we default it to 0. Once loaded, we safely add the increment value to it. Redis does not store NaN values so we use the method from Utils to raise a FloatNaN error if this happens. The new score is then either added or updated with the add method, depending on whether or not member was already in the set, but add knows how to take care of that.

The FloatNaN exception is handled in the command class and returns the appropriate error message in that case, otherwise the result is returned as a string, because RESP2 does not have support for floating numbers.

Conclusion

You can find the code on GitHub

We have now implemented all the native data types, with the exception of Streams. In the next chapter we will add support for the bitmap related commands, which operate on string values but almost behave as a different data type.

Rebuilding Redis in Ruby - Chapter 9 - Adding Set Commands

Pierre Jambet — Thu, 12 Nov 2020 14:13:46 +0000

What we'll cover

Our server now supports Strings, albeit not all string commands are implemented, Lists and Hashes. Redis supports three more native data types, Sets, Sorted Sets and Streams. Streams being a recent addition to Redis, and being a fairly complicated topic, will not be covered in this book. Other data types exists, such as Bitmaps, HyperLogLog and Geospatial items, but all of these are implemented on top of the String native type.

In this chapter we will add support for the Set type. Conceptually a Set is a collection of unique items, and can therefore not contain duplicates. A common operation for Sets, beside the ability to add items to a set, is testing for the presence of an item. Such operation would ideally require constant time, that is a complexity of O(1).

Wikipedia defines a set as:

In computer science, a set is an abstract data type that can store unique values, without any particular order.

Sets, in computer science are very similar to Finite Sets in Mathematics. Redis Sets only store strings.

Given these constraints, it looks like our Dict data structure would be a great fit to implement an API for Sets. The Dict class stores key/value pairs, where keys are strings, so we wouldn't even need to add any values in the dict, adding key/value pairs with nil values would be sufficient.

It is interesting to consider that many data structures can be used to provide a set API. For instance Ruby arrays can be used as set with the following methods:

def new_set
  []
end

def include?(set, member)
  set.include?(member)
end

def add(set, member)
  if include?(set, member)
    false
  else
    set << member
    true
  end
end

listing 9.1 A Set using a Ruby array

These three methods do provide the API for a set, but the performance, especially in the worst case scenario, will degrade quickly as the set grows. The include? method needs to iterate over the entire array to return false, so calling include? for a member that is not in the array, will always require a full iteration of the array.

The List class we created in Chapter 7 could also be used to implement a set but would suffer from the exact same performance issues.

Ruby has a Set class, but we will not use it to follow the "build from scratch" approach we've been using so far.

Redis supports sixteen set commands:

SADD: Add one or more members to a set, creating it if necessary
SCARD: Return the cardinality of the set, the number of members
SDIFF: Compute the difference of sets, from left to right
SDIFFSTORE: Same as SDIFF but store the result in another key instead of returning it
SINTER: Compute the intersection of all given sets
SINTERSTORE: Same as SINTER but store the result in another key instead of returning it
SUNION: Compute the union of all given sets
SUNIONSTORE: Same as SUNION but store the result in another key instead of returning it
SISMEMBER: Return 0 or 1 depending on whether or not the given member is in the given set
SMEMBERS: Return all the members of the given set
SMISMEMBER (new in 6.2.0): Variadic version of SISMEMBER, that is, it accepts multiple arguments and returns an array
SMOVE: Move a member from one set to another
SPOP: Remove a member from the given set
SRANDMEMBER: Return a member from the given set, but leave it in the set
SREM: Remove the given member from the given set
SSCAN: Similar to SCAN, HSCAN & ZSCAN. We will not implement it for the same reason we did not implement HSCAN in Chapter 8

How does Redis do it?

As mentioned in the previous chapter, the set commands are implemented in the t_set.c file, in which Redis uses two different data structures to store the data. As long as the elements, also called members in a set, can be represented as integers, and that the size of the set if below the set-max-intset-entries config value, which has a default value of 512, Redis uses an intset, and otherwise uses the dictionary from dict.c. We already created the Dict class in Chapter 6, but the intset is a new data structure.

Using an IntSet provides two interesting benefits, the memory footprint of the set is smaller, and most set operations will be faster than with a dict when the number of members is small. Note that while sets do not have to be ordered, int sets are actually ordered, which is how it provides a reasonably fast way to check for the existence of an element in a set, through binary search.

Everything is a string in a dict, this means that the number 1 would be stored as the character '1', which uses one byte of memory, eight bits, and the number 1,000 would be stored as the string '1000' and use four bytes, thirty two bits. This means that large numbers, such as, 1,000,000,000,000, one trillion, which is composed of thirteen digits, would use thirteen bytes, one hundred and four bits.

Storing these values as 64-bit integers would improve things for large numbers, one trillion would only require sixty four bits instead of one hundred and four, but it would actually make things worse for small numbers. If we were to store the number one as a 64-bit integer, it would use sixty four bits instead of eight if we stored it as a character.

Redis' intset type uses an encoding variable, which determine the size of the integers stored. Because the intset structure uses an array, it has to use the same underlying type for all elements. This is what allows indexed access in O(1) time. If we want to access the element at index i, we can multiply i by the size of the variables stored in the array, and we'll land on the i-th element. The content of the intset is an array of int8_t, so that each element uses exactly one byte. This allows Redis to store all the integers in this array, in 8-bit chunks, the actual length of each integer is determined by the encoding of the intset.

Redis uses three possible encoding values, int16_t, int_32_t & int64_t, respectively 16-bit integers, 32-bit integers, and 64-bit integers. When an intset is created, its encoding is set to 16-bit integers. Whenever new members are added to the set, Redis tries to find the smallest encoding that can hold the value. int16_t can range from -2^15 to 2^15 - 1, -32,768 to 32,767, int32_t from -2^31 to 2^31 - 1, -2,147,483,648 to 2,147,483,647 and int64_t from -2^63 to 2^63 - 1, -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807.

With this approach, storing numbers between 0 & 9 would still be more expensive, because they would be store as 16-bit integers, but we're already breaking even for negative numbers from -1 to -9, and we're saving space for numbers lower than or equal to -10, which would use three bytes as strings, and only use two as 16-bit integers, and for numbers greater than or equal to 100. The benefits become even greater as the number grow in either direction.

Speed wise, the benefits are the same that what we discussed in the previous chapter in the ziplist vs dict section. Using an intset will gradually become slower as the intset grows, but all operations will be really fast when the number of members is small. Large intset structure suffer from the same issues than ziplists and become slow to manipulate as they grow, because the whole memory chunk needs to be reallocated and items within it moved, to make space for new elements.

The intset functionalities are implemented in intset.c. An intset is essentially a sorted array of integers. We already implemented a sorted array, the SortedArray class, but the requirements are a little bit different here so we will create an IntSet class to mimic the Redis behavior.

As mentioned in the previous section, a dictionary provide a solid foundation to implement the Set API. It turns out that this is exactly what Redis does. When handling the SADD command, if the underlying data structure is a dict, it calls the dictAdd function with the member as the key, and nothing as the value, this is done in the setTypeAdd function.

Our Dict class already exists, and will only require a few changes, to make sure that it handles entries with nil values without any hiccups, on the other hand, we need to build an IntSet class from scratch, so let's get to it.

The IntSet data structure

Because the intset structure uses a sorted array, this makes the time complexity of a lookup O(logn), which will be worse than O(1) when the array gets bigger. Additionally, because arrays are contiguous chunks of memory, inserting an element requires a lot of work, to shift all the elements to the right of the new element. This makes arrays more and more expensive to manipulate as they grow in size, and explains why Redis only uses it if sets contain 512 items or less.

Redis stores all values in little endian in an intset. This means that the number 1, which has the following binary representation as a 16-bit integer:

irb(main):025:0> ('%016b' % 1).chars.each_slice(8).map { |byte| byte.join }.join(' ')
=> "00000000 00000001"

The two bytes, \x00 & \x01 need to be reversed in little endian, because the least significant bytes come first, so the bytes of 1, in little endian are [ "\x01", "\x00" ].

Let's look at a larger example, the representation of 1,000,000 as a 32-bit integer:

irb(main):027:0> ('%032b' % 1_000_000).chars.each_slice(8).map { |byte| byte.join }.join(' ')
=> "00000000 00001111 01000010 01000000"

The big endian bytes are [ "\x00", "\x0F", "\x42", "\x40" ], and [ "\x40", "\x42", "\x0F", "\x00 ] as little endian.

Another way to play with these values is to use the Array#pack method, with the l format, which is the format for signed 32-bit integer, which uses the native endianness of the platform by default, but can be forced to use big endian with l> and little endian with l<:

irb(main):043:0> [1_000_000].pack('l')
=> "@B\x0F\x00"
irb(main):044:0> [1_000_000].pack('l>')
=> "\x00\x0FB@"
irb(main):045:0> [1_000_000].pack('l<')
=> "@B\x0F\x00"

This example illustrates that the default endianness of my machine, a macbook pro, is little endian. Note that some of these bytes don't look like the others, "@" & "B" vs "\x0F" and "\x00". This is because Ruby attempts to display the characters as ASCII by default, and it turns out that "\x42" is the hex representation of the decimal 66, the character 'B' in ASCII and "\x40" is the hex representation of the decimal 64, the character '@' in ASCII.

Our class will implement the following public methods:

add(member)
cardinality
each
contains?(member)
members
pop
random_member
empty?
remove(member)

The methods above will allow us to implement all the Set related commands. All the following methods are implemented in the int_set.rb file, under the BYORedis::IntSet class, let's start with the constructor and the add method:

module BYORedis
  class IntSet

    INT16_MIN = -2**15 # -32,768
    INT16_MAX = 2**15 - 1 # 32,767
    INT32_MIN = -2**31 # -2,147,483,648
    INT32_MAX = 2**31 - 1 # 2,147,483,647
    INT64_MIN = -2**63 # -9,223,372,036,854,775,808
    INT64_MAX = 2**63 - 1 # 9,223,372,036,854,775,807

    # Each of the constant value represents the number of bytes used to store an integer
    ENCODING_16_BITS = 2
    ENCODING_32_BITS = 4
    ENCODING_64_BITS = 8

    def initialize
      @underlying_array = []
      @encoding = ENCODING_16_BITS
    end

    def add(member)
      raise "Member is not an int: #{ member }" unless member.is_a?(Integer)

      # Ruby's Integer can go over 64 bits, but this class can only store signed 64 bit integers
      # so we use this to reject out of range integers
      raise "Out of range integer: #{ member }" if member < INT64_MIN || member > INT64_MAX

      encoding = encoding_for_member(member)

      return upgrade_and_add(member) if encoding > @encoding

      # search always returns a value, either the position of the item or the position where it
      # should be inserted
      position = search(member)
      return false if get(position) == member

      move_tail(position, position + 1) if position < size

      set(position, member)

      true
    end

    private

    def set(position, member)
      @encoding.times do |i|
        index = (position * @encoding) + i
        @underlying_array[index] = ((member >> (i * 8)) & 0xff).chr
      end
    end

    def move_tail(from, to)
      @underlying_array[(to * @encoding)..-1] = @underlying_array[(from * @encoding)..-1]
    end

    def search(member)
      min = 0
      max = size - 1
      mid = -1
      current = -1

      # the index is always 0 for an empty array
      return 0 if empty?

      if member > get(max)
        return size
      elsif member < get(min)
        return 0
      end

      while max >= min
        mid = (min + max) >> 1
        current = get(mid)

        if member > current
          min = mid + 1
        elsif member < current
          max = mid - 1
        else
          break
        end
      end

      if member == current
        mid
      else
        min
      end
    end

    def get(position)
      get_with_encoding(position, @encoding)
    end

    def get_with_encoding(position, encoding)
      return nil if position >= size

      bytes = @underlying_array[position * encoding, encoding]

      # bytes is an array of bytes, in little endian, so with the small bytes first
      # We could iterate over the array and "assemble" the bytes into in a single integer,
      # by performing the opposite we did in set, that is with the following
      #
      # bytes.lazy.with_index.reduce(0) do |sum, (byte, index)|
      #   sum | (byte << (index * 8))
      # end
      #
      # But doing do would only work if the final result was positive, if the first bit of the
      # last byte was a 1, then the number we're re-assembling needs to be a negative number, we
      # could do so with the following:
      #
      # negative = (bytes[-1] >> 7) & 1 == 1
      #
      # And at the end of the method, we could apply the following logic to obtain the value,
      # get the 1 complement, with `~` and add 1. We also need to apply a mask to make sure that
      # the 1 complement result stays within the bounds of the current encoding
      # For instance, with encoding set to 2, the mask would be 0xffff, which is 65,535
      #
      # if negative
      #   mask = (2**(encoding * 8) - 1)
      #   v = -1 * ((~v & mask) + 1)
      # end
      #
      # Anyway, we can use the pack/unpack methods to let Ruby do that for us, calling
      # bytes.pack('C*') will return a string of bytes, for instance, the number -128 is stored
      # in the intset as [ 128, 255 ], calling, `.pack('C*')` returns "\x80\xFF". Next up, we
      # pick the right format, 's' for 16-bit integers, 'l' for 32 and 'q' for 64 and we let
      # Ruby put together the bytes into the final number.
      # The result of unpack is an array, but we use unpack1 here, which is a shortcut to
      # calling unpack() followed by [0]
      #
      # What this whole thing tells us is that we could have used `.pack('s').bytes` in the
      # set method, but using >> 8 is more interesting to understand actually what happens!
      format = case encoding
               when ENCODING_16_BITS then 's'
               when ENCODING_32_BITS then 'l'
               when ENCODING_64_BITS then 'q'
               end

      bytes.join.unpack1(format)
    end

    def encoding_for_member(member)
      if member < INT32_MIN || member > INT32_MAX
        ENCODING_64_BITS
      elsif member < INT16_MIN || member > INT16_MAX
        ENCODING_32_BITS
      else
        ENCODING_16_BITS
      end
    end

    def upgrade_and_add(member)
      current_encoding = @encoding
      current_size = size
      new_size = current_size + 1
      @encoding = encoding_for_member(member)

      prepend = member < 0 ? 1 : 0
      @underlying_array[(new_size * @encoding) - 1] = nil # Allocate a bunch of nils

      # Upgrade back to front
      while (current_size -= 1) >= 0
        value = get_with_encoding(current_size, current_encoding)
        # Note the use of the prepend variable to shift all elements one cell to the right in
        # the case where we need to add the new member as the first element in the array
        set(current_size + prepend, value)
      end

      if prepend == 1
        set(0, member)
      else
        set(size - 1, member)
      end

      true
    end
  end
end

listing 9.2 The IntSet#add method, and all the private methods it requires

The add method starts with a few checks to make sure that we can indeed add the given member to the set. If the value is not an integer or is out of range, we reject it right away. The next step is to find the smallest encoding that can fit member. Once we found the encoding, either 2, 4 or 8, we compare it to the current encoding of the set. If the new encoding is greater, then we know that member is either going to be the smallest entry in the set, or the largest, because it would otherwise have used the same encoding.

The upgrade_and_add method takes care of migrating all current elements to the new encoding, and inserts the new member, either at index 0 or as the last element in the array.

Let's look at the three main steps of the upgrade_and_add method, first it increases the size of the array, with @underlying_array[(new_size * @encoding) - 1] = nil, which pads the array to the right with nil values until the new size, let's look at an example:

irb(main):029:0> l = ["\x01", "\x00", "\x02", "\x00"]

This array is what the @underlying_array of an IntSet storing the values 1 and 2 would be. "\x01" and "\x00" is the little endian representation of the number 1 in a 16-bit integer, 0000 0001 0000 0000. The first byte, 0000 0001, written as 0x01 in the Ruby hex literal syntax is the byte for the number 1, since 2^0 == 1. The second byte, the most significant one only contains zeroes: 0000 0000. If the number had been 258, the two bytes would have been 0000 0010 0000 0001. The first byte is the least significant one, 0000 0010, these are the bits between index 0 and 7, representing the number 2, and the most significant one is 0000 0001, the bits between index 8 and 15, representing the number 1. Putting it all together we get 2^1 + 2^8 == 258. Another way to write this number is "\x02\x01"

"\x02", "\x00", is the little endian representation of the number 2, 0000 0010 0000 0000, written as 0x0002 as a hex literal. Note that hex literals in Ruby follow the "natural" order and assume big-endian. So 0x0002 returns 1, whereas 0x0200 returns 512, because 0000 0010 0000 0000 is 512 in big endian, 2^9 == 512.

If we were to increase the encoding to 32-bit, to store numbers larger than 32,767, each number would now need to use four bytes, so the array would now need to be 12 item long, 4 bytes x 3 members, as we can see, calling l[11] adds all the nil values:

irb(main):030:0> l[11] = nil
irb(main):031:0> l
=> ["\x01", "\x00", "\x02", "\x00", nil, nil, nil, nil, nil, nil, nil, nil]

The next step, upgrading all the elements, is done in the while loop. In our previous example, current_size would be 1. Calling get_with_encoding(2, 2) would read the numbers 2 and 0, and return 2. Calling set(1 + 0, 2) would insert the four bytes representing 2 at the right place in the array. If we were to add 32,768, then prepend would be 0.

The set method splits the number into the number of bytes for the encoding, with the new encoding being 4, to store numbers as 32-bit integer, we would iterate four times with the @encoding.times loop. In the first iteration index would be (1 * 4) + 0, 4 and the value would be "\x01" because ((member >> 0) 0xff).chr returns "\x01" if member is 1.

0xFF acts as a mask here, it is the hex literal for the number 255, a byte of only ones, we could have written 0b11111111, it just happens to be easier to use 0xff, but they're identical. We use 0xff in combination of the bitwise AND operator, &, which effectively only selects the rightmost byte of any numbers.

We call .chr, which is equivalent to calling .pack('C'). This allows us transform the Integer instance into a one-byte string. As we've already discussed, Ruby has a special handling for numbers, which is why it can handle numbers greater than what a 64-bit integer can handle, but it also means that we don't have any direct visibility in what is actually allocated when we have an instance of the Integer class. Using .pack('C') treats the number in the array as an 8-bit integer, a byte. What we get in return is a one-character string, representing the byte value. Let's look at an example:

irb(main):001:0> [1].pack('C')
=> "\x01"
irb(main):002:0> 1.chr
=> "\x01"
irb(main):003:0> [255].pack('C')
=> "\xFF"
irb(main):004:0> 255.chr
=> "\xFF"
irb(main):005:0> [128].pack('C')
=> "\x80"
irb(main):006:0> [1].pack('C')
=> "\x01"
irb(main):007:0> [255].pack('C')
=> "\xFF"
irb(main):008:0> [128].pack('C')
=> "\x80"
irb(main):009:0> [128].pack('C').size
=> 1
irb(main):010:0> [128].pack('C').chars
=> ["\x80"]

Even though it looks like the strings returned by pack/chr contains four characters, a double-quoted Ruby string handles the \x prefix in a special way, and as we can see with the last two examples, it only contains a single character.

The next iteration would give index the value 5, and set the value 0 at index 5. The last two iterations would set the value 0 for index 6 and 7.

Back to upgrade_and_add, the while loop would run one more time with current_size == 0 and upgrade the encoding of 1. After the while loop the array would contain the following, the existing inters have been migrated from 16-bit integers to 32-bit integers, in little endian, which result in right padding them with bytes containing zeroes:

[
  "\x01", "\x00", "\x00", "\x00",
  "\x02", "\x00", "\x00", "\x00",
  nil, nil, nil, nil,
]

The last step of the method is to add the new member, in this example prepend is not set to 1 and it is inserted as the last item with set(size - 1, member), where size == 3. set will split 32,768 into four bytes, which would be represented in big endian as the four bytes: 0x00, 0x00, 0x80, & 0x00:

# '%032b' % 32_768 =>
   MSB                           LSB    (Most Significant Byte/Least Significant Byte)
|-------| |-------| |-------| |-------|
7       0 7       0 7       0 7       0 (indices within each byte)
∨       ∨ ∨       ∨ ∨       ∨ ∨       ∨
0000 0000 0000 0000 1000 0000 0000 0000
∧                   ∧    ∧         ∧  ∧
31                  15   11        3  0 (indices within a 32-bit integer)

The only 1 in the previous binary number is at index 15, 2^15 = 32,768. The four bytes have the decimal values, 0, 0, 128 & 0. The second byte from the right have the value 128 because within that byte, the only 1 is at index 7 and 2^7 = 128.

Redis stores these numbers in little endian, so we store them in reverse order with the least significant bytes first: [ 0, 128, 0, 0 ]. The hex literal representation of 128 is 0x80.

With that, the final value of @underlying_array is:

[
  "\x01", "\x00", "\x00", "\x00",
  "\x02", "\x00", "\x00", "\x00",
  "\x00", "\x80", "\x00", "\x00",
]

The next step in add is to search for member in the array, which we do with the search method. The method always returns an integer, which is either the position of member in the array or the position where it should be inserted.

The search method uses a divide and conquer approach to find the element. First, it checks if the element should go at the front of the array, if the new member is smaller than the element at index 0, or at the end of the array if it is greater than the last element in array.

If neither of these checks are true, we look at the value at index mid, because we know that the array is sorted, we compare that value with member, if it is greater than member, then we know that member will either be on the left side, or not present and we set the max to mid - 1, effectively narrowing the range to the left side of the array. We perform the opposite operation if the value at index mid is lower than member, then we want to keep searching on the right half of the array, and we do so with min = mid + 1. We stop iterating if neither of these checks are true, that is, if current == member, in which case mid is the index of member.

The other condition causing the loop to exit is if min becomes greater than max, which means that we did not find member in the array, in which case min is the index where it should be inserted. Let's look at an example to illustrate this:

array = [10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65]

The variable array contains twelve elements, so min will start at 0, max at 11 and mid & current at -1.

If we were to search for 56, we would enter the while loop and set mid to (min + max) >> 1, which is (0 + 11) >> 1, 5, this is because 11 is 0b1011 in binary, and shifting it to the right one time results in 101, 2^0 + 2^2, 5. array[5] == 35, so member > current and we set min to mid + 1, 6.

Using a bitwise right shift of 1 is a "trick" to divide a number by two. Let's look at an example to illustrate it. With the previous example, min is 0, max is 11. Given that the array has an even number of items, there's no index where there would be the same number of elements to the left and to the right, but we can still use the integer division to find an index that is close to that, 11 / 2 == 5, so far so good.

   1011 (11)
>> 1
   ----
   0101 (5)

This property, that shifting by one to the right is the same as dividing by two happens to be the result from how we represent numbers in binary. A 1 in binary represent a power of two, so 11 is binary is really the sum of 2^0, 2^1 & 2^3. Shifting by one to the right returns the number that is the sum of 2^0 and 2^1, which happens to be the result of an integer division by two.

This also works with even numbers:

   1100 (10)
>> 1
   ----
   0110 (6)

Tada!

If it feels a little bit like magic, I encourage you to experiment with different numbers, either on paper or with irb, it took me a little while until it "clicked".

Back to the search method.

On the next iteration mid becomes 8, because (6 + 11) >> 1 is 0b10001 >> 1, which is 0b1000, 8. array[8] == 50, so we set min to 9.

On the next iteration, mid becomes 10, because (9 + 11) >> 1 is 0b10100 >> 1, 0b1010, 10. array[10] == 60, so this time we set max to mid - 1, 9.

On the next and last iteration, mid is 9, because (9 + 9) >> 1 is 0b10010 >> 1, 0b1001, 9. array[9] == 55, so min becomes 10, and the loop exits because min > max.

9 happens to be the index where 56 should be inserted to maintain the array sorted.

Reading values from the array, which is what the get method does, is more complicated than in a regular array. Our set members span across multiple cells in the array, because each cell contains one byte. So a member spans across two cells with the 16-bit encoding, four with the 32-bit encoding and eight with the 64-bit encoding.

The get_with_encoding method grabs all the bytes and uses the unpack1 method to reconstruct the integer from its byte parts. Ruby knows how to convert the bytes based on the format string, s means signed short integer, int16_t, l means signed long integer, int32_t and q means signed long long integer, int64_t.

These methods give us the foundation to write the remaining methods, let's look at each, members, empty? & size:

module BYORedis
  class IntSet

    # ...

    def empty?
      @underlying_array.empty?
    end

    def each(&block)
      members.each(&block)
    end

    def members
      size.times.map do |index|
        get(index)
      end
    end

    def size
      @underlying_array.size / @encoding
    end
    alias cardinality size
    alias card cardinality

    private

    # ...

  end
end

listing 9.3 The empty?, each, members and size methods in the IntSet class

The empty? method relies on the Array#empty? method, we don't need to change anything to its default behavior. The size method uses Array#size and divides the result by the size of encoding. This is because with the 16-bit encoding each integer will be split over two elements in the array, four elements with 32-bit and eight elements with 64-bit.

members uses size to determine how many times to iterate with the Integer#times method and passes the index to the get method we wrote earlier, which knows how to reassemble multiple array items into integers. Doing this in the block to map returns an array of Integer instances.

Finally the each method forwards its block argument to Array#each on the result of members.

An important method of the IntSet class is the include? method, which we can express in terms of search and get:

module BYORedis
  class IntSet

    # ...

    def include?(member)
      return false if member.nil?

      index = search(member)
      get(index) == member
    end
    alias member? include?

    # ...
  end
end

listing 9.4 The IntSet#include? method, aliased to member?

pop and rand_member are very similar, we use Kernel#rand with size as the exclusive upper boundary, which returns an index we can feed to get to return any elements of the set. In the pop case we do want to remove the item from the set and we do so with Array#slice!. This method takes two argument, the first one is the start index of the range we wish to delete and the second one is the length of the range.

remove uses Array#slice! in a similar way to how pop does it:

module BYORedis
  class IntSet

    # ...

    def pop
      rand_index = rand(size)
      value = get(rand_index)
      @underlying_array.slice!(rand_index * @encoding, @encoding)
      value
    end

    def random_member
      rand_index = rand(size)
      get(rand_index)
    end

    def remove(member)
      index = search(member)
      if get(index) == member
        @underlying_array.slice!(index * @encoding, @encoding)
        true
      else
        false
      end
    end

    private

    # ...
  end
end

listing 9.5 The pop, random_member and remove methods in the IntSet class

And with this the IntSet class is now feature complete.

Adding Set commands

Creating a Set with `SADD`

Let's start the same way we started in the previous in chapters, with the ability to create a new element in the main keyspace. Sets are created with the SADD command, which usually adds members to a set, if necessary, and creates a set if the key is not already used by a value of a different type.

require_relative './redis_set'

module BYORedis

  class SAddCommand < BaseCommand
    def call
      Utils.assert_args_length_greater_than(1, @args)
      key = @args.shift
      new_member_count = 0

      set = @db.lookup_set_for_write(key)

      @args.each do |member|
        added = set.add(member)
        new_member_count += 1 if added
      end

      RESPInteger.new(new_member_count)
    end

    def self.describe
      Describe.new('sadd', -3, [ 'write', 'denyoom', 'fast' ], 1, 1, 1,
                   [ '@write', '@set', '@fast' ])
    end
  end
end

listing 9.6 The SAddCommand class

The structure of the call method for the SAddCommand class is similar to HSet in the previous chapter. The first argument is the key for the hash, and the following arguments are the members to add to the set. Once the set is loaded, we iterate over the argument and use the RedisSet#add method.

In the previous chapter we created the RedisHash class, so let's create the RedisSet class, with the add method:

module BYORedis
  class RedisSet
    attr_reader :underlying

    def initialize
      @underlying = IntSet.new
    end

    def add(member)
      case @underlying
      when IntSet
        int_member = convert_to_int_or_nil(member)
        if int_member
          added = @underlying.add(int_member)

          if added && cardinality + 1 > Config.get_config(:set_max_intset_entries)
            convert_intset_to_dict
          end

          added
        else
          convert_intset_to_dict
          @underlying.set(member, nil)
        end
      when Dict then @underlying.set(member, nil)
      else
        raise "Unknown type for structure: #{ @underlying }"
      end
    end

    private

    def convert_intset_to_dict
      dict = Dict.new
      @underlying.each do |member|
        dict[Utils.integer_to_string(member)] = nil
      end

      @underlying = dict
    end

    def convert_to_int_or_nil(member)
      Utils.string_to_integer(member)
    rescue InvalidIntegerString
      nil
    end
  end
end

listing 9.7 The RedisSet#add method

We are going to use the same case/when pattern we introduced in the previous chapter but for IntSet and Dict instead of List and Dict. In the add method, if @underlying is an IntSet then we start by checking if the new member is a string that represents an integer by calling the convert_to_int_or_nil method. This method uses the Utils.string_to_integer method we introduced in the previous chapter.

If member can be converted to an integer, then we proceed with the IntSet instance and call the IntSet#add method. If the element was added to the set, that is, if it was not already in the set, then we check the cardinality of the set to see if it exceeded the set_max_intset_entries config value. If it did, the set is now too big to be an IntSet and we convert it to a Dict.

Luckily the conversion from IntSet to Dict, which we perform in the convert_intset_to_dict private method, does not require a lot of steps. We create a new Dict instance, and then iterate through all the IntSet members with the IntSet#each method and use the Dict#set method through its []= alias to add the members to the Dict. We only need the keys, so we set the value to nil for each new DictEntry we add.

If @underlying is an IntSet, but the new member cannot be represented as an integer, such as the string 'abc', then we convert the IntSet to a Dict, regardless of its current size, and add the new member with Dict#set. We don't use the []= alias here because doing so ignores the method's return value, and we want RedisSet#add to return the boolean returned by Dict#set, indicating whether or not the member was added to the set.

Finally, if @underlying is a Dict, which would be true in two cases, the set is either too big to be an IntSet, more than 512 members by default, or a member that cannot be converted to an integer was previously added. Regardless, we're now dealing with a Dict and we call Dict#set to add the member to the set.

Back to SAddCommand, we now need to add the lookup_set_for_write method to the DB class:

module BYORedis
  class DB

    # ...

    def lookup_set(key)
      set = @data_store[key]
      raise WrongTypeError if set && !set.is_a?(RedisSet)

      set
    end

    def lookup_set_for_write(key)
      set = lookup_set(key)
      if set.nil?
        set = RedisSet.new
        @data_store[key] = set
      end

      set
    end
  end
end

listing 9.8 The DB#lookup_set_for_write method

Now that we introduced a new type, set, we need to update the TypeCommand:

module BYORedis
  class TypeCommand < BaseCommand

    def call
      Utils.assert_args_length(1, @args)

      key = @args[0]
      ExpireHelper.check_if_expired(@db, key)
      value = @db.data_store[key]

      case value
      when nil       then RESPSimpleString.new('none')
      when String    then RESPSimpleString.new('string')
      when List      then RESPSimpleString.new('list')
      when RedisHash then RESPSimpleString.new('hash')
      when RedisSet  then RESPSimpleString.new('set')
      else raise "Unknown type for #{ value }"
      end
    end

    # ...

  end
end

listing 9.9 Adding set to the TypeCommand class

In the next section we will add the six commands implementing set operations, namely difference, union and intersection, and their *STORE variants.

Set operations with SDIFF, SINTER, SUNION and their *STORE variants

Set difference

module BYORedis

  # ...

  class SDiffCommand < BaseCommand
    def call
      Utils.assert_args_length_greater_than(0, @args)
      sets = @args.map { |other_set| @db.lookup_set(other_set) }

      RESPArray.new(RedisSet.difference(sets).members)
    end

    def self.describe
      Describe.new('sdiff', -2, [ 'readonly', 'sort_for_script' ], 1, -1, 1,
                   [ '@read', '@set', '@slow' ])
    end
  end
end

listing 9.10 The SDiffCommand class

The SDIFF command performs the "set difference" operation, starting with the leftmost set, it then iterates through all the other sets and removes their members from the first set, if it contained them, let's look at some examples:

127.0.0.1:6379> SADD s1 2 4 6 8 10 12
(integer) 6
127.0.0.1:6379> SADD s2 1 2 3 4
(integer) 4
127.0.0.1:6379> SADD s3 5 6 7 8 9 10 11 12 13
(integer) 9
127.0.0.1:6379> SDIFF s1
1) "2"
2) "4"
3) "6"
4) "8"
5) "10"
6) "12"
127.0.0.1:6379> SDIFF s1 s2
1) "6"
2) "8"
3) "10"
4) "12"
127.0.0.1:6379> SDIFF s1 s2 s3
(empty array)
127.0.0.1:6379> TYPE s4
none
127.0.0.1:6379> SDIFF s1 s4
1) "2"
2) "4"
3) "6"
4) "8"
5) "10"
6) "12"

Set difference is also called relative complement in set theory.

The difference of single set always returns the same set, you can compare it to subtracting 0 to any number, the result is always the same number. In the second example, we compute the difference of s1 and s2, in other words we subtract all the elements in s2 from s1, so we remove 1, 2, 3, & 4 from s1. We end up with 6, 8, 10 & 12. 2 & 4 were removed and 1 & 3 were ignored since they were not present in s1.

In the last example, we start with the same operation from the second example, we subtract s2 from s1, and we subtract s3 from that result. Removing 5, 6, 7, 8, 9, 10, 11, 12 & 13 from the intermediary set containing 6, 8, 10 & 12 yields an empty set.

Non existing sets are treated as empty sets, which are ignored in the difference operation, similar to a subtraction by zero.

The difference operation is performed on multiple sets, and return a new set, which makes implementing it as an instance method a little bit odd since it operates on multiple sets, as opposed to a single set like most other instance methods. We therefore implement it as a class method on the RedisSet class:

module BYORedis
  class RedisSet

    # ...

    def self.difference(sets)
      first_set = sets[0]
      return RedisSet.new if first_set.nil?

      # Decide which algorithm to use
      #
      # Algorithm 1 is O(N*M) where N is the size of the element first set
      # and M the total number of sets.
      #
      # Algorithm 2 is O(N) where N is the total number of elements in all
      # the sets.
      algo_one_work = 0
      algo_two_work = 0
      sets.each do |other_set|
        algo_one_work += sets[0].cardinality
        algo_two_work += other_set ? other_set.cardinality : 0
      end
      # Directly from Redis:
      # Algorithm 1 has better constant times and performs less operations
      # if there are elements in common. Give it some advantage:
      algo_one_work /= 2
      diff_algo = (algo_one_work <= algo_two_work) ? 1 : 2

      if diff_algo == 1
        if sets.length > 1
          sets[0..0] + sets[1..-1].sort_by! { |s| -1 * s.cardinality }
        end
        difference_algorithm1(sets)
      else
        difference_algorithm2(sets)
      end
    end
  end
end

listing 9.11 The RedisSet.difference class method

Redis uses two different algorithms to perform the set difference operation, while it not possible to know for sure which one will be more efficient, it tries to guess, depending on the size of the sets, which one will be faster.

The first algorithm works by iterating through the first set, and for each element, we look into each other set, as soon we find the element, we stop and move to the next element in the first set. If we make it through all the other sets without finding the element, we add the member to a new set, acting the result set. In other words, the main criteria dictating the "cost" of this algorithm is the size of the first set. The bigger the first set, the more iteration we'll have to perform.

The second algorithm works by creating a new set with all the elements from the first set and then iterate through all the elements in all the other sets, removing each member from the new set. The "cost" of this algorithm is the sum of the cardinalities of all the sets, given that each set will be iterated once.

Redis gives the first algorithm an edge because it seems like it tends to be faster, what this means in practice is that algorithm 2 will only be picked if the first set is significantly bigger than every other sets, in which case we know that its performance would not be ideal by having to iterate over all the items in the first set no matter what and through all the other sets for each member in the first set, and we have a potential shortcut with algorithm 2.

module BYORedis
  class RedisSet

    # ...

    def self.difference_algorithm1(sets)
      return RedisSet.new if sets.empty? || sets[0].nil?

      dest_set = RedisSet.new

      sets[0].each do |element|
        i = 0
        other_sets = sets[1..-1]
        while i < other_sets.length
          other_set = other_sets[i]
          # There's nothing to do when one of the sets does not exist
          next if other_set.nil?
          # If the other set contains the element then we know we don't want to add element to
          # the diff set
          break if other_set == self

          break if other_set.member?(element)

          i += 1
        end

        if i == other_sets.length
          dest_set.add(element)
        end
      end

      dest_set
    end
    private_class_method :difference_algorithm1

    def self.difference_algorithm2(sets)
      return self if sets.empty? || sets[0].nil?

      dest_set = RedisSet.new

      # Add all the elements from the first set to the new one
      sets[0].each do |element|
        dest_set.add(element)
      end

      # Iterate over all the other sets and remove them from the first one
      sets[1..-1].each do |set|
        set.each do |member|
          dest_set.remove(member)
        end
      end

      dest_set
    end
    private_class_method :difference_algorithm2

    def include?(member)
      return false if member.nil?

      case @underlying
      when IntSet then
        if member.is_a?(Integer)
          member_as_int = member
        else
          member_as_int = Utils.string_to_integer_or_nil(member)
        end

        if member_as_int
          @underlying.member?(member_as_int)
        else
          false
        end
      when Dict then @underlying.member?(member)
      else raise "Unknown type for structure #{ @underlying }"
      end
    end
    alias member? include?

    # ...
  end
end

listing 9.12 The difference_algorithm1 & difference_algorithm2 class methods in RedisSet

Each of these two methods does not need to be exposed and we make them private with private_class_method.

The difference_algorithm1 method implements the first algorithm described above where the first set is iterated over, and all the other sets are iterated over as long as the current member of the first set is not found. We need to add the include? method, aliased to member? to check for the presence of a member in a set.

The IntSet case of include? needs to account for the fact that the argument might already be an Integer or might be a String representing an Integer. We add the Utils.string_to_integer_or_nil method to help for this:

module BYORedis
  module Utils

    # ...

    def self.string_to_integer_or_nil(string)
      begin
        string_to_integer(string)
      rescue InvalidIntegerString
        nil
      end
    end

    # ...
  end
end

listing 9.13 The string_to_integer_or_nil method in the Utils module

In the Dict case of RedisSet#include? we delegate to the Dict#include? method, through its member? alias, which implements the exact interface we need, so let's add the alias:

module BYORedis
  class Dict

    # ...

    def include?(key)
      !get_entry(key).nil?
    end
    alias member? include?

    # ...

  end
end

listing 9.14 Adding the Dict#member? aliased to Dict#include?

The second algorithm intends to provide an alternative in case the first set is significantly larger than the other sets.

The next command, SDIFFSTORE is similar to SDIFF, with the difference being that the result is stored in the given key, and the return value is the cardinality of that set.

We have two other very similar methods coming next with SINTERSTORE & SUNIONSTORE so we're using a helper method with the shared logic:

module BYORedis
  module SetUtils

    # ...

    def self.generic_set_store_operation(db, args)
      Utils.assert_args_length_greater_than(1, args)
      destination_key = args.shift
      sets = args.map { |other_set| db.lookup_set(other_set) }
      new_set = yield sets

      if new_set.empty?
        db.data_store.delete(destination_key)
      else
        db.data_store[destination_key] = new_set
      end

      RESPInteger.new(new_set.cardinality)
    end
  end

  # ...

  class SDiffStoreCommand < BaseCommand
    def call
      SetUtils.generic_set_store_operation(@db, @args) do |sets|
        RedisSet.difference(sets)
      end
    end

    def self.describe
      Describe.new('sdiffstore', -3, [ 'write', 'denyoom' ], 1, -1, 1,
                   [ '@write', '@set', '@slow' ])
    end
  end
end

listing 9.15 The SDiffStoreCommand class

The SDiffStoreCommand class is extremely similar to SDiffCommand, with the exception that it needs to handle an extra argument at the beginning of the argument list, for the destination key. It then stores the result set at that key, or delete what was there before if the result set is empty. Finally, it returns the cardinality of the result set.

Set Union

The next set operation we will implement is set union, which is defined as:

The union of two sets A and B is the set of elements which are in A, in B, or in both A and B

Let's start by creating the SUnionCommand class:

module BYORedis

  # ...

  class SUnionCommand < BaseCommand
    def call
      Utils.assert_args_length_greater_than(0, @args)
      sets = @args.map { |set_key| @db.lookup_set(set_key) }.compact

      RESPArray.new(RedisSet.union(sets).members)
    end

    def self.describe
      Describe.new('sunion', -2, [ 'readonly', 'sort_for_script' ], 1, -1, 1,
                   [ '@read', '@set', '@slow' ])
    end
  end
end

listing 9.16 The SUnionCommand class

We are implementing the union method as a class method on RedisSet for the same reasons we decided to implement difference as a class method.

module BYORedis
  class RedisSet

    # ...

    def self.union(sets)
      if sets.empty?
        RedisSet.new
      else
        union_set = RedisSet.new
        sets.each do |set|
          set.each { |member| union_set.add(member) }
        end

        union_set
      end
    end

    # ...
  end
end

The set union operation is simpler, we iterate over all sets, over all of their members and add each member to a new set, once we're done, we return the set.

Similarly to how we first added the SDIFF command followed by the SDIFFSTORE command, we're now adding the SUnionStoreCommand class.

module BYORedis

  # ...

  class SUnionStoreCommand < BaseCommand
    def call
      SetUtils.generic_set_store_operation(@db, @args) do |sets|
        RedisSet.union(sets)
      end
    end

    def self.describe
      Describe.new('sunionstore', -3, [ 'write', 'denyoom' ], 1, -1, 1,
                   [ '@write', '@set', '@slow' ])
    end
  end
end

listing 9.17 The SUnionStoreCommand class

Set Intersection

The final set operation we're going to implement is set intersection, with the SINTER command.

Set intersection is defined as:

In mathematics, the intersection of two sets A and B, denoted by A ∩ B, is the set containing all elements of A that also belong to B (or equivalently, all elements of B that also belong to A).

module BYORedis
  module SetUtils
    def self.generic_sinter(db, args)
      sets = args.map do |set_key|
        set = db.lookup_set(set_key)

        return RedisSet.new if set.nil?

        set
      end

      RedisSet.intersection(sets)
    end
  end
  # ...

  class SInterCommand < BaseCommand
    def call
      Utils.assert_args_length_greater_than(0, @args)
      intersection = SetUtils.generic_sinter(@db, @args)

      RESPArray.new(intersection.members)
    end

    def self.describe
      Describe.new('sinter', -2, [ 'readonly', 'sort_for_script' ], 1, -1, 1,
                   [ '@read', '@set', '@slow' ])
    end
  end
end

listing 9.18 The SInterCommand class

The logic in SInterCommand shares a lot of logic with SInterStoreCommand, which we'll create next, so we go ahead and create a method with the shared logic BYORedis::SetUtils.generic_sinter. In order to implement this method, we need the intersection class method on the RedisSet class, but before calling this method, we do return early if any of the given keys does not exist. This is an important shortcut, when computing the intersection of sets, if any of the sets is empty, we know that the final result will be an empty sets, and non existing sets are treated as empty sets.

module BYORedis
  class RedisSet

    # ...

    def self.intersection(sets)
      # Sort the sets smallest to largest
      sets.sort_by!(&:cardinality)

      intersection_set = RedisSet.new
      # Iterate over the first set, if we find a set that does not contain it, discard

      sets[0].each do |member|
        present_in_all_other_sets = true
        sets[1..-1].each do |set|
          unless set.member?(member)
            present_in_all_other_sets = false
            break
          end
        end
        # Otherwise, keep
        intersection_set.add(member) if present_in_all_other_sets
      end

      intersection_set
    end

    # ...

  end
end

listing 9.19 The RedisSet.intersection class method

Set intersection is not as straightforward as set union but is not at complicated as set difference. We start by sorting all the sets from smallest to largest. Doing the sorting is a small optimization because we have to iterate over all the elements of at least one set, so we might as well pick the smaller one for that.

Once the sets are sorted, we pick the first one, the smallest one, and iterate over all its members, for each member we check its presence in all other sets, as soon as this check is false, we continue to the next member. This is because the result set of the intersection only contains elements present in all the sets.

If we make it through all the sets, then the member is indeed present in all sets and we add it to the result set.

The last set operation command on our list is SINTERSTORE, we've seen this pattern two times already by now. We use the SetUtils.generic_set_store_operation method the same way we did previously:

module BYORedis

  # ...

  class SInterStoreCommand < BaseCommand
    def call
      SetUtils.generic_set_store_operation(@db, @args) do
        SetUtils.generic_sinter(@db, @args)
      end
    end

    def self.describe
      Describe.new('sinterstore', -3, [ 'write', 'denyoom' ], 1, -1, 1,
                   [ '@write', '@set', '@slow' ])
    end
  end
end

listing 9.20 The SInterStoreCommand class

With SInterStoreCommand completed, we've now implemented all the set operation commands, SUNION, SINTER & SDIFF, and their *STORE variants.

Membership related operations

SMEMBERS

The SMEMBERS command returns all the members of a set. Sets do not guarantee ordering, so we can just return members without having to worry about ordering. Let's create the SMembersCommand:

module BYORedis

  # ...

  class SMembersCommand < BaseCommand
    def call
      Utils.assert_args_length(1, @args)
      set = @db.lookup_set(@args[0])

      RESPArray.new(set.members)
    end

    def self.describe
      Describe.new('smembers', 2, [ 'readonly', 'sort_for_script' ], 1, 1, 1,
                   [ '@read', '@set', '@slow' ])
    end
  end
end

listing 9.21 The SMembersCommand class

We need to add the RedisSet#members command:

module BYORedis
  class RedisSet

    # ...

    def members
      case @underlying
      when IntSet then @underlying.members.map { |i| Utils.integer_to_string(i) }
      when Dict then @underlying.keys
      else raise "Unknown type for structure #{ @underlying }"
      end
    end

    # ...

  end
end

listing 9.22 The RedisSet#members method

In the IntSet case we call the IntSet#members methods, and convert all in Integer instances to String with the Utils.integer_to_string method. In the Dict case we can directly return from the already existing Dict#keys method.

The conversion to strings in the IntSet case is to follow the behavior of the Redis command, as the following example shows, even if the set is an intset, Redis converts the value to RESP strings before returning them:

127.0.0.1:6379> SADD s 1 2 3
(integer) 3
127.0.0.1:6379> DEBUG OBJECT s
Value at:0x7f88e7004130 refcount:1 encoding:intset serializedlength:15 lru:11187844 lru_seconds_idle:3
127.0.0.1:6379> SMEMBERS s
1) "1"
2) "2"
3) "3"

The quotes around the numbers show us that the array members are indeed strings, and redis-cli would prefix them with (integer) otherwise, but we can use nc to confirm the type of the elements of the array:

> echo "SMEMBERS s" | nc -c localhost 6379
*3
$1
1
$1
2
$1
3

SISMEMBER

SISMEMBER returns an integer acting as a boolean, 1 for true and 0 for false, depending on the presence of the given member in the set:

module BYORedis

  # ...

  class SIsMemberCommand < BaseCommand
    def call
      Utils.assert_args_length(2, @args)
      set = @db.lookup_set(@args[0])
      if set
        presence = set.member?(@args[1]) ? 1 : 0
        RESPInteger.new(presence)
      else
        RESPInteger.new(0)
      end
    end

    def self.describe
      Describe.new('sismember', 3, [ 'readonly', 'fast' ], 1, 1, 1,
                   [ '@read', '@set', '@fast' ])
    end
  end
end

listing 9.23 The SIsMemberCommand class

We delegate the actual work of checking if the set contains the given member to the RedisSet#member? method which we added earlier when adding the SDIFF command.

SMISMEMBER (New in 6.2.0)

module BYORedis

  # ...

  class SMIsMemberCommand < BaseCommand
    def call
      Utils.assert_args_length_greater_than(1, @args)
      set = @db.lookup_set(@args.shift)
      members = @args

      if set.nil?
        result = Array.new(members.size, 0)
      else
        result = members.map do |member|
          set.member?(member) ? 1 : 0
        end
      end

      RESPArray.new(result)
    end

    def self.describe
      Describe.new('smismember', -3, [ 'readonly', 'fast' ], 1, 1, 1,
                   [ '@read', '@set', '@fast' ])
    end
  end
end

listing 9.24 The SMIsMemberCommand class

This command uses the same method from RediSet we used in SIsMemberCommand, but inside the map method, which we use to return an array of integers acting as booleans, 1 if the member is in the set, 0 if it isn't.

SCARD

module BYORedis

  # ...

  class SCardCommand < BaseCommand
    def call
      Utils.assert_args_length(1, @args)
      set = @db.lookup_set(@args[0])

      cardinality = set.nil? ? 0 : set.cardinality
      RESPInteger.new(cardinality)
    end

    def self.describe
      Describe.new('scard', 2, [ 'readonly', 'fast' ], 1, 1, 1,
                   [ '@read', '@set', '@fast' ])
    end
  end
end

listing 9.25 The SCardCommand class

The command delegates to the RedisSet#cardinality method:

module BYORedis
  class RedisSet

    # ...

    def cardinality
      case @underlying
      when IntSet then @underlying.cardinality
      when Dict then @underlying.used
      else raise "Unknown type for structure #{ @underlying }"
      end
    end

    # ...

  end
end

listing 9.26 The RedisSet#cardinality method

Both IntSet and Dict already have methods that return what we need here, so we call them and return their results directly.

SRANDMEMBER

module BYORedis

  # ...

  class SRandMemberCommand < BaseCommand
    def call
      Utils.assert_args_length_greater_than(0, @args)
      raise RESPSyntaxError if @args.length > 2

      count = Utils.validate_integer(@args[1]) if @args[1]
      set = @db.lookup_set(@args[0])

      if set
        if count.nil?
          random_members = set.random_member
        else
          random_members = set.random_members_with_count(count)
        end

        RESPSerializer.serialize(random_members)
      elsif count.nil?
        NullBulkStringInstance
      else
        EmptyArrayInstance
      end
    end

    def self.describe
      Describe.new('srandmember', -2, [ 'readonly', 'random' ], 1, 1, 1,
                   [ '@read', '@set', '@slow' ])
    end
  end
end

listing 9.27 The SRandMemberCommand class

Getting a random element from an IntSet does not require as many steps as it does for a Dict. Given that the structure behind an IntSet is an array, we can pick a random index, between 0 and size - 1, and return the element at that index. As long as the random function we use is "random enough", the element returned will be random. We're using quotes here because the topic of randomness if fairly complicated, luckily Ruby does a lot of the heavy lifting for us, with the Kernel.rand method and the SecureRandom module. The difference between the two different approaches is a little bit out of scope for now, and we'll be using Kernel.rand since it is sufficient for our needs.

On the other hand, getting a random entry for a set using a Dict is way more complicated. It might seem similar at first glance, the data structure powering a hash table is also an array, but some buckets might be empty, so we cannot "just" pick a random index and return the value stored there, given that it could be empty. This problem is not insurmountable, we can add some form of retry logic, until a non empty bucket is found. We also need to take into account that the Dict might be in the middle of the rehashing process, in which case the buckets will be split across two different hash tables. Finally, even if we dealt with these issues, there is still a major issue, buckets contain zero or more entries, and given the nature of the SipHash algorithm, there is no way to expect the distribution of entries across buckets.

In practical terms, this means for a hash of size 8, containing 6 entries, it is entirely possible that five buckets are empty, two contain one element each and that the last bucket contain the other four elements. This means that even after finding a non empty bucket, we now need to select a random entry within that bucket if it contains more than one. Doing so works in the sense of being able to potentially return any of the values contained in the hash, but it completely obliterates the distribution of the result.

Ideally, as we call SRANDMEMBER multiple times, the distribution of the returned element should trend towards the perfect proportion. Using the example from above, if a set contains six elements, each element should have a probability of 1/6 to be returned, but the approach described above might be completely off, let's look at an example.

Note that these examples assume that the Kernel.rand has a perfect distribution, which is not the case, computers cannot be perfectly random, but it is pretty close as the following example shows:

distribution = Hash.new { |h, k| h[k] = 0 }
times = 100_000

times.times do
  distribution[rand(1..6)] += 1
end

p Hash[distribution.map { |k, v| [k, v / times.to_f] }]

The previous script returned the following on my machine, the result should be different but pretty close on another machine, or as you run it multiple times:

{1=>0.17011, 2=>0.16645, 6=>0.16604, 5=>0.16696, 4=>0.1657, 3=>0.16474}

The perfect distribution would be 0.166666667, so it's not that far. For example, running the same example with times = 1_000_000 instead returned closer results as expected, the more we run it, the more the results will get closer to the perfect distribution:

{3=>0.166954, 4=>0.16658, 5=>0.16693, 2=>0.166722, 1=>0.166075, 6=>0.166739}

Back to the distribution problem in a hash table, the previous example we mentioned looked like the following, five empty buckets, two with one member each and one with four.

s = { nil, nil, nil, nil, nil, < 1 >, < 2 >, < 3, 4, 5, 6 > }

Each of the bucket has about 1/6 chance of getting picked, but we'll retry as long we pick an empty bucket, so given that there are three non-empty buckets, each of these has about 1/3 change of getting picket. The problem is if the last one get picked, then we need to roll the die one more time, and this time each item will have about 1/4 chance of getting picked, bringing the probabilities of each elements to the following:

1 => 1/3 ~= 0.333333333
2 => 1/3 ~= 0.333333333
3 => 1/3 * 1/4 = 1/24 ~= 0.083333333
4 => 1/3 * 1/4 = 1/24 ~= 0.083333333
5 => 1/3 * 1/4 = 1/24 ~= 0.083333333
6 => 1/3 * 1/4 = 1/24 ~= 0.083333333

To attempt addressing this problem Redis uses two functions, dictGetSomeKeys and dictGetFairRandomKey. It also uses a third function, dictGetRandomKey, which implements the logic we described previously, only as a backup, in case dictGetSomeKeys fails to return any keys.

This approach alleviates some of the issues we described earlier by first randomly selecting keys through the dictionary, putting them in a flat array, and only then picking one random index in this array.

Let's add these methods to our Dict class

module BYORedis
  class Dict

    # ...

    GETFAIR_NUM_ENTRIES = 15
    def fair_random_entry
      entries = get_some_entries(GETFAIR_NUM_ENTRIES)

      if entries.empty?
        random_entry
      else
        entries[rand(0...entries.size)]
      end
    end

    private

    def get_some_entries(count)
      entries = []
      stored = 0
      count = used if count > used
      maxsteps = count * 10

      count.times { rehash_step } if rehashing?

      tables = rehashing? ? 2 : 1
      maxsizemask = main_table.sizemask
      if tables > 1 && rehashing_table.sizemask > maxsizemask
        maxsizemask = rehashing_table.sizemask
      end

      i = rand(0..maxsizemask)
      empty_len = 0
      while stored < count && maxsteps
        iterate_through_hash_tables_unless_rehashing do |hash_table|
          # If we're in the process of rehashing, up to the indexes already visited in the main
          # table during the rehashing, there are no populated buckets so we can skip in the
          # main table, all the indexes between 0 and @rehashidx - 1
          if rehashing? && hash_table == main_table && i < @rehashidx
            if i >= rehashing_table.size
              i = @rehashidx
            else
              next
            end
          end

          next if i >= hash_table.size # Out of range for this table

          hash_entry = hash_table.table[i]

          # Count contiguous empty bucket and jump to other locations if they reach 'count'
          # with a minimum of 5
          if hash_entry.nil?
            empty_len += 1
            if empty_len >= 5 && empty_len > count
              i = rand(0..maxsizemask)
              empty_len = 0
            end
          else
            empty_len = 0
            while hash_entry
              entries << hash_entry
              hash_entry = hash_entry.next
              stored += 1
              return entries if stored == count
            end
          end
        end

        i = (i + 1) & maxsizemask # increment and wraparound if needed
        maxsteps -= 1
      end

      entries
    end

    def random_entry
      return if used == 0

      rehash_step if rehashing?

      hash_entry = nil

      if rehashing?
        # There are no elements indexes from 0 to rehashidx-1 so we know the only places we can
        # find an element are in main_table[rehashidx..-1] and anywhere in the rehashing table
        # We generate the random_index between the total number of slots (the two sizes), minus
        # the rehashing index. An example, we're growing from 8 to 16 buckets, that's 24 total
        # slots, now let's imagine that @rehashidx is 4, we generate an index between 0 and 20
        # (excluded), and we add 4 to it, that means that we _never_ have a value under 4.
        # If the random index is 8 or more, we need to look in the rehashing table, but we need
        # adjust it by removing 8, the size of the main table to it, so say it was initially 19,
        # plus four, that' 23, minus 8, that's 15, the last bucket in the rehashing table.
        # If the random index is between 4 and 7, then we look directly in the main table
        while hash_entry.nil?
          max = slots - @rehashidx
          random_index = @rehashidx + SecureRandom.rand(max)
          hash_entry =
            if random_index >= main_table.size
              rehashing_table.table[random_index - main_table.size]
            else
              main_table.table[random_index]
            end
        end
      else
        while hash_entry.nil?
          random_index = SecureRandom.rand(main_table.size)
          hash_entry = main_table.table[random_index]
        end
      end

      # Now that we found a non empty bucket, we need to pick a random element from it, but if
      # there's only one item, we can save some time and return right away
      return hash_entry if hash_entry.next.nil?

      list_length = 0
      original_hash_entry = hash_entry
      while hash_entry
        list_length += 1
        hash_entry = hash_entry.next
      end
      random_list_index = SecureRandom.rand(list_length)
      hash_entry = original_hash_entry
      random_list_index.times do
        hash_entry = hash_entry.next
      end

      hash_entry
    end
  end
end

listing 9.28 The new random methods in the Dict class

The three methods we just added to the Dict class implement this better approach to retrieving a random key/value pair. fair_random_entry is the only public one and attempts to use the result from get_some_entries, but due to the nature of the hash table, it is possible that get_some_keys fails to return any keys, in which case we fall back to random_entry, which suffers from the distribution above outlined above, but still returns a random element, just not that random.

Now that Dict is updated, we can implement random_member and random_members_with_count in the RedisSet class:

module BYORedis
  class RedisSet

    # How many times bigger should be the set compared to the requested size
    # for us to don't use the "remove elements" strategy? Read later in the
    # implementation for more info.
    # See: https://github.com/antirez/redis/blob/6.0.0/src/t_set.c#L609-L612
    SRANDMEMBER_SUB_STRATEGY_MUL = 3

    # ...

    def random_members_with_count(count)
      return [] if count.nil? || count == 0

      # Case 1: Count is negative, we return that many elements, ignoring duplicates
      if count < 0
        members = []
        (-count).times do
          members << random_member
        end

        return members
      end

      # Case 2: Count is positive and greater than the size, we return the whole thing
      return self if count >= cardinality

      # For both case 3 & 4 we need a new set
      new_set_content = Dict.new
      # Case 3: Number of elements in the set is too small to grab n random distinct members
      # from it so we instead pick random elements to remove from it
      # Start by creating a new set identical to self and then remove elements from it
      if count * SRANDMEMBER_SUB_STRATEGY_MUL > cardinality
        size = cardinality
        each { |member| new_set_content.add(member, nil) }
        while size > count
          random_entry = new_set_content.fair_random_entry
          new_set_content.delete(random_entry.key)
          size -= 1
        end
        return new_set_content.keys
      end

      # Case 4: The number of elements in the set is big enough in comparison to count so we
      # do the "classic" approach of picking count distinct elements
      added = 0
      while added < count
        member = random_member
        added += 1 if new_set_content.add(member, nil)
      end

      new_set_content.keys
    end

    def random_member
      case @underlying
      when IntSet then Utils.integer_to_string(@underlying.random_member)
      when Dict then @underlying.fair_random_entry.key
      else raise "Unknown type for structure #{ @underlying }"
      end
    end

    # ...

  end
end

listing 9.29 The random_member and random_member_with_count methods in the RedisSet class

The random_members_with_count method breaks down the possible scenarios in four different possibilities. "Case 1" is for a negative count value, in which case duplicates are allowed, so we can iterate as many times as needed and select a random member at each step, without having to worry about anything else.

"Case 2" is if count is greater than the size of the set, in which case we don't actually need to select any random members and we can return the whole set.

We differentiate between "Case 3" and "Case 4" depending on how close count is to the size of the set. The idea being that because we can't return duplicates, it might take a while to pick n random members if n is really close to the size of the set. Let's look at an example with a set of size 10, the numbers 1 to 10. If we call SRANDMEMBER set 9, we want to return nine random members, but doing so would require many tries.

Each of the member has a 1/10 change of getting picked, so the first pick will get a random member, but on the second try we have a 1/10 chance of picking the same element, in which case we'd need to try again. As we fill up the result dict, the odds of picking only one of the few non selected members will be really small.

This case if avoided with the if count * SRANDMEMBER_SUB_STRATEGY_MUL > cardinality condition. The constant is set to 15, so with a count value of 10 and a set containing 15 elements, the condition will be true, 10 * 15 > 15.

The condition would only be false with a large enough set and a small enough count value, such as 100 and 2, 2 * 15 > 100 == false.

So, in "Case 3", we instead create a new dict with all the elements from the set, and remove random members until its size reaches count.

"Case 4" is the most straightforward approach, we keep looping and picking random members, until we've picked count unique members.

Removing elements

SPOP

module BYORedis

  # ...

  class SPopCommand < BaseCommand
    def call
      Utils.assert_args_length_greater_than(0, @args)
      raise RESPSyntaxError if @args.length > 2

      if @args[1]
        count = Utils.validate_integer(@args[1])
        return RESPError.new('ERR index out of range') if count < 0
      end
      key = @args[0]
      set = @db.lookup_set(key)

      if set
        popped_members = @db.pop_from_set(key, set, count)

        RESPSerializer.serialize(popped_members)
      elsif count.nil?
        NullBulkStringInstance
      else
        EmptyArrayInstance
      end
    end

    def self.describe
      Describe.new('spop', -2, [ 'write', 'random', 'fast' ], 1, 1, 1,
                   [ '@write', '@set', '@fast' ])
    end
  end
end

listing 9.30 The SPopCommand class

Similarly to SRANDMEMBER, SPOP accepts a count argument, describing how many items can be returned, contrary to SRANDMEMBER, SPOP does not accept negative count values, and a value of 0 is effectively a no-op, no members are popped.

Let's start with the DB#pop_from_set method:

module BYORedis
  class DB

    # ...

    def pop_from_set(key, set, count)
      popped_members = if count.nil?
                         set.pop
                       else
                         set.pop_with_count(count)
                       end

      @data_store.delete(key) if set.empty?

      popped_members
    end

    # ...

  end
end

listing 9.31 The DB#pop_from_set method

We now need to add RedisSet#pop and RedisSet#pop_with_count:

module BYORedis
  class RedisSet

    # ...

    # How many times bigger should be the set compared to the remaining size
    # for us to use the "create new set" strategy? Read later in the
    # implementation for more info.
    # See: https://github.com/antirez/redis/blob/6.0.0/src/t_set.c#L413-416
    SPOP_MOVE_STRATEGY_MUL = 5

    # ...

    def pop
      case @underlying
      when IntSet then @underlying.pop.to_s
      when Dict then
        random_entry = @underlying.fair_random_entry
        @underlying.delete(random_entry.key)
        random_entry.key
      else raise "Unknown type for structure #{ @underlying }"
      end
    end

    def pop_with_count(count)
      return [] if count.nil? || count == 0

      # Case 1: count is greater or equal to the size of the set, we return the whole thing
      if count >= cardinality
        all_members = members
        clear
        return all_members
      end

      remaining = cardinality - count
      if remaining * SPOP_MOVE_STRATEGY_MUL > count
        # Case 2: Count is small compared to the size of the set, we "just" pop random elements

        count.times.map { pop }
      else
        # Case 3: count is big and close to the size of the set, and remaining is small, we do
        # the reverse, we pick remaining elements, and they become the new set
        new_set = RedisSet.new
        remaining.times { new_set.add(pop) }
        # We have removed all the elements that will be left in the set, so before swapping
        # them, we store all the elements left in the set, which are the ones that will end up
        # popped
        result = members

        # Now that we have saved all the members left, we clear the content of the set and copy
        # all the items from new_set, which are the ones left
        clear
        new_set.each { |member| add(member) }

        result
      end
    end

    # ...

  end
end

listing 9.32 The pop & pop_with_count methods in RedisSet

pop is straightforward in the IntSet case given that we already implemented the IntSet#pop method, we can call it and directly return from it.

On the other hand, the Dict case is trickier and we now use the newly created Dict#fair_random_entry method, to find which entry to delete from the set.

pop_with_count is optimized to handle a few different edge cases elegantly. We label "Case 1" the case where the count value is greater than or equal to the cardinality of the set, and can return all the members, and clear the set.

In the case where count is anywhere between 1 and cardinality - 1, we do need to find random elements to remove from the set and return them. While it may seem like a simple problem at first, just iterate count times and pop a random element at each iteration, the reality is a little bit more complicated. The problem is that the process of popping random element can get expensive if the count number is very large, so instead we use an arbitrary threshold and if count is big enough that it is too close to the cardinality of the set, we reverse the process. We only pop the number of elements that should be left in the set, set them aside, extract all the remaining elements from the set, return them and add back the small set of remaining elements in the set.

Looking at a set of 10 elements as an example, if count is 9, then remaining * SPOP_MOVE_STRATEGY_MUL is 5, which is not greater than 9, so we fall in "Case 3". In the case where count is 8, then 2 * 5 = 10, which is greater than count, so we pop eight times from the set, that's "Case 2".

SREM

module BYORedis

  # ...

  class SRemCommand < BaseCommand
    def call
      Utils.assert_args_length_greater_than(1, @args)
      set = @db.lookup_set(@args.shift)
      remove_count = 0

      if set
        @args.each do |member|
          remove_count += 1 if @db.remove_from_set(key, set, member)
        end
      end

      RESPInteger.new(remove_count)
    end

    def self.describe
      Describe.new('srem', -3, [ 'write', 'fast' ], 1, 1, 1,
                   [ '@write', '@set', '@fast' ])
    end
  end
end

listing 9.33 The SRemCommand class

SREM relies on DB#remove_from_set, let's add it:

module BYORedis
  class DB

    # ...

    def remove_from_set(key, set, member)
      removed = set.remove(member)
      @data_store.delete(key) if set.empty?

      removed
    end

    # ...

  end
end

Let's now add RedisSet#remove to complete the SREM implementation:

module BYORedis
  class RedisSet

    # ...

    def remove(member)
      case @underlying
      when IntSet
        member_as_integer = Utils.string_to_integer_or_nil(member)
        if member_as_integer
          @underlying.remove(member_as_integer)
        else
          false
        end
      when Dict then !@underlying.delete_entry(member).nil?
      else raise "Unknown type for structure #{ @underlying }"
      end
    end

    # ...

  end
end

listing 9.34 The RedisSet#remove method

Up until now the Dict#remove method used to return the value of the deleted pair, or nil if the dict did not contain the given key. The problem with this approach is that if the value stored in the Dict was nil, the caller would not be able to differentiate the successful deletion of a pair, or a "miss" if the dictionary did not contain the key.

Similarly to how we initially implemented the get method by using the lower level method get_entry, we are adding the delete_entry method, which returns the full DictEntry instance for a successful deletion. With this change, callers can now know that a nil value is indeed a miss whereas a DictEntry instance, regardless of what the value of the value field is, is a successful deletion.

module BYORedis
  class Dict

    # ...

    def delete_entry(key)
      return if main_table.used == 0 && rehashing_table.used == 0

      rehash_step if rehashing?

      hash_key = SipHash.digest(RANDOM_BYTES, key)
      iterate_through_hash_tables_unless_rehashing do |hash_table|
        index = hash_key & hash_table.sizemask
        entry = hash_table.table[index]
        previous_entry = nil

        while entry
          if entry.key == key
            if previous_entry
              previous_entry.next = entry.next
            else
              hash_table.table[index] = entry.next
            end
            hash_table.used -= 1
            return entry
          end
          previous_entry = entry
          entry = entry.next
        end
      end

      nil
    end

    def delete(key)
      delete_entry(key)&.value
    end

    # ...

  end
end

listing 9.35 The delete & delete_entry methods in the Dict class

The delete method is now written in terms of delete_entry. The &. operator allows us to concisely return nil from delete if delete_entry returned nil itself, in the case where the Dict does not contain key. If the value field of the DictEntry is nil, then callers of delete have no way of differentiating a miss, when the Dict does not contain key, and a successful deletion. If this is important to the callers, which in the case in SRemCommand, then callers can use delete_entry instead.

SMOVE

SMOVE is used to remove a member from a set and add it to another

module BYORedis

  # ...

  class SMoveCommand < BaseCommand
    def call
      Utils.assert_args_length(3, @args)
      source_key = @args[0]
      source = @db.lookup_set(source_key)
      member = @args[2]
      destination = @db.lookup_set_for_write(@args[1])

      if source.nil?
        result = 0
      else
        removed = @db.remove_from_set(source_key, source, member)
        if removed
          destination.add(member)
          result = 1
        else
          result = 0
        end
      end

      RESPInteger.new(result)
    end

    def self.describe
      Describe.new('smove', 4, [ 'write', 'fast' ], 1, 2, 1,
                   [ '@write', '@set', '@fast' ])
    end
  end
end

listing 9.36 The SMoveCommand class

If the source set does not exist, there is nothing to do since we have no set to remove member from. If it does exist, then we call DB#remove_from_set, which we added earlier for the SREM command, if it returns true then we add the member in destination. Note that we return 1 as long as member was found in source, even if it was already in destination.

This wraps up the last set command!

Conclusion

You can find the code on GitHub.

With set commands implemented, we now have one last native data type left, sorted sets, and this is what Chapter 10 will cover.

Appendix A: A more idiomatic `IntSet`

The IntSet class we created earlier in the Chapter was trying to replicate as much as possible the logic used in intset.c in Redis, at the cost of not being really idiomatic Ruby. If we were to set aside the encoding, and use the Ruby Integer class as provided, we can end up with a simpler implementation:

module BYORedis
  class IntSet

    def initialize
      @underlying_array = []
    end

    def add(member)
      raise "Member is not an int: #{ member }" unless member.is_a?(Integer)

      index = @underlying_array.bsearch_index { |x| x >= member }

      if index.nil?
        @underlying_array.append(member)

        true
      elsif @underlying_array[index] == member
        false
      else
        @underlying_array.insert(index, member)

        true
      end
    end

    # ...
  end
end

listing 9.37 The initialize & add methods in the IntSet class

The first thing we do in add is check that the argument is indeed an Integer, and abort if it isn't. It is up to callers of these methods to do the due diligence of calling this method with the correct argument type.

The next step is to find where in the array should the new element be added, and we use the bsearch_index method for that. By giving the block { |x| x >= member } block to the method, it will return the smallest index of an element that is greater than or equal to member. If no elements are greater than or equal to member, then it returns nil.

Based on these cases, there are three cases we need to consider, the first one, if index is nil, means that no elements in the array are greater than or equal to member, in other words, member is now the member with the largest value, and we should add it at the end of the array. This is what we do with the Array#append method.

Next is the case where index is not nil, and in this case there are two options, the value at index is either equal to member, or greater than member. If the value is equal to member, it means that there is already an Integer in the array with the same value as member, and it means that we have nothing to do, the member is already present.

The last case is if the value at index is greater than member. bsearch_index guarantees that if returned the smallest index of all the values greater than member, so we need to add member right before this element, and this is what Array#insert does, it inserts the given value before the element with the given index.

The next main method is the ability to check for the presence of a member in the set, this is what the include? method does:

def include?(member)
  return false if member.nil?

  !@underlying_array.bsearch { |x| member <=> x }.nil?
end
alias member? include?

listing 9.38 The IntSet#include? method

We're following Ruby's Set class naming convention here, naming the method include?, also accessible through the member? alias. We'll use the member? method throughout this chapter to use the same language used by the Redis commands such as SISMEMBER.

The include? methods relies almost exclusively on the bsearch method, which we've indirectly explored previously through its close sibling bsearch_index. Both methods use a similar API, the difference being that bsearch returns an element from the array whereas bsearch_index returns the index of an element. Both could have been used here given that the returned value itself is not that important, we already really care whether or not the result is nil.

These bsearch methods have two modes of operation, which can be a bit confusing these the mode is decided implicitly depending on the return value of the block passed to the method. The two modes are find-minimum & find-any. We've only used the find-minimum mode so far, by passing the block { |x| member <=> x } to the method, we end up using the find-any mode, because the Integer#<=> method returns an integer, -1, 0 or 1. The Ruby documentation of the method is surprisingly not clear at describing the behavior of this mode, on the other hand the man page of bsearch(3), accessible with man 3 bsearch, which is what the find-any mode is based after is a little bit more helpful:

The contents of the array should be in ascending sorted order according to the comparison function referenced by compar.

The compar routine is expected to have two arguments which point to the key object and to an array member, in that order. It should return an integer which is less than, equal to, or greater than zero if the key object is found, respectively, to be less than, to match, or be greater than the array member.

We need to "translate" this description given that it describes the C function, not the Ruby one, but the compar function is essentially very similar to the block argument, and the key object is the argument to the method, member in the include? method.

With that said, what the documentation tells us is that the block should return 0 if both values are equal, a negative value if member is less than x, and a positive value if member is greater than x. The <=>, often called "spaceship operator", does exactly that!

Using this block, bsearch will return the value if it finds it, or nil if it can't find it.

Let's now add the ability to remove an element from a set with the remove method:

def remove(member)
  index = @underlying_array.bsearch_index { |x| member <=> x }
  if index
    @underlying_array.delete_at(index)
    true
  else
    false
  end
end

listing 9.39 The IntSet#remove method

The method method uses the bsearch_index method in a way almost identical to how the include? method uses the bsearch method. By using the Interger#<=> method, we will either receive the index of member in the array, or nil. If index is nil, there's nothing to remove, so we can return false and call it a day. On the other hand, if index is not nil, we use the Array#delete_at method, which deletes the element at the given index.

Let's now add the pop and random_member methods, which both behave very similarly, with the exception that random_member does not remove any elements from the array:

# ...
def pop
  rand_index = rand(@underlying_array.size)
  @underlying_array.delete_at(rand_index)
end

def random_member
  rand_index = rand(@underlying_array.size)
  @underlying_array[rand_index]
end

listing 9.40 The pop, & random_member methods in the IntSet class

Finally, we need a few more methods to provide an easy to use API for the IntSet class, namely, the methods empty? to check if the sets is empty or not, members, to return all the members in the set, cardinality, to return the size of the set, and each, to provide a way to iterate over all the members in the set. Ruby gives us tools that allow us to provide these methods without having to explicitly define them. We're using the Forwardable module to delegate some methods directly to the Array instance, @underlying_array. We're also using the alias keyword to provide some of these methods through the same naming conventions used in Redis. We also use the attr_reader approach to create an accessor for the @underlying_array instance variable, and alias it to members to provide a more explicit method name:

require 'forwardable'

module BYORedis
  class IntSet
    extend Forwardable

    attr_reader :underlying_array
    def_delegators :@underlying_array, :empty?, :each, :size

    alias cardinality size
    alias card cardinality
    alias members underlying_array

    def initialize
      @underlying_array = []
    end

    # ...

  end
end

listing 9.41 The members, empty?, each & cardinality methods in the IntSet class

And with these aliases, we now have completed the IntSet class, let's now use it, in combination with the Dict class, to implement the Set commands, starting with the one allowing us to create a new set.

Rebuilding Redis in Ruby - Chapter 8 - Adding Hash Commands

Pierre Jambet — Fri, 06 Nov 2020 13:38:30 +0000

What we'll cover

Now that our server supports Lists, the next data type we will add support for is Hashes. We've covered the concept of a Hash, also called Map or Dictionary in Chapter 6 where we built our own Dict class, implemented as a hash table, to store all the database data in memory. It turns out that within this Dict, the @data_store instance variables in the DB class, values can also be hashes.

This allows clients to store multiple key/value pairs for a top-level key. Say that for instance you wanted to store product data in Redis, where a product has an id, and a set of attributes, such as a name, a price, and an image URL. You could do this with good old GET/SET, but that would require you to use as many keys in the top level dictionary as you need attributes. It is simpler to use a hash in this case:

127.0.0.1:6379> HSET product:1 name "Product One" price 25 image_url https://...
(integer) 3
127.0.0.1:6379> HSET product:123 name "Product 123" price 100 image_url https://...
(integer) 3
127.0.0.1:6379> HGETALL product:1
1) "name"
2) "Product One"
3) "price"
4) "25"
5) "image_url"
6) "https://..."
127.0.0.1:6379> HGETALL product:123
1) "name"
2) "Product 123"
3) "price"
4) "100"
5) "image_url"
6) "https://..."
127.0.0.1:6379> HGET product:1 name
"Product One"
127.0.0.1:6379> HGET product:123 price
"100"

With the HSET command we can set as many key/value pairs as we want for the given key, product:1 and product:123 in the example above. Note that since RESP does not support a dictionary type, the returned value of HGETALL, which returns all the pairs, is a flat array of field names and values, it is up to the client to read this array and wrap it in a more appropriate data type, such as Hash in Ruby, Object or Map in JavaScript, or dict in Python.

There are 15 commands for the Hash data type:

HDEL: Delete one or more fields from a hash
HEXISTS: Check for the existence of a field in a hash
HGET: Return the value for the given field
HGETALL: Return all the key/value pairs
HINCRBY: Increment the value for the given field, by the given integer, positive or negative
HINCRBYFLOAT: Increment the value for the given field, by the given float, positive or negative
HKEYS: Return all the keys
HLEN: Return the number of pairs
HMGET: Return all the values for the given keys
HMSET: This command is deprecated, it was necessary before HSET gained the capability to set multiple key/value pairs at once
HSCAN: Return a subset of key/value pairs as well as a scan cursor. This is similar to the SCAN command
HSET: Set one or more key/value pairs, creating the hash if it does not already exist
HSETNX: Set the value for the given field in the hash, only if the field does not already exist
HSTRLEN: Return the length of the string stored for the given field
HVALS: Return all the values

We will only implement thirteen of these fifteen commands, we will not implement HMSET, because as noted above, it was made obsolete when HSET was updated in 4.0.0 to become variadic. That's just a fancy word to say that it accepts one or more key/value pairs. Prior to that it would only accept a single pair.

We will also ignore HSCAN, it behaves very similarly to SCAN, which operates on the top-level dictionary, SSCAN, which works on sets and ZSCAN which works on sorted sets. The idea behind each of these commands is that retrieving all the values is an O(n) operation, where n is the number of elements in the database for SCAN, the number of fields for HSCAN and the number of members for SSCAN & ZSCAN. In practical terms, it means that calling HSCAN on large hashes, which have no limit on the number of pairs, the memory available is the only limit, will be very slow if that number is really high.

The *SCAN commands "solve" this problem by breaking the iteration in multiple steps, calling HSCAN only returns a subset of the key/value pairs, and includes a cursor that can be used to keep iterating until the cursor is 0, indicating that the iteration is over.

SCAN is the alternative to KEYS, HSCAN is the alternative to HKEYS, SSCAN is the alternative to SMEMBERS and ZSCAN is the alternative to ZRANGE zset 0 -1 WITHSCORES.

All four *SCAN commands work very similarly and are fairly complex, the C implementation spans over a few hundred lines and code, and for reference the documentation of dictScan, one of the main functions, is over 80 lines long. A big part of the complexity comes from the fact that the *SCAN commands are stateless, the server does not store any data with the status of the iteration, and it also needs to be smart to know how to iterate over the underlying dict if it is being rehashed.

The *SCAN commands might be implemented in a later chapter but this is not currently planned

It is important to note that despite the existence of the HINCRBY & HINCRBYFLOAT commands, all keys and values in a Hash are strings. We will see in details how these two commands are implemented and how the data is converted from a string to an integer or a float later in this chapter.

How does Redis do it

We already know, to some extent, how Redis handles dictionaries, we explored how it implemented a hash table in the dict.c file in Chapter 6, but what we built so far is missing a few elements, which we'll look into later.

Additionally, Redis uses a really interesting approach where it uses a different underlying structure to store the hash data depending on the size of the hash and the length of the keys and values. These values can be configured through the configuration file, the default values are 512 for the maximum number of items stored as a ziplist & 64 for the maximum length of a key or value stored as a ziplist. As long as the number of keys is lower or equal to 512 and that the strings stored in the hash are shorter than 64 characters, Redis will use a ziplist to store the Hash. Once any of these two conditions break, it will convert the ziplist to a dict.

We will implement a similar approach because it illustrates a crucial point with regard to time complexity and O-notation. Most of the operations important to a hash, such as HGET, have an O(n) time complexity when using a list. This is because in the case where the element we're looking for is the last element in the list or is not present, we'd have to browse the whole list to find it. On the other hand, as we've seen in Chapter 6, a hash table, such as the one we implemented in the Dict class, can perform this operation with a O(1) complexity.

That being said, O(n) does not mean "slow", and O(1) does not mean "fast". What they mean is that a dict lookup using a hash table will always require the same number of steps and therefore take roughly the same amount of time, that's what O(1) means, the time it takes is constant. On the other hand a hash lookup using a list will require one more step, in the worst case scenario, for each new items in the list, this is a linear growth.

Now, back to our hash table, what we want, or what the users of our database want, is for it to be as fast as possible. If a hash only contains a single key/value pair, it seems very likely that a list implementation will actually be faster than a hash table. There's no hashing, no internal table allocation, a single check of the element at the head of the list and that's it.

If a list is more efficient than a hash table for one element, it seems reasonable to assume it will also be faster for two, three, four and all "small" hashes. But how do we define "small" in concrete terms. Well, this is when you have to measure things. The process here would be to run benchmarks, to measure the performance of different operations, against each implementation and see at what point the list implementation starts to slow down to a point where a hash table would be faster.

The developers of Redis did this work, and we the default value is 512 entries, as we can see in the redis.conf file. This value means that Redis will use a ziplist for the first 512 pairs added to the hash, and adding a 513th one will start using a hash table. Let's look at it in practice:

127.0.0.1:6379> HSET h 1 1
(integer) 1
127.0.0.1:6379> DEBUG OBJECT h
Value at:0x7ffe00804650 refcount:1 encoding:ziplist serializedlength:16 lru:10150943 lru_seconds_idle:5

The DEBUG OBJECT command returns information about the given key, including its encoding, and we can see that the hash at key h is encoded as a ziplist. Let's add 511 items to it and see what happens, we could do this with redis-cli, but it would take a while, so let's use irb, with the redis gem:

irb(main):001:0> require 'redis'
=> true
irb(main):002:0> redis = Redis.new
irb(main):003:0> 511.times { |i| redis.hset('h', i, i) }
=> 511

Back in redis-cli:

127.0.0.1:6379> debug object h
Value at:0x7ffdfec194a0 refcount:1 encoding:ziplist serializedlength:2836 lru:10150882 lru_seconds_idle:5

The hash is still a ziplist, and contains 512 pairs, let's add another one:

127.0.0.1:6379> HSET h f 1
(integer) 1
127.0.0.1:6379> debug object h
Value at:0x7ffdfec194a0 refcount:1 encoding:hashtable serializedlength:2825 lru:10150894 lru_seconds_idle:2

And voila! Redis updated the encoding of the hash from a ziplist to a hashtable, the changes are in practice invisible to the user, the commands are the same, but internally Redis uses what it believes is the most efficient implementation.

The hash length is not the only factor Redis uses to decide which underlying implementation to use, it also uses the length of the values, with a default value of 64 in the redis.conf file. If either a key or a value in the hash has a length longer than 64, Redis will start using a hash table instead of a ziplist. This is a consequence of the ziplist data structure, which we explored in Appendix A in the previous chapter. Ziplists are represented as a chunk of contiguous memory, making them more and more expensive to manipulate at they grow, as the whole list needs to be reallocated when a new element is inserted for instance. When using small key and value strings, the whole chunk of memory allocated will stay relatively small, but if we were to store any strings until the size of the hash reaches 512 entries, we might still end up with a very big and slow ziplist if a client started using long strings as keys or values.

Adding Hash Commands

It's interesting to consider the fact that in essence the hash type in Redis does not require any new concrete data structures, it is a layer of abstraction on top of ziplists and dicts. We have not reimplemented ziplists so we will instead use our List class for small hashes, and use the Dict class for large hashes.

We first need to add the ability to create hashes, that's what HSET & HSETNX do.

Creating a Hash with HSET & HSETNX

HSET's behavior is fairly similar to LPUSH & RPUSH from the previous chapter. If no objects exist for the given key, a new hash is created, and all the given key/value pairs are added to it. If an object already exists and is not a hash, an error is returned, and if a hash already exists, the new pairs are added to it. The command returns the number of fields added to the hash. Updating elements does not count as adding new elements since we're not adding a new pair to the hash. Let's look at some examples:

127.0.0.1:6379> HSET h field-1 value-1 one 1 2 two
(integer) 3
127.0.0.1:6379> HGETALL h
1) "field-1"
2) "value-1"
3) "one"
4) "1"
5) "2"
6) "two"
127.0.0.1:6379> HSET h field-1 new-value one something-else a-new-key a-new-pair
(integer) 1
127.0.0.1:6379> HGETALL h
1) "field-1"
2) "new-value"
3) "one"
4) "something-else"
5) "2"
6) "two"
7) "a-new-key"
8) "a-new-pair"

The second HSET command returned one because only one of the three given keys did not already exists, the other two, field-1 and one were updated. As mentioned above, note that in a hash, everything is a string.

Config values

We want our hash implementation to behave similarly to Redis and choose the best underlying structure, between List and Dict, depending on the size of the hash. To achieve this we are creating a Config module, which will keep the value of all the supported configuration options. For now we only need two configs, hash_max_ziplist_entries & hash_max_ziplist_value:

module BYORedis
  module Config

    UnsupportedConfigParameter = Class.new(StandardError)
    UnknownConfigType = Class.new(StandardError)

    DEFAULT = {
      hash_max_ziplist_entries: 512,
      hash_max_ziplist_value: 64,
    }

    @config = DEFAULT.clone

    def self.set_config(key, value)
      key = key.to_sym
      existing_config = @config[key]
      raise UnsupportedConfigParameter, key unless existing_config

      case existing_config
      when Integer
        @config[key] = Utils.string_to_integer(value)
      else
        raise UnknownConfigType, "#{ key }/#{ value }"
      end
    end

    def self.get_config(key)
      @config[key.to_sym]
    end
  end
end

listing 8.1 The Config class

Similarly to what we did in the previous chapter, we're going to create a new file, hash_commands.rb, where we'll add all the command classes related to the Hash data type, let's start with HSetCommand:

module BYORedis
  class HSetCommand < BaseCommand
    def call
      Utils.assert_args_length_greater_than(1, @args)
      key = @args.shift
      raise InvalidArgsLength unless @args.length.even?

      hash = @db.lookup_hash_for_write(key)
      count = 0

      @args.each_slice(2).each do |pair|
        key = pair[0]
        value = pair[1]

        count += 1 if hash.set(key, value)
      end

      RESPInteger.new(count)
    end

    def self.describe
      Describe.new('hset', -4, [ 'write', 'denyoom', 'fast' ], 1, 1, 1,
                   [ '@write', '@hash', '@fast' ])
    end
  end
end

listing 8.2 The HSetCommand class

In order for the Server class to respond to the HSET command we need to add a require statement in server.rb for the new hash_commands.rb file, as well as adding an entry in the COMMANDS dictionary. This is a repetitive process, so we will stop showing it from now on, but remember that for each class that we add, we actually need to "enable" it in the Server class.

We have to use a new type of validation for the number of arguments with the HSET command, all arguments after the hash's key come in pairs, so we need to validate that we have an even number of arguments after the key.

We now need a class to represent the hash, in the Redis source code, the file that implements the hash logic is called t_hash.c. The main data types are all implemented in files starting with t_, which I assume stands for Type. String commands are implemented in t_string.c, List commands in t_list.c, Set commands in t_set.c, Sorted Sets commands in t_zset.c, Hash commands in t_hash.c and Stream commands in t_stream.c. We could follow this pattern and name our class THash, but this is not a very explicit name, so instead we'll go with RedisHash, to be more explicit. We are not calling it Hash because Ruby already ships a Hash class, and even though nothing technically prevents up from "reopening" the class and adding our own methods, overriding existing ones, this would likely become problematic. For instance, we might accidentally use methods defined in the Ruby Hash class, it is easier to start fresh, with our own class.

module BYORedis
  class RedisHash

    ListEntry = Struct.new(:key, :value)

    def initialize
      @underlying = List.new
    end

    def set(key, value)
      max_string_length = Config.get_config(:hash_max_ziplist_value)
      convert_list_to_dict if @underlying.is_a?(List) &&
                              (key.length > max_string_length || value.length > max_string_length)

      case @underlying
      when List then
        added = set_list(key, value)
        if @underlying.size + length > Config.get_config(:hash_max_ziplist_entries)
          convert_list_to_dict
        end
        added
      when Dict then @underlying.set(key, value)
      else raise "Unknown structure type: #{ @underlying }"
      end
    end
    alias []= set

    private

    def set_list(key, value)
      iterator = List.left_to_right_iterator(@underlying)
      while iterator.cursor && iterator.cursor.value.key != key
        iterator.next
      end

      if iterator.cursor.nil?
        @underlying.right_push(ListEntry.new(key, value))

        true
      else
        iterator.cursor.value.value = value

        false
      end
    end

    def convert_list_to_dict
      dict = Dict.new
      iterator = List.left_to_right_iterator(@underlying)

      while iterator.cursor
        dict[iterator.cursor.value.key] = iterator.cursor.value.value
        iterator.next
      end

      @underlying = dict
    end
  end
end

listing 8.3 The RedisHash class

The RedisHash class starts with a List as the data structure backing the hash, it will be converted to a Dict as needed, when the hash grows.

Our implementation differs slightly from Redis with regards to how data is stored in the list. Redis stores the keys and values, as flat elements, one after the other. This means that adding one pair to the hash results in two elements being added to the list. Our approach is bit different, we create a struct, ListEntry, to store the pairs in the list. This allows us to use our List class in a slightly more idiomatic way. One pair represents conceptually one element. This allows us to directly use the size attribute of the list, instead of having to divide it by two to obtain the number of elements in the hash.

The set method, which we alias to []=, to provide a similar API to the Dict & Hash classes, is the method used by the HSET command. We first lookup the current value of the hash_max_ziplist_value config, and convert the list to a dict if either the key or the value we're adding are longer than the config.

Once this check is performed, we use a pattern we'll see a lot in this chapter, a case/when statement to check the type of @underlying. There are three branches, it is either a List, a Dict, or anything else, in which case we want to crash the server as this is never supposed to happen.

The code for the List branch requires a few more lines of code, so we extract it to the set_list private method, below in the class. In the Dict case, it only requires three lines of code. We start by checking how many items are present in the Dict, we then use the Dict#set method, which we slightly modify to return a boolean:

module BYORedis
  class Dict
    # ...
    def set(key, value)
      entry = get_entry(key)
      if entry
        entry.value = value

        false
      else
        add(key, value)

        true
      end
    end
    alias []= set
    # ...
    end
  end
end

listing 8.4 Updates to the set method in the Dict class

The Dict#set method we introduced in Chapter 6 used to return the new value, which was not really helpful since the caller knows what the value is, it is the second argument to the method. We're now returning a boolean instead, indicating whether the pair was added or not.
The method either updates the value if the key is already in the hash, or add it altogether. We're not using the Dict#[]= alias here because doing so will not return anything, and we do care about the boolean value returned, to increment the count in the HSetCommand class.

The set_list method creates a list iterator and starts iterating from the head, as long as the node's key is different from the given key, we keep iterating. If we encounter a node with the same key, the iteration stops, and we update the value for that node. Otherwise, we add a new node at the end of the list. As noted above, we are using a new class to store node values, ListEntry. We could have used a "tuple approach", by storing a two-element array, such as [ 'key', 'value' ], but the ListEntry struct makes things a little bit more explicit, the items stored in the list can be read with clear methods, key & value, instead of using [0] & [1] respectively with the tuple approach.

Now that we have enough of RedisHash implemented, we need to add the new lookup_hash_for_write method to the DB class:

module BYORedis
  class DB

    # ...

    def lookup_hash(key)
      hash = @data_store[key]
      raise WrongTypeError if hash && !hash.is_a?(RedisHash)

      hash
    end

    def lookup_hash_for_write(key)
      hash = lookup_hash(key)
      if hash.nil?
        hash = RedisHash.new
        @data_store[key] = hash
      end

      hash
    end
  end
end

listing 8.5 The lookup_hash_for_write method in the DB class

This method is very similar to the one we create for lists in the previous chapter, except that it expects a Hash instance and creates one if necessary.

And with all this, the server can now response to HSET commands, let's now add the HSetNXCommand:

module BYORedis
  # ...

  class HSetNXCommand < BaseCommand
    def call
      Utils.assert_args_length(3, @args)
      key = @args[0]
      field = @args[1]
      value = @args[2]
      hash = @db.lookup_hash_for_write(key)

      if hash[field]
        RESPInteger.new(0)
      else
        hash[field] = value
        RESPInteger.new(1)
      end
    end

    def self.describe
      Describe.new('hsetnx', 4, [ 'write', 'denyoom', 'fast' ], 1, 1, 1,
                   [ '@write', '@hash', '@fast' ])
    end
  end
end

listing 8.6 The HSetNX class

This new command uses existing methods, if the given field already exists in the hash, we directly return 0 and leave the hash untouched. On the other hand, if the field is not already present, we add it, using the RedisHash#[]= this time, since we know it will add the element, and return 1.

Reading Hash values with HGET, HMGET & HGETALL

Now that we can create Hash instances in our database, we need to add the ability to read data from these hashes for them to be actually useful. Redis hash three commands to do so, HGET, to retrieve a single value, HMGET, to retrieve multiple values at once, and HGETALL to retrieve all the key/value pairs.

Let's start with adding the HGetCommand:

module BYORedis
  # ...
  class HGetCommand < BaseCommand
    def call
      Utils.assert_args_length(2, @args)

      hash = @db.lookup_hash(@args[0])

      if hash.nil?
        NullBulkStringInstance
      else
        key = @args[1]
        value = hash[key]
        if value.nil?
          NullBulkStringInstance
        else
          RESPBulkString.new(value)
        end
      end
    end

    def self.describe
      Describe.new('hget', 3, [ 'readonly', 'fast' ], 1, 1, 1,
                   [ '@read', '@hash', '@fast' ])
    end
  end
end

listing 8.7 The HGetCommand class

If the hash does not exist, or if the hash exists but does not contain the field, we return a null string, otherwise, we return the string stored for that field. We need to add the ability to find a key/value pair to the RedisHash class:

module BYORedis
  class RedisHash

    # ...

    def get(field)
      case @underlying
      when List then get_list(field)
      when Dict then @underlying[field]
      else raise "Unknown structure type: #{ @underlying }"
      end
    end
    alias [] get

    private

    # ...

    def get_list(field)
      iterator = List.left_to_right_iterator(@underlying)

      while iterator.cursor
        return iterator.cursor.value.value if iterator.cursor.value.key == field

        iterator.next
      end
    end
  end
end

listing 8.8 The RedisHash#get method

Once again, the Dict branch is simpler, so we perform it inline, we call the Dict#[] method, and return its result, a string or nil. In the List case, we go to the get_list private method. The approach here is very similar to the set_list method we wrote earlier, we iterate through the list, starting at the head, and stop if we find a ListEntry for which the key attribute matches the field parameter. If no entry matches, the method returns nil. Note that this is a perfect example of the worst case scenario time complexity we previously discussed. If the field is not present in the hash, we still have to iterate through the entire list to check every ListEntry instances.

Let's continue with the HMGetCommand:

module BYORedis
  # ...
  class HMGetCommand < BaseCommand
    def call
      Utils.assert_args_length_greater_than(1, @args)

      key = @args.shift
      hash = @db.lookup_hash(key)

      if hash.nil?
        responses = Array.new(@args.length)
      else
        responses = @args.map do |field|
          hash[field]
        end
      end

      RESPArray.new(responses)
    end

    def self.describe
      Describe.new('hmget', -3, [ 'readonly', 'fast' ], 1, 1, 1,
                   [ '@read', '@hash', '@fast' ])
    end
  end
end

listing 8.9 The HMGetCommand class

The HMGET command is very similar to HGET, the only difference is that it accepts multiple fields as its input, and returns an array. The implementation uses the same method from RedisHash, get, which we use through its alias, [], and call it in the block passed to Array#map. Using map here allows us to maintain the order of the results, we create an array where the n-th item will be the value for the n-th field passed as command argument after the hash key itself.

If the hash does not exist, we use the Array.new method to create an array of nil values with the same length as the number of fields passed to the command.

The last read command we need to add is HGetAllCommand. Because RESP2 does not have support for a map type, the result is an even-numbered array, containing an alternating sequence of keys and values.

module BYORedis
  # ...
  class HGetAllCommand < BaseCommand
    def call
      Utils.assert_args_length(1, @args)

      hash = @db.lookup_hash(@args[0])

      if hash.nil?
        pairs = []
      else
        pairs = hash.get_all
      end

      RESPArray.new(pairs)
    end

    def self.describe
      Describe.new('hgetall', 2, [ 'readonly', 'random' ], 1, 1, 1,
                   [ '@read', '@hash', '@slow' ])
    end
  end
end

listing 8.10 The HGetAllCommand class

This time we need a new method in RedisHash, get_all:

module BYORedis
  class RedisHash

    # ...

    def get_all
      case @underlying
      when List then get_all_list
      when Dict then get_all_dict
      else raise "Unknown structure type: #{ @underlying }"
      end
    end

    private

    # ...

    def get_all_list
      iterator = List.left_to_right_iterator(@underlying)
      pairs = []
      while iterator.cursor
        pairs.push(iterator.cursor.value.key, iterator.cursor.value.value)
        iterator.next
      end

      pairs
    end

    def get_all_dict
      pairs = []

      @underlying.each do |key, value|
        pairs.push(key, value)
      end

      pairs
    end
  end
end

The implementation for both data structures requires a few lines of code, so we move it to two private methods. In get_all_list, we follow the tried and true pattern we've used so far. We start iterating from the head, and accumulate the keys and values in an array.

In the get_all_dict method, we rely on the Dict#each method, which iterates through all the pairs in the dictionary, and for each pair we push both the key and the value to an array, and return it.

We now have a solid foundation for the Hash commands, we can add elements to the hash and read them back. Next on the list is the ability to increment values, if and only if the strings represent numeric values.

Incrementing numeric values with HINCRBY & HINCRBYFLOAT

Redis supports two commands to increment or decrement numeric values, HINCRBY & HINCRBYFLOAT. Decrement operations are performed using these commands with a negative argument. To decrement a value in a hash by 1, you would call HINCRBY h key -1.

These commands are very similar to INCRBY and INCRBYFLOAT which operates on Strings at the top-level. The *INCRBY commands only accept integer increments, and will reject floats, the *INCRBYFLOAT commands accept both integers and floats.

Even though you could use integer values with HINCRBYFLOAT, HINCRBY is still useful for two reasons, well really, only the first one actually matters:

Exactness: because the float based commands use floating point arithmetic, you're not guaranteed to get the result you'd expect, unexpected results can (and will) happen. Let's look at an example, imagine that you're building a bidding platforms where you store prices. It would be a fair requirement to increment the price of a product after a bid:

127.0.0.1:6379> HSET product price 166.92
(integer) 1
127.0.0.1:6379> HINCRBYFLOAT product price 402.22
"569.14000000000000001"

Yikes, yeah, that's close, but that's not really what we'd expect, which is 569.14. Floating point errors happen very often, for instance, many languages fail to return 3.3 for 1.1 + 2.2, you can try it with Ruby, Python, Elixir, Scala, Haskell and Javascript, they all pretty much return the same thing: 3.3000000000000003. The website 0.30000000000000004.com shows even more examples across most programming languages and explains in more details the cause of this unexpected result.

The bottom line is that floating point arithmetic suffers from precision issues, whereas integer operations do not. The only caveat to be aware of regarding integer arithmetic is around overflows, which we'll cover later when we implement the HINCRBY command.

A little bit less memory used: Redis uses long double variables for the floating point numbers, which use 16 bytes of memory, whereas it uses long long for integers, which use 8 bytes of memory. That being said, note these types are only used while the command is processed, the data in the hash, whether it is a list or a dict is a string, which uses one byte per digit. The string '1' representing the integer 1 uses 1 byte, the string '1.1' representing the float 1.1 uses 3 bytes, and so on.

Important note about prices

If you're working on any systems that handle prices, avoid at all cost using floating point numbers. A common approach is to always manipulate prices in cents, or whatever is the smallest currency unit, and use integers. In the example above, we would have done the following:

127.0.0.1:6379> HSET product price 16692
(integer) 1
127.0.0.1:6379> HINCRBY product price 40222
(integer) 56914

With this approach, you only transform the price from cents to the "regular" unit, dollar, yuan, pound, euro by doing the appropriate division only when displaying it to the user in the expected unit. By doing so, you guarantee that addition and subtraction operations will never result in loss of precision, as long as they don't overflow.

HINCRBY

Let's start with HINCRBY, and before writing any code, let's play with it quickly in the repl:

127.0.0.1:6379> HINCRBY h an-int 1
(integer) 1
127.0.0.1:6379> HINCRBY h an-int a
(error) ERR value is not an integer or out of range
127.0.0.1:6379> HSET h not-an-int a
(integer) 1
127.0.0.1:6379> HINCRBY h not-an-int 1
(error) ERR hash value is not an integer
127.0.0.1:6379> HINCRBY h an-int 9223372036854775806
(integer) 9223372036854775807
127.0.0.1:6379> HINCRBY h an-int 1
(error) ERR increment or decrement would overflow
127.0.0.1:6379> HINCRBY h an-int -9223372036854775807
(integer) 0
127.0.0.1:6379> HINCRBY h an-int -9223372036854775807
(integer) -9223372036854775807
127.0.0.1:6379> HINCRBY h an-int -1
(integer) -9223372036854775808
127.0.0.1:6379> HINCRBY h an-int -1
(error) ERR increment or decrement would overflow

As we can see in the previous example, calling HINCRBY on a non existing hash creates one, initializes the field's value to 0 and apply the increment afterwards. An error is return is the increment value is not an integer, and a different error is return if the value we're trying to increment cannot be represented an integer.

The other examples show the behavior around integer overflows. Redis attempts to convert the stored string values as long long, which are represented as signed 64 bit integers. The maximum value of a long long is 2^63 - 1, 9,223,372,036,854,775,807 and the minimum value is -(2^63), -9,223,372,036,854,775,808. One might expect that the minimum should equal the inverse of the maximum, that is min = -max, but as we just saw that's not the case, we have min = -(max + 1).

This is a result of the representation of signed integers as two's complement. With this representation the first bit is used to represent the sign, 1 means negative, 0 means positive. The other 63 bits are used to encode the actual integer value, which is why the max and min values are around 2^63. The biggest value that can be encoded is a zero, for the positive sign, followed by sixty-three 1s, which is 2^63 - 1. Let's look at an example with less bits, for the sake of simplicity. Imagine a three bit integer, the max value would be 2^2 - 1, 3 and the min value would be -(2^2), -4. As previously mentioned the max value is a zero followed by ones, 011. To obtain 3 from this, we start from the right, with the first digit, a 1, and use the index, starting at zero, as the power value, we get 2^0, which is 1, we then continue, another 1, at index 1, which gives us 2^1, 2. 2 + 1 = 3 so far so good. Another way to get to this number is with 2^2 - 1, also 3. In plain English, "Two to the power of the number of bits minus 1, minus 1". The same exact approach can be applied to 63 bits instead of 2, 2^0 + 2^1 + 2^2 + ... + 2^62 = 2^63 - 1.

In order to confirm the min value, we need to look at how negative numbers are represented in two's complement representation.

Two's complement is defined as:

The two's complement of an N-bit number is defined as its complement with respect to 2^N; the sum of a number and its two's complement is 2^N

Let's take 2 as an example, represented in binary as 010, using three bits. So, with N set to three, 2^N is 8, so to get to 8 from 2, we need 6, since 8 - 2 = 6, 6 is 110 as a three bit integer, 2^2 + 2^1. This tells us that the complement of 2, is 6, so -2 is represented the same way we'd represent 6, as 110.

Another, and probably easier, way to obtain the two's complement of a number is by inverting the digits and adding one.
Using this definition, let's see how we would represent -1. 1 is 001, because 2^0 = 1, so to represent -1, we first flip all the bits, 110, and add one, 111. Let's do the same thing for -2, 2 is 010, because 2^1 = 2, so flipping the bits gives 101, and adding one is 110, using the same process, we get to 101 for -3. We have the representation of 7 numbers so far, -3 (101), -2 (110), -1 (111), 0 (000), 1 (001), 2 (010) & 3 (011), but there are eight possible values with three bits, indeed, none of these numbers use 100.

For the previous numbers, we started from their decimal representation, but to show that 100 represents -4, we can use the opposite approach, convert a number from its two's representation, to its decimal version. So let's start with 100 and show that we end up with -4. To do so, we can start from the left, and the leftmost digit is treated differently from others, if 1, it is negative, if zero, well, it's zero, there's nothing to do, we then proceed to add all the following power of twos, so for 100, which is the third digit from the right, so index 2 in a 0-based system, 2^2 = 4, but since it's a 1, we start start at -4, we then add 0^1, we use 1 as the power here because the second digit, has an index of 1, and we finally add 0^0, for the rightmost digit, 0, at index 0, -4 + 0 + 0 = -4!

We can illustrate this approach with the numbers we previously arrived at, let's look at -3/101 for instance -(2^2) + 0^1 + 1^1, which can be expanded to -4 + 0 + 1, -3!

It's important to note that with two's complement, it is impossible to represent negative zero, which does not exist in ordinary arithmetic, zero does not have a sign.

It's time to create the HIncrByCommand class:

module BYORedis

  # ...

  class HIncrByCommand < BaseCommand
    def call
      Utils.assert_args_length(3, @args)
      incr = Utils.validate_integer(@args[2])

      key = @args[0]
      field = @args[1]
      hash = @db.lookup_hash_for_write(key)

      value = hash[field]
      if value.nil?
        value = 0
      else
        value = Utils.string_to_integer(value)
      end

      if (incr < 0 && value < 0 && incr < (LLONG_MIN - value)) ||
         (incr > 0 && value > 0 && incr > (LLONG_MAX - value))
        raise IntegerOverflow
      else
        new_value = value + incr
      end

      hash[field] = Utils.integer_to_string(new_value)

      RESPInteger.new(new_value)
    rescue InvalidIntegerString
      RESPError.new('ERR hash value is not an integer')
    rescue IntegerOverflow
      RESPError.new('ERR increment or decrement would overflow')
    end

    def self.describe
      Describe.new('hincrby', 4, [ 'write', 'denyoom', 'fast' ], 1, 1, 1,
                   [ '@write', '@hash', '@fast' ])
    end
  end
end

listing 8.11 The HIncrByCommand class

Once the validations are done, we look at the value for the given field and initialize it to 0 if the field does not exist. If the field does exist, we want to convert the string to an integer, returning an error if it cannot be converted. We use a new method in the Utils module to do so, string_to_integer.

The next step in an integer overflow check. This check is artificial in a language like Ruby that supports overflowing numbers, but, in order to both keep compatibility with Redis as well as understand how integer arithmetic works, we're imposing these arbitrary constraints on ourselves here.

We want to check that the operation will not result in an overflow. An overflow would happen if the sum of the old value and the new value were to be greater than the max value that can be represented by a signed 64-bit integer, 2^63-1 or lower than the minimum value, -2^63. We created two constants to hold these values, LLONG_MIN & LLONG_MAX, which happen to be defined in the climits.h header file in C.

We could have written these lines in a way that might be considered easier to read, with:

new_value = value + incr
if new_value > LLONG_MAX || new_value < LLONG_MIN

While this would work, this is a little bit of a chicken and egg problem, we'd be relying on the fact that the operation did overflow to raise an exception, but we wouldn't be able to know that the operation overflowed in a system where such situations can happen, like in C, because it would have overflowed. In other words, the condition could never have been true because no signed integer can be greater than LLONG_MAX and no signed integer can be lower than LLONG_MIN.

So far we were relying on the Kernel#Integer method to parse strings to integers. While this worked well until now, doing this is a little bit like "cheating". As a matter of fact, Redis uses its own function to transform a string to a long long: string2ll.

Let's now add the string_to_integer method to the Utils module:

module BYORedis

  ULLONG_MAX = 2**64 - 1 # 18,446,744,073,709,551,615
  ULLONG_MIN = 0
  LLONG_MAX = 2**63 - 1 # 9,223,372,036,854,775,807
  LLONG_MIN = 2**63 * - 1 # -9,223,372,036,854,775,808

  IntegerOverflow = Class.new(StandardError)
  InvalidIntegerString = Class.new(StandardError)

  module Utils

    # ...

    def self.string_to_integer(string)
      raise InvalidIntegerString, 'Empty string' if string.empty?

      bytes = string.bytes
      zero_ord = '0'.ord # 48, 'a'.ord == 97, so

      return 0 if bytes.length == 1 && bytes[0] == zero_ord

      if bytes[0] == '-'.ord
        negative = true
        bytes.shift
        raise InvalidIntegerString, 'Nothing after -' if bytes.empty?
      else
        negative = false
      end

      unless bytes[0] >= '1'.ord && bytes[0] <= '9'.ord
        raise InvalidIntegerString
      end

      num = bytes[0] - zero_ord

      1.upto(bytes.length - 1) do |i|
        unless bytes[i] >= zero_ord && bytes[i] <= '9'.ord
          raise InvalidIntegerString, "Not a number: '#{ bytes[i] }' / '#{ [ bytes[i] ].pack('C') }'"
        end

        raise IntegerOverflow, 'Overflow before *' if num > ULLONG_MAX / 10

        num *= 10
        raise IntegerOverflow, 'Overflow before +' if num > ULLONG_MAX - (bytes[i] - zero_ord)

        num += bytes[i] - zero_ord
      end

      if negative && num > -LLONG_MIN
        # In Redis, the condition is:
        #
        # if (v > ( (unsigned long long) (-(LLONG_MIN+1)) +1) )
        #
        # But used to be (-(unsigned long long)LLONG_MIN) until this commit:
        # https://github.com/redis/redis/commit/5d08193126df54405dae3073c62b7c19ae03d1a4
        #
        # Both seem to be similar but the current version might be safer on different machines.
        # Essentially it adds one to LLONG_MIN, so that multiplying it by -1 with the - operator
        # falls within the boundaries of a long long, given that min can be -9...808 while max
        # is always 9...807, we then cast the positive value to an unsigned long long, so that
        # we can add 1 to it, turning it into 9...808
        # The C standard does not seem to be very specific around the exact value of LLONG_MIN
        # it seems to either be -9..807 or, as it is on my machine, a mac, -9...808, which is
        # because it uses Two's Complement.
        raise IntegerOverflow, 'Too small for a long long'
      elsif negative
        -num
      elsif num > LLONG_MAX
        raise IntegerOverflow, 'Too big for a long long'
      else
        num
      end
    end
  end
end

listing 8.12 The string_to_integer method

There's a lot going on in this method, so let's take it one step at a time. The overall approach is to look at all the characters in the string, starting from the left, and converting them to a number, and accumulate it to the final result. The accumulated number is an unsigned number, and the last step is to make sure that the parsed number can fit within a signed number. Let's dive right in:

If the string is empty, there's no need to continue, we raise an InvalidIntegerString, which is rescued in the command class to return the value is not an integer or out of range error. The next step is to get all the bytes in the string, which is what we'll be iterating over. Note that an array of bytes is how strings are represented in C, with the type char buf[], a char is an 8-bit type, a byte. While it could have been tempting to use the String#[] method, as we've shown in Chapter 5, the Ruby String class performs a few tricks under the hood to deal with characters spanning over more than one byte. The following is an example using the wave emoji:

irb(main):102:0> s = '👋'
irb(main):103:0> s[0]
=> "👋"
irb(main):104:0> s.bytes
=> [240, 159, 145, 139]

As we can see the String#[] method makes it look like there's only one character, when there are actually four bytes in that string. The bytes are returned as numbers, between 0 and 255, the range of an 8-bit integer:

irb(main):110:0> 'abc'.bytes
=> [97, 98, 99]
irb(main):111:0> '123'.bytes
=> [49, 50, 51]

The next step is a small helper variable we'll need throughout the method. In Redis, they use '0' which can be used for integer arithmetic, and is replaced by its ASCII representation, 48. Because we will need this value a lot, we store it in a variable to avoid having to use '0'.ord throughout the method. The String#ord method returns the 'ordinal' value, that is its value in the ASCII encoding, 'a' is 97, 'b' is 98, '1' is 49, '2' is 50 and so on. We can see that these values correspond to what the String#bytes method returns.

If the string only contains one byte and that byte is equal to 48, the value of the zero character, then we return 0 right away, the work is done.

In the case where the first character is equal to 45, which is the value returned by '-'.ord, then we set the boolean variable negative to true, so that we know to return a negative number at the end of the method. We also call bytes.shift to remove the negative sign from the byte array. If the array is empty after that, that is, we only received a string containing a negative sign, we raise an InvalidIntegerString error, the input is invalid.

The first digit of a number cannot be 0, it can only be between 1 and 9, so we raise an error if the first byte is not in that range, between 49 ('1'.ord) and 57 ('9'.ord).

We then initialize the num variable, which we'll be our accumulator throughout the method, to num = bytes[0] - zero_ord. The operation bytes[0] - zero_ord returns the integer value of a character representing a digit between 0 and 9. This is because the value of the character '0' in ASCII is 48, and only goes up from there, so the character '5', which is 53 in ASCII, will return 5 when doing '5'.ord - zero_ord.

Now that the first digit was converted, we need to convert the rest of the string, where this time 0 is now an acceptable value, since numbers cannot start with a zero but can contain a zero afterwards. In the loop, we start by checking that the current byte is within 48 & 57 and raise an error if it is isn't. We then need to perform a few overflow checks. For the sake of simplicity, let's imagine that we were dealing with 8-bit numbers, where the maximum value would be 255 for an unsigned number, 2^8 - 1.

Let's manually go through the steps for the number 254, in this case, by the time we enter the loop, num would have been set to '2'.ord - zero_ord, 2. In the loop, we'd start with i set to 1, giving us bytes[1] == '5'. The range check would pass, '5'.ord is 53, it is between 48 and 57. The next check is num > ULLONG_MAX / 10, ULLONG_MAX is actually 2^64 - 1, but in our simplified 8-bit example, it is 2^8 - 1, 255. 255 / 10 would return 25, because this is an integer division, and num, 2, is not greater than that, so the check would pass, we then multiply num by 10, now that we know that multiplying it by 10 will not overflow.

We then check that num > ULLONG_MAX - (bytes[i] - zero_ord), which is essentially a way to check that adding the next digit from the string to num will not overflow. bytes[i] is '5', subtracting zero_ord returns 5, so we check that 20 > 255 - 5, 20 is not greater than 250, so we can perform the addition, num is now 25.

Repeating these steps in the next iteration, we look at the next character, '4', 25 is not greater than 25, so we multiply num by 10, giving us 250, and 250 is not greater than 255 - 4, so we add 4 to num and get the final result, 254.

Let's quickly look at two examples that would have raised overflow errors. If we had attempted to parse 256, we would have entered the loop with num set to 2, similarly as above, left the iteration at 25, multiplied by 10, and then failed the check 250 > 255 - 6. 255 is greater than 249, so we cannot add 6 to 255, it does not fit.

The other failure happens with numbers greater than 299, let's try with 300. We would enter the loop with num set to 3, exited the first iteration with num set to 30, and in the second iteration, we would fail the check 30 > 25. This tells us that we cannot multiply 30 by 10 and stay within the bounds of the integers we can represent.

The next steps take care of handling the sign of the number.

If negative is true, then we need to multiple num by -1, but before doing so, we need to check it is within the limits of a signed integer. So far we've parsed the numbers as an unsigned integer, which can go up to 2^64-1, but the minimum value of a signed integer is -2^63, so if num is greater than -LLONG_MIN, 9,223,372,036,854,775,808, then we raise an overflow error, otherwise, we can safely multiple num by -1. In other words, if num is 9,223,372,036,854,775,808 or lower, we can multiply it by -1, if it is 9,223,372,036,854,775,809 or more, then it would not fit.

We perform a similar check if negative is false and the final number should be positive, in that case, we check that num is not greater than LLONG_MAX, 2^63 - 1, 9,223,372,036,854,775,807, and if it is, we raise an overflow error.

Finally, if all the checks passed, we return num!

We can now rewrite validate_integer to use our own string_to_integer method:

Up until now we were using the OptionUtils.validate_integer method, which used the ruby Integer class to transform a String instance to an Integer instance, we can now use string_to_integer instead.

So, let's delete the option_utils file altogether and only use the Utils module:

module BYORedis
  module Utils

    # ...

    def self.validate_integer(str)
      string_to_integer(str)
    rescue IntegerOverflow, InvalidIntegerString
      raise ValidationError, 'ERR value is not an integer or out of range'
    end
  end
end

listing 8.13 The validate_integer method

We catch both exceptions here IntegerOverflow and InvalidIntegerString, and raise a ValidationError instead. This allows us to keep using the code in BaseCommand we introduced in the previous chapter.

The last method we need to add to the Utils module is integer_to_string, which we need to convert the new value back to a string before updating the value in the hash.

module BYORedis
  module Utils

    # ...

    def self.integer_to_string(integer)
      return '0' if integer == 0

      v = integer >= 0 ? integer : -integer
      zero_ord = '0'.ord
      bytes = []

      until v == 0
        bytes.prepend(zero_ord + v % 10)
        v /= 10
      end

      bytes.prepend('-'.ord) if integer < 0
      bytes.pack('C*')
    end
  end
end

listing 8.14 The validate_integer method

We start this method by converting the input to a positive integer in case it was negative, this allows us to process it regardless of its sign, and we prepend the '-' character at the end if the input was indeed negative.

We create an empty array, which we'll use to accumulate all the bytes representing all the characters of the string. We then loop until v reaches 0. In each iteration of the loop we get the character value by getting the modulo 10 of the input. Getting the modulo 10 essentially returns the right most digit. 255 % 10 = 5, 36 % 10 = 6. Once we have the decimal value of the rightmost digit, we add it to zero_ord, 48, to get the ASCII value of that number. The last step of the loop is to divide v by 10, to shift the whole number to the right, with the previous two examples, 255 would become 25, and 36 would become 3.

Once the loop exits, we have an array of number, representing the byte values of the string. We can now use the pack method to transform it a Ruby String. The C format tells Ruby to treat each number in the array as an 8-bit value representing a character, so it knows that 48 will be '0', 49, '1' and so on.

And with that, we have a working implementation of the HINCRBY command.

HINCRBYFLOAT

Ruby has a Float class, which, quoting the official documentation:

represent inexact real numbers using the native architecture's double-precision floating point representation.

While we could use the Float class to implement the HINCRBYFLOAT command, its precision is inferior to the implementation in Redis. Redis always use 17 digits precision, and with Ruby's Float class, we'd be stuck with at most a double-precision floating point number, which is a double in C. Redis uses long double, which offer greater precision. Ruby does not provide an easy way to use long double numbers, but it provides another class to handle number with decimal digits, BigDecimal:

provides arbitrary-precision floating point decimal arithmetic

The BigDecimal class provides so many features that it could be considered "cheating" according to the rules I set for this book, but dealing with floating point is a really complicated topic, and, to some extent, is not that central to how Redis works. That being said, even the Redis codebase does not perform all the float operations "from scratch". The conversion from a string to a long double is performed with the strtold function provided by the C standard library. This function takes care of a lot of the heavy lifting, such as parsing numbers using the E notation, like 1.1234e5 being parsed to 112,340.0. The conversion back from a long double to a string is then performed with snprintf("%.17Lf"), which is another function provided by the C standard library.

Let's look at how BigDecimal works:

irb(main):001:0> require 'bigdecimal'
=> true
irb(main):002:0> 0.1 + 0.2
=> 0.30000000000000004
irb(main):003:0> BigDecimal(0.1, 1) + BigDecimal(0.2, 1)
=> 0.3e0

The BigDecimal constructor, which weirdly enough is the method BigDecimal on the Kernel class, requires two arguments if the first argument is a float, to determine how many significant digits should be considered. In this example, one is enough, but let's look at an example with other numbers to see its impact:

irb(main):008:0> BigDecimal(0.123, 2)
=> 0.12e0

We passed the float 0.123, but because we told BigDecimal to only consider two significant digits, the result is a BigDecimal object representing the number 0.12.

Let's just check what Redis returns for the same operation:

127.0.0.1:6379> HINCRBYFLOAT h 1 0.1
"0.1"
127.0.0.1:6379> HINCRBYFLOAT h 1 0.2
"0.3"

The example above illustrates that some rounding errors we observe in Ruby, and other languages, with double-precision floating numbers are avoided with the long double type.

So now that we settled on the BigDecimal class, it's time to create the command class:

module BYORedis

  # ...

  class HIncrByFloatCommand < BaseCommand
    def call
      Utils.assert_args_length(3, @args)
      incr = Utils.validate_float(@args[2], 'ERR value is not a valid float')

      key = @args[0]
      field = @args[1]
      hash = @db.lookup_hash_for_write(key)

      value = hash[field]
      if value.nil?
        value = BigDecimal(0)
      else
        value = Utils.validate_float(value, 'ERR hash value is not a float')
      end

      new_value = value + incr

      raise FloatOverflow if new_value.nan? || new_value.infinite?

      new_value_as_string = Utils.float_to_string(new_value)
      hash[field] = new_value_as_string

      RESPBulkString.new(new_value_as_string)
    rescue InvalidFloatString
      RESPError.new('ERR hash value is not a float')
    rescue FloatOverflow
      RESPError.new('ERR increment would produce NaN or Infinity')
    end

    def self.describe
      Describe.new('hincrbyfloat', 4, [ 'write', 'denyoom', 'fast' ], 1, 1, 1,
                   [ '@write', '@hash', '@fast' ])
    end
  end
end

listing 8.15 The HIncrByFloatCommand class

In the previous chapter we introduced the OptionUtils.validate_float method, but as we just did with validate_integer, we are going to use a new method in the Utils package, and use BigDecimal instead of the Float class as we used to:

module BYORedis
  module Utils

    # ...

    def self.validate_float(str, error_message)
      case str
      when '+inf', 'inf', 'infinity', '+infinity' then BigDecimal::INFINITY
      when '-inf', '-infinity' then -BigDecimal::INFINITY
      else
        parsed = BigDecimal(str)
        if parsed.nan?
          raise ArgumentError
        else
          parsed
        end
      end
    rescue ArgumentError, TypeError
      raise ValidationError, error_message
    end
  end
end

listing 8.16 The validate_float method

The method mainly relies on Kernel#BigDecimal to do the heavy lifting, but we have to add a few custom pieces of logic. The first one is to translate the Redis representation of infinity to the BigDecimal one.

Redis recognizes the strings 'inf', '+inf', 'infinity', '+infinity', '-inf' & '-infinity' as special values representing the positive and negative infinity values, which are valid floats.

The constraint of the HINCRBYFLOAT regarding infinity are interesting given that HSET can be used to set the value of a field to either infinity or -infinity but the result of HINCRBYFLOAT cannot be either of these values:

127.0.0.1:6379> HSET h valid-inf inf
(integer) 1
127.0.0.1:6379> HSET h invalid-inf infi
(integer) 1
127.0.0.1:6379> HINCRBYFLOAT h valid-inf inf
(error) ERR increment would produce NaN or Infinity
127.0.0.1:6379> HINCRBYFLOAT h invalid-inf inf
(error) ERR hash value is not a float

This example shows that setting the value to inf means that Redis considers it to be a valid float, but rejects the operation because the result of inf + inf would result in infinity, and it refuses to do so. On the other hand, we can see that if the value in the hash was set to infi, which is just a regular string, then it fails with a different error, telling us that it can't perform the operation because the value in the hash is not a valid float.

The first error message also mentions NaN, which stands for Not A Number. NaN can happen for operations that cannot result in a valid result, such as the following:

irb(main):122:0> BigDecimal::INFINITY - BigDecimal::INFINITY
=> NaN

In order to replicate this logic, we use the BigDecimal#infinite? and BigDecimal#nan? methods to raise an exception if the result of the operation is not valid for Redis. The last step is similar to the one in HINCRBY, we convert the value back to a string, store it in the hash and return it. Let's look at float_to_string:

module BYORedis
  module Utils

    # ...

    def self.float_to_string(big_decimal)
      if big_decimal == BigDecimal::INFINITY
        'inf'
      elsif big_decimal == -BigDecimal::INFINITY
        '-inf'
      elsif (truncated = big_decimal.truncate) == big_decimal
        # Remove the .0 part of the number
        integer_to_string(truncated)
      else
        big_decimal.to_s('F')
      end
    end
  end
end

listing 8.17 The float_to_string method

If the value is either INFINITY or -INFINITY, we transform it to the valid Redis representation, inf & -inf. This is not necessary for now, but will become useful with other commands.

In the case where the value could be represented as an integer, that is, there are only zeroes on the right side, we want to only return the left side. That is, if the value is 2.0, we want to return 2. Checking if the truncated number is the same as the number is a way to test whether or not the number is an integer, in which case we use the integer_to_string method. We have to do this because the to_s method would otherwise return the number 2.0.

Finally, we use the to_s method on BigDecimal with the F argument, which returns the number using "conventional floating point notation", it would otherwise use the E notation:

irb(main):124:0> BigDecimal('1.2345').to_s('F')
=> "1.2345"
irb(main):125:0> BigDecimal('1.2345').to_s
=> "0.12345e1"

We can now use the validate_float method to create the validate_timeout method, which we can use for the blocking methods we created in the previous chapter:

module BYORedis
  module Utils

    # ...

    def self.validate_timeout(str)
      timeout = validate_float(str, 'ERR timeout is not a float or out of range')
      raise ValidationError, 'ERR timeout is negative' if timeout < 0 || timeout.infinite?

      timeout
    end
  end
end

listing 8.18 The validate_timeout method

And we can now update the blocking list commands:

module BYORedis
  # ...

  module ListUtils
    # ...

    def self.common_bpop(db, args, operation)
      Utils.assert_args_length_greater_than(1, args)

      timeout = Utils.validate_timeout(args.pop)
      list_names = args
      # ...
    end

    # ...
  end

  # ...

  class BRPopLPushCommand < BaseCommand
    def call
      Utils.assert_args_length(3, @args)

      source_key = @args[0]
      source = @db.lookup_list(source_key)
      timeout = Utils.validate_timeout(@args[2])
      destination_key = @args[1]
      # ...
    end
    # ...
  end
end

listing 8.19 Updates to the blocking list commands

Utility commands

We have six more commands to add, which happen to be simpler than the ones we added earlier. Let's start with a really useful one, HDEL.

HDEL

HDEL allows clients to delete one or more fields in a hash:

module BYORedis

  # ...

  class HDelCommand < BaseCommand
    def call
      Utils.assert_args_length_greater_than(1, @args)
      key = @args.shift
      hash = @db.lookup_hash(key)

      delete_count = 0
      if hash
        delete_count += @db.delete_from_hash(key, hash, @args)
      end

      RESPInteger.new(delete_count)
    end

    def self.describe
      Describe.new('hdel', -3, [ 'write', 'fast' ], 1, 1, 1,
                   [ '@write', '@hash', '@fast' ])
    end
  end
end

listing 8.20 The HDelCommend class

We call the DB#delete_from_hash method, so let's create this method now:

module BYORedis
  class DB

    # ...

    def delete_from_hash(key, hash, fields)
      delete_count = 0
      fields.each do |field|
        delete_count += (hash.delete(field) == true ? 1 : 0)
      end
      @data_store.delete(key) if hash.empty?

      delete_count
    end
  end
end

listing 8.21 The delete_from_hash method

The method iterates over all the given fields and calls RedisHash#delete, incrementing a counter for all successful deletions, returning this count at the end of the process. The method also takes care of deleting the hash from the database if the hash is empty after deleting all fields. Let's look at the delete method:

module BYORedis
  class RedisHash
    # ...

    def delete(field)
      case @underlying
      when List then was_deleted = delete_from_list(field)
      when Dict then
        was_deleted = !@underlying.delete(field).nil?
        if was_deleted && length - 1 == Config.get_config(:hash_max_ziplist_entries)
          convert_dict_to_list
        elsif @underlying.needs_resize?
          @underlying.resize
        end
      else raise "Unknown structure type: #{ @underlying }"
      end

      was_deleted
    end

    private

    # ...

    def convert_dict_to_list
      list = List.new
      @underlying.each do |key, value|
        list.right_push(ListEntry.new(key, value))
      end

      @underlying = list
    end

    def delete_from_list(field)
      was_deleted = false
      iterator = List.left_to_right_iterator(@underlying)

      while iterator.cursor
        if iterator.cursor.value.key == field
          @underlying.remove_node(iterator.cursor)

          return true
        end

        iterator.next
      end

      was_deleted
    end
  end
end

listing 8.22 The RedisHash#delete method

The deletion process for a list is delegated to the private method delete_from_list, while it is inlined for a Dict. For the latter, we call the Dict#delete method, which returns nil if nothing was deleted, or the value for the key it is was found and deleted. We perform two additional checks, first, if the size of the hash is now below the threshold, we convert the dict back to a list, through the private method convert_dict_to_list. Finally, we check whether or not the Dict instance needs resizing, Dict instances automatically grow but do not automatically shrink, so this check will make sure that a Hash can reduce its memory footprint and avoid waste.

The delete_from_list method should look pretty familiar at this point, we iterate starting from the head, and keep going until we find the element we're trying to delete. When we do find the list entry we need to remove, we call a new method on the List class: List#remove_node:

module BYORedis
  class List

    ListNode = Struct.new(:value, :prev_node, :next_node) do
      def remove
        if prev_node
          prev_node.next_node = next_node
        end

        if next_node
          next_node.prev_node = prev_node
        end

        self.next_node = nil
        self.prev_node = nil
      end
    end

    # ...

    def remove_node(node)
      if @head == node
        @head = node.next_node
      end

      if @tail == node
        @tail = node.prev_node
      end

      node.remove
      @size -= 1
    end

    # ...
  end
end

listing 8.23 The List#remove_node & ListNode#remove methods

The remove_node method removes the given node from the list, while updating the @head and @tail variables if needed. It uses the ListNode#remove method, which delegates all the next_node/prev_node handling to the struct itself. The whole process is very mechanical and reminiscent of the previous chapter, all the node pointers have to be updated, while being careful to check for nil values at each step of the way.

HEXISTS

The HEXISTS commands is used to check for the existence of a key inside a hash. Note that because RESP does have a boolean type, it returns a boolean, 1 if the key exists, 0 otherwise.

module BYORedis

  # ...

  class HExistsCommand < BaseCommand
    def call
      Utils.assert_args_length(2, @args)

      hash = @db.lookup_hash(@args[0])

      if hash.nil?
        RESPInteger.new(0)
      else
        value = hash[@args[1]]
        if value.nil?
          RESPInteger.new(0)
        else
          RESPInteger.new(1)
        end
      end
    end

    def self.describe
      Describe.new('hexists', 3, [ 'readonly', 'fast' ], 1, 1, 1,
                   [ '@read', '@hash', '@fast' ])
    end
  end
end

listing 8.24 The HDelCommend class

The command uses the RedisHash#get, through its [] alias, to check for the existence of the field, and return the appropriate number, acting as a boolean.

HKEYS

The HKEYS command is used to list all the keys inside a hash:

module BYORedis

  # ...

  class HKeysCommand < BaseCommand
    def call
      Utils.assert_args_length(1, @args)

      hash = @db.lookup_hash(@args[0])

      if hash.nil?
        EmptyArrayInstance
      else
        RESPArray.new(hash.keys)
      end
    end

    def self.describe
      Describe.new('hkeys', 2, [ 'readonly', 'sort_for_script' ], 1, 1, 1,
                   [ '@read', '@hash', '@slow' ])
    end
  end
end

listing 8.25 The HKeysCommand class

The command uses the new RedisHash#keys method:

module BYORedis
  class RedisHash
    # ...

    def keys
      case @underlying
      when List then keys_list
      when Dict then @underlying.keys
      else raise "Unknown structure type: #{ @underlying }"
      end
    end

    # ...

    def keys_list
      iterator = List.left_to_right_iterator(@underlying)
      keys = []

      while iterator.cursor
        keys << iterator.cursor.value.key

        iterator.next
      end

      keys
    end

    # ...
  end
end

listing 8.26 The RedisHash#keys method

When @underlying is a Dict, we can delegate directly to the Dict#keys method, on the other hand, if it is a List, we need to manually iterate through all the pairs in the list and accumulate the keys in an array.

HVALS

HVALS is very similar to HKEYS, except that it returns all the values:

module BYORedis

  # ...

  class HValsCommand < BaseCommand
    def call
      Utils.assert_args_length(1, @args)
      hash = @db.lookup_hash(@args[0])

      if hash.nil?
        EmptyArrayInstance
      else
        RESPArray.new(hash.values)
      end
    end

    def self.describe
      Describe.new('hvals', 2, [ 'readonly', 'sort_for_script' ], 1, 1, 1,
                   [ '@read', '@hash', '@fast' ])
    end
  end
end

listing 8.27 The HValsCommand class

This implementation is also very similar to HKeysCommand, except that we call RedisHash#values:

module BYORedis
  class RedisHash
    # ...

    def values
      case @underlying
      when List then values_list
      when Dict then @underlying.values
      else raise "Unknown structure type: #{ @underlying }"
      end
    end

    private

    # ...

    def values_list
      iterator = List.left_to_right_iterator(@underlying)
      values = []

      while iterator.cursor
        values << iterator.cursor.value.value

        iterator.next
      end

      values
    end
  end
end

listing 8.28 The RedisHash#values method

Similarly to RedisHash#keys, in the Dict case we call Dict#values, and in the List case we iterate through the list and accumulate all the values in an array.

HLEN

HLEN returns the number of key/value pairs in the hash:

module BYORedis

  # ...

  class HLenCommand < BaseCommand
    def call
      Utils.assert_args_length(1, @args)

      hash = @db.lookup_hash(@args[0])
      hash_length = 0

      unless hash.nil?
        hash_length = hash.length
      end

      RESPInteger.new(hash_length)
    end

    def self.describe
      Describe.new('hlen', 2, [ 'readonly', 'sort_for_script' ], 1, 1, 1,
                   [ '@read', '@hash', '@slow' ])
    end
  end
end

listing 8.29 The HLenCommand class

We use the RedisHash#length method to return the length of the hash:

module BYORedis
  class RedisHash
    # ...

    def length
      case @underlying
      when List then @underlying.size
      when Dict then @underlying.used
      else raise "Unknown structure type: #{ @underlying }"
      end
    end

    # ...
  end
end

listing 8.30 The RedisHash#length method

The length method is pretty succinct, it either calls List#size or Dict#used, which both return the number of elements they contain.

HSTRLEN

Finally, HSTRLEN returns the length of the value for the given key inside a hash:

module BYORedis

  # ...

  class HStrLenCommand < BaseCommand
    def call
      Utils.assert_args_length(2, @args)
      key = @args[0]
      field = @args[1]

      hash = @db.lookup_hash(key)
      value_length = 0

      unless hash.nil?
        value = hash[field]
        value_length = value.length unless value.nil?
      end

      RESPInteger.new(value_length)
    end

    def self.describe
      Describe.new('hstrlen', 3, [ 'readonly', 'fast' ], 1, 1, 1,
                   [ '@read', '@hash', '@fast' ])
    end
  end
end

listing 8.31 The HStrLenCommand class

This command does not need any new methods from the RedisHash class, it obtains the string stored at field with the RedisHash#get method and uses the Ruby String#length method to return its length.

Refactoring the test utilities

We introduced the Config class in this chapter, but there's no way to change the default values. We are going to add the CONFIG GET & CONFIG SET commands, in order to update config values at runtime and test the RedisHash class behavior with both a List and a Dict as the underlying data structure.

Let's first add the ConfigCommand class. The CONFIG command is different from all the other commands we've implemented so far in that it supports sub-commands. We are only adding support for the GET & SET sub-commands here, but the real Redis also supports CONFIG RESETSTAT & CONFIG REWRITE.

module BYORedis
  class ConfigCommand < BaseCommand

    def call
      if @args[0] != 'SET' && @args[0] != 'GET'
        message =
          "ERR Unknown subcommand or wrong number of arguments for '#{ @args[0] }'. Try CONFIG HELP."
        RESPError.new(message)
      elsif @args[0] == 'GET'
        Utils.assert_args_length(2, @args)
        value = Config.get_config(@args[1].to_sym)
        return RESPBulkString.new(Utils.integer_to_string(value))
      elsif @args[0] == 'SET'
        Utils.assert_args_length_greater_than(2, @args)
        @args.shift # SET
        @args.each_slice(2) do |key, _|
          raise RESPSyntaxError if key.nil? || value.nil?

          Config.set_config(key, value)
        end
      end

      OKSimpleStringInstance
    end

    def self.describe
      Describe.new('config', -2, [ 'admin', 'noscript', 'loading', 'stale' ], 0, 0, 0,
                   [ '@admin', '@slow', '@dangerous' ])
    end
  end
end

listing 8.32 The ConfigCommand class

The version of CONFIG GET we implemented is a simplified version of the one in Redis which supports glob-style patterns, with *.

With these two new commands, we can now update the config values in our test, which will allow us to lower the value of hash_max_ziplist_entries so that we don't have to add 513 items to hash for it to be converted to a Dict. Ideally we'll want to run all our tests under different combinations of configuration values.

The problem with the current approach to testing is that we spin up a new server for each test, which adds quite some time to each test as forking a new process and start a Ruby process within it takes some time. We will instead start a single process, and reuse it across our tests.

In order to do so, we need to do a little bit of work to make sure that the state of the server is clean for each tests. For instance, if a tests sends the BRPOPLPUSH a b 1 command, we want to make sure that if a next test runs within a second, that the first client correctly disconnected.

We also need to make sure that the database is in a clean state, and for that we will implement the FLUSHDB command:

module BYORedis
  class FlushDBCommand < BaseCommand

    def initialize(db, args)
      @db = db
      @args = args
    end

    def call
      Utils.assert_args_length(0, @args)
      @db.flush

      OKSimpleStringInstance
    end

    def self.describe
      Describe.new('flushdb', 1, [ 'write' ], 1, -1, 1, [ '@keyspace', '@write', '@slow' ])
    end
  end
end

listing 8.33 The FlushDBCommend class

Let's add the DB#flush method:

module BYORedis
  class DB

    # ...

    def initialize
      @logger = Logger.new(STDOUT)
      @logger.level = LOG_LEVEL
      flush
    end

    def flush
      @data_store = Dict.new
      @expires = Dict.new
      @ready_keys = Dict.new
      @blocking_keys = Dict.new
      @client_timeouts = SortedArray.new(:timeout)
      @unblocked_clients = List.new
    end

    # ...
  end
end

listing 8.34 The DB#flush method

Ruby makes our lives really easy here, to flush the database, we can simply instantiate a few fresh Dict, List and SortedArray and call it a day, the garbage collector will take care of freeing the memory of the previous ones now that they're not referenced anymore.

We now need to make some changes to the test_helper.rb file.

# test_helper.rb

require 'timeout'
require 'stringio'
require 'logger'

ENV['LOG_LEVEL'] = 'FATAL' unless ENV['LOG_LEVEL']

require_relative '../server'

$child_process_pid = nil
$socket_to_server = nil

def restart_server
  kill_child
  $child_process_pid = nil
  start_server
  $socket_to_server = nil
end

def start_server
  if $child_process_pid.nil?

    if !!ENV['DEBUG']
      options = {}
    else
      options = { [ :out, :err ] => '/dev/null' }
    end

    start_server_script = <<~RUBY
    begin
      BYORedis::Server.new
    rescue Interrupt
    end
    RUBY

    $child_process_pid =
      Process.spawn('ruby', '-r', './server', '-e', start_server_script, options)
  end
end

start_server

# Make sure that we stop the server if tests are interrupted with Ctrl-C
Signal.trap('INT') do
  kill_child
  exit(0)
end

require 'minitest/autorun'

def do_teardown
  with_server do |socket|
    socket.write(to_query('FLUSHDB'))
    read_response(socket)
    args = BYORedis::Config::DEFAULT.flat_map do |key, value|
      [ key.to_s, value.to_s ]
    end
    socket.write(to_query('CONFIG', 'SET', *args))
    read_response(socket)
  end
end

class MiniTest::Test
  def teardown
    with_server do
      do_teardown
    end
  rescue Errno::EPIPE, IOError => e
    $socket_to_server&.close
    $socket_to_server = nil
    connect_to_server
    do_teardown
    p "Exception during teardown: #{ e.class }/ #{ e }"
  end
end

def kill_child
  if $child_process_pid
    Process.kill('INT', $child_process_pid)
    begin
      Timeout.timeout(1) do
        Process.wait($child_process_pid)
      end
    rescue Timeout::Error
      Process.kill('KILL', $child_process_pid)
    end
  end
rescue Errno::ESRCH
  # There was no process
ensure
  if $socket_to_server
    $socket_to_server.close
    $socket_to_server = nil
  end
end

MiniTest.after_run do
  kill_child
end

def connect_to_server

  return $socket_to_server if !$socket_to_server.nil? && !$socket_to_server.closed?

  # The server might not be ready to listen to accepting connections by the time we try to
  # connect from the main thread, in the parent process. Using timeout here guarantees that we
  # won't wait more than 1s, which should more than enough time for the server to start, and the
  # retry loop inside, will retry to connect every 10ms until it succeeds
  connect_with_timeout
rescue Timeout::Error
  # If we failed to connect, there's a chance that it's because the previous test crashed the
  # server, so retry once
  p "Restarting server because of timeout when connecting"
  restart_server
  connect_with_timeout
end

def connect_with_timeout
  Timeout.timeout(1) do
    loop do
      begin
        $socket_to_server = TCPSocket.new 'localhost', 2000
        break
      rescue StandardError => e
        $socket_to_server = nil
        sleep 0.2
      end
    end
  end
  $socket_to_server
end

def with_server
  server_socket = connect_to_server

  yield server_socket

  server_socket.close
end

listing 8.35 Updates to the test_helper.rb file

Bare with me for a minute, I know that global variables are frowned upon, but we're only using them to make our lives easier.
I would not describe global variables as something to never use, but instead, as something to be extremely careful with. They can indeed become really problematic if they're used a lot throughout a codebase, especially if the value they hold changes a lot. Doing so could require a lot of headache . By using a global variable, we make it easier to maintain a single instance of the child process, without having to create a class, instantiate it, and burying the logic, what we want is actually not that much:

At the beginning of the test, spawn a new process in which we start the server, keep the pid of this process
For each test, create a socket and connect it to the server. At the end of the test, disconnect the socket
If the server crashes, we want to restart it so that subsequent tests work
At the end of each test, we want to run the FLUSHDB command so that next tests start with a clean database

This big refactor of the test context now allows us to use the following helper. Using this approach, which creates many more tests, would have been really slow with the "start a new process for each test approach", but now, each of these tests only generates a few round trips to the server, which is really fast, in the sub millisecond range.

# test_helper.rb
def test_with_config_values(combinations)
  # This line goes from a hash like:
  # { config_1: [ 'config_1_value_1', 'config_2_value_2' ],
  #   config_2: [ 'config_2_value_1', 'config_2_value_2' ] }
  # to:
  # [ [ [:config_1, "config_1_value_1"], [:config_1, "config_2_value_2"] ],
  #   [ [:config_2, "config_2_value_1"], [:config_2, "config_2_value_2"] ] ]
  config_pairs = combinations.map { |key, values| values.map { |value| [ key, value ] } }

  # This line combines all the config values into an array of all combinations:
  # [ [ [ :config_1, "config_1_value_1"], [:config_2, "config_2_value_1" ] ],
  #   [ [ :config_1, "config_1_value_1"], [:config_2, "config_2_value_2" ] ],
  #   [ [ :config_1, "config_2_value_2"], [:config_2, "config_2_value_1" ] ],
  #   [ [ :config_1, "config_2_value_2"], [:config_2, "config_2_value_2" ] ] ]
  all_combinations = config_pairs[0].product(*config_pairs[1..-1])

  # And finally, using the Hash.[] method, we create an array of hashes and obtain:
  #  [ { :config_1=>"config_1_value_1", :config_2=>"config_2_value_1" },
  #    { :config_1=>"config_1_value_1", :config_2=>"config_2_value_2" },
  #    { :config_1=>"config_2_value_2", :config_2=>"config_2_value_1" },
  #    { :config_1=>"config_2_value_2", :config_2=>"config_2_value_2" } ]
  all_combination_hashes = all_combinations.map { |pairs| Hash[pairs] }

  all_combination_hashes.each do |config_hash|
    with_server do |socket|
      socket.write(to_query('FLUSHDB'))
      resp = read_response(socket)
      assert_equal("+OK\r\n", resp)

      config_parts = config_hash.flat_map { |key, value| [ key.to_s, value.to_s ] }
      socket.write(to_query('CONFIG', 'SET', *config_parts))
      resp = read_response(socket)
      assert_equal("+OK\r\n", resp)
    end

    yield
  end
end

listing 8.36 the test_with_config_values helper in test_helper.rb

You can find all the tests on GitHub, but here is an example of the tests we can now write with the test_with_config_values helper:

describe 'HVALS' do
  it 'returns an array of all the values in the hash' do
    test_with_config_values(hash_max_ziplist_entries: [ '512', '1' ]) do
      assert_command_results [
        [ 'HSET h f1 v1 f2 v2', ':2' ],
        [ 'HVALS h', unordered([ 'v1', 'v2' ]) ],
      ]
    end
  end
end

listing 8.37 Example of a test using test_with_config_values for the HVALS command

The implementation of the HVALS command is different depending on whether the RedisHash instance is using a List or Dict to store the key/value pairs, so ideally we'd want to test both cases. Given that the test themselves are identical, at the end of the day, we do want to test the same output, but with two different implementation, it would be really repetitive to write the tests twice.

This approach allows us to wrap the tests we want to run with the different config values, and the helper will use FLUSHDB and CONFIG SET to prepare the context before running the tests.

Conclusion

As usual, you can find the code on GitHub. In the next chapter we will implement Sets, see you there!

Rebuilding Redis in Ruby - Chapter 7 - Adding List Commands

Pierre Jambet — Mon, 05 Oct 2020 15:17:56 +0000

What we'll cover

So far we've mainly worked with Strings, and arrays of Strings. A command is received as an array of Strings. GET accepts a String and return a String, or a nil String. SET works with two Strings, a key and a value. We've also handled a few integers here and there, like for the result value of the TTL & PTTL commands.

We are going to add full support for List related commands in this chapter.

Redis Data Types

When it comes to key/value pairs stored in the main keyspace, that is, the @data_store Dict instance, Redis supports other data types beside Strings:

Lists: An ordered collection of one or more Strings, sorted by order of insertion.
Sets: An unordered collection of unique Strings
Sorted Sets: An ordered collection of unique Strings. Strings are always added with a score, a float value, and the set orders them by their score. Uniqueness is enforced for the String value, multiple elements can have the same score. An example would be a list of user names, sorted by their age. The usernames are unique, but not the age: { <'pierre', 30>, <'james', 31>, <'jane', 32>, <'john', 32> }
Hashes: An unordered collection of key/value pairs, where keys are unique. Keys and values are Strings. This is essentially the same data type we implemented in Chapter 6.
Bit arrays (or bitmaps): This one is slightly different than the other data types as it uses Strings under the hood, but through dedicated commands allows users to perform operations using said Strings as arrays of bits. An example of such command is BITOP AND dest-key k1 k2, which will perform an AND operation between the values at keys k1 and k2 and store the value in dest-key.
HyperLogLogs: A HyperLogLog (HLL for short) is a data structure optimized to trade space for efficiency when counting elements in a set. The main operation of an HLL is to count the number of unique elements. The result might not be exact, but the storage used by HLLs will always be extremely small regardless of how many items were counted. Similarly to Bit arrays, HLLs are implemented using Strings. A Set could be used to achieve a similar count operation, the difference being that the set will store each element, allowing it to return an accurate count, at the cost of its memory footprint being proportional to the size of the set. A HyperLogLog in Redis cannot grow larger than 12,288 (12k) bytes.
Streams: An append-only collection, conceptually similar to a log structure. Elements are key/value pairs and are by default ordered by the time they were added to the stream. Streams are a fairly complicated data structure, well documented on the official website and will not be covered in this book.

Note about empty collections: Lists, Sets, Sorted Sets & Hashes cannot be empty, once the last element is deleted, the collection is removed from the keyspace.

Note about integers: Redis handles integers in a way that can seem confusing. The type of a value is never specified, it is always a String. Whether you send the command SET a-string-key a-string or SET an-integer-key 100, the value is sent as a string. Internally Redis knows whether or not what was received as a String can be treated as an integer. In some cases it will optimize the way it stores the data based on whether or not it is an integer. There are even commands that will only work if a value can be treated as a number, such as the INCR command.

List related terms

Let's clarify some list related terms first. A list is often represented as a left to right sequence of elements. The left-most element is called the head and the right-most element is called the tail.

It is also common to use the term "tail" to describe all the elements after the head. According to this definition, the tail is another list, empty if the list itself has zero or one element, non empty otherwise. This is not the definition we'll use throughout this book.

An empty list has no head and no tail, or it could be said that they're both nil. For a one-element list, the head and the tail are equal.

Elements can be added or removed from either side, left or right, as well as from within the list. Adding elements is commonly referred to as a push and removing elements as a pop. The terms shift and unshift are also used, where shift commonly means popping an element to the left of the list and unshift pushing an element to the left.

Finally, the terms append & prepend can also be used. In this context append is similar to a right push, and prepend is similar to a left push, or unshift.

Lists and other data structures

Lists can be used as the backing data structures for various more specific structures such as queues and stacks.

A list can be used as a queue by implementing the push operation as a "left push", that is, new elements are added to the left of the list, and the new element becomes the head of the list. Elements are removed with a "right pop", that is, elements are removed from the right of the list, the tail is removed and the element that preceded the tail becomes the new tail. This satisfies the First In First Out constraint of a queue.

A list can be used as a stack by implementing the push operation as a "right push", that is, new elements are added to the right of the list, and the new element becomes the new tail. Elements are removed as a "right pop", similarly to how we described a queue above. This satisfies the Last In First Out constraint of a stack.

The choice of right and left in these two examples is arbitrary but also very common within western countries where languages are read and written left to right.

Identical data structures could be implemented by reversing the side of each operations. A queue could be implemented by pushing new elements with a right push and removing them with a left pop. A stack could be implemented by pushing new elements with a left push and removing them with a left pop.

Redis List commands

Redis supports eighteen list related commands:

LINDEX: Gets an element by its index
LINSERT: Inserts an element
LLEN: Returns the length of the list
LPOP: Removes an element on the left of the list
LPOS: Returns the position of one or elements
LPUSH: Adds an element to the left of the list, creating it if needed
LPUSHX: Same as LPUSH, but does not create a new list if it doesn't already exist
LRANGE: Returns elements from the list
LREM: Removes one or more elements from the list
LSET: Replaces an element in a list
LTRIM: Keeps a subset of the list
RPOP: Removes an element on the right of the list
RPOPLPUSH: Removes an element to the right of a list and adds it to the left of another list
RPUSH: Adds an element to the right of the list, creating it if needed
RPUSHX: Same as RPUSH, but does not create a new list if it doesn't already exist
BLPOP: Blocking variant of LPOP
BRPOP: Blocking variant of RPOP
BRPOPLPUSH: Blocking variant of RPOPLPUSH

How does Redis do it

We've actually already implemented a list in the previous chapter to handle hash collisions and store multiple pairs in the same bucket.

We could use a similar structure to implement all the list commands detailed previously but the list we used for the Dict implementation has a few limitations with regards to performance that would be problematic in some cases.

The need for a doubly linked list

The following is what we used to create a list of entries in a bucket:

module BYORedis
  class DictEntry

    attr_accessor :next, :value
    attr_reader :key

    def initialize(key, value)
      @key = key
      @value = value
      @next = nil
    end
  end
end

listing 7.1 The linked list used for the Dict class

We used the class syntax to control which getters and setters we wanted to be generated, with attr_accessor & attr_reader, but it could be simplified as the following:

Node = Struct.new(:value, :next)

listing 7.2 The Node struct

Given the nature of Ruby's type system, value could be anything, and could therefore be a key/value pair if we passed a two-element array such as [ 'a-key', 'a-value' ].

This implementation is a "singly linked list". Each node has one "link" to its successor in the list, the next attribute in the previous example. If the link value is empty, then there's no successor and the element is the last one in the list.

One problem with this approach is that adding or deleting an element at the end of list requires to iterate through the whole list, which will become slower and slower as the list grows. We covered this problem in the previous chapter when we explained the need for growing the hash table.

A solution to this problem is to use what is commonly called a sentinel node, essentially a way to hold a reference to the tail of the list, for easy access to it, this would look like this:

class List
  def initialize
    @head = nil
    @tail = nil
  end

  def prepend(element)
    # The new node is created with its next value set to @head, if @head was nil, the list was
    # empty, and new_node is now the head, its next value will also be nil.
    # If @head was not nil, new_node now points to what used to be the head
    new_node = Node.new(element, @head)
    # The head of the list is now the new node
    @head = new_node
    if @tail.nil?
      # If @tail is nil, the list was empty, and has now one element, the head and the tail are
      # the same node
      @tail = new_node
    end

    new_node
  end

  def append(element)
    # new_node will be the new tail, so its next value must be nil
    new_node = Node.new(element, nil)
    if @tail
      @tail.next = new_node
    else
      # If the list is empty, both @head and @tail are nil
      @head = @tail
    end

    # Update the @tail value to now point to new_node
    @tail = new_node
    new_node
  end
end

listing 7.3 A singly linked list with prepend & append operations

The append operation is now O(1) with this implementation, it will require the same amount of steps regardless of the size of the list, it runs in constant time.

While this is a big win, another common list operation, removing the tail, also known as right pop, cannot be optimized from O(n) to O(1) with this approach:

def right_pop
  # If @tail is nil, the list is empty
  return nil if @tail.nil?

  # When the list has only one element, @head will be equal to @tail. In this case, popping from
  # the right or the left is effectively the same, and we can do so by setting both variables to
  # nil. The list is now empty
  if @head == @tail
    @head = nil
    @tail = nil

    nil
  elsif @head.next == @tail
    # If the second element, the one pointed at by @head.next is equal to @tail, the list has
    # only two elements
    @head.next = nil
    tail = @tail
    @tail = @head

    tail
  else
    # We now need a reference to the element pointing at @tail, so we can set its next value to
    # nil, instead of @tail, and set @tail to it, doing so will require iterating through all
    # the element in the list:
    cursor = @head.next

    # The number of steps this loop will go through is proportional to the size of the list,
    # making this method O(n)
    while cursor.next != @tail
      cursor = cursor.next
    end

    # We exited the loop, so the condition cursor.next == @tail is now true
    cursor.next = nil
    tail = @tail
    @tail = cursor

    tail
  end
end

listing 7.4 right_pop operation for a singly linked list

We could keep going down this road, and optimize the right_pop method by adding a third instance variable to the List class, one that always holds a value the second to last node. With such a value we would not have to iterate through the list in the else branch. Maintaining the value would require a few more steps in the append, prepend & pop operations to always keep it pointing at the second to last element.

This would still not allow us to efficiently implement all operations. The LRANGE command is used to return elements in the list, given two values start & stop. LRANGE list 0 1 means "return the first two items of the list list, the one at index 0 and the one at index 1. LRANGE list 0 3 means "return the first four items, the ones at indices 0, 1, 2 & 3, or put differently, all items between index 0 and 3. But the LRANGE command also supports a different syntax, with negative indices, to count elements starting from the tail. -1 means the last element in the list, -2, the second to last, and so on. Using this syntax we can return all the elements in the list with LRANGE 0 -1. We could also return the last three elements of the list with LRANGE -3 -1.

The main benefit of negative indices is that you don't need to know the size of the list to return the last n elements. In the previous examples, if we knew the list had 10 elements, we could have used LRANGE 7 9 to return the last three elements. But if an 11th element is added to the list, we would need to now send LRANGE 8 10. Using the negative index syntax, we can always use LRANGE -3 -1, regardless of the size of the list.

As we've already discussed a few times, Redis should be able to handle really big data sets, in the case of lists, ideally ones with thousands, and even millions of elements. Being able to return the last n elements of large lists should ideally not require to iterate through the whole list, which, sadly, with a singly linked list, we'd be forced to do.

A solution to this problem is to use a doubly linked list, one where each node contains a link to the next element but also one to the previous element. It's important to address right away that this approach comes with an important trade-off, we're now storing twice as much metadata per node, in exchange for potential speedups.

A doubly linked list can be implemented with the following Ruby struct:

DoublyLinkedNode = Struct.new(:value, :prev_node, :next_node)

listing 7.5 The Doubly Linked List Struct

We can now use this DoublyLinkedNode class in combination with the sentinel approach to implement the LRANGE command in an efficient manner:

def range(start, stop)
  # The complete range method, which handles all the possible edge cases is implemented later
  # on, we're only showing a simplified version here, to highlight the benefits of the doubly
  # linked list
  # The "real" method should handle the case where only one of the two bounds is negative, such
  # as LRANGE 0 -1
  list_subset = []
  if start < 0 && stop < 0 && start <= stop
    current_index = -1
    cursor = @tail
    while current_index >= start
      if current_index <= stop
        list_subset.prepend(cursor.value)
      end
      cursor = cursor.prev_node
      current_index -= 1
    end
  end

  list_subset
end

listing 7.6 A partial range method implementation leveraging a doubly linked list

As detailed in the comment, the previous example is not complete, but it shows how we can use the prev_node field to iterate from right to left instead of being forced to iterated from left to right with a singly linked list.

We can also simplify the right_pop method. The prev_node field on each node removes the need for a third instance variable to hold the second to last node.

def right_pop
  return if @tail.nil?

  tail = @tail
  @tail = @tail.prev_node
  if @tail
    @tail.next_node = nil
  else
    # If the list had only element, we removed it and the list is now empty, @tail is now nil
    # and we also need to set @head to nil
    @head = nil
  end

  tail
end

listing 7.7 The right_pop operation leveraging a doubly linked list

Redis uses a doubly linked list approach, for the reasons we mentioned above, but it uses a more optimized version to reduce memory usage.

On a 64-bit CPU, which is what most modern CPUs use, a pointer is a 64-bit integer, which uses 8 bytes (8 * 8 = 64 bits). So regardless of what the node actually stores, you need two pointers per node, one for the next node, one for the previous node, that's 16 bytes, per node. In a common case where each node would store an integer, which in C can have different sizes, as small as one byte and as big as 8, it means that even if you're storing a large integer, one that takes 8 bytes, you end up with 16 bytes of metadata, for 8 bytes of actual data. The problem is potentially even worse if you're storing smaller integer, say that the integers stored are small, 2 bytes for instance, you would end up with a node of 18 bytes where only 2 bytes are the actual data. This problem might not seem too bad with small lists but as the list grows, it would cause the server to waste a lot of memory to store these pointers.

Redis uses two data structures to optimize for this problem. At the top level, it uses a structure it calls Quicklist, which is a doubly linked list where each element is a Ziplist. A Ziplist is a compact optimization of a doubly linked list that does not suffer from the metadata storage problem we described.

Briefly summarizing a Ziplist is a complicated task, but it is conceptually close to how arrays work. It allocates a contiguous chunk of memory with malloc(3). A key difference is that all elements in an array have the same size, so there's not need for metadata beyond the size of the array. If you're trying to get the 5th element in an array, you do so by multiplying the size of an element in the array by five, and use the value as an offset from the beginning of the array, and the value you found at that location is the fifth element in the array.

Elements can have different sizes with a Ziplist, small integers, with a value between 0 and 12 will only use one byte, whereas larger integers, with values greater than 2,147,483,647 - the maximum value of an int32_t, which uses 4 bytes - will require 9 total bytes, 8 bytes for the value, an int64_t, and one byte of metadata. These variations in element sizes, which allows for greater optimizations, small elements take a small amount of space, means that the list needs to store some metadata to allow traversal of the list and access to the different elements. Appendix A explains in more details about Ziplist works, but does not provide a Ruby implementation.

The Ziplist approach also shares some similarities with how SQL database systems organize data into pages. For reference this page of the Postgres documentation explains in details the page layout. A key difference is that pages have a fixed size, whereas a ziplist will grow until it reaches its max size. The similarity is in the fact that they both operate on a set amount of memory which starts with some metadata followed by the actual data, organized in a compact sequence.

The reason why Redis does not exclusively uses Ziplists is because as Ziplists grow, update operations such as adding or removing elements, become more and more costly. This mixed approach allows Redis to benefit from the best of both worlds, the quicklist maintains a list of ziplists, which actually hold the elements in the list. Each ziplist has a maximum size, and when it reaches it, a new ziplist node is created in the parent quicklist.

This implementation is fairly complex so for the sake of simplicity we will use the basic doubly linked list approach shown above in this chapter.

The quicklist/ziplist approach might be implemented in a future chapter

It is now time to add support for all the list commands to our server.

Handling Incorrect Types

Now that our server is about to handle more than one type, Strings and Lists, we need to consider the type of each key/value pair when processing commands. For instance, if a pair is first created as a String, with the SET command, calling LLEN on it, which attempts to return the length of a List, is invalid:

127.0.0.1:6379> SET a-string a-value
OK
127.0.0.1:6379> LLEN a-string
(error) WRONGTYPE Operation against a key holding the wrong kind of value

Some commands work regardless of the type of the pair, such as TTL and PTTL:

127.0.0.1:6379> TTL a-string
(integer) -1
127.0.0.1:6379> RPUSH a-list 1
(integer) 1
127.0.0.1:6379> TTL a-list
(integer) -1

Redis has a TYPE command that can be used to return the type of a pair, the possible return values are: string, list, set, zset, hash and stream. If the key does not exist, it returns none.

Let's add support for the TYPE command.

So far we've repeated some code between all the commands, let's improve this by defining a BaseCommand class:

module BYORedis
  class BaseCommand

    def initialize(data_store, expires, args)
      @logger = Logger.new(STDOUT)
      @logger.level = LOG_LEVEL
      @data_store = data_store
      @expires = expires
      @args = args
    end

    def call
      raise NotImplementedError
    end
  end
end

listing 7.8 The BaseCommand class, parent of all command classes

We can now define the TypeCommand class, which inherits from BaseCommand:

module BYORedis
  class TypeCommand < BaseCommand

    def call
      Utils.assert_args_length(1, @args)

      key = @args[0]
      ExpireHelper.check_if_expired(@db, key)
      value = @db.data_store[key]

      case value
      when nil
        RESPSimpleString.new('none')
      when String
        RESPSimpleString.new('string')
      else
        raise "Unknown type for #{ value }"
      end
    end
  end
end

listing 7.9 The TypeCommand class

The command is not that useful for now, it either returns none or string. We will add a new branch to the case/when statement when we add the List type.

The next step is to add it to the COMMANDS constant in the Server class:

# ...
require_relative './pttl_command'
require_relative './type_command'
require_relative './command_command'

module BYORedis

  class Server

    COMMANDS = Dict.new
    COMMANDS.set('command', CommandCommand)
    # ...
    COMMANDS.set('type', TypeCommand)

    MAX_EXPIRE_LOOKUPS_PER_CYCLE = 20
    # ...
  end
end

listing 7.10 Updates to the Server class to support the TYPE command

All the command classes we've created need describe class method, which is in turn used by the CommandCommand class to return the description of the command. The approach worked well so far but it is a bit verbose, and as we add more commands, becomes more and more annoying.

Let's simplify this process with a struct, Describe, defined in BaseCommand:

module BYORedis
  class BaseCommand

    Describe = Struct.new(:name, :arity, :flags, :first_key_position, :last_key_position,
                          :step_count, :acl_categories) do
      def serialize
        [
          name,
          arity,
          flags.map { |flag| RESPSimpleString.new(flag) },
          first_key_position,
          last_key_position,
          step_count,
          acl_categories.map { |category| RESPSimpleString.new(category) },
        ]
      end
    end
  end

  # ...
end

listing 7.11 The Describe Struct defined in the BaseCommand class

We need to update the CommandCommand class as well, to use this new class:

module BYORedis
  class CommandCommand < BaseCommand

    # ...

    def call
      RESPArray.new(SORTED_COMMANDS.map { |command_class| command_class.describe.serialize })
    end

    def self.describe
      Describe.new('command', -1, [ 'random', 'loading', 'stale' ], 0, 0, 0,
                   [ '@slow', '@connection' ])
    end
  end
end

listing 7.12 The CommandCommand class using the Describe Struct

We can now define the self.describe method in the TypeCommand with the following:

module BYORedis
  class TypeCommand < BaseCommand

    ...

    def self.describe
      Describe.new('type', 2, [ 'readonly', 'fast' ], 1, 1, 1,
                   [ '@keyspace', '@read', '@fast' ])
    end
  end
end

listing 7.13 The TypeCommand class using the Describe struct

We also need to update all the existing command classes: DelCommand, GetCommand, SetCommand, TtlCommand, & PTtlCommand. Doing so is fairly mechanical and is therefore not included here. You can find all the code from this chapter on GitHub

Adding List Commands

Now that we've explored the landscape of lists in Redis, it's time to add these functionalities to our server. We're going to start with the two commands that allow us to create a list, LPUSH & RPUSH.

Creating a list with LPUSH & RPUSH

There are two commands that allow clients to create a list, LPUSH & RPUSH. If the given key does not already exist and only one new element is passed to the command, they behave identically, they create a list and add the given element to it. The difference with this two commands occurs when the list already exists and has some elements in it or if multiple elements are passed. LPUSH adds them to the Left and RPUSH adds them to the Right.

Both commands accepts one or more values,

127.0.0.1:6379> LPUSH a-list a b c d
(integer) 4
127.0.0.1:6379> LRANGE a-list 0 -1
1) "d"
2) "c"
3) "b"
4) "a"

In the previous example we can see that LPUSH added four elements to the newly created list a-list. It first added a, then b to the left, as the new head, and then c and d to the left as well, the final list is [ 'd', 'c', 'b', 'a' ].

We can try the same example but with RPUSH:

127.0.0.1:6379> DEL a-list
(integer) 1
127.0.0.1:6379> RPUSH a-list a b c d
(integer) 4
127.0.0.1:6379> LRANGE a-list 0 -1
1) "a"
2) "b"
3) "c"
4) "d"

The subsequent elements, b, c & d were added as new tails, to the right, and the final list is: [ 'a', 'b', 'c', 'd' ].

Finally, if the key already exists and is not a List, Redis returns an error:

127.0.0.1:6379> SET a b
OK
127.0.0.1:6379> LPUSH a a
(error) WRONGTYPE Operation against a key holding the wrong kind of value
127.0.0.1:6379> RPUSH a a
(error) WRONGTYPE Operation against a key holding the wrong kind of value

Because most of the list commands will be small and related to each other, instead of creating one file per command as we've done so far, this time we'll create a single file list_commands.rb and define all the command classes in there. Let's start with LPUSH.

Before creating the LPushCommand, we will start by reorganizing how the Server class and the different command classes interact with each other. So far we've passed the @data_store and @expires Dict instances, as well as the command arguments, to each command class.

We will need to create more data structures to handle various list related commands, so to simplify this, we will wrap all these data structures inside a DB class. It's worth noting that this approach is conceptually similar to how the code is organized in the Redis codebase. Redis defines a C struct for a DB type.

The process of a looking up a list is slightly different than it is from a String. For commands such as RPUSH & LPUSH, we want to see if a list exists for the given key, but if it doesn't exist, we want to create a new list. Let's create a method in the DB class for this purpose:

module BYORedis
  class DB

    attr_reader :data_store, :expires
    attr_writer :ready_keys

    def initialize
      @logger = Logger.new(STDOUT)
      @logger.level = LOG_LEVEL
      @data_store = Dict.new
      @expires = Dict.new
    end

    def lookup_list(key)
      list = @data_store[key]
      raise WrongTypeError if list && !list.is_a?(List)

      list
    end

    def lookup_list_for_write(key)
      list = lookup_list(key)
      if list.nil?
        list = List.new
        @data_store[key] = list
      end

      list
    end
  end
end

listing 7.14 The new DB class

We'll define the WrongTypeError shortly, when we add a few other utility methods.

Let's now refactor the BaseCommand class to use the DB class:

module BYORedis
  class BaseCommand

    # ...

    def initialize(db, args)
      @logger = Logger.new(STDOUT)
      @logger.level = LOG_LEVEL
      @db = db
      @args = args
    end
  end
end

listing 7.15 Updates to the BaseCommand class to support the DB class

And finally, let's update the Server class:

# ...
module BYORedis

  class Server
    # ...
    def initialize
      @logger = Logger.new(STDOUT)
      @logger.level = LOG_LEVEL

      @clients = Dict.new
      @db = DB.new
      @server = TCPServer.new 2000
      # ...
    end

    # ...
    def handle_client_command(command_parts)
      @logger.debug "Received command: #{ command_parts }"
      command_str = command_parts[0]
      args = command_parts[1..-1]

      command_class = COMMANDS[command_str.downcase]

      if command_class
        command = command_class.new(@db, args)
        command.call
      else
        # ...
      end
    end
    # ...
  end
end

listing 7.16 Updates to the Server class to support the DB class

Both LPUSH & RPUSH accept at least two arguments, the key and one or more elements to add. The validation of the number of arguments is very similar across commands, so let's create a helper method for that. Let's first create a Utils module and add it there:

module BYORedis

  InvalidArgsLength = Class.new(StandardError) do
    def resp_error(command_name)
      RESPError.new("ERR wrong number of arguments for '#{ command_name }' command")
    end
  end
  WrongTypeError = Class.new(StandardError) do
    def resp_error
      RESPError.new('WRONGTYPE Operation against a key holding the wrong kind of value')
    end
  end

  module Utils
    def self.assert_args_length(args_length, args)
      if args.length != args_length
        raise InvalidArgsLength, "Expected #{ args_length }, got #{ args.length }: #{ args }"
      end
    end

    def self.assert_args_length_greater_than(args_length, args)
      if args.length <= args_length
        raise InvalidArgsLength,
              "Expected more than #{ args_length } args, got #{ args.length }: #{ args }"
      end
    end
  end
end

listing 7.17 The new Utils module

We can now call Utils.assert_args_length_greater_than(1, @args) to validate the number of arguments for the LPUSH & RPUSH commands. Doing so will raise a InvalidArgsLength exception. Let's add a rescue statement for this exception, as well as for the WrongTypeError we used earlier in the DB class. Let's do this in the BaseCommand so that all classes benefit from it:

module BYORedis
  class BaseCommand

    # ...
    def execute_command
      call
    rescue InvalidArgsLength => e
      @logger.debug e.message
      command_name = self.class.describe.name.upcase
      e.resp_error(command_name)
    rescue WrongTypeError => e
      e.resp_error
    end

    def call
      raise NotImplementedError
    end
  end
end

listing 7.18 Updates to the BaseCommand class to catch shared exceptions

This is an application of the template method pattern. The base class defines logic around the call method, and it is up to the subclasses to define the actual logic of the call method. We need to update the Server class to now call the execute_command method instead:

module BYORedis
  class Server

    # ...

    def handle_client_command(command_parts)
      @logger.debug "Received command: #{ command_parts }"
      command_str = command_parts[0]
      args = command_parts[1..-1]

      command_class = COMMANDS[command_str.downcase]

      if command_class
        command = command_class.new(@db, args)
        command.execute_command
      else
        # ...
      end
    end
  end
end

listing 7.19 Updates to the Server class to use the execute_command method

We already know we are going need a very similar feature soon, for the RPUSH command, so we're first defining a set of shared methods:

module BYORedis
  module ListUtils

    def self.common_lpush(list, elements)
      elements.each { |element| list.left_push(element) }
      RESPInteger.new(list.size)
    end

    def self.common_rpush(list, elements)
      elements.each { |element| list.right_push(element) }
      RESPInteger.new(list.size)
    end

    def self.common_find(args)
      Utils.assert_args_length_greater_than(1, args)

      yield args[0]
    end

    def self.find_or_create_list(db, args)
      common_find(args) do |key|
        db.lookup_list_for_write(key)
      end
    end
  end
end

With all this refactoring out of the way, we can now define the LPushCommand:

require_relative './list'

module BYORedis

  class LPushCommand < BaseCommand
    def call
      list = ListUtils.find_or_create_list(@db, @args)
      values = @args[1..-1]
      ListUtils.common_lpush(list, values)
    end

    def self.describe
      Describe.new('lpush', -3, [ 'write', 'denyoom', 'fast' ], 1, 1, 1,
                   [ '@write','@list', '@fast' ])
    end
  end
end

listing 7.20 Adding list_commands.rb and the LPushCommand class

We first use Utils.assert_args_length_greater_than to validate that we have at least two arguments. We then lookup a list, creating it if doesn't exist and failing if the key already exists and is of a different type. Finally, we iterate over all the remaining arguments and push them to the left of the list with list.left_push and return the size of list.

There's a major problem, the List class does not exist yet! Let's address this now:

module BYORedis

  class List

    ListNode = Struct.new(:value, :prev_node, :next_node)

    attr_accessor :head, :tail, :size

    def initialize
      @head = nil
      @tail = nil
      @size = 0
    end

    def left_push(value)
      new_node = ListNode.new(value, nil, @head)

      if @head.nil?
        @tail = new_node
      else
        @head.prev_node = new_node
      end

      @head = new_node
      @size += 1
    end
  end
end

listing 7.21 The List class

The class defines a doubly linked list, similar to what we shown earlier in the chapter. The list is initialized with a size of 0, and nil values for @head and @tail. The left_push method behaves differently depending on whether the list was empty or not. We start by creating a new node, with the given value, and setting the prev_node attribute to nil, given that the new element will be the new head, it does not have a previous element in the list. The next element is set to @head, which may be nil if the list was empty.

If the list was empty, which we check with if @head.nil?, then the new element is also the tail of the list, and we update the @tail attribute as such. Otherwise, we update the old head's previous node, which is still the value held by @head, to the new node. In other words, the old head now has a previous element in the list.

Finally, we update @head to the new node and increment the size of the list.

We need to require the list_commands file in the server.rb file as well as adding LPushCommand to the COMMANDS constant:

# ...
require_relative './list_commands'
# ...
module BYORedis

  class Server

    COMMANDS = Dict.new
    COMMANDS.set('command', CommandCommand)
    COMMANDS.set('del', DelCommand)
    COMMANDS.set('get', GetCommand)
    COMMANDS.set('set', SetCommand)
    COMMANDS.set('ttl', TtlCommand)
    COMMANDS.set('pttl', PttlCommand)
    COMMANDS.set('lpush', LPushCommand)
    # ...
  end
end

listing 7.22 Updates to the Server class to support the LPUSH command

Now that lists can be added to the keyspace, let's handle List values in the ListCommand class:

module BYORedis
  class TypeCommand < BaseCommand

    def call
      # ...
      case value
      when nil
        RESPSimpleString.new('none')
      when String
        RESPSimpleString.new('string')
      when List
        RESPSimpleString.new('list')
      else
        raise "Unknown type for #{ value }"
      end
    end
    # ...
  end
end

listing 7.23 Update to the TypeCommand to support the List type

The RPushCommand is very similar to the LPushCommand:

module BYORedis
  # ...

  class RPushCommand < BaseCommand
    def call
      list = ListUtils.find_or_create_list(@db, @args)
      values = @args[1..-1]
      ListUtils.common_rpush(list, values)
    end

    def self.describe
      Describe.new('rpush', -3, [ 'write', 'denyoom', 'fast' ], 1, 1, 1,
                   [ '@write', '@list', '@fast' ])
    end
  end
end

listing 7.24 The RPushCommand class

The class is almost identical to LPushCommand, except that we call right_push on the list instead, so let's add this method to the List class:

module BYORedis

  class List

    # ...

    def right_push(value)
      new_node = ListNode.new(value, @tail, nil)

      if @head.nil?
        @head = new_node
      else
        @tail.next_node = new_node
      end

      @tail = new_node
      @size += 1
    end
  end
end

listing 7.25 The List#right_push method

The right_push method is also similar to left_push and also behaves differently whether or not the list was empty. We start by creating the new node, where the previous node value is set to what the tail was. If the list was empty, this value will be nil and the new element will be the only element in the list, otherwise its previous node value will point at the old tail. If the list was empty, we also need to update the head value. Otherwise, we need to update the next node value of the old tail to point to the new node.

Finally, we update the value of @tail and increment the size of list.

The *X variants, LPUSHX & RPUSHX

There are two more commands, almost identical to LPUSH & RPUSH, LPUSHX & RPUSHX. The only difference is that these two commands only push new elements to the list if it already exists. If the list does not already exist, it returns 0, otherwise it returns the new size of the list, like LPUSH & RPUSH do.

Given that these commands share a good amount of logic, let's add a few more helpers to the ListUtils module:

module BYORedis

  # ...

  module ListUtils
    # ...
    def self.find_list(db, args)
      common_find(args) do |key|
        db.lookup_list(key)
      end
    end

    def self.common_xpush(list)
      if list.nil?
        RESPInteger.new(0)
      else
        yield
      end
    end
  end
  # ...
end

listing 7.26 The ListUtils module

We can now create the LPushXCommand and RPushXCommand classes:

module BYORedis

  # ...
  class LPushXCommand < BaseCommand
    def call
      list = ListUtils.find_list(@db, @args)
      values = @args[1..-1]
      ListUtils.common_xpush(list) do
        ListUtils.common_lpush(list, values)
      end
    end

    def self.describe
      Describe.new('lpushx', -3, [ 'write', 'denyoom', 'fast' ], 1, 1, 1,
                   [ '@write', '@list', '@fast' ])
    end
  end
end

listing 7.27 The LPushXCommand

The only difference is that we call @db.lookup_list(key) instead of @db.lookup_list_for_write(key).

module BYORedis

  # ...
  class RPushXCommand < BaseCommand
    def call
      list = ListUtils.find_list(@db, @args)
      values = @args[1..-1]
      ListUtils.common_xpush(list) do
        ListUtils.common_rpush(list, values)
      end
    end

    def self.describe
      Describe.new('rpushx', -3, [ 'write', 'denyoom', 'fast' ], 1, 1, 1,
                   [ '@write', '@list', '@fast' ])
    end
  end
end

listing 7.28 The RPushXCommand

We now have four List related commands, which allow us to add elements to a list, but we don't have any way to read the content of the list. This is what the LRANGE command was created for.

Reading a list with LRANGE

The LRANGE command accepts three arguments, a key, a start index and a stop index. It returns all the elements in the list stored at the given key, between the start and stop indices. As discussed earlier in the chapter, it supports a special syntax, with negative indices, to index element from the right instead of from the left.

Another important piece of the LRANGE command is that it does not return errors for out of bound indices or for out of order start and stop values. Let's look at few examples first.

127.0.0.1:6379> RPUSH a-list a b c d e f
(integer) 6

The a-list list contains the following elements, [ 'a', 'b', 'c', 'd', 'e', 'f' ]. The index of the element, 'f' is 5, but LRANGE silently ignores this if we ask for all the elements up to index 10 for instance:

127.0.0.1:6379> LRANGE a-list 0 10
1) "a"
2) "b"
3) "c"
4) "d"
5) "e"
6) "f"

It returned all the elements in the given range, and ignored the fact that there are no elements between index 6 and 10. The same logic works the other way, with negative indices, 'f' is at index -1, and 'a' is at index -6. There is no element at index -7 and beyond:

127.0.0.1:6379> LRANGE a-list -10 -6
1) "a"

And Redis returns empty arrays if the whole request is outside the list:

127.0.0.1:6379> LRANGE a-list -10 -7
(empty array)
127.0.0.1:6379> LRANGE a-list 6 10
(empty array)

Finally, the following commands don't make any sense, since the start value is after the stop value, and Redis returns an empty array in this case:

127.0.0.1:6379> LRANGE a-list 2 1
(empty array)
127.0.0.1:6379> LRANGE a-list -1 -2
(empty array)

Let's implement the logic for the LRANGE command:

module BYORedis
  # ...

  class LRangeCommand < BaseCommand
    def call
      Utils.assert_args_length(3, @args)

      key = @args[0]
      start = OptionUtils.validate_integer(@args[1])
      stop = OptionUtils.validate_integer(@args[2])
      list = @db.lookup_list(key)

      if list.nil?
        EmptyArrayInstance
      else
        ListSerializer.new(list, start, stop)
      end
    end

    def self.describe
      Describe.new('lrange', 4, [ 'readonly' ], 1, 1, 1, [ '@read', '@list', '@slow' ])
    end
  end
end

listing 7.29 The LRange Command

We use another argument length validator, in this case we always require three arguments. We then need to validate that the second and third arguments are integers, we do so with a new validator method, validate_integer:

module BYORedis

  ValidationError = Class.new(StandardError) do
    def resp_error
      RESPError.new(message)
    end
  end

  module OptionUtils

    def self.validate_integer(str)
      Integer(str)
    rescue ArgumentError, TypeError
      raise ValidationError, 'ERR value is not an integer or out of range'
    end

    def self.validate_float(str, field_name)
      Float(str)
    rescue ArgumentError, TypeError
      raise ValidationError, "ERR #{ field_name } is not a float or out of range"
    end
  end
end

listing 7.30 The OptionsUtils module

The module also includes a validate_float method, which is very similar to the validate_integer method. We will need the validate_float method later on in this chapter.

Similarly to InvalidArgsLength, we need to handle ValidationError in the execute_command method in BaseCommand:

module BYORedis
  class BaseCommand
    # ...
    def execute_command
      call
    rescue InvalidArgsLength => e
      @logger.debug e.message
      command_name = self.class.describe.name.upcase
      e.resp_error(command_name)
    rescue WrongTypeError, ValidationError => e
      e.resp_error
    end
  end
end

listing 7.31 Updates to the BaseCommand class to support ValidationError exceptions

We delegate the actual serialization logic to a dedicated class, ListSerializer, given that the logic requires the handling of many edge cases. Let's look at the class:

module BYORedis
  # ...

  class ListSerializer

    attr_reader :start, :stop, :list
    attr_writer :start, :stop

    def initialize(list, start, stop)
      @list = list
      @start = start
      @stop = stop
    end

    def serialize
      @stop = @list.size + @stop if @stop < 0
      @start = @list.size + @start if @start < 0

      @stop = @list.size - 1 if @stop >= @list.size
      @start = 0 if @start < 0

      return EmptyArrayInstance.serialize if @start > @stop

      response = ''
      size = 0
      distance_to_head = @start
      distance_to_tail = @list.size - @stop

      if distance_to_head <= distance_to_tail
        iterator = List.left_to_right_iterator(@list)
        within_bounds = ->(index) { index >= @start }
        stop_condition = ->(index) { index > @stop }
        accumulator = ->(value) { response << RESPBulkString.new(value).serialize }
      else
        iterator = List.right_to_left_iterator(@list)
        within_bounds = ->(index) { index <= @stop }
        stop_condition = ->(index) { index < @start }
        accumulator = ->(value) { response.prepend(RESPBulkString.new(value).serialize) }
      end

      until stop_condition.call(iterator.index)
        if within_bounds.call(iterator.index)
          accumulator.call(iterator.cursor.value)
          size += 1
        end

        iterator.next
      end

      response.prepend("*#{ size }\r\n")
    end
  end
end

listing 7.32 ListSerializer in list.rb

The class implements an interface similar to the classes in resp_types.rb, which allows us to return it from the call method, and let the Server class call .serialize on the value returned by execute_command.

The serialize method is fairly long, so let's look at it one line at a time:

The first two lines take care of negative indices. If either start or stop are negative, we convert them to a 0-based index by adding the negative value to the size of list. Let's illustrate this with an example.

list = [ 'a', 'b', 'c' ] # list.size == 3
stop = -1
stop = 3 + stop # stop == 2
list[stop] == 'c' # stop is the last index of the array
start = -2
start = 3 + start # start == 1
list[start] == 'b' # start is the second to last index of the array

The next two lines take care of out of bound indices. If stop is greater than the last index of the list, size - 1, then we set it to that value. There's no need to keep iterating once we reached the last element. Similarly, if stop is lower than 0, we set it to 0, there is no element before the one at index 0, so there's no need to look there.

Once the indices have been sanitized, we can return early if start > stop, as we've shown above, this is nonsensical and we return an empty array right away.

The next step is an optimization to speed up the iteration process. Depending on the values of start & stop, it might be more interesting to start iterating from the head of the list, or from the tail. Imagine a list with one million elements it would be great if LRANGE -1 -1, which returns the tail of the list, would run in O(1) time, and not in O(n) time. Our List class holds a reference to the tail, so it is feasible to return it without iterating through the list.

We achieve this with the distance_to_head & distance_to_tail variables. If start is 0, there is no need for "empty iterations" to reach the first element we need to return, but if start were, say, 100, we would need to iterate 100 times to reach the first element we need to return.

The same goes for distance_to_tail, if list.size - stop is equal to 0, the last element of the list needs to be returned, so there's no need for empty iterations, but if stop were, say, 2, in a list of 100 elements, we'd need 98 empty iterations from the right to reach the last element that needs to be returned.

If distance_to_head is smaller than distance_to_tail, it'll be faster to reach the sublist that needs to be returned if we start from the head, and if distance_to_tail is smaller, then it'll be faster to start from the tail.

We could have written two while/until loops in each branch of the if/else condition, but instead we chose to define a few procs that will determine the direction of the iteration, so that we can write a single loop below.

Because this is a common pattern across list commands, we define an Iterator struct as well as two helpers to return the two common iterators:

module BYORedis
  class List
    # ...
    Iterator = Struct.new(:cursor, :index, :cursor_iterator, :index_iterator) do
      def next
        self.cursor = cursor_iterator.call(cursor)
        self.index = index_iterator.call(index)
      end
    end
    # ...
    def self.left_to_right_iterator(list)
      # cursor, start_index, iterator, index_iterator
      Iterator.new(list.head, 0, ->(node) { node.next_node }, ->(i) { i + 1 })
    end

    def self.right_to_left_iterator(list)
      # cursor, start_index, iterator, index_iterator
      Iterator.new(list.tail, list.size - 1, ->(node) { node.prev_node }, ->(i) { i - 1 })
    end
    # ...
  end
end

listing 7.33 The new Iterator class in the List class

We define the starting index for the iteration, 0 in the left to right iteration, size - 1 for a left to right iteration.

The cursor attribute of the Iterator instance will be set to each node of the list, its initial value is list.head if we start from the left and list.tail if we start from the right.

The cursor_iterator attribute is a proc that determines how to get the following node in the list. If we're iterating from left to right, we need to get the node at next_node, and if we're iterating from right to left, we need to get the node at prev_node.

Similarly, index_iterator is a proc that updates the current index value, it increments it for a left to right iteration and decrements it for a right to left iteration.

within_bounds is a proc that returns a boolean for an index, which indicates if the current index is within the bounds of the requested range. If we're iterating from left to right, as soon as the current index reaches the value of start, we need to start accumulating the elements we find. The condition is different for a right to left iteration, we need to start accumulating elements as soon as the current index is lower or equal to stop. Let's look at few examples to illustrate this:

list = [ 'a', 'b', 'c', 'd', 'e', 'f', 'g' ] # size is 7, last index is 6

If start is 1 and stop is 2, the returned list should be [ 'b', 'c' ] and we'll start iterating from the head. index will be initialized at 0, on the next iteration it'll be 1, which will cause the within_bounds proc to return true since 1 >= 1. On the next iteration index will be two and within_bounds will return true again. On the next iteration index will be 3, which it outside the bounds of the given range, so we could return at this point.

We achieve this with the stop_condition proc. In this example it'll be index > stop, and since 3 > 2, the until loop will stop.

The accumulation process works with the response string. In a left to right iteration we append new values to this string with << and in a right to left operation we prepend to it with the prepend method.

The last step of the method is to add the size of the RESP array at the beginning of the string, to follow the RESP protocol. We do this with response.prepend("*#{ size }\r\n").

Now that we can add elements to a list, and read elements from a list, it's time to add commands to remove elements.

Removing items with LPOP & RPOP

The main two commands to remove elements from a list are LPOP & RPOP. LPOP pops an element from the Left and RPOP pops an element from the Right. Both commands accept a single arguments, the key for the list. If a pair exists for this key, but is not a list, an error is returned. If no pairs exist, a nil string is returned. Redis does not keep empty lists in memory, as can be shown in the next example, so if the last element is popped form the list, we need to remove the pair from @db.data_store:

127.0.0.1:6379> RPUSH a-list a b
(integer) 2
127.0.0.1:6379> LPOP a-list
"a"
127.0.0.1:6379> LPOP a-list
"b"
127.0.0.1:6379> TYPE a-list
none

Let's create the LPopCommand class, but first, similarly to how we approached the LPUSH & RPUSH commands, we know we're going to need very similar functionality with LPOP & RPOP, so let's define shared methods in ListUtils first:

module BYORedis
  # ...
  module ListUtils
    # ...
    def self.common_pop(db, args)
      Utils.assert_args_length(1, args)
      key = args[0]
      list = db.lookup_list(key)

      if list.nil?
        NullBulkStringInstance
      else
        value = yield key, list
        RESPBulkString.new(value)
      end
    end
  end
  # ...
  class LPopCommand < BaseCommand

    def call
      ListUtils.common_pop(@db, @args) do |key, list|
        @db.left_pop_from(key, list)
      end
    end

    def self.describe
      Describe.new('lpop', 2, [ 'write', 'fast' ], 1, 1, 1, [ '@write', '@list', '@fast' ])
    end
  end
end

listing 7.34 The LPopCommand using a shared method from ListUtils

We could call the not implemented yet left_pop method from the call method directly, but because the process of checking if the list is now empty and removing it from the keyspace is so common, we wrap this logic in the DB#left_pop_from method. Let's create this method:

module BYORedis
  class DB
    # ...
    def left_pop_from(key, list)
      generic_pop_wrapper(key, list) do
        list.left_pop
      end
    end

    private

    def generic_pop_wrapper(key, list)
      popped = yield
      @data_store.delete(key) if list.empty?

      if popped
        popped.value
      else
        @logger.warn("Unexpectedly popped from an empty list or a nil value: #{ key }")
        nil
      end
    end
  end
end

listing 7.35 The proxy method in the DB class to handle deletions of empty lists

left_pop_from starts by calling generic_pop_wrapper with the key & list values as well as with a block calling left_pop on the list.

generic_pop_wrapper allows us to define logic that is agnostic of which pop operations we're performing. It starts by calling yield, which is assuming to pop a value from the list, either from the left or from the right. It then proceeds to delete the list from @data_store if the list is now empty. It is expected that popped will never be nil, since we only keep non-empty lists in @data_store, but as a way to be extra cautious, we log a warning and return nil if popped happened to be nil.

We can now define RPopCommand:

module BYORedis
  # ...
  class RPopCommand < BaseCommand
    def call
      ListUtils.common_pop(@db, @args) do |key, list|
        @db.right_pop_from(key, list)
      end
    end

    def self.describe
      Describe.new('rpop', 2, [ 'write', 'fast' ], 1, 1, 1, [ '@write', '@list', '@fast' ])
    end
  end
end

listing 7.36 The RPopCommand class

Let's also define right_pop_from in DB:

module BYORedis
  class DB
    # ...
    def right_pop_from(key, list)
      generic_pop_wrapper(key, list) do
        list.right_pop
      end
    end
  end
end

listing 7.37 The proxy method in the DB class to handle empty list deletions after a right pop

Removing from a list and adding to another with RPOPLPUSH

A common use case for Redis is to use lists as queues, where elements are added from the left, with LPUSH, and removes from the right, with RPOP, this makes the list a FIFO, First In First Out, queue, where the order of insertion defines the order of processing.

For an extra layer of reliability, it can be convenient to keep the popped element in another list to make sure that even if the process in charge of processing the message fails to complete it, the message won't be lost and will still be in the processing queue.

A use case for RPOPLPUSH

An example of such use case is an e-commerce site with a subscription component. Let's imagine that there is a system in charge of processing subscriptions, each day at 1am it processes the subscriptions for the day, creating orders for customers and charging them for the amount of the order. We could implement this process by calling LPUSH for each subscription that needs to be processed that day and having one or worker processes continuously calling RPOP for that list. Each message would contain enough information to process the order.

This approach of queuing messages for processing by separate workers is fairly common as it allows for easy scaling of the number of workers. It only needs a single process to run on a schedule, find all the subscriptions that need to be processed and adds them to the queue. On the other end of the queue, there could be one or more workers, picking up messages and processing them. There could even be an auto-scaling systems that adds workers based on the size of the queues. If tens of thousand of subscriptions need to be processed, many workers could be started to speed up the process.

Now, imagine that the system in charge of handling the payment portion is having some issues and is causing some errors, we would still want to process these orders later on, when the payment system is back online. This is where the idea of a queue dedicated to keeping the items being processed is useful.

The worker processes would call RPOP, and before processing the message, would call LPUSH to add the message to the processing queue. With this approach, regardless of what happens to the worker, the message is safe in the processing list. There might be another process in charge of scanning the processing list and retrying the message at a later point. If no errors happen, the worker must delete the message from the processing queue, to signify that it is fully processed.

The problem with this approach is that things can still go wrong, networks encounter issues, and there is no guarantee that the LPUSH operation will work, even if the RPOP operation was successful. In other words, with RPOP, the messages leaves the Redis server completely, and even if we try to put it back with RPUSH, this is not guaranteed to succeed.

Another worst-case scenario, which is not that uncommon in my experience, is that something goes wrong on the client side, after receiving the message from RPOP but before sending it back with RPUSH. There are different ways in which something could go wrong, like an unhandled exception for instance. That being said, many workers are setup with some try/catch/begin/rescue wrappers to handle such cases, but the likelihood of a bug is still there.

Additionally, with some platforms such as the JVM, it is also possible to observe exceptions such as OutOfMemoryError. These errors are usually rare, but still likely to happen, at some point, for any large enough project.

This is the problem that RPOPLPUSH solves, in one operation, it pops an element from the right of a list, and pushes it to the left of another. The message never leaves Redis.

The RPopLPushCommand class

If the source key is nil, a nil string is returned, and if it exists but is not a list, an error is returned. Otherwise, it pops an element, again deleting the list from the keyspace if the list ends up empty, and pushes the element to the destination. Destination is created if it does not already exist, and an error is returned if it already exists and is not a list:

module BYORedis
  class RPopLPushCommand < BaseCommand
    def call
      Utils.assert_args_length(2, @args)

      source_key = @args[0]
      source = @db.lookup_list(source_key)

      if source.nil?
        NullBulkStringInstance
      else
        destination_key = @args[1]

        if source_key == destination_key && source.size == 1
          source_tail = source.head.value
        else
          destination = @db.lookup_list_for_write(destination_key)
          source_tail = @db.right_pop_from(source_key, source)
          destination.left_push(source_tail)
        end

        RESPBulkString.new(source_tail)
      end
    end

    def self.describe
      Describe.new('rpoplpush', 3, [ 'write', 'denyoom' ], 1, 2, 1,
                   [ '@write', '@list', '@slow' ])
    end
  end
end

listing 7.38 The RPopLPushCommand class

We do not use any new methods here. We initially call lookup_list for the source list, and return early if it doesn't exist. We then call lookup_list_for_write for the destination, creating the list if it does not already exist.

We then pop from the source, with the right_pop_from method, which deletes the list if it is now empty, and then call left_push to the destination list. The returned value is the element that was popped and pushed.

It's worth nothing that RPOPLPUSH can be used with the same list as source and destination, which ends up moving the tail of the list to the head. This creates an edge case if the list has a single element, if so, we don't have to do anything, and we can simply return the head of the list, or the tail, since they're the same value. We have to do this because otherwise calling @db.right_pop_from(source_key, source) would cause the deletion of the list, which we want to avoid.

A bunch of useful utility commands

Redis defines a few more list commands:

LINDEX - The element at a given index

LINDEX accepts two arguments, a key and the index of the element we want to return. Similarly to LRANGE, LINDEX accept negative indices. If there is an existing pair and it is not a list, an error is returned. If the index is out of bounds, a nil string is returned.

module BYORedis
  # ...
  class LIndexCommand < BaseCommand
    def call
      Utils.assert_args_length(2, @args)

      key = @args[0]
      index = OptionUtils.validate_integer(@args[1])
      list = @db.lookup_list(key)

      if list.nil?
        NullBulkStringInstance
      else
        value_at_index = list.at_index(index)
        if value_at_index
          RESPBulkString.new(value_at_index)
        else
          NullBulkStringInstance
        end
      end
    end

    def self.describe
      Describe.new('lindex', 3, [ 'readonly' ], 1, 1, 1, [ '@read', '@list', '@slow' ])
    end
  end
end

listing 7.39 The LIndex class

Let's add the List#at_index method:

module BYORedis
  class List
    # ...
    def at_index(index)
      index += @size if index < 0
      return if index >= @size || index < 0

      distance_to_head = index
      distance_to_tail = @size - index

      if distance_to_head <= distance_to_tail
        iterator = List.left_to_right_iterator(self)
      else
        iterator = List.right_to_left_iterator(self)
      end

      while iterator.cursor
        return iterator.cursor.value if iterator.index == index

        iterator.next
      end
    end
  end
end

listing 7.40 The List#index method

The at_index method makes use of the Iterator helpers, since we also want to iterate from the right side if the given index is closer to the tail. We also perform the same sanitation step to transform a negative index into a positive 0-based index.

LPOS - The index (or indices) of element(s) matching the argument

The LPOS command supports three options: COUNT, MAXLEN & RANK. COUNT determines the maximum number of elements that can be returned if multiple elements match the argument. By default LPOS returns one or zero index, for the first element being equal to the argument, starting from the left, but if COUNT is given, it returns an array, empty if no elements were found, or containing all the indices of element being equal to the argument.

RANK can be used to skip some matches and return the n-th one, from the left with a positive rank and from the right with a negative rank. A rank value cannot be zero or negative, and a rank value of 1 is the default, return the first match starting from the head. A rank value of -1 will return the last match. A rank value of 2 will return the second match, from the left, and -2, the second to last element, or second element from the right, and so on.

The last option, MAXLEN, can be used to limit the number of elements that will be scanned. By default LPOS will scan up to the whole list, but with MAXLEN n, it will stop after n elements.

The options can be combined, for instance LPOS a-list element RANK -1 MAXLEN 10 will only look at the last ten elements of the list when trying to find the index of element.

module BYORedis

  ZeroRankError = Class.new(StandardError) do
    def message
      'ERR RANK can\'t be zero: use 1 to start from the first match, 2 from the second, ...'
    end
  end
  NegativeOptionError = Class.new(StandardError) do
    def initialize(field_name)
      @field_name = field_name
    end

    def message
      "ERR #{ @field_name } can\'t be negative"
    end
  end

  # ...
  class LPosCommand < BaseCommand

    def initialize(db, args)
      super
      @count = nil
      @maxlen = nil
      @rank = nil
    end

    def call
      Utils.assert_args_length_greater_than(1, @args)

      key = @args.shift
      element = @args.shift
      list = @db.lookup_list(key)

      parse_arguments unless @args.empty?

      if list.nil?
        NullBulkStringInstance
      else
        position = list.position(element, @count, @maxlen, @rank)
        if position.nil?
          NullBulkStringInstance
        elsif position.is_a?(Array)
          RESPArray.new(position)
        else
          RESPInteger.new(position)
        end
      end
    rescue ZeroRankError, NegativeOptionError => e
      RESPError.new(e.message)
    end

    def self.describe
      Describe.new('lpos', -3, [ 'readonly' ], 1, 1, 1, [ '@read', '@list', '@slow' ])
    end

    private

    def parse_arguments
      until @args.empty?
        option_name = @args.shift
        option_value = @args.shift
        raise RESPSyntaxError if option_value.nil?

        case option_name.downcase
        when 'rank'
          rank = OptionUtils.validate_integer(option_value)
          raise ZeroRankError if rank == 0

          @rank = rank
        when 'count'
          count = OptionUtils.validate_integer(option_value)
          raise NegativeOptionError, 'COUNT' if count < 0

          @count = count
        when 'maxlen'
          maxlen = OptionUtils.validate_integer(option_value)
          raise NegativeOptionError, 'MAXLEN' if maxlen < 0

          @maxlen = maxlen
        else
          raise RESPSyntaxError
        end
      end
    end
  end
end

listing 7.41 The LPosCommand class

The LPosCommand class takes care of parsing the list of arguments and storing the values in the @count, @maxlen & @rank instance variables. We perform a few validations to make sure that if a rank is given it is an integer, but not zero and that MAXLEN & COUNT are positive integers.

We add the RESPSyntaxError class to the utils.rb file now that we need to use it from more than one command, for SET & LPOS.

module BYORedis
  # ...
  RESPSyntaxError = Class.new(StandardError) do
    def message
      'ERR syntax error'
    end
  end
  # ...
end

listing 7.42: Adding RESPSyntaxError to utils.rb

We also need to add it to the list of exceptions rescued in BaseCommand.

module BYORedis
  class BaseCommand
    # ...
    def execute_command
      call
    rescue InvalidArgsLength => e
      @logger.debug e.message
      command_name = self.class.describe.name.upcase
      e.resp_error(command_name)
    rescue WrongTypeError, RESPSyntaxError, ValidationError => e
      e.resp_error
    end
  end
end

listing 7.43 Updates to the BaseCommand class to support RESPSyntaxError exceptions

The actual logic is in the List#position method:

module BYORedis
  class List
    # ...
    def position(element, count, maxlen, rank)
      return if count && count < 0
      return if @size == 0
      return if rank && rank == 0

      match_count = 0
      maxlen = @size if maxlen == 0 || maxlen.nil?
      indexes = [] if count

      if rank.nil? || rank >= 0
        iterator = List.left_to_right_iterator(self)
      else
        iterator = List.right_to_left_iterator(self)
      end

      while iterator.cursor
        if (rank.nil? || rank >= 0) && iterator.index >= maxlen
          break
        elsif (rank && rank < 0) && (@size - iterator.index - 1) >= maxlen
          break
        end

        if element == iterator.cursor.value
          match_count += 1

          reached_rank_from_head = rank && rank > 0 && match_count >= rank
          reached_rank_from_tail = rank && rank < 0 && match_count >= (rank * -1)

          if rank.nil? || reached_rank_from_head || reached_rank_from_tail
            return iterator.index if indexes.nil?

            indexes << iterator.index
          end

          return indexes if indexes && indexes.size == count
        end

        iterator.next
      end

      indexes
    end
  end
end

listing 7.44 The List#position method

There's a lot going on in List#position, let's break it down one step at a time:

The first three lines are sanity checks for the three optional arguments, if any of these values are invalid, there's no need to continue.
We initialize a variable to count the number of matches, match_count. We also give maxlen a default value of the size of the list if it wasn't set. If a count value is given then we instantiate an array to store all the indices that need to be returned.

If no rank is given or if rank is positive, we create a left to right iterator, otherwise we create a right to left iterator, we then iterate through the list with the newly created iterator.

If we are in a left to right iteration, checked with the if (rank.nil? || rank >= 0) condition, we can stop iterating once the index of the iterator reached maxlen, we've seen enough elements, there's no need to continue. We perform a similar check in the case of a right to left iteration, checked with if (rank && rank < 0), but this time we know that we've seen enough elements if @size - iterator.index - 1 is greater than or equal to maxlen. Using a list of size 10 as an example, if maxlen is set to 3, and rank is negative, we need to inspect the last three values, at index 9, 8 & 7. So once we reach index 6, 10 - 6 - 1 is equal to 3, which is the value of maxlen and the loop will exit. If neither of these conditions match, we need to keep going through the list.

If the current element does not equal element, we can jump to the next element in the list, and ignore it, otherwise, we found a match, and the steps to take depend on the rank and maxlen arguments.

Regardless of the arguments value, we found a new match, so we increment match_count.

If a rank value was given, we need to check if the current match should be accounted for based on the rank value. In other words, we might have to ignore the match. For instance, if rank is 3, and this is the first match, we need to ignore the match, neither reached_rank_from_head or reached_rank_from_tail will be initialized. On the other hand, if we're dealing with the third match, then reached_rank_from_head will be true. The reached_rank_from_tail variable works the same way but for negative ranks.

If no rank was given, or if either of the two previous variables was set to true, we have found a valid match. If indexes is nil, there's no array to accumulate the values into, no count was given and we can return right away. Otherwise, we add the match to the indexes variable.

Once these checks have been performed for the rank value, we need to check if we can return right away, which is the case if no count was given, or if there is a count and we have found enough matches.

LINSERT - Add a new element before or an after an element in the list

LINSERT inserts a new element before or after a given pivot. If no values in the list match the pivot, it does nothing and return -1.

module BYORedis
  class LInsertCommand < BaseCommand
    def call
      Utils.assert_args_length(4, @args)

      if ![ 'before', 'after' ].include?(@args[1].downcase)
        raise RESPSyntaxError
      else
        position = @args[1].downcase == 'before' ? :before : :after
      end

      pivot = @args[2]
      element = @args[3]
      list = @db.lookup_list(@args[0])

      return RESPInteger.new(0) if list.nil?

      new_size =
        if position == :before
          list.insert_before(pivot, element)
        else
          list.insert_after(pivot, element)
        end
      RESPInteger.new(new_size)
    end

    def self.describe
      Describe.new('linsert', 5, [ 'write', 'denyoom' ], 1, 1, 1,
                   [ '@write', '@list', '@slow' ])
    end
  end
end

listing 7.45 The LInsertCommand

The LInsertCommand class parses the list of arguments to see if the new element needs to be added before or after the pivot and delegates the operation to the List class.

We now need to add the List#insert_before and List#insert_after methods:

module BYORedis
  class List
    # ...
    def insert_before(pivot, element)
      generic_insert(pivot) do |node|
        new_node = ListNode.new(element, node.prev_node, node)
        if @head == node
          @head = new_node
        else
          node.prev_node.next_node = new_node
        end

        node.prev_node = new_node
      end
    end

    def insert_after(pivot, element)
      generic_insert(pivot) do |node|
        new_node = ListNode.new(element, node, node.next_node)
        if @tail == node
          @tail = new_node
        else
          node.next_node.prev_node = new_node
        end

        node.next_node = new_node
      end
    end

    private

    def generic_insert(pivot)
      cursor = @head

      while cursor
        break if cursor.value == pivot

        cursor = cursor.next_node
      end

      if cursor.nil?
        -1
      else
        @size += 1

        yield cursor

        @size
      end
    end
  end
end

listing 7.46 The List#insert_before & List#insert_after methods

The first step is the same for each method, and we share the logic in the generic_insert method. We iterate from the left, until we find the pivot, the difference occurs when we do find the pivot.

In the insert_before case, we create a new node with the new value, with its prev_node value to the node that was before the pivot, node.prev_node and its next_node value to the pivot, node. If the pivot was not the head, it will have a prev_node value, and we need to update the next_node value on that node to now point to the new node. Otherwise, if it was the head, then we need to update the @head reference to be the new node.
The element preceding the pivot is now the new node with node.prev_node = new_node.

The logic in insert_after is quite similar. We start by creating a new node, where prev_node is set to the pivot, node, and its next_node is set to the element that follows the pivot, node.next_node. If the pivot was the tail, then we need to update the @tail reference, otherwise, we need to update the prev_node value of the element following pivot to now point to the new node, node.next_node.prev_node = new_node.
The element following the pivot is now the new node with node.next_node = new_node.

LLEN - Return the length of a list

The LLenCommand class is more straightforward than the previous ones. It takes a single argument, the key for the list, and returns its length, or 0 if the list does not exist:

module BYORedis
  class LLenCommand < BaseCommand

    def call
      Utils.assert_args_length(1, @args)
      key = @args[0]
      list = @db.lookup_list(key)

      if list.nil?
        RESPInteger.new(0)
      else
        RESPInteger.new(list.size)
      end
    end

    def self.describe
      Describe.new('llen', 2, [ 'readonly', 'fast' ], 1, 1, 1, [ '@read', '@list', '@fast' ])
    end
  end
end

listing 7.47 The LLenCommand class

LREM - Remove one more element from a list

The LRemCommand commands accepts three arguments, the key for the list, a count value and the element that we intend to remove. The meaning of the count argument is the following:

count > 0: Remove elements equal to element moving from head to tail.
count < 0: Remove elements equal to element moving from tail to head.
count = 0: Remove all elements equal to element.

module BYORedis
  class LRemCommand < BaseCommand
    def call
      Utils.assert_args_length(3, @args)

      key = @args[0]
      count = OptionUtils.validate_integer(@args[1])
      element = @args[2]
      list = @db.lookup_list(key)

      if list.nil?
        RESPInteger.new(0)
      else
        RESPInteger.new(list.remove(count, element))
      end
    end

    def self.describe
      Describe.new('lrem', 4, [ 'write' ], 1, 1, 1, [ '@write', '@list', '@slow' ])
    end
  end
end

listing 7.48 The LRemCommand class

The LRemCommand classes performs the necessary validations, such as validating that the value for count is an integer, and then delegates the actual removal operation to the List#remove method:

module BYORedis
  class List
    def remove(count, element)
      delete_count = 0
      if count >= 0
        iterator = List.left_to_right_iterator(self)
      else
        iterator = List.right_to_left_iterator(self)
      end

      while iterator.cursor
        cursor = iterator.cursor
        if cursor.value == element
          if @head == cursor
            @head = cursor.next_node
          else
            cursor.prev_node.next_node = cursor.next_node
          end

          if @tail == cursor
            @tail = cursor.prev_node
          else
            cursor.next_node.prev_node = cursor.prev_node
          end

          delete_count += 1
          @size -= 1

          if count != 0 && (delete_count == count || delete_count == (count * -1))
            break
          end
        end

        iterator.next
      end

      delete_count
    end
  end
end

listing 7.49 The List#remove method

The first steps should look familiar at this point. Because of the difference in logic between a negative or positive/zero count, we will need to iterate from the left or the right, and create the necessary iterator.

The main loop does nothing if the current element does not match the given element we want to remove. If there is a match, we always start by deleting the node. The deletion step in a doubly linked list requires a few steps:

If the cursor is the head, then we update the head to be the element following the cursor
Otherwise we update the next_node value of the element preceding the cursor, to point at the element following the cursor
If the cursor is the tail, then we update the list to be the element preceding the cursor
Otherwise we update the prev_node value of the element following the cursor to the element preceding the cursor

Finally, we increment the delete_count variable, and decrement the size of the list. The last step of the loop is to check if we should exit based on the count value. If count is zero, we have to iterate through the whole list to delete all matches, so we do not break early. Otherwise, we stop if we have deleted enough elements.

LSET - Update the value of the element at the given index

The LSET command takes three arguments, the key for the list, the index of the element to be updated, and the new value. ERR index out of range is returned if the index is out of range. If the key does not exist, it returns the ERR no such key error.

Like many of the previous commands, LSET supports negative indices.

module BYORedis
  class LSetCommand < BaseCommand
    def call
      Utils.assert_args_length(3, @args)

      key = @args[0]
      index = OptionUtils.validate_integer(@args[1])
      new_value = @args[2]
      list = @db.lookup_list(key)

      if list.nil?
        RESPError.new('ERR no such key')
      elsif list.set(index, new_value)
        OKSimpleStringInstance
      else
        RESPError.new('ERR index out of range')
      end
    end

    def self.describe
      Describe.new('lset', 4, [ 'write', 'denyoom' ], 1, 1, 1, [ '@write', '@list', '@slow' ])
    end
  end
end

listing 7.50 The LSetCommand class

The LSetCommand class performs the necessary validations, and delegates the update process to the List#set method. If the method returns nil, then LSetCommand#call returns the out of range error back to the Server class.

module BYORedis
  class List
    # ...
    def set(index, new_value)
      # Convert a negative index
      index += @size if index < 0

      return if index < 0 || index >= @size

      distance_from_head = index
      distance_from_tail = @size - index - 1

      if distance_from_head <= distance_from_tail
        iterator = List.left_to_right_iterator(self)
      else
        iterator = List.right_to_left_iterator(self)
      end

      while iterator.index != index
        iterator.next
      end

      iterator.cursor.value = new_value
    end
  end
end

listing 7.51 The List#set method

Like the previous commands supporting negative indices, we first convert the index to a 0-based index and return nil if the index is out of bounds. We then perform the same optimizations we performed earlier to decide if we should initiate the iteration from the head or from the tail.

Finally, we iterate until we reach the desired index and update the value in place.

LTRIM - Keep a subset of the list and discard the rest

The last utility command is LTRIM, it accepts three arguments, the key for the list, a start index and a stop index. It only keeps the elements in the range delimited by the start and stop indices, all other elements are discarded. If the range is empty, then the list is deleted. Like other commands, it supports negative indices.

module BYORedis
  class LTrimCommand < BaseCommand
    def call
      Utils.assert_args_length(3, @args)

      key = @args[0]
      start = OptionUtils.validate_integer(@args[1])
      stop = OptionUtils.validate_integer(@args[2])
      list = @db.lookup_list(key)

      if list
        @db.trim(key, list, start, stop)
      end
      OKSimpleStringInstance
    end

    def self.describe
      Describe.new('ltrim', 4, [ 'write' ], 1, 1, 1, [ '@write', '@list', '@slow' ])
    end
  end
end

listing 7.52 The LTrimCommand class

The LTrimCommand class validates that the start and stop indices are integer, and delegates the trim operation to the DB#trim method. It always returns +OK, even if the list does not exist.

We now need to add the DB#trim & List#trim methods:

module BYORedis
  class DB
    # ...

    def trim(key, list, start, stop)
      list.trim(start, stop)
      @data_store.delete(key) if list.empty?
    end
  end
end

listing 7.53 The DB#trim proxy method to handle deletions of empty lists

The DB#trim method deletes the list from the database if the resulting list is empty.

module BYORedis
  class List
    # ...
    def trim(start, stop)
      current_head = @head

      # Convert negative values
      stop = @size + stop if stop < 0
      stop = @size - 1 if stop >= @size
      start = @size + start if start < 0
      start = 0 if start < 0

      if start >= @size || start > stop
        @size = 0
        @head = nil
        @tail = nil
        return
      end

      return if start == 0 && stop == @size - 1

      distance_to_start = start
      distance_to_stop = @size - stop - 1

      if distance_to_start <= distance_to_stop
        iterator = List.left_to_right_iterator(self)
        target_index = start
      else
        iterator = List.right_to_left_iterator(self)
        target_index = stop
      end

      new_head = nil
      new_tail = nil

      while iterator.index != target_index
        iterator.next
      end

      # We reached the closest element, either start or stop
      # We first update either the head and the nail and then find the fastest way to get to the
      # other boundary
      if target_index == start
        new_head = iterator.cursor
        target_index = stop
        # We reached start, decide if we should keep going right from where we are or start from
        # the tail to reach stop
        if distance_to_stop < stop - iterator.index
          iterator = List.right_to_left_iterator(self)
        end
      else
        new_tail = iterator.cursor
        target_index = start
        # We reached stop, decide if we should keep going left from where we are or start from
        # the head to reach start
        if distance_to_start < iterator.index - start
          iterator = List.left_to_right_iterator(self)
        end
      end

      while iterator.index != target_index
        iterator.next
      end

      # We now reached the other boundary
      if target_index == start
        new_head = iterator.cursor
      else
        new_tail = iterator.cursor
      end

      @head = new_head
      @head.prev_node = nil

      # If start == stop, then there's only element left, and new_tail will not have been set
      # above, so we set here
      if start == stop
        new_tail = new_head
        @size = 1
      else
        # Account for the elements dropped to the right
        @size -= (@size - stop - 1)
        # Account for the elements dropped to the left
        @size -= start
      end

      @tail = new_tail
      @tail.next_node = nil
    end
  end
end

listing 7.54 The List#trim method

We perform the usual steps with negative indices, converting to 0-based indices. We then check if they're out of order and if so, clear the list right away, there's no need to iterate in this case. We also check that if start == 0 && stop == -1, in this case, there is nothing to do, we can keep the list as is.

This method is pretty long and pretty verbose in order to minimize the number of empty iterations required to reach the nodes at indices start & stop. The algorithm can be described as:

First, find the fastest node to get to. It is start if the distance between the head and start is lower than the distance between stop and the tail. It is stop otherwise.
Once the closest node is reached, try to reach the second boundary in the most efficient manner, there are four options:
- if the node we reached is start, we can reach stop in two ways:
- continue with the same iterator if stop is closer to the iterator than it is from the tail
- iterate from the tail with a new iterator otherwise
- if the node we reached is stop, we can reach start in two ways:
- continue with the same iterator if start is closer to the iterator than it is from the head
- iterate from the head with a new iterator otherwise
Once we found each nodes, we update @head & @tail accordingly and adjust the size to accommodate for the dropped nodes.

This algorithm means that calling LTRIM 997 998 on a 1,000 element list will start from the right, reach the stop node in one iteration, and continue with the same iterator to reach the start node.

Likewise, calling LTRIM 1 2 on a 1,000 element list will start from the left, reach the start node in one iteration and continue with the same iterator to reach the stop node.

Finally, calling LTRIM 1 998 on a 1,000 element list will start from the left, reach the start node in one iteration and create a new iterator, to start from the tail to reach the stop node in one iteration.

We've now implemented most of the commands, there are only three more commands, that implement existing logic, but in a blocking manner.

The blocking variants, BLPOP, BRPOP & BRPOPLPUSH

The last three commands we need to implement are almost identical to LPOP, RPOP & RPOPLPUSH, with the difference that they are Blocking.

Let's start with the BLPOP command. It accepts two arguments or more. The first one is the key for a list, and the last one must be a number, integer or float, which is the timeout the command can block for. It accepts more than one list keys. The following are valid BLPOP commands: BLPOP a 1, BLPOP a b c 1.2. But the following is invalid BLPOP a b, b is not a valid number and cannot be used to describe a timeout. It's worth noting that since keys can be numbers, which Redis represents at Strings internally, the following is valid BLPOP 1 1. It means: "Block for up to 1s for the key '1'".

A timeout value of 0 means an infinite timeout. The server will only unblock the client if a new element is added to one of the lists it is blocked on.

Redis looks at all the keys from left to right, and as soon as it finds a list, it will pop an element from the left and return it, alongside the key, so that the client knows which list the element was popped from:

127.0.0.1:6379> LPUSH b b1
(integer) 1
127.0.0.1:6379> BLPOP a b 1
1) "b"
2) "b1"

In the previous example, if the list at key a had contained at least one element, it would have been returned instead.

The blocking behavior happens when all the lists are empty, in this case Redis will not send a response to the client, effectively blocking it. Any other commands received while blocked will be accumulated and replied to once the blocking operation is unblocked.

There are three conditions that can unblock a client:

One of the keys the client is blocked on receives a push before the timeout expires
The timeout expires before any of the list receives a push
The client disconnects before any of the two previous conditions are met

In the first case, the return value is the same as if the command returned right away without blocking, it returns the key of the list and the element that was popped.

In the second case, it returns a nil array.

In the last example, the client disconnects, so whether one of the list receives a push, or the timeout expires, nothing will happen, the server needs to clear its internal state to make sure it doesn't actually pop any elements since there's no clients to send the data to. Let's look at an example:

Let's block for 100 seconds in one session and close it with Ctrl-C right after starting it:

127.0.0.1:6379> BLPOP a 100
^C

Reopen another redis-cli shell and push a value to the list a:

127.0.0.1:6379> RPUSH a a1
(integer) 1
127.0.0.1:6379> LRANGE a 0 -1
1) "a1"

If all went well, we did perform all these operations under 100s, but the element we pushed to the list a was not popped, because when the client disconnected, the server knew that there was no client blocked anymore.

If we'd perform the same operations with two sessions, without closing the first one, this is what would happen:

127.0.0.1:6379> BLPOP a 100

In a second session:

127.0.0.1:6379> RPUSH a a1
(integer) 1

And back in the first session, we see the result of BLPOP alongside the amount of time it was blocked for thanks to redis-cli:

127.0.0.1:6379> BLPOP a 100
1) "a"
2) "a1"
(19.50s)

If we let the timeout expire, we can see that the server returns a nil array:

127.0.0.1:6379> BLPOP a 1
(nil)
(1.05s)

Note that redis-cli displays nil strings and nil arrays the same way, with (nil), we can see which one it is with nc:

> nc -c localhost 6379
BLPOP a 1
*-1

Blocking operations use cases

Before implementing the command, it's worth stopping to discuss the benefits of the blocking variants.

There are two main benefits to the blocking variants of the LPOP, RPOP & RPOPLPUSH commands:

It reduces the latency between the moment when an element is pushed to a list and when it is read by a client
It reduces the number of potential "empty"/"useless" commands between clients and the server

Latency improvement

The latency improvement aspect is interesting if there is a benefit to elements added to the list being popped and sent to a client as fast as possible. A common example of such system is a web application processing jobs in the background.

Let's use a password reset flow as an example. It is not unusual for web applications to offer a password reset flow, so that users can reset their passwords if they forgot it. The idea is that a user provides their email address and they'll receive an email providing information about the steps to take to reset their password.

While it could be considered over-engineered, it's not unusual for web applications to use a background worker process system to implement such flow. The motivation behind this approach is that web applications often rely on third party providers to actually send emails, such as SendGrid, MailChimp or Braze. While often reliable, it is always possible that the APIs provided by these services occasionally fail, but once the email address has been read, the requests can be retried until it finally succeeds, this is where a background worker can be useful. The web server receives the email from the client, queues the request in Redis with an LPUSH command, and returns a successful response to the client. The user will see a successful response even though the email might not have yet been actually sent. It is then up to a background worker to fetch the message from the list, with RPOP and attempt to actually send the email, retrying if it fails.

In this scenario, we'd like the worker to pick up the message as soon as the message is pushed, but without a blocking variant, the only option we have is to use a polling mechanism. The worker issues sends RPOP commands, if something is returned, it processes it, otherwise, it optionally waits and issues another RPOP command. Rinse & repeat.

The problem with this approach is that if we program our workers to wait after receiving an empty response, to prevent spamming the server with unnecessary work, we could end up with a delay between when a message is pushed and when it is popped. Let's imagine that workers wait thirty seconds after each empty pop, if we're unlucky, a worker might send a pop command right before the message is pushed and only process it thirty seconds later.

While the impact in this example might be minimal, a thirty second delay for an email could be considered negligible, the fastest we process the email, the more likely we are to prevent confusion for the user. Any delays might cause confusion on their end, wondering if the process worked as expected.

Empty responses reduction

One approach that could improve the problem mentioned in the previous section would be to reduce the wait time between each poll. We started with thirty seconds in the previous example, and could decide to reduce is to one hundred milliseconds.

With such a short interval, we would we be guarantee to process the email within 100 milliseconds of the user requesting it. The problem is that the worker is now sending up to 10 requests per second, for a system that might see less than one message per minute.

This approach would create a lot of empty responses and effectively send a lot of useless commands. While Redis is optimized to process responses fast, no commands to process will always be faster that some.

The blocking variants avoid these issues by blocking the clients until an element is available, or until the timeout expires. With this approach, and with a long enough timeout, the number of empty responses can be reduced to almost zero. A worker would send the RPOP email-reset-queue 60 command, to either get directly a message from the email-reset-queue or wait up to 60s until a message is added to the queue. An empty response would only happen if no customers request a password reset within 60s.

The blocking commands also solve the latency issue since there's now no delay between the moment when the message is pushed to the queue and when it is received by a worker.

Blocking commands caveats

There are some caveats that should be considered when using blocking commands. Redis will accumulate any further commands sent while the client that initiated the command is blocked. This means that if the application that communicates with Redis needs to perform other related operations, it would need to create a new connection.

This can be a problem for applications using connection pooling. The idea behind connection pooling is that an application will create a bunch of connections, the pool, and keep them available for future use. This removes the need for opening and closing connections, which is unnecessary since once a response has been sent, the same connection can be reused to send a different command.

This approach is particularly useful for web application where one or more processes or threads might be running to serve all the incoming HTTP requests. Let's use a multi-threaded web server as an example, running 100 threads. Let's imagine that about a tenth of the incoming requests require a connection to Redis.

A naive approach would be give each thread a connection, but this would be wasteful, given that most connections would be idle most of the time based on the 10% example we defined above. Instead, we could create a smaller number of connections, in a pool, and make each thread take a connection from the pool, use it and return it to the pool. Concurrency access concerns are really important here as only one thread should use a connection to prevent unexpected things to happen, such as two threads writing their own command to the connection, it would be impossible to read the responses and assign them to each thread.

Now, while connection pools are great to avoid creating too many connections and reuse existing connections to prevent unnecessary work with closing and creating connections, there might be issues if the pool has a fixed size and cannot grow. Different connection pool libraries might provide different features, but it is very common to have a maximum size of connections. The connection_pool gem is a common library in Ruby, and HikariCP is a very popular one in Java, both use fixed size pools.

In our example, if the pool was created with ten connections, and nine threads send a BLPOP command with a long timeout, only one connection is left available for the other ninety threads, which would cause delay as each of these threads would need to wait for the previous command to complete.

The BLPopCommand class

Implementing the blocking behavior will require some changes to the Server class, but let's start by adding the BLPopCommand class in list_commands.rb:

module BYORedis
  module ListUtils
    # ...
    def self.timeout_timestamp_or_nil(timeout)
      if timeout == 0
        nil
      else
        Time.now + timeout
      end
    end

    def self.common_bpop(db, args, operation)
      Utils.assert_args_length_greater_than(1, args)

      timeout = OptionUtils.validate_float(args.pop, 'timeout')
      list_names = args

      list_names.each do |list_name|
        list = db.lookup_list(list_name)

        next if list.nil?

        popped = yield list_name, list
        return RESPArray.new([ list_name, popped ])
      end

      BYORedis::Server::BlockedState.new(ListUtils.timeout_timestamp_or_nil(timeout),
                                         list_names, operation)
    end
  end
  # ...
  class BLPopCommand < BaseCommand
    def call
      ListUtils.common_bpop(@db, @args, :lpop) do |list_name, list|
        @db.left_pop_from(list_name, list)
      end
    end

    def self.describe
      Describe.new('blpop', -3, [ 'write', 'noscript' ], 1, -2, 1,
                   [ '@write', '@list', '@slow', '@blocking' ])
    end
  end
end

listing 7.55 The BLPopCommand using a shared method from ListUtils

Most of the code in the common_bpop method uses existing methods, the only difference is if none of the lists in list_names exist, we return an instance of BlockedState. This is a new struct defined in the Server class:

module BYORedis
  class Server
    # ...
    BlockedState = Struct.new(:timeout, :keys, :operation, :target, :client)
    # ...
  end
end

listing 7.56 The BlockedState Struct

The BlockedState struct allows us to store information about blocked clients in the Server class. This is necessary to handle both timeouts and unblocking clients when lists they are blocked on are pushed to and a response should be sent to the blocked clients. We also need to remember which type of operation was initially sent, so that we use left_pop or right_pop when it is time to unblock the client.

The process_poll_events method is starting to get big, so we're extracting logic from it when processing client buffers.

module BYORedis
  class Server
    # ...
    def process_poll_events(sockets)
      sockets.each do |socket|
        if socket.is_a?(TCPServer)
          socket = safe_accept_client
          next unless socket

          @clients[socket.fileno.to_s] = Client.new(socket)
        elsif socket.is_a?(TCPSocket)
          client = @clients[socket.fileno.to_s]
            # ...
          if client_command_with_args.nil?
            # ...
          else
            client.buffer += client_command_with_args

            process_client_buffer(client)
          end
        else
          raise "Unknown socket type: #{ socket }"
        end
      end
    end

    def process_client_buffer(client)

      split_commands(client.buffer) do |command_parts|
        return if client.blocked_state

        response = handle_client_command(command_parts)
        if response.is_a?(BlockedState)
          block_client(client, response)
        else
          @logger.debug "Response: #{ response.class } / #{ response.inspect }"
          serialized_response = response.serialize
          @logger.debug "Writing: '#{ serialized_response.inspect }'"
          unless Utils.safe_write(client.socket, serialized_response)
            disconnect_client(client)
          end

          handle_clients_blocked_on_keys
        end
      end
    rescue IncompleteCommand
      # Not clearing the buffer or anything
    rescue ProtocolError => e
      client.socket.write e.serialize
      disconnect_client(client)
    end

    def block_client(client, blocked_state)
      if client.blocked_state
        @logger.warn "Client was already blocked: #{ blocked_state }"
        return
      end

      blocked_state.client = client

      # Add the state to the client
      client.blocked_state = blocked_state
      if blocked_state.timeout
        @db.client_timeouts << blocked_state
      end

      # Add this client to the list of clients waiting on this key
      blocked_state.keys.each do |key|
        client_list = @db.blocking_keys[key]
        if client_list.nil?
          client_list = List.new
          @db.blocking_keys[key] = client_list
        end
        client_list.right_push(client)
      end
    end
    # ...
  end
end

listing 7.57 The blocking related method in the Server class

The blocked_state instance is only added to client_timeouts if there is a timeout. If the client specified a value of 0, then the server will never unblock the client after a timeout and there is therefore no need to keep track of it here.

We added a new method to the Utils module, to wrap the logic around writing to a socket, while handling potential exceptions if the socket was closed and the write operation fails:

module BYORedis
  module Utils
    # ...
    def self.safe_write(socket, message)
      socket.write(message)
      true
    rescue Errno::ECONNRESET, Errno::EPIPE
      false
    end
  end
end

listing 7.58 The Utils.safe_write helper method

The block_client method is making use of a few new elements. First it stores the BlockedState instance in the client, so we need to update the Client struct to add a new attribute:

module BYORedis
  class Server
    # ...
    Client = Struct.new(:socket, :buffer, :blocked_state) do
      attr_reader :id

      def initialize(socket)
        @id = socket.fileno.to_s
        self.socket = socket
        self.buffer = ''
      end
    end
    # ...
  end
end

listing 7.59 Updates to the Client Struct to support

We also need to add a few more data structures to the DB class, namely: ready_keys, :blocking_keys, client_timeouts & unblocked_clients.

The first two, ready_keys and blocking_keys, are Dict instances. With ready_keys we use a Dict even though we only care about keys, and will use nil for all values. The purpose of this dictionary is essentially to be a set, a collection that does not store duplicates. Whenever a list receives a push, we will add the list's key to ready_keys if it was blocked on. There is no need to store this information more than once, so using a Dict as a set works perfectly. We will be able to know if it was blocked by inspecting blocking_keys. blocking_keys is also a dictionary where keys will be keys of lists that are being blocked on, and the values will be lists of clients blocked. By using a list as the value and appending to the list, we can maintain an order so that the first client to block for a key will be the first client to receive a response when elements will be pushed to the list.

client_timeouts is necessary to handle clients that should receive an empty response if the timeout expired before elements were pushed to the list. Without an extra data structure to store timeouts, we would have to iterate over all the connected clients, and manually inspect their blocked_state attribute to check if it expired or not. This would be extremely inefficient, especially given that we need to perform this check on each iteration of the event loop. This operation would have an O(n) complexity, where n is the number of connected clients. If no clients are blocked, even if thousands of clients are connected, we want to be able to know that there's no need to check for expired clients.

This is the problem that the client_timeouts attribute on DB solves. It is a sorted array, which we provide an implementation for in Appendix B

Whenever a blocked command is received and there is no element to return right away, a BlockedState instance will be pushed to client_timeouts unless the timeout was 0. Using a regular array to store these BlockedState instances would already be an improvement compared to the scenario described above. We would only check if clients are expired within the set of blocked clients, but we would still have to check all the clients, one by one, to see if they are expired or not. This is also a O(n) operation, where n is the number of blocked clients.

Using a sorted array turns the operation into an O(1) operation, we can inspect the first element in the sorted array, if it is not expired, there's no need to inspect any other clients, we know their expiration is later than the first one. If the first client in the sorted array is expired, we need to handle it as such and look at the next element, and stop as soon as we find a non expired client.

Using a sorted array also helps with the cleanup tasks required once a client was unblocked after the list it was blocked on received a push. When this happens, we need to remove the client from the client_timeouts list since it is not blocked anymore. The sorted array class guarantees that this operation happens in O(logn) time, where n is the number of blocked clients, instead of O(n) if the array was not sorted.

This implementation is a deviation from the Redis one. Redis uses a Radix Tree to store client timeouts. A Radix tree provides similar time complexity for the iteration of clients from the ones with the earliest expirations, but it provides a better time complexity for the deletion use case.

We decided to not implement a Radix tree here given the complexity of such implementation. A sorted array provides a good middle ground approach, by being fairly short to implement and still being a great improvement compared to a regular array.

Finally, unblocked_clients is a List instance, which will be appended to when clients are unblocked, either because their timeouts expired or if the list they were blocked on received a push. Regardless of the reason, we need to check if the client sent commands while it was blocked, and process them.

module BYORedis
  class DB

    attr_reader :data_store, :expires, :ready_keys, :blocking_keys, :client_timeouts,
                :unblocked_clients
    attr_writer :ready_keys

    def initialize
      @logger = Logger.new(STDOUT)
      @logger.level = LOG_LEVEL
      @data_store = Dict.new
      @expires = Dict.new
      @ready_keys = Dict.new
      @blocking_keys = Dict.new
      @client_timeouts = SortedArray.new(:timeout)
      @unblocked_clients = List.new
    end
    # ...
  end
end

listing 7.60 New data structures in the DB class

At this point, once we're done processing a BLPOP command, we stored the BlockedState information in client_timeouts and in the Client instance. We also added a new entry if necessary in blocking_keys, where the value is a list containing the client that issued the BLPOP command.

We're now ready to handle the two possible outcomes for a blocked client. Either its timeout expires, in which case we return a nil array, and process any commands sent while blocked. If one of the lists the client was blocked on receives a push before the timeout expires, we return the head of the list to the client, alongside the list key and also process any commands sent while blocked.

Handling timeouts

Let's first add two new steps to the event loop, handle_blocked_clients_timeout and process_unblocked_clients:

module BYORedis
  class Server
    # ...
    def start_event_loop
      loop do
        handle_blocked_clients_timeout
        process_unblocked_clients

        timeout = select_timeout
        @logger.debug "select with a timeout of #{ timeout }"
        result = IO.select(client_sockets + [ @server ], [], [], timeout)
        sockets = result ? result[0] : []
        process_poll_events(sockets)
        process_time_events
      end
    end
    # ...
  end
end

listing 7.61 Updates to the event loop in the Server class to handle blocked client timeouts

We now start each iteration of the event loop by first checking if any of the blocked clients are expired, and then processing any commands sent while blocked. As mentioned in the previous section, the time complexity of these two methods is important since they would otherwise add a delay at the beginning of each event loop iteration.

module BYORedis
  class Server
    # ...

    def handle_blocked_clients_timeout
      @db.client_timeouts.delete_if do |blocked_state|
        client = blocked_state.client
        if client.blocked_state.nil?
          @logger.warn "Unexpectedly found a non blocked client in timeouts: #{ client }"
          true
        elsif client.blocked_state.timeout < Time.now
          @logger.debug "Expired timeout: #{ client }"
          unblock_client(client)

          unless Utils.safe_write(client.socket, NullArrayInstance.serialize)
            @logger.warn "Error writing back to #{ client }: #{ e.message }"
            disconnect_client(client)
          end

          true
        else
          # Impossible to find more later on since client_timeouts is sorted
          break
        end
      end
    end

    def unblock_client(client)
      if client.socket.closed?
        @logger.warn 'RETURNING EARLY'
        return
      end
      @db.unblocked_clients.right_push client

      return if client.blocked_state.nil?

      # Remove this client from the blocking_keys lists
      client.blocked_state.keys.each do |key2|
        list = @db.blocking_keys[key2]
        if list
          list.remove(1, client)
          @db.blocking_keys.delete(key2) if list.empty?
        end
      end
      client.blocked_state = nil
    end

    def process_unblocked_clients
      return if @db.unblocked_clients.empty?

      cursor = @db.unblocked_clients.left_pop

      while cursor
        client = cursor.value

        if @clients.include?(client.id)
          process_client_buffer(client)
        else
          @logger.warn "Unblocked client #{ client } must have disconnected"
        end

        cursor = @db.unblocked_clients.left_pop
      end
    end
  end
end

listing 7.62 Updates to the Server class to process timeouts for blocked clients

We need to add one more piece of logic to the server with regards to timeouts. If a client disconnects before its timeout expired, we want to clean things of up so that we don't keep track of this client anymore. We need to more that deleting the client from the @clients array in process_poll_events.

module BYORedis
  class Server
    # ...
    def safe_accept_client
      @server.accept
    rescue Errno::ECONNRESET, Errno::EPIPE => e
      @logger.warn "Error when accepting client: #{ e }"
      nil
    end

    def safe_read(client)
      client.socket.read_nonblock(1024, exception: false)
    rescue Errno::ECONNRESET, Errno::EPIPE
      disconnect_client(client)
    end

    def process_poll_events(sockets)
      sockets.each do |socket|
        if socket.is_a?(TCPServer)
          socket = safe_accept_client
          next unless socket

          @clients[socket.fileno.to_s] = Client.new(socket)
        elsif socket.is_a?(TCPSocket)
          client = @clients.find { |client| client.socket == socket }
          client_command_with_args = socket.read_nonblock(1024, exception: false)

          client = @clients[socket.fileno.to_s]
          client_command_with_args = safe_read(client)

          if client_command_with_args.nil?
            disconnect_client(client)
          elsif client_command_with_args == :wait_readable
            # There's nothing to read from the client, we don't have to do anything
            next
          elsif client_command_with_args.empty?
            @logger.debug "Empty request received from #{ socket }"
          else
            # ...
          end
        end
      end
    end

    def disconnect_client(client)
      @clients.delete(client.id)
      @db.unblocked_clients.remove(1, client)

      if client.blocked_state
        @db.client_timeouts.delete(client.blocked_state)
        client.blocked_state.keys.each do |key|
          list = @db.blocking_keys[key]
          if list
            list.remove(1, client)
            @db.blocking_keys.delete(key) if list.empty?
          end
        end
      end

      client.socket.close
    end
    # ...
  end
end

listing 7.63 Updates to the Server class to handle clients disconnection

Handling blocked clients before timeout

We need another handler, for when a list that is being blocked on, for instance either a or b after receiving BLPOP a b 1, receives a push and is created. New lists are always created from DB#lookup_list_for_write, so let's add a condition there, that will notify that one key that is being blocked on can now be used to unblock one or more clients:

module BYORedis
  class DB
    def lookup_list_for_write(key)
      list = lookup_list(key)
      if list.nil?
        list = List.new
        @data_store[key] = list

        if @blocking_keys[key]
          @ready_keys[key] = nil
        end
      end

      list
    end
  end
end

listing 7.64 Updates to the DB#lookup_list_for_write to notify for list creation

blocking_keys is populated when we process the result of a blocking command. It is a dictionary that contains a list of clients blocked for that key. When a new list is created, we inspect blocking_keys, if there's no entry, then no clients are blocked on this key, on the other hand, if there is one or more clients, then we know that the key is being blocked on and we add it to @ready_keys. The value of the pair does not matter here, we only care about having an entry in the dictionary.

module BYORedis
  class Server
    # ...
    def initialize
      @logger = Logger.new(STDOUT)
      @logger.level = LOG_LEVEL

      @clients = Dict.new
      @db = DB.new
      @blocked_client_handler = BlockedClientHandler.new(self, @db)
      @server = TCPServer.new 2000
      @time_events = []
      @logger.debug "Server started at: #{ Time.now }"
      add_time_event(Time.now.to_f.truncate + 1) do
        server_cron
      end

      start_event_loop
    end
    # ...
    def handle_clients_blocked_on_keys
      return if @db.ready_keys.used == 0

      @db.ready_keys.each do |key, _|
        unblocked_clients = @blocked_client_handler.handle(key)

        unblocked_clients.each do |client|
          unblock_client(client)
        end
      end

      @db.ready_keys = Dict.new
    end
  end
end

listing 7.65 Blocked client handling in the Server class

handle_clients_blocked_on_keys is called from process_poll_events, after processing the commands from the clients. If the command did not result in the creation of a list, then nothing will happen, no clients can be unblocked. If the command did create a list, then @db.ready_keys will have received elements and @blocked_client_handler.handle will be called for each of these keys.

The code in charge of handling blocked client lives in its own class, BlockedClientHandler.

module BYORedis
  class BlockedClientHandler
    def initialize(server, db)
      @server = server
      @db = db
      @logger = Logger.new(STDOUT)
      @logger.level = LOG_LEVEL
    end

    def handle(key)
      clients = @db.blocking_keys[key]
      unblocked_clients = []

      list = @db.data_store[key]

      if !list || !list.is_a?(List)
        @logger.warn "Something weird happened, not a list: #{ key } / #{ list }"
        raise "Unexpectedly found nothing or not a list: #{ key } / #{ list }"
      end

      raise "Unexpected empty list for #{ key }" if list.empty?

      cursor = clients.left_pop

      while cursor
        client = cursor.value

        if handle_client(client, key, list)
          unblocked_clients << client
        end

        if list.empty?
          break
        else
          cursor = clients.left_pop
        end
      end

      @db.blocking_keys.delete(key) if clients.empty?

      unblocked_clients
    end

    def handle_client(client, key, list)
      blocked_state = client.blocked_state

      # The client is expected to be blocked on a set of keys, we unblock it based on the key
      # arg, which itself comes from @db.ready_keys, which is populated when a key that is
      # blocked on receives a push
      # So we pop (left or right) from the list at key, and send the response to the client
      if client.blocked_state

        response = pop_operation(key, list, blocked_state.operation, blocked_state.target)

        serialized_response = response.serialize
        @logger.debug "Writing '#{ serialized_response.inspect } to #{ client }"

        unless Utils.safe_write(client.socket, serialized_response)
          # If we failed to write the value back, we put the element back in the list
          rollback_operation(key, response, blocked_state.operation, blocked_state.target)
          @server.disconnect_client(client)
          return
        end
      else
        @logger.warn "Client was not blocked, weird!: #{ client }"
        return
      end

      true
    end

    private

    def pop_operation(key, list, operation, target)
      case operation
      when :lpop
        RESPArray.new([ key, @db.left_pop_from(key, list) ])
      when :rpop
        RESPArray.new([ key, @db.right_pop_from(key, list) ])
      when :rpoplpush
        raise "Expected a target value for a brpoplpush handling: #{ key }" if target.nil?

        ListUtils.common_rpoplpush(@db, key, target, list)
      else
        raise "Unknown pop operation #{ operation }"
      end
    end

    def rollback_operation(key, response, operation, target_key)
      list = @db.lookup_list_for_write(key)
      case operation
      when :lpop
        element = response.underlying_array[1]
        list.left_push(element)
      when :rpop
        element = response.underlying_array[1]
        list.right_push(element)
      when :rpoplpush
        target_list = @db.lookup_list(target_key)
        element = target_list.left_pop
        @db.data_store.delete(target_key) if target_list.empty?
        list.right_push(element.value)
      else
        raise "Unknown pop operation #{ operation }"
      end
    end
  end
end

listing 7.66 The BlockedClientHandler class

This is the final step required to handle blocked clients. We are processing all the keys that can be used to unblock clients. A command can push more than one element, so we might be able to unblock more than one client. We start this process by getting the list of blocked clients for this key, in @db.blocked_clients. Clients are added with right push operations, so we use left_pop here to get them in order of insertions. The first client to block on this list will be processed first. We also get the recently created list that will pop elements from to unblock clients.

For each client, we call handle_client. In handle_client we pop an element from the list, according to the operation that the client blocked with, either left_pop or right_pop, and we send the response back to the client. In case of a failure, we return the element back to the list and disconnect the client.

Back in BlockedClientHandler#handler, we accumulate a list of clients that were unblocked, which will in turn be returned to the Server class. This list is used to call unblock_client, which will flag the client to be processed in case there were accumulated commands sent while blocked.
We then check if the list is empty, if it is, we cannot unblock anymore clients and exit the iteration. Otherwise we continue and unblock the next client. unblock_client also takes care of cleaning up the server state. Now that a client is unblocked, it should not be in any of the lists in @db.blocking_keys. We also need remove the entry in the sorted array of timeouts.

If we were able to unblock all the clients, we remove the key from blocking_keys.

With all these changes, we can implement the similar BRPopCommand class:

module BYORedis
  # ...
  class BRPopCommand < BaseCommand
    def call
      ListUtils.common_bpop(@db, @args, :rpop) do |list_name, list|
        @db.right_pop_from(list_name, list)
      end
    end

    def self.describe
      Describe.new('brpop', -3, [ 'write', 'noscript' ], 1, -2, 1,
                   [ '@write', '@list', '@slow', '@blocking' ])
    end
  end
end

The implementation is almost identical to BLPopCommand. The last class to add is BRPopLPushCommand:

module BYORedis
  module ListUtils
    # ...
    def self.common_rpoplpush(db, source_key, destination_key, source)
      if source_key == destination_key && source.size == 1
        source_tail = source.head.value
      else
        destination = db.lookup_list_for_write(destination_key)
        source_tail = db.right_pop_from(source_key, source)
        destination.left_push(source_tail)
      end

      RESPBulkString.new(source_tail)
    end
  end
  # ...
  class RPopLPushCommand < BaseCommand
    def call
      Utils.assert_args_length(2, @args)

      source_key = @args[0]
      source = @db.lookup_list(source_key)

      if source.nil?
        NullBulkStringInstance
      else
        destination_key = @args[1]
        ListUtils.common_rpoplpush(@db, source_key, destination_key, source)
      end
    end
    # ...
    end
  end
  # ...
  class BRPopLPushCommand < BaseCommand
    def call
      Utils.assert_args_length(3, @args)

      source_key = @args[0]
      source = @db.lookup_list(source_key)
      timeout = OptionUtils.validate_float(@args[2], 'timeout')
      destination_key = @args[1]

      if source.nil?
        BYORedis::Server::BlockedState.new(ListUtils.timeout_timestamp_or_nil(timeout),
                                           [ source_key ], :rpoplpush, destination_key)
      else
        ListUtils.common_rpoplpush(@db, source_key, destination_key, source)
      end
    end

    def self.describe
      Describe.new('brpoplpush', 4, [ 'write', 'denyoom', 'noscript' ], 1, 2, 1,
                   [ '@write', '@list', '@slow', '@blocking' ])
    end
  end
end

The code here is almost identical to the one in RPopLPushCommand, with the difference that it returns BlockedState instance if the source list does not exist. One difference with the previous two commands is that we need to store the target of the pop, the list in which we'll push the element to.

We also extracted some shared logic between RPopLPushCommand & BRPopLPushCommand to ListUtils.common_rpoplpush to prevent code repetition. This shared code is important as it handles the edge case where both source & destination are the same list and only contains one element.

Tests

We added a few more test files in this chapter, and placed all of them in the test/ sub folder, we also added and a task in Rakefile to run all the tests:

require 'rake/testtask'

Rake::TestTask.new do |t|
  t.pattern = 'test/*test.rb'
end

listing 7.67 Rakefile content

The new test files allow us to easily run a sub section of the test suite, the following is a list of the new files:

list_test.rb
dict_unit_test.rb
command_test.rb
sorted_array_unit_test.rb
list_unit_test.rb

We can run all the tests with rake test, or single files with ruby list_test.rb for instance.

Conclusion

In this chapter we added full list support to our Server. We also added the TYPE command which allows us to inspect the type of values added to the database. The current types are string, list and none if the given key does not exist.

We also made a lot of changes to the server to support blocking commands.

In the next chapter we'll add support for another Redis data type, Hashes.

Code

You can find the code on GitHub

Appendix A: Ziplist Deep Dive

Ziplists are implemented in the ziplist.h & ziplist.c files. The ziplist.c file contains a great explanation of how ziplists work. In this appendix we provide a few more details as well as some examples.

Ziplists were added to Redis in 2014 by Twitter engineers who apparently needed a more compact data structure to store Twitter timelines, essentially lists of ids. You can read more about it on the Pull Request page on GitHub and on the blog of one of the authors here and there.

A ziplist is a contiguous chunk of memory, that can store integers and strings, with the following layout:

<zlbytes> <zltail> <zllen> <entry> <entry> ... <entry> <zlend>

zlbytes is an unsigned 32-bit integer (uint32_t), which describes the number of bytes used by the ziplist, including itself.
ztail is also an unsigned 32-bit integer, which is an offset to the last element in the list. It serves a purpose similar to @tail in the implementation used in this chapter.
zllen is an unsigned 16-bit integer (uint16_t), it keeps count of the number of entries. Because it is an unsigned 16 bit integer, its maximum value is 65,535 (2^16 - 1). In order to allow ziplist to hold more elements, this ziplist implementation knows that the maximum value that can be described by this integer is 65,534 (2^16 - 2), and if the value of the integer is 65,535, it will scan the whole list to count the number of items. In practice, Redis caps the size of the ziplists it creates, the default is 2Kb, and this scenario is extremely unlikely to happen.
zlend is a single byte (uint8_t), and is set to 255, the maximum value of a byte, 2^8 - 1, or 1111 1111 / FF in binary and hexadecimal representations respectively.

The following elements are the actual entries in the list. Each entry has the following layout:

<prevlen> <encoding> <entry-data>

prevlen represents the length of the previous entry, allowing for right to left. It serves a purpose similar to the prev_node attribute in the list used in this chapter, but uses a dynamic size and will often be smaller. If the length of the previous entry in bytes is smaller than 254 bytes, prevlen will be stored in a single byte (uint8_t). The maximum value of a uint8_t is 255 (2^8 - 1). Otherwise, it will consumes 5 bytes, the first byte will be set to 254, the maximum value of a byte, and the following. As a reminder, 255 cannot be used since this value is used to flag the end of the list. This means that an entry cannot have a length greater than what can fit in a 32 bit integer, since the length is stored over 4 bytes, 32 bits. This sets the maximum length of a string stored by a ziplist to 4,294,967,295 (2^32 - 1). Which is, admittedly, a very large string.
encoding describes how the entry data is actually stored. It will take either one, two or five bytes. The first two bits of the encoding determines the type, 11 means an integer, and anything else, that is either 00, 01 or 10, means a string.

String encoding

If the encoding starts with 00, then the encoding itself will only occupy a single byte. This means that we can only use the remaining 6 bits of the encoding byte to encode the length of the string. The maximum value that can be encoded with 6 bits is 63 (2^6 - 1). This means that if the encoding byte is 0000 0001, the string that is stored has a length of 1. With 0000 0100, the string has a length of 4, and with 0011 1111, it has a length of 63.
If the encoding starts with 01, then the encoding occupies two bytes, which leaves us with 14 bits, the 6 bits of the first byte, and the 8 bits of the second bytes. This encoding can describe a string with a length up to 16,383 (2^14 - 1). As an examples, 0100 0001 0000 0000 describes a string of length 256.
If the encoding starts with 10, then the encoding will use five bytes. The 6 extra bits of the first byte are not used since we can use the other four bytes to hold a 32-bit value, which is the maximum length of a string in a ziplist. This encoding will only be used if strings have a length greater than 16,383.

Integer encoding

There are six variants of the integer encoding:

If the encoding is 1100 0000, it describes a signed integer that occupies 2 bytes (int16_t), which value can go up to 32,767 (2^15 - 1)
If the encoding is 1101 0000, it describes a signed integer that occupies 4 bytes (int32_t), which value can go up to 2,147,483,648 (2^31 - 1)
If the encoding is 1110 0000, it describes a signed integer that occupies 8 bytes (int64_t), which value can go up to 18,446,744,073,709,551,616 (2^63 - 1)
If the encoding is 1111 0000, it describes a signed integer that occupies 3 bytes (24 bits), which value can go up to 8,388,607 (2^23 - 1)
If the encoding is 1111 1110, it describes a signed integer that occupies 1 byte (8 bits), which value can go up to 128 (2^7 - 1)
Finally, if the first four bits are 1111, then the last four bits are used to hold an unsigned integer value itself. This means that technically we'd be bound to storing integers from 0 up to 15 (2^4 - 1), but the reality is actually more restrictive. We cannot store 0 because 1111 0000 means the 3 byte encoding described above. We also cannot store 15, because 1111 would make the whole byte 1111 1111 and this value, 255, is reserved for the end of list marker. We also cannot store 14, 1110, because 1111 1110 means the 1 byte encoding described above. This narrows the range of possible values from 1 (0001) to 13 (1101). Redis actually applies an offset here, to allow numbers from 0 to 12 to be stored with this encoding, so if you store 0 in a Ziplist, it actually stores 1, if you store 12, it actually stores 13. Storing -1 or 13 would require the next encoding type, the one using a full extra byte.

The DEBUG ZIPLIST command

Redis provides a few commands for debug purposes. One of them prints details about the ziplist if the key is itself stored as a ziplist. As it turns out, a list is never stored as a ziplist, it is always a quicklist of ziplists. But other Redis data types can use ziplists, for instance, a hash with a few items, less than 512 entries and the values are shorter than 64 bytes. Note that depending on how you run Redis on your machine, things will be different. The DEBUG ZIPLIST command does not print the results in the REPL but instead prints them on standard out. I often run a manually built from source Redis, which makes it easy to get the logs since they are by default printed to the terminal where you started the server, but if you run Redis through homebrew, the logs will be located somewhere else.

Let's try it:

127.0.0.1:6379> hset foo name pierre
(integer) 1
127.0.0.1:6379> DEBUG ZIPLIST foo
Ziplist structure printed on stdout

And on stdout:

{total bytes 25} {num entries 2}
{tail offset 16}
{
        addr 0x7f8f1350d8ca,
        index  0,
        offset    10,
        hdr+entry len:     6,
        hdr len 2,
        prevrawlen:     0,
        prevrawlensize:  1,
        payload     4
        bytes: 00|04|6e|61|6d|65|
        [str]name
}
{
        addr 0x7f8f1350d8d0,
        index  1,
        offset    16,
        hdr+entry len:     8,
        hdr len 2,
        prevrawlen:     6,
        prevrawlensize:  1,
        payload     6
        bytes: 06|06|70|69|65|72|72|65|
        [str]pierre
}
{end}

The previous shows an example of a ziplist containing two items, each is a string. The first entry occupies 6 bytes, and the second one eight. Let's look closer at all the bytes:

The first byte, 00, represents the length of the previous item, which is zero since this is the head of the list. The second byte is 04, which is the first encoding byte. The encoding can be between one and 5 bytes, so we need to look at the values, bit by bit, to determine which encoding it is, which will tell us how the data is stored. 04 has the following binary representation: 0000 0100. Looking at the "String encoding" section above, this fits in the first bullet point, a string described with a single byte, because it starts with 00. We then need to look at the other six bytes to determine the length of the string, 000100 is the number four, so the string has a length of four.

The last four bytes in the first entry are 00, 04, 6e, 61, 6d & 65. The string stored is name, which holds four bytes: 6e, 61, 6d & 65. The values are the ascii representation of the letters n, a, m & e.

We need to continue as long as we don't encounter the FF byte, or 255. The next six bytes are 06, 06, 70, 69, 65, 72, 72 & 65.

The first byte is the length of the previous item, 06, the hex representation of the number 6, which is what we found when looking at the previous entry, so far so good. The second byte is the first byte of the encoding, 06, or 0000 0110 in binary. Similarly as for the previous string, 00 tells us it's a string with a length lower than 63, and we can see it has a length of 6. The next six bytes represent the letters p, i, e, r, r, & e in ASCII and the whole entry has a length of 8.

We can now see why the total bytes value is 25, 4 for the zlbytes field, 4 for the zltail field, 2 for the zllen field, 6 for the first entry and 8 for the second entry and 1 for the end of list marker:
4 + 4 + 2 + 6 + 8 + 1 = 25. A basic doubly linked list would have required 20 bytes for the first node, 2 8-byte pointers and 4 bytes of data, 22 bytes for the second node, 2 8-byte pointers and 6 bytes of data, as well as 16 bytes for the head and tail pointers, for a grand total of 62. In this small example a ziplist is less than half the size!

The problem with ziplists is that adding and removing element require a reallocation of memory, and insertions/deletions in the middle of the list become expensive as the list grows since the whole list needs to be rearranged.

The tail offset field is 16 since the list header occupies 10 bytes, 4, 4 & 2, and the first item takes 6, that's 16.

Let's look at two more examples, one with an integer, and one with a string using a different encoding:

127.0.0.1:6379> hset foo age 30
(integer) 1
127.0.0.1:6379> DEBUG ZIPLIST foo

The following is the debug information, the first two entries are the same as previously and were omitted.

{total bytes 33} {num entries 4}
{tail offset 29}
...
{
        addr 0x7f8681004338,
        index  2,
        offset    24,
        hdr+entry len:     5,
        hdr len 2,
        prevrawlen:     8,
        prevrawlensize:  1,
        payload     3
        bytes: 08|03|61|67|65|
        [str]age
}
{
        addr 0x7f868100433d,
        index  3,
        offset    29,
        hdr+entry len:     3,
        hdr len 2,
        prevrawlen:     5,
        prevrawlensize:  1,
        payload     1
        bytes: 05|fe|1e|
        [int]30
}
{end}

Looking at the new bytes, 08 is the length of the previous entry, 8, 03 is the encoding for age, a string of length 3, and the three following bytes are the three letters. For the last entry, the length of the previous entry is 5, and the encoding is fe, or 1111 1110 in binary. This is the one byte integer encoding, which makes sense given that 30 is too big to use the "encoded within encoding header" approach, where the max value is 12 and is small enough to fit within a single byte, where the max value is 127. If we had set the age to 128, the bytes would have been 05, c0,80 & 00. c0 is the hex representation of 192/1100 0000, which represents the 2 byte signed integer encoding, which is why it is followed by two bytes, 80 & 00. 80 is the hex representation of 1000 0000, aka 128. Note that 128 is encoded as 80 00 and not 00 80. This is because Redis represents integers as little endian in Ziplists, the least significant bytes come first. There are two exceptions where integers are stored in big endian, for two of the different string encoding, the one using two bytes and the one using five bytes.

Let's now look at a sixty-four byte long string:

127.0.0.1:6379> hset foo 64-char-string aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
(integer) 0
127.0.0.1:6379> DEBUG ZIPLIST foo

The following is the debug information:

{total bytes 116} {num entries 6}
{tail offset 48}
...
{
        addr 0x7f867f705880,
        index  4,
        offset    32,
        hdr+entry len:    16,
        hdr len 2,
        prevrawlen:     3,
        prevrawlensize:  1,
        payload    14
        bytes: 03|0e|36|34|2d|63|68|61|72|2d|73|74|72|69|6e|67|
        [str]64-char-string
}
{
        addr 0x7f867f705890,
        index  5,
        offset    48,
        hdr+entry len:    67,
        hdr len 3,
        prevrawlen:    16,
        prevrawlensize:  1,
        payload    64
        bytes: 10|40|40|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|61|
        [str]aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...
}
{end}

Looking at the last entry, we see that the length of the previous entry is 10, which the hexadecimal representation of the number 16. As a quick reminder, this is because the two characters represent 4 bits each, 1 represents 0001 and 0 is 0000, and the byte 0001 0000 is the value 16, 2^4. The string "64-char-string" has 14 characters, and there are two more bytes, one of the prevlen field, and one for the encoding. That's 16. The encoding takes two bytes for the last entry, 40 & 40. 40 is the representation of the number 64, or 0100 0000 in binary.

The first two bits of the encoding are 01, so we know that the overall encoding will use two bytes, and the length of the string will use 14 of the 16 bytes. These 14 bits are: 00 0000 0100 000, which is the number 64, the length of the string coming up next.

Finally, we can see that if we set a string with a length longer than 64, Redis will convert the hash from a Ziplist to a Hash table

127.0.0.1:6379> hset foo 64-char-string aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa11
(integer) 0
127.0.0.1:6379> DEBUG ZIPLIST foo
(error) ERR Not an sds encoded string.
127.0.0.1:6379> DEBUG OBJECT foo
Value at:0x7f867f62f8e0 refcount:1 encoding:hashtable serializedlength:47 lru:7660156 lru_seconds_idle:36

Note that the error message is incorrect, it should say: "Not a ziplist encoded object". This was fixed in a more recent version of Redis.

Appendix B: Sorted Array

The following is a sorted array implementation that provides O(logn) push/<< & delete. The implementation uses a plain Ruby array as the underlying storage, which it delegates most of its operations to.

Elements are kept sorted in the push/<< method by using the built-in bsearch_index method. Using this method, we compute the index at which the element should be added in the array and maintain the ordering.

The delete method also relies on bsearch_index to find the index of the elements it needs to remove. Given that multiple elements could meet the conditions, it needs to find all the indices for the elements that should be removed. bsearch_index will return the leftmost index of an element that should be deleted, and we keep looking to the right as look as we find matching elements and mark them for deletion.

The SortedArray is meant to be used to hold classes holding multiple attributes, such as Structs, and order them based on one their fields. Its main use case in the BYORedis codebase is to store BlockedState instances, sorted by their timeouts.

module BYORedis
  class SortedArray
    def initialize(field)
      @underlying = []
      @field = field.to_sym
    end

    def push(new_element)
      if @underlying.empty?
        index = 0
      else
        index = @underlying.bsearch_index do |element|
          element.send(@field) >= new_element.send(@field)
        end
      end

      index = @underlying.size if index.nil?
      @underlying.insert(index, new_element)
    end
    alias << push

    def [](index)
      @underlying[index]
    end

    def size
      @underlying.size
    end

    def shift
      @underlying.shift
    end

    def delete_if(&block)
      @underlying.delete_if(&block)
    end

    def delete(element)
      index = @underlying.bsearch_index { |x| x.send(@field) >= element.send(@field) }
      return if index.nil?

      element_at_index = @underlying[index]
      indices_for_deletion = []
      while element_at_index
        if element_at_index == element
          indices_for_deletion << index
        end

        index += 1
        next_element = @underlying[index]
        if next_element && next_element.send(@field) == element.send(@field)
          element_at_index = next_element
        else
          break
        end
      end

      indices_for_deletion.each { |i| @underlying.delete_at(i) }
    end
  end
end

listing 7.68 The SortedArray class

The following is a test suite for the two main methods of the SortedArray class, push/<< & delete:

# coding: utf-8

require_relative './test_helper'
require_relative './sorted_array'

describe BYORedis::SortedArray do
  TestStruct = Struct.new(:a, :timeout)

  describe 'push/<<' do
    it 'appends elements while keeping the array sorted' do
      sorted_array = new_array(:timeout)

      sorted_array << TestStruct.new('a', 1)
      sorted_array << TestStruct.new('b', 2)
      sorted_array << TestStruct.new('c', 10)
      sorted_array << TestStruct.new('d', 20)
      sorted_array << TestStruct.new('e', 15)
      sorted_array << TestStruct.new('f', 8)

      assert_equal(6, sorted_array.size)
      assert_equal(1, sorted_array[0].timeout)
      assert_equal(2, sorted_array[1].timeout)
      assert_equal(8, sorted_array[2].timeout)
      assert_equal(10, sorted_array[3].timeout)
      assert_equal(15, sorted_array[4].timeout)
      assert_equal(20, sorted_array[5].timeout)
    end
  end

  describe 'delete' do
    it 'deletes the element from the array' do
      sorted_array = new_array(:timeout)

      sorted_array << TestStruct.new('a', 10)
      sorted_array << TestStruct.new('b1', 20)
      sorted_array << TestStruct.new('b2', 20)
      sorted_array << TestStruct.new('b3', 20)
      sorted_array << TestStruct.new('c', 30) # array is now a, b3, b2, b1, c

      sorted_array.delete(TestStruct.new('d', 40)) # no-op
      sorted_array.delete(TestStruct.new('b1', 20))

      assert_equal(4, sorted_array.size)
      assert_equal(10, sorted_array[0].timeout)
      assert_equal(TestStruct.new('b3', 20), sorted_array[1])
      assert_equal(TestStruct.new('b2', 20), sorted_array[2])
      assert_equal(30, sorted_array[3].timeout)
    end
  end

  def new_array(field)
    BYORedis::SortedArray.new(field)
  end
end

listing 7.69 Basic unit tests for the SortedArray class

Rebuilding Redis in Ruby - Chapter 6 - Building a Hash Table from scratch

Pierre Jambet — Mon, 31 Aug 2020 19:28:32 +0000

What we'll cover

So far we've been using the Ruby Hash class as the main storage mechanism for the key/value pairs received through the SET command. We also use it for the secondary dictionary necessary to implement the TTL related options of the SET command. We store the expiration timestamp of keys with TTLs, which allows us to know whether a key is expired or not.

Redis is written in C, which does not provide a collection similar to Ruby's Hash. In C, you only get one collection out of the box, arrays.

Redis implements its own dictionary collection, in dict.c. Because the dictionary data structure is so central to how Redis functions, we will replace the use of the Ruby Hash class with a Dict class we will build from scratch.

We will also add the DEL command, the ability to delete keys is really important to Redis. Having the DEL command will also allow us to easily play with our Dict class to make sure that it handles deletion operations correctly.

We are not adding any new features in this chapter, beside the DEL command, we're rewriting a key part of the system with lower level elements. Given that Redis' dict data structure relies on arrays, we will still use the Ruby Array class. We could have reimplemented the Array class, and you'll find an example in Appendix A, but arrays in C are not specific to Redis. On the other hand, the structure defined in dict.c is.

Let's get to it.

Maps, Dictionaries, Associative Arrays

This chapter covers hash tables, which is one way of implementing a data structure commonly called Map, Dictionary or Associate Array. I will use the term "Dictionary" as I find that "Map" can be confusing, especially when working with languages providing a map function/method, such as Ruby! From my experience the term associative array, while very explicit, is not as common.

The basic definition of such data structure is one that holds zero or more key/value pairs, where a key cannot appear more than once.

A key operation (pun intended!) of a dictionary is the ability to retrieve a value given a key. The returned value is either empty if no element with such keys exists in the map, or the value mapped to the key if it exists.

Some definitions also include the ability to add, update or delete key/value pairs, which is not provided in immutable versions, where such operations would result in the creation of a new dictionary. The immutable versions will implement similar operations returning a new structure instead of modifying it.

There are multiple ways of implementing a data structure providing these operations. A naive and fairly inefficient version could be to use an array where each element is a key value pair:

def set(map, key, value)
  map.each do |pair|
    pair_key = pair[0]
    # Override the value if the key is already present
    if key == pair_key
      pair[1] = value
      return pair
    end
  end
  pair = [key, value]
  map << pair
  pair
end

def lookup(map, key)
  map.each do |pair_key, pair_value|
    return pair_value if key == pair_key
  end
  return
end

listing 6.1: A basic dictionary using an array

Given these functions, set & lookup, we could use them with an array:

map = []
set(map, "key-1", "value-1") # => ["key-1", "value-1"]
set(map, "key-2", "value-2") # => ["key-2", "value-2"]
set(map, "key-2", "value-3") # => ["key-2", "value-3"]

lookup(map, "key-1") # => "value-1"
lookup(map, "key-2") # => "value-2"
lookup(map, "key-3") # => nil

This approach works from an API standpoint, but it would show performance issues as we keep adding elements to the array. Because we must prevent duplicated keys, we need to iterate through the whole array every time we attempt to add a new pair if the key is not already present. This set implementation is an O(n) operation, where n is the number of pairs in the dictionary. The amount of time required to add an element is proportional to the size of the collection.

A lookup might not always require a complete scan of the array, if we're lucky and find the key before the end, but it might, in the worst case scenario. The lookup operation is therefore also an O(n) operation.

For Redis, which should be able to handle hundreds of thousand of keys, even millions, and potentially billions, these performance issues are not acceptable.

One common implementation that addresses these performance issues is a hash table. Another possible implementation is a tree map, which uses a tree structure to store elements. For instance, the Java TreeMap class uses a Red-Black tree to maintain the underlying tree balanced. One of the benefits of a tree map compared to a hash table is that it stores elements in order, whereas a hash table does not.

In the next section we will learn how hash tables implement these operations in a more time efficient manner.

Before moving on and abandoning this implementation, it's really important to note that while this implementation would not perform well with large collections, it might actually be one of the most efficient options for very small collections, such as with one or two pairs, thanks to its simplicity. If the array is small, finding an element requires very few steps and little memory overhead.

As a matter of fact, the Scala standard library does something similar for dictionaries with up to four pairs, it has special case classes meant to handle these fixed sized dictionaries, allowing them to be really fast as there's no need for hashing or anything else.

Hash Tables

Hash tables are available in many programming languages as part of their standard libraries. Python has dict, Java has HashMap, Scala has Map, Elixir has Map, Rust has HashMap, Ruby's Hash class is a hash table implementation too. You get it, they're almost everywhere.

Hash tables can be implemented in different ways, the wikipedia article shows a few different examples. The one we'll explore in this chapter uses a collision resolution called separate chaining. But why do we need collision resolution? To answer this we first need to look at the central element of a hash table, its hash function.

A hash function must obey a few properties, one of the most important ones being determinism, in other words, identical inputs should result in identical outputs. To explain why, let's look at how the hashing function is used through a pseudo code implementation of a hash table:

function new_node(key, value, next_node)
    return Node(key, value, next_node)

function update_node(node, new_value)
    return Node(node.key, new_value)

function create_hash_table()
    table = allocate_array_of_arbitrary_initial_size()
    return table

function add_or_update_key_value_pair(table, key, value)
    hash = hash_function(key)
    index = hash % table.size
    node = table[index]
    if node == null
        table[index] = new_node(key, value, null)
    else
        while node != null && node.key != key
            node = node.next_node

        if node.nil?
            existing_node = table[index]
            table[index] = new_node(key, value, existing_node)
        else
            update_node(node, value)

function lookup_key(table, key)
    hash = hash_function(key)
    index = hash % table.size
    if table[index] == null
        return null
    else
        node = table[index]
        while node != null
            if node.key == key
                return key
            else
                node = node.next_node

        return null

listing 6.2: Pseudo-code hash table

The new_node function acts as the entry point of a linked list. A node contains a key, a value, and a next node value. If the next node value is null, the element is the last one in the list.

Prepending — a fancy word for "adding at the beginning" — an element to such list is done by first creating a single node list and then a second one, with the next_node value set to the first one:

node1 = new_node(key1, value1, null)
node2 = new_node(key2, value2, node1)

In this example node1 is a list with a single node, and node2 is a list with two nodes. The first node is the one with the key key2 and the value value2, its next_node value is equal to node1, which has the key key1 and value value1, it does not have a next_node value and is the last element of the list.

update_node works with an existing node and changes its value. It is a useful function when we find an existing pair with a matching key in add_or_update_key_value_pair. We explore this other function in more details below.

create_hash_table does only one thing, it allocates an array of arbitrary size. We purposefully do not define this function here. The size is not really important, as long as it creates a non empty array. The implementation of the allocation is also not really relevant to this example. Most operating systems provide such features, it's therefore fair to assume that it would use the allocation operations provided by the operating system. malloc is a function in the C standard library that does provide the ability to manually allocate memory.

add_or_update_key_value_pair does more work and let's walk through it, one line at a time. It takes three parameters, the table we want to insert the pair into, the key and the value.
We first call hash_function with key. We'll dive deeper into what an implementation of hash_function looks like later, but for now, let's assume it returns an integer. Because the hash function is unaware of the size of the array, the returned value might be larger than the size of the array.

We use the modulo operation to convert the hash value returned by hash_function into a number between 0 and table.size - 1. We can now use the result of the modulo operation as an index. That's why we have the create_hash_table function, to make sure that table is initialized with empty slots. These slots are often called buckets in hash table lingo.

If the bucket is empty, then we create a new node, add it to the bucket and we're done. If the bucket is not empty, then things are more complicated.

There are two distinct cases to consider if there is already an item at the location obtained through the hash function. One of the nodes in the bucket might have the same key, in this case we want to override its value with the new value. This the case where we want to update an element in the array instead of adding it.

The other one is that the nodes already present might all have a different key, in which case we want to keep all the existing nodes and add the new one. This the case where we want to add a new pair, and is called a collision.

Let's illustrate with an example. Let's set the initial size of the array to 4, all the buckets are empty:

table = [nil, nil, nil, nil]

Let's define a hash function, that returns the length of the input string:

function hash_function(string)
    return string.length

If we first call add_or_update_key_value_pair with the key "a" and the value "b", hash_function will return 1, the length of the string "a". 1 % 4 returns 1, so we add the pair at index 1:

table = [nil, Node("a", "b", nil), nil, nil]

Let's call add_or_update_key_value_pair with the pair "cd"/"e", the length of "cd" is 2, 2 % 4 is 2, we insert the pair at index 2:

table = [nil, Node("a", "b", nil), Node("cd", "e", nil), nil]

Let's now call add_or_update_key_value_pair with "fg"/"h". The length of "fg" is 2, but there's already a pair at index 2. Because we want to keep all pairs we need a solution to resolve this collision. There are different strategies available to us here, and the one we're going to use is called "separate chaining".

The essence of this strategy is that each bucket contains a linked list of values. So in the previous example, we insert the new element at the beginning of the list at index 2. Note that prepending an element to a linked list is an O(1) operation, it takes the same amount of time regardless of the size of the list. This is the list once the operation is completed:

table = [nil, Node("a", "b", nil), Node("fg", "h", "Node("cd", "e", nil)), nil]

lookup_key is very similar to add_or_update_key_value_pair. We use the same process to find which bucket the key should be in. If the bucket is empty, we return null, the key is not present in the dictionary. On the other hand, if the bucket is not empty, we need to look through the list until we find the node with the key argument.

If we don't find any, the key is not present in the dictionary.

Trying to avoid collisions

The hash_function we used in the previous works well as an example because of its simplicity but it would not be practical in the real world. To keep hash tables efficient, we want to reduce the number of collisions as much as possible. This is because iterating through the linked list is inefficient, if there are a lot of collisions, it could take a long time to loop through all the items in the bucket.

This is where the uniformity property of a hash function is really important. Uniformity helps reduce the likelihood of collision. In the previous example, if a hypothetical hash function had returned the values 1, 2 & 3, respectively, instead of 1, 2 & 2, there wouldn't have been any conflicts.

Collisions are also related to the size of the underlying array. Regardless of the uniformity of the hash function, if the underlying array has a size n, storing n + 1 items cannot happen without at least one collision.

One approach would be to start by allocating a very large amount of memory, but this can be wasteful, because there could be a lot of memory allocated, but unused. Many hash table implementation have mechanisms to adjust the size as needed, and it turns out that Redis does this too, as we'll see in the next section.

A good hash function that provides uniformity means that both operations add/update & lookup have an O(1) time complexity, meaning that the number of steps is always the same regardless of the number of elements already present. We first hash the value, transform it to an index and use the matching bucket.

On the other hand, a bad hash function without uniformity would make these operations O(n). In the absolute worst case scenario, all keys would land in the same bucket, and the number of operations required would depend on the number of elements already present in the linked list in the bucket.

Back to determinism

Now that we know how the result of a hash function is used, that is, it determines the location of a key/value pair in the underlying array, let's go back to the determinism element of a hash function.

Let's demonstrate why we need determinism by showing what would happen with a hash function that is not deterministic.

In Ruby, each object is given an object id, in the following examples, the two variables str1 & str2 are different instances, each holding the same value, and are therefore considered equal, but have different object_id values:

str1 = "hello"
str2 = "hello"
str1.object_id # => 180
str2.object_id # => 200
# The object_id values might be different on your machine, they are assigned at runtime
# and will therefore differ if you've created more Ruby objects beforehand for instance
str1 == str2 # => true

Let's define a wrong hash function, which returns its input's object_id:

def hash_function(object)
  object.object_id
end

Let's manually walk through a small example, let's start by creating a hash table of size 3 and add the pair a-key/a-value to it. Let's re-use the same object_id from the previous example, and assume that a-key would have returned 180. 180 % 3 = 0, so we insert the new node at index 0:

table = [Node("a-key", "a-value", nil), nil, nil]

And let's now call the lookup function with a different string holding the same value, and, reusing the previous example data again, assume that its object id is 200, 200 % 3 = 2. The lookup would look at the bucket at index 2, find a nil value and return nil, whereas the table does contain a pair with the key a-key.

A deterministic hash function prevents this.

Common Hash Functions

In order for a hash table implementation to be efficient, it needs a good hash function. Hash functions come in different flavors, as shown on wikipedia:

Cyclic redundancy checks
Checksums
Universal hash function families
Non-cryptographic
Keyed cryptographic
Unkeyed cryptographic

Some of the functions in the "Unkeyed cryptographic hash functions" category are pretty common. MD5 used to be very common to verify the integrity of a file downloaded over the internet. You would download the file, compute the md5 of the file locally and compare it against the md5 published by the author of the file. It is common to see sha256 used instead nowadays. This is what the Downloads page on ruby-lang.org does!

For a long time sha1 was the default algorithm used by git to hash commits and other objects. It now supports multiple algorithms such as sha256. This change was required after researchers proved that it was possible to forge two different inputs resulting in the same sha1 hash.

Redis uses SipHash which is in the "Keyed cryptographic hash functions" category. We will look closer at the SipHash algorithm below.

All Ruby objects implement a hash method, which happens to use the Siphash algorithm, the same algorithm Redis uses!

str1 = "hello"
str2 = "hello"
# Note that the hash value is partially computed from a random value and will therefore be different
# on your machine
# It will also be different if you restart irb
str1.hash # => 2242191710387986831
str2.hash # => 2242191710387986831

Now that we know what a hash function is, how it used to implement a hash table, let's look at how things work in Redis.

How does Redis do it?

Redis uses three data structures to implement a dictionary, dict, dictht & dictEntry, the following diagram, from wjin.org, shows how they each relate to each other:

It's important to note that dictionaries are used in multiple places in the Redis codebase, but there are two main ones for each database, the one holding all the top-level key/value pairs, such as the ones added with SET and other commands creating pairs, and the expires dictionary, used to store key TTLs.

If you're not used to C, don't worry too much about it for now, we're not going to look too closely at pointers and other C specific features.

Our implementation supports a single database, but Redis can handle multiple databases. A database in Redis represents a set of key/value pairs it is defined as the following C struct:

typedef struct redisDb {
    dict *dict;                 /* The keyspace for this DB */
    dict *expires;              /* Timeout of keys with a timeout set */
    dict *blocking_keys;        /* Keys with clients waiting for data (BLPOP)*/
    dict *ready_keys;           /* Blocked keys that received a PUSH */
    dict *watched_keys;         /* WATCHED keys for MULTI/EXEC CAS */
    int id;                     /* Database ID */
    long long avg_ttl;          /* Average TTL, just for stats */
    unsigned long expires_cursor; /* Cursor of the active expire cycle. */
    list *defrag_later;         /* List of key names to attempt to defrag one by one, gradually. */
} redisDb;

listing 6.3: The C struct for redisDB defined in dict.c

We will ignore all the fields but the first two for now. We can see that the two fields, dict & expires are both of the same type: dict.

dict is defined in dict.h:

typedef struct dict {
    dictType *type;
    void *privdata;
    dictht ht[2];
    long rehashidx; /* rehashing not in progress if rehashidx == -1 */
    unsigned long iterators; /* number of iterators currently running */
} dict;

listing 6.4: The C struct for dict

Once again, in order to keep things as simple as possible, we will ignore some fields, specifically, privdata & iterators.

The dictType struct is used to configure the behavior of a dict instance, such as using a different hash function for instance. It is defined as:

typedef struct dictType {
    uint64_t (*hashFunction)(const void *key);
    void *(*keyDup)(void *privdata, const void *key);
    void *(*valDup)(void *privdata, const void *obj);
    int (*keyCompare)(void *privdata, const void *key1, const void *key2);
    void (*keyDestructor)(void *privdata, void *key);
    void (*valDestructor)(void *privdata, void *obj);
} dictType;

listing 6.5: The C struct for dictType defined in dict.c

The syntax used in this struct is different because the members are function pointers. That's about as far as we'll go with C in this chapter. Redis does this to allow a form of configuration of a dict instance. It has the ability to create two dictionaries with potentially two different hash function implementation. We don't need this level of flexibility at the moment so we will not implement these features for now.

The most interesting element of the dict struct for us is the dictht array. ht here stands for Hash Table. ht[2] means that the struct member is named ht and is an array of size two. Essentially, each dict instance has two hash tables, ht[0] & ht[1].

dictht is defined as follows:

/* This is our hash table structure. Every dictionary has two of this as we
 * implement incremental rehashing, for the old to the new table. */
typedef struct dictht {
    dictEntry **table;
    unsigned long size;
    unsigned long sizemask;
    unsigned long used;
} dictht;

listing 6.6: The C struct for dictht defined in dict.c

The comment tells us why a dict has two tables, for rehashing. To explain rehashing, we first need to explain the first member of dictht: dictEntry **table. The double star syntax, a pointer to pointer, is not that interesting to us at the moment, it is one way of defining an array of dictEntry with dynamic size, one that can be set at runtime. We then need to look at the dictEntry struct:

typedef struct dictEntry {
    void *key;
    union {
        void *val;
        uint64_t u64;
        int64_t s64;
        double d;
    } v;
    struct dictEntry *next;
} dictEntry;

listing 6.7: The C struct for dictEntry defined in dict.c

dictEntry is a linked list, a common term for a structure like this one is "a node". It contains a key, key, a value, v and a link to the next element in the list, next.

Because dictEntry **table is an array with a dynamic size, we also need the size member. used is a counter, that starts at 0 and that is incremented when items are added, and decremented when items are removed.

sizemask is an integer value, which is initialized at 0 if size is also 0, but is otherwise always set to size - 1.

To understand the need for the sizemask member, let's look back at our pseudo code implementation from above. We can see that a very common operation is to use hash_value % array_size. This operation converts a value, potentially larger than the array size, to one that is between 0 and size - 1, allowing us to use the result as an index for the underlying array.

The modulo operation, %, is not that costly, but it does require a few steps, an integer division, followed by a multiplication and a subtraction: c - (c/m*m). Let confirm with an example:

c = 469513
m = 143317
c % m # => 39562
c - (c/m*m) # => 39562

Given how crucial this operation is to the performance of a hash table, every operation relies on it to find the bucket index, it is valuable to attempt to optimize it.

It turns out that if, and only if, the modulus (the second part of the modulo operation, b in a % b, the size of the array in the hash table) is a power of two, then the modulo can be computed in a single operation with the bitwise AND/& operator:

a & (b-1)

Let's illustrate this with a few examples, if the backing array of the hash table is of size 4, then:

0 % 4 = 0, 0 & 3 = 0
1 % 4 = 1, 1 & 3 = 1
2 % 4 = 2, 2 & 3 = 2
3 % 4 = 3, 3 & 3 = 3
4 % 4 = 0, 4 & 3 = 0
5 % 4 = 1, 5 & 3 = 1
...

It might help to visualize the number as binary, let's use 4 bit integers for readability. The binary representation of 4 is 0100 and 3 is 0011, so 0 & 3 can be represented as:

The following shows the visualization of 1 & 3, 2 & 3, 3 & 3, 4 & 3 & 5 & 3.

 0001 (1)  |  0010 (2) |  0011 (3) |  0100 (4) |  0101 (5)
&0011      | &0011     | &0011     | &0011     | &0011
    =      |     =     |     =     |     =     |     =
 0001 (1)  |  0010 (2) |  0011 (3) |  0000 (0) |  0001 (1)

In order to take advantage of this property, Redis always picks a size that is a power of two for the backing array. By setting sizemask to size - 1, Redis can efficiently compute the index of any keys once it obtained its hash value. This is a part of the code for dictFind:

dictEntry *dictFind(dict *d, const void *key)
{
    // ...
    h = dictHashKey(d, key);
    for (table = 0; table <= 1; table++) {
        idx = h & d->ht[table].sizemask;
        he = d->ht[table].table[idx];
        // ...
    }
    return NULL;
}

listing 6.8: excerpt of dictFind

h is the value returned by the hash function and d->ht[table].sizemask is how Redis accesses the sizemask value for its hash table. idx is the index indicating the location of the bucket. Redis then looks into the array to inspect the bucket with he = d->ht[table].table[idx] (he stands for hash entry).

Rehashing

Now that we looked at the data structures that Redis uses to implement its dict type, we need to look at the rehashing process. A new dictionary in Redis is always empty, the backing table, the table member, is set to NULL and the size, sizemask and used members are all set to 0:

// https://github.com/antirez/redis/blob/6.0.0/src/dict.c#L102-L108
static void _dictReset(dictht *ht)
{
    ht->table = NULL;
    ht->size = 0;
    ht->sizemask = 0;
    ht->used = 0;
}

// https://github.com/antirez/redis/blob/6.0.0/src/dict.c#L121-L131
int _dictInit(dict *d, dictType *type,
        void *privDataPtr)
{
    _dictReset(&d->ht[0]);
    _dictReset(&d->ht[1]);
    d->type = type;
    d->privdata = privDataPtr;
    d->rehashidx = -1;
    d->iterators = 0;
    return DICT_OK;
}

listing 6.9: C code for _dictReset & _dictInit

Whenever Redis adds a new key/value pair to a dictionary, it first checks if the dictionary should be expanded. The main reason causing a dict to expand is if the number of items in it, the used member, is greater than or equal to the size of the dict, the size member. This will always be true for an empty dictionary since both are initialized to 0. This will also be true every time the number of items reaches the size of the dict. When the dict is of size 4, once 4 items are added, the next addition will trigger a resize.

This is necessary to limit the likelihood of collisions, because as we discussed, collisions slow down the hash table. If the array contains four buckets and the table needs to store five items, it will at best have one collision. If we resize the array to contain eight buckets, then it is possible to store the five items without any collisions.

As mentioned earlier, in order to take advantage of the "fast modulo for a power of two value through bitwise AND" property, Redis will always choose a power of two for the size. The smallest non empty size is 4, and it will grow through power of twos from there on: 8, 16, 32, 64 and so on. All the way up to LONG_MAX + 1, 9,223,372,036,854,775,808, also written as 9.223372036854776e+18 in the exponential notation. That's 9.2 billion billions, Yes it's a huge number!

Why LONG_MAX + 1

Redis uses LONG_MAX + 1 as the maximum value for a dict size. LONG_MAX is a constant defined in the C header file limits.h and is set to 9,223,372,036,854,775,807. We can see that it's not a power of two by looking at the right most digit, 7, it's not even, so we already know that it's not a power of two. All power of two are even numbers since by definition, we obtain them by multiplying 2s.

This big number is a 64-bit integer, where all bits are 1s, except the first one:

0111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111

The first bit is a 0 because LONG_MAX is a signed integer, that is it can be negative of positive. Signed integers use the first bit to determine the sign, 0 for positive, 1 for negative. The other bound is LONG_MIN, set to -9,223,372,036,854,775,808, which has the following binary representation:

1000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

This representation of signed numbers is called Two's complement, there are other representations but two's complement is a very common one.

The only larger number that can fit in a 64-bit integer, is the unsigned max value, where all bits, even the first one, are used to encode the integer value, it's called ULONG_MAX, U stands for usigned here, and is set to: 18,446,744,073,709,551,615, 2^64 - 1 . As we did before, we can see that it's not a power of two, 5, the last digit, is not even. This is its binary representation, 64 1s:

1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111

It means that if we want to use power of two numbers, then the largest one we can get to is 2^63. The max value we can encode in a 64 bit integer is 2^64 - 1, one short, so we have to stop one power of two below, hence setting the upper bound to LONG_MAX + 1, aka, 2^63.

Back in Chapter 4 we talked about Big O notation and time complexity. The bottom line being that since Redis processes incoming commands sequentially, a slow operation would effectively back the queue. You can think of it as someone taking a long time to go through checkout at a grocery store. The longer they take, the more likely it is that the queue of customers waiting in line increases.

Resizing a hash table is essentially an O(n) operation, the time it takes to do it is proportional to n, the number of elements in the hash table. In other words, the more elements in the table, the longer it'll take to resize it. And as we just saw, Redis hash tables can get big, really big! Forcing all the clients in the queue to wait while we resize the table is far from desirable.

Enter rehashing!

Rehashing is the process Redis uses to incrementally, in small steps, resize the table, while still allowing other operations to be processed, and this is why it uses two hash tables per dictionary. Let's look at how rehashing works through an example.

Note that resizing the array is technically not necessary to store the items given that each entry in the array, the buckets, are linked lists. This means that even an array of size 4 could store millions and billions of key/value pairs. The problem is that the performance would suffer drastically, iterating through that many items in a linked list would take a very long time. With millions of items, it could easily take multiple seconds.

The Redis server starts, the main dict is initialized with two hash tables, both empty
The server receives a SET command, it needs to expand the dictionary.
It finds the next power of two that will be enough to fit all the elements, it needs to store one element, so it uses the minimum value 4.
It allocates the main array, ht[0], with a size of 4, and adds the first key/value pair
The second, third & fourth values are added without any issues. used is now set to 4
A fifth SET command is received, Redis decides to resize the dict.
The resize process allocates a new table, big enough to store all the items, for which the size is a power of two. It selects the next power of two, 8.
The new table is assigned to the secondary table, the rehashing one, ht[1]
The dict is now a rehashing state. In this state, all new keys are added to the rehashing table.
The dict has now 2 tables, where 4 keys are in the first table and 1 is in the second one.
While in this state, many operations, such as key lookups will look at both tables. For instance a GET command will first look at the first table, and if it doesn't find the item, will look at the rehashing table, and only if the items can't be find in either table, will return NULL.
While in rehashing state, many commands, such as key lookups or key additions, will perform a single step of rehashing. The server_cron time event we looked at in Chapter 3 also attempts to rehash dictionaries that needs to.
The rehashing process starts at index 0, looks at the first table, and if it finds an item, moves it to the second table. It then moves to index 1, until it iterated through the entire table
Once it's done, it makes the rehashing table the primary, resets the rehashing table to an empty table and exits the rehashing state

This process allows Redis to resize dictionaries in small steps, while not preventing clients to send commands in the meantime.

Rehashing is also used to reduce the size of a dictionary. If the number of keys, the used member, goes below 1/10th of the size value, that is, only 10% of the dictionary is used, Redis will try to find a better size to fit the keys. That is, it will find the smallest power of two that is greater than or equal to used.
This is also performed in server_cron, as a time event and prevents Redis from unnecessarily using memory.

If you want to dig deeper in the Redis implementation, here are few interesting functions you can start with:

The SipHash hash function

The last thing we need to look at before building our own hash table is the hash function. Redis has been using the SipHash algorithm since version 5.0. Before that it had been using the MurmurHash2 algorithm as of version 2.5. And before that, it used a simple version from Dan Bernstein called djb2.

The SipHash algorithm is described in this paper: SipHash: a fast short-input PRF paper.

One of the benefits of SipHash is that it offers strong protection against attacks such as hash flooding.

The implementation of the SipHash algorithm is quite complicated. The one used by Redis is in the siphash.c file and a Ruby implementation is provided in Appendix B. What is important to note is that Siphash requires a key, usually coming from random bytes, to compute a hash value.

This means that unlike md5 or sha1, which always return the same value for the same input, siphash will return the same value, if, and only if, the key is the same.

This is essentially a simplified explanation of how the hash flooding protection works. If Redis were to use md5 as its hashing function, I could try to guess what the hash value used to compute the index would be. Let's see that with an example:

The md5 hash for the string a is the string 0cc175b9c0f1b6a831c399e269772661:

Digest::MD5.hexdigest("a") # => 0cc175b9c0f1b6a831c399e269772661

The result is a 32 character string representing a 128-bit (16 bytes) result. Because most CPUs use 64-bit integers as their largest types, the result we just saw is actually the hex representation of two 64 bit integers. Let's illustrate this with the pack and unpack method.

The string is a hex string, so we need to look at each pair of characters. We call hex on each pair, which returns the integer value. For instance '00'.hex returns 0, 'ff'.hex returns 255, the maximum value of an 8-bit integer — a byte. We then call .pack('c16') which returns a string representing all the bits concatenated together. We use 'c16' because the result of .map(&:hex) is an array of 16 bytes.

Finally .unpack('QQ') looks at the string of bytes and tries to convert to two 64 bit integers. We use 'QQ', which is identical to 'Q2', because a string of 16 bytes can be unpacked to two 64-bit integer. One 64-bit integer is composed of 8 bytes — a byte contains 8 bits, so 8 bytes contain 64 bits — so 16 bytes can be unpacked to two 64-bit integers.

bytes = "0cc175b9c0f1b6a831c399e269772661".scan(/../).map(&:hex).pack('c16') # => "\f\xC1u\xB9\xC0\xF1\xB6\xA81\xC3\x99\xE2iw&a"
bytes.unpack("QQ") # => [12157170054180749580, 7000413967451013937]

The previous code was adapted from this blog post

A simpler example might help illustrate this:

"ffff0000ffff0000ffff0000ffff0000".scan(/../).map(&:hex)
=> [255, 255, 0, 0, 255, 255, 0, 0, 255, 255, 0, 0, 255, 255, 0, 0]

[255, 255, 0, 0, 255, 255, 0, 0, 255, 255, 0, 0, 255, 255, 0, 0].pack('c16')
=> "\xFF\xFF\x00\x00\xFF\xFF\x00\x00\xFF\xFF\x00\x00\xFF\xFF\x00\x00"

"\xFF\xFF\x00\x00\xFF\xFF\x00\x00\xFF\xFF\x00\x00\xFF\xFF\x00\x00".unpack('QQ')
=> [281470681808895, 281470681808895]

We can play with pack and unpack a little bit more to confirm that these two 64-bit integers we got are the two elements of the md5 result:

[12157170054180749580].pack("Q").unpack("H16")
=> ["0cc175b9c0f1b6a8"]
[7000413967451013937].pack("Q").unpack("H16")
=> ["31c399e269772661"]

12157170054180749580 represents the first 64 bits of the md5 value, by calling .pack('Q') we convert it to a string representing all these bits back to back, and convert it back to a string of 16 hex characters with .unpack('H16'). We can confirm that 0cc175b9c0f1b6a is the first half of 0cc175b9c0f1b6a831c399e269772661 and that 31c399e269772661 is the second half.

We can also look at the actual 64 bits with unpack('B64')

[12157170054180749580].pack('Q').unpack('B64')
=> ["0000110011000001011101011011100111000000111100011011011010101000"]
[7000413967451013937].pack('Q').unpack('B64')
=> ["0011000111000011100110011110001001101001011101110010011001100001"]

Back to our hypothetical use of md5 as a hash function in Redis. Given that we would only use a single integer to apply the modulo to, we could pick either the first one or last one, let's arbitrarily pick the second one, just because.

If I sent the command SET a-key a-value, the hash value of a-key is the 64 bit integer 7000413967451013937. This knowledge can be used to forge special requests and maximize the chances of collisions, potentially causing performance issues to the hash table.

With a keyed algorithm such as Siphash, it's impossible to infer what the hash value would be if the server uses random bytes as the key. We can demonstrate this with Ruby, which also uses Siphash by running "a".hash in an irb shell, closing and reopening irb, the hash value will be different. This is because Ruby initializes random bytes at startup that it then uses to compute siphash values.

Siphash was added to Redis in this commit. SipHash returns a 64 bit integer, so we can use the value directly instead of going trough all the steps we had to go through with md5.

Our own `Dict` class

`Dict`, `HashTable` & `DictEntry`

We previously looked at the main data structures used in the Redis code base to implement a hash table. We will reimplement a simplified version of those, let's start with the two small ones, DictEntry & HashTable:

module BYORedis
  class DictEntry

    attr_accessor :next, :value
    attr_reader :key

    def initialize(key, value)
      @key = key
      @value = value
      @next = nil
    end
  end
end

listing 6.10: The DictEntry class

module Redis
  class HashTable

    attr_reader :table, :size, :sizemask
    attr_accessor :used

    def initialize(size)
      @table = size == 0 ? nil : Array.new(size)
      @size = size
      @sizemask = size == 0 ? 0 : size - 1
      @used = 0
    end

    def empty?
      @size == 0
    end

    def each
      return unless @table

      @table.each
    end
  end
end

listing 6.11: The HashTable class

@table in HashTable will contain instances of DictEntry, which is our linked list implementation to handle collisions.

We could have defined DictEntry as a Struct, but the explicit class has one small benefit, it allows us to not define a setter for the key attribute.

As already discussed, because Ruby is Ruby, where most things are mutable, it does not mean that a key attribute will always be the same, but at least it prevents reassignment. The value can still be changed by updating the value in place:

dict_entry = BYORedis::DictEntry.new("a-key", "a-value") # => nil
dict_entry # => #<BYORedis::DictEntry:0x00007fd577a17c60 @key="a-key", @value="a-value", @next=nil>
dict_entry.key.gsub!("a", "A") # => "A-key"
dict_entry # => #<BYORedis::DictEntry:0x00007fd577a17c60 @key="A-key", @value="a-value", @next=nil>
dict_entry.key # => "A-key"

Let's now create the main class, Dict:

require_relative './siphash'
require_relative './dict_entry'
require_relative './hash_table'

module BYORedis
  class Dict

    INITIAL_SIZE = 4
    MAX_SIZE = 2**63

    attr_reader :hash_tables

    def initialize
      @hash_tables = [ HashTable.new(0), HashTable.new(0) ]
      @rehashidx = -1
    end

    # ...
  end
end

listing 6.12: The Dict class

Our Dict class does not require any arguments, it is initialized with two empty hash tables, and a rehash index set to -1, indicating that it is not in a rehashing state. The rehashing index is used to keep track of the progress throughout rehashing. Its value will change from 0 to @hash_tables[0].size - 1 as we rehash the table, and reset to -1 when completed.

The first method we need to add to the Dict class is set, it will be used to add or update an element to a dictionary. This is needed for the SET command.

# dict.rb
module BYORedis
  class Dict

    # ...

    def set(key, value)
      index = key_index(key)

      rehash_step if rehashing?

      hash_table = rehashing? ? rehashing_table : main_table
      entry = hash_table.table[index]

      entry = entry.next while entry && entry.key != key

      if !entry.nil?
        entry.value = value
      else
        entry = DictEntry.new(key, value)
        entry.next = hash_table.table[index]
        hash_table.table[index] = entry
        hash_table.used += 1
      end
    end
    alias []= set
  end
end

listing 6.13: The set method in the Dict class

The method is very similar to the pseudo code we looked at earlier in add_or_update_key_value_pair. We first obtain the index for the key, aka the location of the bucket the key should go into. We then perform a rehash step if we're in rehashing state.

We select which table the key will end up in depending on whether or not we are in rehashing state. If we are, the key will be added to the rehashing table, otherwise it is added to the main table.

The next step is to inspect the bucket at the index position, if it is not empty we look through all the items in the bucket to check if one of the entries has the same key.

If we find an entry with the same key, we don't need to create a new entry, we can just update the existing entry with the new value. The number of elements in the dictionary does not change.

On the other hand, if we didn't find an entry with the same key, we need to create a new entry for the new key/value pair. First, we instantiate DictEntry with key & value, then we need to add it to the linked list.

There are essentially two valid options for us here, either append the element to the list or prepend it. Prepending to a linked list is a constant time operation O(1), whereas appending is proportional to the size of list, it is an O(n) operation. Let's illustrate with an example, we'll start by creating a list with three elements

entry1 = BYORedis::DictEntry.new(1, 'value-1') # => nil
entry2 = BYORedis::DictEntry.new(2, 'value-2') # => nil
entry3 = BYORedis::DictEntry.new(3, 'value-3') # => nil
entry1.next = entry2 # => nil
entry2.next = entry3 # => nil
entry1 # => #<BYORedis::DictEntry:0x00007fd57403dbc0 @key=1, @value="value-1", @next=#<BYORedis::DictEntry:0x00007fd577a3eb58 @key=2, @value="value-2", @next=#<BYORedis::DictEntry:0x00007fd5779c3d90 @key=3, @value="value-3", @next=nil>>>

Now that the list is created, let's assume we only have a single variable for the list, list, which is equal to entry1 in the previous example, prepending a new node is done as follows:

new_entry = BYORedis::DictEntry.new(99, 'value-99') # => nil
new_entry.next = list # => nil
list = new_entry # => nil
list # => #<BYORedis::DictEntry:0x00007fd5779b8878 @key=99, @value="value-99", @next=#<BYORedis::DictEntry:0x00007fd57403dbc0 @key=1, @value="value-1", @next=#<BYORedis::DictEntry:0x00007fd577a3eb58 @key=2, @value="value-2", @next=#<BYORedis::DictEntry:0x00007fd5779c3d90 @key=3, @value="value-3", @next=nil>>>>

This process can be repeated regardless of the size of the list. On the other hand, to append an element, we first need to find the last element, here is how we do it:

new_entry = BYORedis::DictEntry.new(99, 'value-99') # => nil
current_entry = list # => nil
current_entry = current_entry.next while current_entry.next != nil # => nil
current_entry.next = new_entry # => nil
list # => #<BYORedis::DictEntry:0x00007fd57403dbc0 @key=1, @value="value-1", @next=#<BYORedis::DictEntry:0x00007fd577a3eb58 @key=2, @value="value-2", @next=#<BYORedis::DictEntry:0x00007fd5779c3d90 @key=3, @value="value-3", @next=#<BYORedis::DictEntry:0x00007fd5779b8878 ...>>>>

The third line iterates through all the elements in the list. The more elements, the longer it'll take. Some linked list implementations work around this issue by always maintaining a pointer to the last element.

Back to our collision handling, even though appending has a worst time complexity compared to prepending, O(n) vs O(1), given that we need to iterate through all the items in the list anyway, to make sure the key does not already exist, we could have kept a variable pointing at the last element.

Redis chooses the prepend option with the assumption that it might be common for recently added items to be accessed more frequently. If this assumption holds true, future lookups would find the key/value pair early in the list, instead of having to iterate through the end of the list.

The lines entry.next = hash_table.table[index] & hash_table.table[index] = entry insert the new entry as the head of the list. The first one makes the new entry point at the old head of the list. If it was null, then the next value will still be null and the new entry will be the only element in the list. If it wasn't null, then next in the new entry now points at the old head, which itself, points at the rest of the list. We're not losing anything, great!
The second line is necessary so that the head of list in the bucket points at the new entry.

The last line, hash_table.used += 1, increments the counter of how many items are in the dictionary.

Finally, we use the Ruby alias keyword to create an alias for set to []=. This allows us to use a Dict instance similarly to how we use a Ruby Hash:

dict = Dict.new
# The following two are equivalent, the same method is called, Dict#add:
dict.set(1, 2)
dict[1] = 2

We used a few private methods that we should now look more closely at: main_table, rehashing_table & key_index

# dict.rb
module BYORedis
  class Dict

    # ...
    private

    def main_table
      @hash_tables[0]
    end

    def rehashing_table
      @hash_tables[1]
    end

    def key_index(key)
      expand_if_needed
      hash = SipHash.digest(RANDOM_BYTES, key)
      index = nil

      iterate_through_hash_tables_unless_rehashing do |hash_table|
        index = hash & hash_table.sizemask
      end

      index
    end

    # In the Redis codebase, they extensively use the following pattern:
    # for (table = 0; table <= 1; table++) {
    #   ...
    #   if (!dictIsRehashing(d)) break;
    # }
    # This is common for many operations, such as finding or deleting an item in the dict,
    # we first need to look at the main table, the first table, but we haven't found in the
    # first one, we should look in the rehashing table, the second one, but only if we're in
    # the process of rehashing.
    # Taking advantage of Ruby blocks, we can write this helper method instead
    def iterate_through_hash_tables_unless_rehashing
      @hash_tables.each do |hash_table|
        yield hash_table
        break unless rehashing?
      end
    end
  end
end

listing 6.14: helper methods in the Dict class

main_table and rehashing_table are used as aliases of @hash_tables[0] and @hash_tables[1]. key_index is used to return the index representing the location of the bucket for the given key. It first calls expand_if_needed, which we'll explore in the next section about rehashing. Once the dictionary has been resized if needed, it computes the hash value using the SipHash class. The code for the siphash algorithm is available in Appendix B.

Once we obtained the hash value, we need to convert it to an index within the boundaries of the backing array, with the modulo operation, or as we discussed earlier, the bitwise AND operation. Before doing that, we need to take into account which table the bucket should go into, if we're not in a rehashing state, it should go in the main table, if we are, it should go to the rehashing table.

This process of first inspecting the main table, and the rehashing table, only if we're in rehashing state is so common that we added a helper method for that, iterate_through_hash_tables_unless_rehashing.

This method replaces the common pattern in the Redis C codebase using a for loop and a conditional break statement at the end of the first iteration. We instead leverage the Ruby block syntax to always yield back the main table to the caller, and only yield the rehashing table if we're in a rehashing state.

The implication for key_index is that if we're in a rehashing state, we'll first find the index in the first table, but iterate_through_hash_tables_unless_rehashing will yield a second time and index will end up being an index for the rehashing table.

The digest method in the SipHash class requires a 16-byte key composed of random bytes. Redis does generates these with the getRandomBytes function, which attempts to use /dev/urandom/ and defaults to a weaker seed based on the current time and the pid of the server if /dev/urandom is not accessible. Ruby's SecureRandom module provides a random_bytes method which uses /dev/urandom under the hood, so let's use it:

# server.rb
require 'socket'
require 'logger'
require 'strscan'
require 'securerandom'

LOG_LEVEL = ENV['DEBUG'] ? Logger::DEBUG : Logger::INFO
RANDOM_BYTES = SecureRandom.bytes(16)
# ...

listing 6.15: Initialization of the random byte in server.rb

Now that we implemented the set method, and its alias, []=, we need to add the get method, which will be used by the GET command, to retrieve an element from a dictionary based on its key.

module BYORedis
  class Dict
    def get(key)
      return if main_table.used == 0 && rehashing_table.used == 0

      rehash_step if rehashing?

      hash = SipHash.digest(RANDOM_BYTES, key)

      iterate_through_hash_tables_unless_rehashing do |hash_table|
        index = hash & hash_table.sizemask

        entry = hash_table.table[index]

        while entry
          return entry.value if entry.key == key

          entry = entry.next
        end
      end
    end
    alias [] get

    def include?(key)
      !get(key).nil?
    end

    def each
      return if main_table.used == 0 && rehashing_table.used == 0

      start_index = rehashing? ? @rehashidx : 0
      main_table.table[start_index..-1].each do |bucket|
        next if bucket.nil?

        yield bucket.key, bucket.value until bucket.next.nil?
      end
      return unless rehashing?

      rehashing_table.each do |bucket|
        next if bucket.nil?

        yield bucket.key, bucket.value until bucket.next.nil?
      end
    end
  end
end

listing 6.16: get, include? and each method in Dict.rb

The get method starts with an early return statement if both tables are empty. If that's the case, there's no need to continue, we know the key is not present in the table.

The next step is similar to add, we perform a single rehash step if we're in a rehashing state. The approach allows Redis to incrementally work its way through the rehashing process, without affecting too much the performance of other operations. A single rehashing step does not require a lot of work, and will have a negligible impact on the performance of get, but it has the important benefits of advancing the rehashing process.

Once again, we follow a pattern similar to the pseudo code lookup_key from earlier in the chapter, we start by computing the hash value of the key. Once we have the hash value, we first need to look at the main table, and if we're in a rehashing state, at the second table as well. We do this with the same helper method as previously, iterate_through_hash_tables_unless_rehashing. hash & hash_table.sizemask returns the location of the bucket for the key. There might be more than one item in the bucket, because of potential collisions, so we need to iterate through all of them and compare their key with the key argument. We do this with the while loop. If we do find a matching key, we abort early and return the value associated with that key.

Similarly to what we did with set and []=, we alias get to [], which allows us to use the same syntax we use for Hash instances:

dict = Dict.new
# The following are equivalent, the same method is called:
dict.get(1)
dict[1]

There is a use case where we only care about the presence of a key in a dictionary, but not about the value, this is what the include? method does. We do not need to reimplement the logic, we reuse the get method, discard its value and return a boolean instead.

Finally we add the each method, which will yield all the key/value pairs in the hash table. This time we do not use the iterate_through_hash_tables_unless_rehashing. This is because we're using a small optimization technique to avoid iterating over buckets we know are empty.

If we are in rehashing state, then we know that all buckets between 0 and the value in @rehashidx excluded have been rehashed and the values are empty. In a table of size 16, if @rehashidx is set to 5, it means that the buckets at index 0, 1, 2, 3 & 4 have been migrated and we don't need to inspect them.

For the rehashing table, we iterate through the backing array with the each method on HashTable, which itself delegates each to its Array instance. There is no need to yield back empty buckets, so we continue to the next bucket if we find an empty one. For each non empty bucket, we iterate through the linked list of entries and yield all of them.

Resizing and rehashing

When Dict is initialized, it creates two instances of HashTable, each with an empty array. We need to implement the rehashing mechanism to allow the tables to grow and shrink as needed. The add method we discussed in the previous section starts by calling expand_if_needed:

# dict.rb
module BYORedis
  class Dict

    INITIAL_SIZE = 4
    # ...

    def expand_if_needed
      return if rehashing?

      if main_table.empty?
        expand(INITIAL_SIZE)
      elsif main_table.used >= main_table.size
        expand(main_table.used * 2)
      end
    end
    # ...
  end
end

listing 6.17: expand_if_needed in the Dict class

If we are already in the process of rehashing the dict, then we can abort early, the process will continue incrementally, there's nothing else we should do until rehashing is over.
If the table is empty we call expand with INITIAL_SIZE, which is set to 4, otherwise we call it with a capacity set to twice the current size of the dictionary.

Note that the used member can be greater than the size member. size determines the size of the array in the hash table, in other words how many buckets are in the table, and each bucket can contain more than one value because it uses a linked list to store elements. This means that we might increase by more than twice the current size, since we rely on the number of items in the dict, not the current size, to determine the new size.

Let's now take a look at the expand method:

# dict.rb
def expand(size)
  return if rehashing? || main_table.used > size

  real_size = next_power(size)

  return if real_size == main_table.size

  new_hash_table = HashTable.new(real_size)

  # Is this the first initialization? If so it's not really a rehashing
  # we just set the first hash table so that it can accept keys.
  if main_table.table.nil?
    @hash_tables[0] = new_hash_table
  else
    @hash_tables[1] = new_hash_table
    @rehashidx = 0
  end
end

def next_power(size)
  # Ruby has practically no limit to how big an integer can be, because under the hood the
  # Integer class allocates the necessary resources to go beyond what could fit in a 64 bit
  # integer.
  # That being said, let's still copy what Redis does, since it makes sense to have an
  # explicit limit about how big our Dicts can get
  i = INITIAL_SIZE
  return MAX_SIZE if size >= MAX_SIZE

  loop do
    return i if i >= size

    i *= 2
  end
end

def rehashing?
  @rehashidx != -1
end

listing 6.18: expand, next_power and rehashing? methods in the Dict class

Similarly to resize, if we're already in the process of rehashing, we can abort early. We also abort early if the number of items in the array is greater than the new size. In this case, there's no point in resizing the table, it would be too small.

We use the next_power method to find the next power of two that is greater or equal to the new size. This method is used to maintain the size of the array as of power of two to leverage the bitwise AND operation to compute a modulo.

We then perform another sanity check, if the "real size", the power of two, is equal to the current size, then there's no point in going through a rehashing process, the table is already at the desired size.

Now that we performed all the checks, we create a new hash table of the desired size and make it the rehashing table by assigning it to @hash_tables[1]. We do have to consider the case where this is the first call to expand, in which case both tables are empty, and there's nothing to rehash so we directly assign the newly created hash table to the main table, @hash_tables[0].

If there are elements to rehash, we flag the dict as being in a rehashing state by setting @rehashidx to 0. This instance variable acts as a pointer to the next element that should be rehashed in the main table. By being different than -1, the dict is now considered in rehashing state. This is what the rehashing? method does.

In the case where there are elements to rehash, the expand method does not actually rehash any of them. It only prepares the dictionary for rehashing. The actual rehashing is performed in two different places.

Many operations, such as set and get, will perform a single rehashing step before performing the rest of the operation. The other place where rehashing happens is through the server_cron time event we discussed in Chapter 3.

# dict.rb
def rehash_step
  rehash(1)
end

def rehash(n)
  empty_visits = n * 10
  return 0 unless rehashing?

  while n > 0 && main_table.used != 0
    n -= 1
    entry = nil

    while main_table.table[@rehashidx].nil?
      @rehashidx += 1
      empty_visits -= 1
      return 1 if empty_visits == 0
    end

    entry = main_table.table[@rehashidx]

    while entry
      next_entry = entry.next
      idx = SipHash.digest(RANDOM_BYTES, entry.key) & rehashing_table.sizemask

      entry.next = rehashing_table.table[idx]
      rehashing_table.table[idx] = entry
      main_table.used -= 1
      rehashing_table.used += 1
      entry = next_entry
    end
    main_table.table[@rehashidx] = nil
    @rehashidx += 1
  end

  # Check if we already rehashed the whole table
  if main_table.used == 0
    @hash_tables[0] = rehashing_table
    @hash_tables[1] = HashTable.new(0)
    @rehashidx = -1
    0
  else
    # There's more to rehash
    1
  end
end

listing 6.19: rehashing related methods in the Dict class

rehash_step calls rehash with the parameter 1. The parameter to rehash dictates how many items it will rehash, that is, move from the main table to the rehashing one. Let's look at the method one line at a time.

We start by initializing the empty_visits variables to n * 10. n, the method parameter is the number of elements we want to move to the rehashing table. Because of the nature of a hash table, when iterating through the buckets, from the one at index 0 until the one at index size - 1, we don't know how many empty buckets we'll find. In order to prevent any potential slowness in the rehashing process, Redis uses an upper bound for the number of empty visits.

This is especially important when reducing the size of a hash table, which we'll explore next. In this case the table we're rehashing might be very scarcely populated and there might be many consecutive empty buckets. By using empty_visits, we ensure that the rehashing step has a known upper bound.

The next step is a sanity check, if we are not rehashing, we should exit right away, there's nothing else to do. Note that we're using the same semantics as Redis here, a return value of 0 indicates that all elements are rehashed, 1 indicates that there are more elements to rehash.

The while loop is the main piece of the method, it iterates as long a n is greater than 0 and as long as there are items left in the main table. The second part of the condition is a quick shortcut we attempt to take at the end of the rehashing process. After rehashing the last element in the main table, there might still be a few extra empty buckets in the array, but since we know the table does not have any more elements left by looking at the used variable, we stop instead of iterating through the last empty buckets.

The first step of the loop is to decrement n, this is a very important part of the loop that dictates the number of iterations. We also initialize an entry variable that will be used to iterate through the entries of a bucket.

@rehashidx is initialized at 0 when the rehashing process starts, so we use it as the index to look through the buckets in the main table. As long as we find empty buckets, we increment the empty_visits variable and increment @rehashidx to keep moving along in the main table. If empty_visits reaches 0, we stop the rehashing process, we're already spent too much time iterating through empty buckets. Note that the next call to rehash will be able to continue where we left off since it will use @rehashidx.

This inner while loop will stop as soon as we encounter a non empty bucket, and we set entry to the first entry of the bucket when that happens. As we already established, we might be dealing with more than one entries, so we use another inner while loop to iterate through all the entries in the linked list.

For each of these entries, we first store the following entry in the next_entry variable and then compute the index of the entry in the rehashing table.

The next few lines perform the actual move of the entry from one table to the other, similarly to how we inserted an entry in the add method earlier. We make the next attribute of the entry variable point at the bucket in the rehashing table with entry.next = rehashing_table.table[idx]. This might either set the value to nil if the bucket in the rehashing table was empty, or would otherwise be the head of the list if it was not empty.

rehashing_table.table[idx] = entry is taking care of making the bucket in the rehashing table point at the entry variable, making it the new head of the list.

We then decrement used in the main table and increment it in the rehashing table.

Finally, in order to keep iterating through the entries in the bucket from the main table, we set entry to next_entry, which will repeat these steps with the next entry in the list.

Once we've moved all the entries in the bucket, one rehashing step is completed, we explicitly empty the bucket with main_table.table[@rehashidx] = nil and increment @rehashidx so that the next iteration of the outer while loop continues iterating through the main table.

Once n rehashing steps have been performed, we check a few final details. If used reached 0 in the main table, we're done with the rehashing process, the main table is empty. We set @hash_tables[0] to the new rehashing table, effectively promoting it from rehashing table to main table, and reset the rehashing table to an empty table. Resetting @rehashidx to -1 marks the end of the rehashing process.

We return 1 otherwise, indicating that the rehashing process is not done.

The use of rehash_step makes sure that rehashing keeps happening, one step at a time, but large tables might require many rehash steps, up to millions of steps. Redis uses the server_cron time event to perform rehashing in larger steps, in order to speed up the rehashing process. This is especially useful if the server is idle for a while. If Redis would only perform rehashing steps when receiving commands, it would not make any progress while idle. This approach prevents this and takes advantage of the idle time to do so cleanup work.

Let's add a call to a new method, databases_cron in server_cron in server.rb:

# server.rb

def server_cron
  # ...

  databases_cron

  1000 / DEFAULT_FREQUENCY
end

def databases_cron
  @data_store.resize if ht_needs_resize(@data_store)
  @expires.resize if ht_needs_resize(@expires)

  @data_store.rehash_milliseconds(1)
  @expires.rehash_milliseconds(1)
end

def slots(dict)
  dict.hash_tables[0].size + dict.hash_tables[1].size
end

def size(dict)
  dict.hash_tables[0].used + dict.hash_tables[1].used
end

def ht_needs_resize(dict)
  size = slots(dict)
  used = size(dict)

  size > Dict::INITIAL_SIZE && ((used * 100) / size < HASHTABLE_MIN_FILL)
end

listing 6.20: databases_cron method in the Server class

databases_cron performs two operations on the two dictionaries in the Server class, @data_store, that holds all the key/value pairs, and @expires which keeps track of the keys with TTLs. For each of these dictionaries, if first calls resize, which we've explored in the previous section, only if it thinks the dictionary needs to be resized.

A dictionary needs to be resized if the number of items in the dictionary is less than 10% of the size of the dictionary. Beside the EX & PX options, we have not yet implemented commands to delete keys, such as the DEL command, but we will soon. Once the DEL command is implemented, key/value pairs can be removed from a dictionary, meaning that a dictionary that grew to a given size might become too big at some point. Let's look at an example:

dict = BYORedis::Dict.new
dict.hash_tables[0].size # => 0
dict.hash_tables[1].size # => 0
dict.add("1", "2") # => 1
dict.hash_tables[0].size # => 4
# We're calling to_s because Dict only works with strings and i is an Integer
100.times { |i| dict.add(i.to_s, i.to_s) } # => 100
dict.hash_tables[0].size # => 128

In the previous examples, the main table grew to a size of 128, if we were to delete all items but one, we would end up with only one non empty bucket, while the other 127 would be empty.

With small values like those it might not seem like a big difference, but as we saw earlier, Redis' dict can grow to billions of keys, so efficiently using memory is important.

If the dictionaries need resizing, calling resize will find a good size for the dictionary and start the rehashing process.

The next step is to call rehash_milliseconds. We've already explained why it's important for Redis to maintain an upper bound to all the operations that blocks other clients in the queue. By calling rehash_milliseconds(1), Redis tries to do as much rehashing as possible and stops once one millisecond has elapsed. This means that in the most pessimistic scenario possible, a client would send a command right after Redis enters the rehash_milliseconds(1) call, effectively blocking this client for at least one millisecond. This behavior is considered acceptable by Redis, but it can be disabled through the activerehashing config value. We have not yet implemented any form of configuration so we will assume that all clients of our server are ok with this behavior.

# dict.rb
def rehash_milliseconds(millis)
  start = Time.now.to_f * 1000
  rehashes = 0
  while rehash(100) == 1
    rehashes += 100
    time_elapsed = Time.now.to_f * 1000 - start

    break if time_elapsed > millis
  end
  rehashes
end

def resize
  return if rehashing?

  minimal = main_table.used
  minimal = INITIAL_SIZE if minimal < INITIAL_SIZE

  expand(minimal)
end

listing 6.21: rehash_milliseconds and resize methods in the Dict class

No more `Hash` & `{}`

Equipped with our Dict class, we can now remove all instances of Hash and replace them with Dict. We lose a little bit syntax wise, we cannot use the Hash literal syntax with { key: value }, but overall, things are not that different:

# server.rb
module BYORedis
  class Server

    COMMANDS = Dict.new
    COMMANDS.set('command', CommandCommand)
    COMMANDS.set('del', DelCommand)
    COMMANDS.set('get', GetCommand)
    COMMANDS.set('set', SetCommand)
    COMMANDS.set('ttl', TtlCommand)
    COMMANDS.set('pttl', PttlCommand)
    # ...

    def initialize
      @logger = Logger.new(STDOUT)
      @logger.level = LOG_LEVEL

      @clients = []
      @data_store = Dict.new
      @expires = Dict.new

      # ...
    end

    # ...
  end
end

listing 6.22: Replacing usages of Hash with Dict in the Server class

The SetCommand class was using a Hash to store the configuration values of the possible options. We replace it similarly to what we did for the COMMANDS constant in server.rb:

# set_command.rb
module BYORedis
  class SetCommand
    OPTIONS = Dict.new
    OPTIONS.set(
      'ex',
      CommandOptionWithValue.new(
        'expire',
        ->(value) { validate_integer(value) * 1000 },
      )
    )
    OPTIONS.set(
      'px',
      CommandOptionWithValue.new(
        'expire',
        ->(value) { validate_integer(value) },
      )
    )
    OPTIONS.set(
      'xx', CommandOption.new('presence')
    )
    OPTIONS.set(
      'nx', CommandOption.new('presence')
    )
    OPTIONS.set(
      'keepttl', CommandOption.new('expire')
    )

    # ...
    def initialize(data_store, expires, args)
      # ...
      @options = Dict.new
    end
    # ...
  end
end

listing 6.23: Replacing usages of Hash with Dict in the SetCommand class

Adding the `DEL` command

As mentioned earlier in this chapter, the only option we currently have for keys to be deleted is to set them with a TTL with the EX & PX options. It will be more convenient to manually test the behavior of our new Dict class with a more explicit option. The DEL command is very useful in that aspect, it accepts one or more keys as its arguments and attempts to delete them. It returns an integer representing the number of keys that were deleted:

module BYORedis
  class DelCommand

    def initialize(data_store, expires, args)
      @data_store = data_store
      @expires = expires
      @args = args
    end

    def call
      if @args.empty?
        RESPError.new("ERR wrong number of arguments for 'GET' command")
      else
        keys = @args
        deleted_count = 0
        keys.each do |key|
          entry = @data_store.delete(key)
          if entry != nil
            # If there was a key, then we need to delete its TTL if it had one:
            @expires.delete(key)
            deleted_count += 1
          end
        end

        RESPInteger.new(deleted_count)
      end
    end

    def self.describe
      [
        'del',
        -2, # arity
        # command flags
        [ RESPSimpleString.new('write') ],
        1, # position of first key in argument list
        -1, # position of last key in argument list
        1, # step count for locating repeating keys
        # acl categories: https://github.com/antirez/redis/blob/6.0.0/src/server.c#L161-L166
        [ '@keyspace', '@write', '@slow' ].map { |s| RESPSimpleString.new(s) },
      ]
    end
  end
end

listing 6.24: The new DelCommand class

The DelCommand class implements the behavior of the command, as well as defining the data for the COMMAND command, but it mainly relies on a non existing method on the Dict class, delete. Let's add it:

# dict.rb
def delete(key)
  return if main_table.used == 0 && rehashing_table.used == 0

  rehash_step if rehashing?

  hash_key = SipHash.digest(RANDOM_BYTES, key)
  iterate_through_hash_tables_unless_rehashing do |hash_table|
    index = hash_key & hash_table.sizemask
    entry = hash_table.table[index]
    previous_entry = nil

    while entry
      if entry.key == key
        if previous_entry
          previous_entry.next = entry.next
        else
          hash_table.table[index] = entry.next
        end
        hash_table.used -= 1
        return entry
      end
      previous_entry = entry
      entry = entry.next
    end
  end
end

listing 6.25: The delete method in the Dict class

The logic in delete is similar to the add method. We start with a shortcut we've already seen, if the table is empty, there's no need to go any further, we can stop right there.

We perform a rehashing step if needed, to keep the rehashing process moving and then we get the hash value for the key. The key we're attempting to delete might be in the main table or the rehashing table, so we use our helper method iterate_through_hash_tables_unless_rehashing to help with this.

For each hash table we look at, we obtain the index and inspect the bucket at that location. If the bucket is empty, then there's nothing else to do, the key is not present in this table. If the bucket is not empty, we need to iterate through all the entries and compare the keys of these entries with the method argument. If there's a match we found the entry we need to delete, this is what the while loop is doing.

If there is a match, the if entry.key == key condition, we need to handle two cases, whether or not the entry we want to delete is the first entry in the bucket. We use the previous_entry variable to keep track of the previous entry as we iterate through the entry list in the bucket. Once we find a match, this allows us to know if the entry we're trying to delete was the first one or not, if previous_entry is nil, then it is the first one.

In this case the deleting process is to make the head of the bucket, hash_table.table[index], point at the next element in the list. If entry was the only element, its next value would be nil, and the bucket would now be empty. If entry was the head of list with more than one element, then the bucket now starts with what was the second element in the list, the entry which next is to in entry.

If previous_entry is not nil, then entry is not at the head of the list and the deletion process requires us to change previous_entry.next to now skip over entry since we want to remove it from the list, and point to the entry in entry.next. This entry might be nil or not, and it does not matter to us. If it was nil, then it means that entry was the last element in the list, and by changing previous_entry.next to nil, previous_entry is now the last entry. If it wasn't nil, then previous_entry now points to the element entry is pointing to, and no entries point at entry anymore, it's now floating on its own and is not part of the list anymore.

In a language with manual memory management, such as C, we would also need to explicitly free the memory, but we're in Ruby, so we'll let the garbage collector do its job.

Whether or not the entry was the first one in the bucket, it's now deleted, we decrement the used counter and return.

Conclusion

In this chapter we dived into hash tables, how Redis uses them and how to implement one in Ruby, in order to replace the Hash class.

In the next Chapter we will add new commands to handle a new data type, Lists. See you there.

Appendix A: Array from scratch in Ruby

This implementation use the fiddle gem, that comes with Ruby. Fiddle provides an interface to libffi, which is a C library that allows code in one language to call code in another language. Here we use the Fiddle::Pointer class with the malloc method to explicitly allocate memory.

This implementation only works with strings and does not support resizing.

PTR_SIZE is constant set to ::Fiddle::SIZEOF_LONG, which is 8 on most modern machines. This size is in bytes, and since a byte holds 8 bits, we can see that a long is 64 bits here.

When initialized, we call Fiddle::Pointer.malloc(PTR_SIZE * size) to allocate enough memory to store n pointers, where n is the given size.

Adding an element, beside the sanity checks around the size and the type requires the following step:

We obtain a pointer to the given string with Pointer.to_ptr(str)
We compute the location of the next element in the array and save it in offset. If the array is empty it's gonna be 0, if there's one element, it'll be 8, etc ...
We use the []= notation of the Pointer class to write the 8 bytes representing the address of the given string, which is what the value of ptr.ref. This is similar to the & operator used to return the address of variable.
We increment the size by one.

The get method is similar, we transform the index to an offset by multiplying it by the size of a pointer, 8. And we read a whole pointer, 8 bytes, from there, with the [] method of the Pointer class.

We use .unpack('Q') on the result because the [] method of Pointer returns a string of 8 bytes, representing the 64-bit integer holding the address of the string stored in the array. But we need the actual integer value, to pass to Pointer.new, and that's what unpack('Q') does, it "unpacks" the string to a long long, a 64-bit integer, and returns that as an array, so we grab the first element with [0].

to_s on Pointer is smart enough to read all the bytes until it finds a null byte and return the result as a Ruby string, great, so we don't have to deal with that!

require 'fiddle'

class BYOArray < BasicObject

  PTR_SIZE = ::Fiddle::SIZEOF_LONG

  def initialize(max_size)
    @max_size = max_size
    @current_size = 0
    @beginning_address = ::Fiddle::Pointer.malloc(PTR_SIZE * max_size)
  end

  def add(str)
    ::Kernel.raise 'Array is full' if @current_size == @max_size
    ::Kernel.raise 'Expected a string' unless str.is_a?(::String)

    ptr = ::Fiddle::Pointer.to_ptr(str)
    offset = @current_size * PTR_SIZE # 0 at first, then 8, 16, etc ...

    @beginning_address[offset, PTR_SIZE] = ptr.ref

    @current_size += 1
    self
  end

  def get(i)
    return nil if i < 0 || i >= @current_size

    address = @beginning_address[i * PTR_SIZE, PTR_SIZE].unpack('Q')[0]
    ::Fiddle::Pointer.new(address).to_s
  end

  def to_s
    "Size: #{ @current_size }"
  end
end

listing 6.26: A Ruby implementation of an Array structure

ary = BYOArray.new(2)
::Kernel.puts ary.add("foo")
::Kernel.puts ary.add("bar")

::Kernel.puts ary.get(0)

::Kernel.puts ary.get(1)
::Kernel.puts ary.get(2)
::Kernel.puts ary.add("bar")

Appendix B: A Ruby implementation of SipHash

The following is adapted from the siphash gem. The gem implements the 2-4 version, and the version below implements the same one Redis uses, 1-2. These two numbers, named c & d in the siphash paper represent the number of compression steps and finalization steps. Variants with higher number of compression and finalization steps are supposed to provide a higher security at the cost of being slower. The following is a quote from the Redis implementation:

We use SipHash 1-2. This is not believed to be as strong as the suggested 2-4 variant, but AFAIK there are not trivial attacks against this reduced-rounds version, and it runs at the same speed as Murmurhash2 that we used previously, why the 2-4 variant slowed down Redis by a 4% figure more or less.

# Credit to https://github.com/emboss/siphash-ruby/blob/master/lib/siphash.rb
class SipHash

  # Ruby's Integer class allows numbers by going passed the max value of a 64 bit integer,
  # by encoding the value across multiple 64-bit integers under the hood. In order to make
  # sure that the values we deal with stay within the 64-bit range, we use the following
  # constant as the right operand of an AND bitwise operation
  MASK_64 = 0xffffffffffffffff

  def self.digest(key, msg, compress_rounds: 1, finalize_rounds: 2)
    new(key, msg, compress_rounds, finalize_rounds).digest
  end

  def initialize(key, msg, compress_rounds, finalize_rounds)
    @msg = msg
    @compress_rounds = compress_rounds
    @finalize_rounds = finalize_rounds

    # These are the four 64-bit integers of internal state. The initial values are based on the
    # arbitrary string: "somepseudorandomlygeneratedbytes"
    # "somepseu".split('').map(&:ord).map { |b| b.to_s(16) }.join # => "736f6d6570736575"
    # The 64-bit value is 8317987319222330741, its binary representation is:
    # 0111 0011 0110 1111 0110 1101 0110 0101 0111 0000 0111 0011 0110 0101 0111 0101
    # Which we can obtain with
    # "736f6d6570736575".scan(/../).map { |h| h.hex } # => [115, 111, 109, 101, 112, 115, 101, 117]
    # [115, 111, 109, 101, 112, 115, 101, 117].pack('c8') # => "somepseu"
    # "somepseu".unpack('Q>') # => 8317987319222330741
    # '%064b' % 8317987319222330741 # =>
    # 0111 0011 0110 1111 0110 1101 0110 0101 0111 0000 0111 0011 0110 0101 0111 0101
    #
    # Note that we used 'Q>' which tells unpack to assume big-endianness. Using the default of
    # Q< would have returned the same 8 bytes but in the opposite order, from right to left:
    # "somepseu".unpack('Q<') # => 8459294401660546931
    # '%064b' % 8459294401660546931 # =>
    # 0111 0101 0110 0101 0111 0011 0111 0000 0110 0101 0110 1101 0110 1111 0111 0011
    # These are the same bytes but in different order, the character s is the following 8 bits:
    # ('%08b' % 's'.ord).scan(/..../).join(' ') # => "0111 0011"
    # And the character o is:
    # ('%08b' % 'o'.ord).scan(/..../).join(' ') # => "0110 1111"
    #
    # We can see that in the big endian version, these two bytes are on the left side, and
    # they're on the right side with the little endian version
    #
    # "dorandom".split('').map(&:ord).map { |b| b.to_s(16) }.join # => "646f72616e646f6d"
    # "lygenera".split('').map(&:ord).map { |b| b.to_s(16) }.join # => "6c7967656e657261"
    # "tedbytes".split('').map(&:ord).map { |b| b.to_s(16) }.join # => "7465646279746573"
    @v0 = 0x736f6d6570736575
    @v1 = 0x646f72616e646f6d
    @v2 = 0x6c7967656e657261
    @v3 = 0x7465646279746573

    # The key argument is a 16 byte string, which we want to unpack to two 64-bit integers.
    # A byte contains 8 bits, so one 64-bit integer is composed of 8 bytes, and one 16 byte
    # string can be unpacked to two 64-bit integers.
    # The first line grabs the first 8 bytes with the slice method, and calls unpack with the
    # Q< argument. Q means that Ruby will attempt to unpack as a long, and < is the explicit
    # way of telling it to unpack it as little endian, which is the default on many modern CPUs
    k0 = key.slice(0, 8).unpack('Q<')[0]
    # This line does the same thing with the last 8 bytes of the key
    k1 = key.slice(8, 8).unpack('Q<')[0]

    # The ^ Ruby operator is the XOR bitwise operation, a ^= b is equivalent to a = a ^ b
    # These four lines initialize the four 64-bit integers of internal state with the two
    # parts of the secret, k0 and k1
    @v0 ^= k0
    @v1 ^= k1
    @v2 ^= k0
    @v3 ^= k1
  end

  def digest
    iter = @msg.size / 8

    # Compression step
    iter.times do |i|
      m = @msg.slice(i * 8, 8).unpack('Q<')[0]
      compress(m)
    end

    # Compression of the last characters
    m = last_block(iter)
    compress(m)

    # Finalization step
    finalize

    # Digest result
    @v0 ^ @v1 ^ @v2 ^ @v3
  end

  private

  # This function might look a little bit complicated at first, it implements this section of
  # the compression part of paper, 2.2:
  # where m w-1 includes the last 0 through 7 bytes of m followed by null bytes and ending with a
  # byte encoding the positive integer b mod 256
  def last_block(iter)
    # We initialize last as a 64-bit integer where the leftmost byte, the 8 bits to the left are
    # set to the length of the input. 8 bits can only encode a value up to 255, so if the length
    # is more than 255, the extra bites are discarded, for a length of 11, last would look like
    # 0000 1011 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 000 00000
    # 11 is encoded as 1011 as binary, calling << 56 on it pushes it 56 bits to the left
    last = (@msg.size << 56) & MASK_64

    left = @msg.size % 8
    off = iter * 8

    # At this point, we've applied the compress step to all the 8 byte chunks in the input
    # string, but if the length was not a multiple of 8, there might be between 1 & 7 more bytes
    # we have not compressed yet.
    # For instance, if the string was 'hello world', the length is 11. The digest method would
    # have computed an iter value of 1, because 11 / 8 => 1, and called compress once, with the
    # first 8 bytes, obtained with the slice method, 'hello world'.slice(0, 8) => 'hello wo'
    # We still need to compress the last 3 bytes, 'rld'
    # In this example, left will be 3, and off 8
    # Note that this case/when statement is written to optimize for readability, there are ways
    # to express the same instructions with less code
    case left
    when 7
      last |= @msg[off + 6].ord << 48
      last |= @msg[off + 5].ord << 40
      last |= @msg[off + 4].ord << 32
      last |= @msg[off + 3].ord << 24
      last |= @msg[off + 2].ord << 16
      last |= @msg[off + 1].ord << 8
      last |= @msg[off].ord
    when 6
      last |= @msg[off + 5].ord << 40
      last |= @msg[off + 4].ord << 32
      last |= @msg[off + 3].ord << 24
      last |= @msg[off + 2].ord << 16
      last |= @msg[off + 1].ord << 8
      last |= @msg[off].ord
    when 5
      last |= @msg[off + 4].ord << 32
      last |= @msg[off + 3].ord << 24
      last |= @msg[off + 2].ord << 16
      last |= @msg[off + 1].ord << 8
      last |= @msg[off].ord
    when 4
      last |= @msg[off + 3].ord << 24
      last |= @msg[off + 2].ord << 16
      last |= @msg[off + 1].ord << 8
      last |= @msg[off].ord
    when 3
      # In the example documented above, this is the branch of the case/when we would
      # end up in. last is initially set to:
      # 0000 1011 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 000 00000
      # @msg[off + 2] is the character d, calling ord returns its integer value, 100
      # We shift it 16 bits to the left, effectively inserting as the 6th byte starting
      # from the left, or third from the right:
      # 0000 1011 0000 0000 0000 0000 0000 0000 0000 0000 0110 0100 0000 0000 0000 0000
      last |= @msg[off + 2].ord << 16
      # @msg[off + 1]is the character l, ord returns 108, and we insert it as the 7th byte
      # from the left, or second from the right:

      last |= @msg[off + 1].ord << 8
      # Finally, @msg[off] is the character r, with the ord value 100, and we insert it as the
      # byte from the left, or first from the right:
      # 0000 1011 0000 0000 0000 0000 0000 0000 0000 0000 0110 0100 0110 1100 0111 0010
      last |= @msg[off].ord
    when 2
      last |= @msg[off + 1].ord << 8
      last |= @msg[off].ord
    when 1
      last |= @msg[off].ord
    when 0
      last
    else
      raise "Something unexpected happened with the r value: #{ r }, should be between 0 & 7"
    end

    # Last is now a 64 bit integer containing the length of the input and the last bytes, if any:
    last
  end

  # rotl64 is the left rotation, also called circular shift:
  # https://en.wikipedia.org/wiki/Circular_shift
  def rotl64(num, shift)
    ((num << shift) & MASK_64) | (num >> (64 - shift))
  end

  # This is the main step of the siphash algorithm. A big difference with the C implementation
  # in the Redis codebase is the use of & MASK_64. As explained above, it is used to apply an
  # upper bounds to the results as Ruby would allow them to go past the max value of 64-bit
  # integer: 2^64 - 1
    def sip_round
    @v0 = (@v0 + @v1) & MASK_64
    @v2 = (@v2 + @v3) & MASK_64
    @v1 = rotl64(@v1, 13)
    @v3 = rotl64(@v3, 16)
    @v1 ^= @v0
    @v3 ^= @v2
    @v0 = rotl64(@v0, 32)
    @v2 = (@v2 + @v1) & MASK_64
    @v0 = (@v0 + @v3) & MASK_64
    @v1 = rotl64(@v1, 17)
    @v3 = rotl64(@v3, 21)
    @v1 ^= @v2
    @v3 ^= @v0
    @v2 = rotl64(@v2, 32)
  end

  def compress(m)
    @v3 ^= m
    @compress_rounds.times { sip_round }
    @v0 ^= m
  end

  def finalize
    @v2 ^= 0xff
    @finalize_rounds.times { sip_round }
  end
end

listing 6.27: A Ruby implementation of the siphash 1-2 algorithm

Rebuilding Redis in Ruby - Chapter 5 - Redis Protocol Compatibility

Pierre Jambet — Mon, 17 Aug 2020 18:45:43 +0000

What we'll cover

By the end of this chapter RedisServer will speak the Redis Protocol, RESP v2. Doing this will allow any clients that was written to communicate with the real Redis to also communicate with our own server, granted that the commands it uses are within the small subset of the ones we implemented.

One such client is the redis-cli utility that ships with Redis, it'll look like this:

RESP v2 has been the protocol used by Redis since version 2.0, to quote the documentation:

1.2 already supported it, but Redis 2.0 was the first version to talk only this protocol)

As of version 6.0, RESP v2 is still the default protocol and is what we'll implement in this chapter.

RESP3

RESP v2 is the default version, but not the latest one. RESP3 has been released in 2018, it improves many different aspects of RESP v2, such as adding new types for maps — often called dictionary — and a lot more. The spec is on GitHub and explains in details the background behind it.
RESP3 is supported as of Redis 6.0, as indicated in the release notes:

Redis now supports a new protocol called RESP3, which returns more semantical replies: new clients using this protocol can understand just from the reply what type to return to the calling program.

The HELLO command can be used to switch the connection to a different protocol version. As we can see below, only two versions are currently supported, 2 & 3. We can also see the new map type in action, hello 2 returned an array with 14 items, representing 7 key/value pairs, whereas hello 3 leveraged the new map type to return a map with 7 key/value pairs.

127.0.0.1:6379> hello 2
 1) "server"
 2) "redis"
 3) "version"
 4) "6.0.6"
 5) "proto"
 6) (integer) 2
 7) "id"
 8) (integer) 6
 9) "mode"
10) "standalone"
11) "role"
12) "master"
13) "modules"
14) (empty array)

127.0.0.1:6379> hello 3
1# "server" => "redis"
2# "version" => "6.0.6"
3# "proto" => (integer) 3
4# "id" => (integer) 6
5# "mode" => "standalone"
6# "role" => "master"
7# "modules" => (empty array)

127.0.0.1:6379> hello 1
(error) NOPROTO unsupported protocol version

127.0.0.1:6379> hello 4
(error) NOPROTO unsupported protocol version

Support for the HELLO command and RESP3 might be added in a later chapter but it's not currently on the roadmap of this online book.

Back to RESP v2

The official specification goes into details about the protocol and is still reasonably short and approachable, so feel free to read it, but here are the main elements that will drive the changes to our server.

The 5 data types

RESP v2 defines five data types:

Simple Strings
Errors
Integers
Bulk Strings
Arrays

The type of a serialized RESP data is determined by the first byte:

Simple Strings start with +
Errors start with -
Integers start with :
Bulk Strings start with $
Arrays start with *

The data that follows the type byte depends on each type, let's look at each of them one by one.

Simple Strings

A Simple String cannot contain a new line. One of its main use cases is to return OK back to the client. The full format of a Simple String is "A + character, followed directly by the content of the string, followed by a carriage return (often written as CR or \r) and a line feed (often written as LF or \n).

This is why Simple Strings cannot contain multiples lines, a newline would create confusion given that it is also use a delimiter.

The "OK" string, here shown in its JSON form, returned by the SET command upon success is therefore serialized as +OK\r\n.

redis-cli does the work of detecting the type of the response and only shows us the actual string, OK, as we can see in the example below:

127.0.0.1:6379> SET 1 2
OK

Using nc, we can see what the full response sent back from Redis is:

> nc -v localhost 6379
SET 1 2
+OK

nc does not explicitly display invisible characters such as CR & LF, so it is hard to know for sure that they were returned, beside the newline printed after +OK. The hexdump command is useful here, it allows us to see all the bytes:

echo "SET 1 2" | nc localhost 6379 | hexdump -C
# ...
00000000  2b 4f 4b 0d 0a                                    |+OK..|
00000005

The interesting part is the middle one, 2b 4f 4b 0d 0a, these are the 5 bytes returned by Redis. The part to the right, between pipe characters (|) is their ASCII representation. We can see five characters there, + is the ASCII representation of 2b, O is for 4f, K is for 4d, and the last two bytes do not have a visual representation so they're displayed as ..

2b is the hex notation of 43 ('2b'.to_i(16) in irb), and 43 maps to + in the ASCII table. 4f is the equivalent of 79, and the capital letter O, 4b, the number 75 and the capital letter K.

0d is the equivalent of the number 13, and the carriage return character (CR), and finally, 0a is 10, the line feed character (LF).

Redis follows the Redis Protocol, that's a good start!

Errors

Errors are very similar to Simple Strings, they also cannot contain new line characters. The main difference is that clients should treat them as errors instead of successful results. In languages with exceptions, a client library might decide to throw an exception when receiving an error from Redis. This is what the official ruby library does.

Similarly to Simple Strings, errors end with a carriage return and a line feed, let's see it in action:

❯ echo "GET 1 2" | nc localhost 6379 | hexdump -C
00000000  2d 45 52 52 20 77 72 6f  6e 67 20 6e 75 6d 62 65  |-ERR wrong numbe|
00000010  72 20 6f 66 20 61 72 67  75 6d 65 6e 74 73 20 66  |r of arguments f|
00000020  6f 72 20 27 67 65 74 27  20 63 6f 6d 6d 61 6e 64  |or 'get' command|
00000030  0d 0a                                             |..|
00000032

There are more bytes here, they represent the string: "Err wrong number of arguments for 'get' command", but we can see that the response starts with the 2d byte. Looking at the ASCII table, we can see that 45, the numeric equivalent of 2d, maps to -, so far so good.

And finally, the response ends with 0d0a, respectively CR & LF.

Integers

Integers have a similar representation to Simple Strings and errors. The actual integer comes after the : character and is followed by the CR & LF characters.

An example of integer reply is with the TTL and PTTL commands

The key key-with-ttl was set with the command: SET key-with-ttl value EX 1000.

> echo "TTL key-with-ttl" | nc localhost 6379 | hexdump -C
# ...
00000000  3a 39 38 38 0d 0a                                 |:988..|
00000006

The key not-a-key does not exist.

> echo "TTL not-a-key" | nc localhost 6379 | hexdump -C
# ...
00000000  3a 2d 32 0d 0a                                    |:-2..|
00000005

The key key-without-ttl was set with the command: SET key-without-ttl value.

> echo "TTL key-without-ttl" | nc localhost 6379 | hexdump -C
# ...
00000000  3a 2d 31 0d 0a                                    |:-1..|
00000005

All of these responses start with the 3a byte, which is equivalent to 58, aka :. In the two cases where the response is a negative value, -2 for a non existent key and -1 for an existing key without a ttl, the next byte is 2d, equivalent to 45, aka -.

The rest of the data, before the 0d & 0a bytes, is the actual integer data, in ASCII format, 31 is the hex equivalent to 49, which is the character 1, 32 is the hex equivalent to 50, which is the character 2. 39 & 38 are respectively the hex equivalent to 57 & 56, the characters 9 & 8.

A ruby client parsing this data would extract the string between : and \r\n and call to_i on it: '988'.to_i == 988.

Bulk Strings

In order to work for any strings, Bulk Strings need to first declare their length, and only then the actual data. This lets the receiver know how many bytes to expect, instead of reading anything until it finds CRLF, the way it does for a Simple String.

The length of the string is sent directly after the dollar sign, and is delimited by CRLF, the following is the actual string data, and another CRLF to end the string.

The RESP Bulk String representation of the JSON string "GET" is: $3\r\nGET\r\n.

Interestingly, it seems like Redis does not care that much about the final CRLF, as long as it finds two characters there, it assumes it's the end of the Bulk String and tries to process what comes after.

In the following example, we first send the command GET a to Redis over port 6379, as a an array of Bulk Strings, followed by the non existent command NOT A COMMAND. The response first contains the -1 integer, followed by the error.

irb(main):001:0> require 'socket'
=> true
irb(main):002:0> socket = TCPSocket.new 'localhost', 6379
irb(main):004:0> socket.write("*2\r\n$3\r\nGET\r\n$1\r\na\r\n*1\r\n$13\r\nNOT A COMMAND\r\n")
=> 35
irb(main):005:0> socket.read_nonblock(1024, exception: false)
=> "$-1\r\n-ERR unknown command `NOT`, with args beginning with: `A`, `COMMAND`, \r\n"

The following is handled identically by Redis, despite the fact the a Bulk String is not terminated by CRLF. We can see that Redis ignored the b and c characters and proceeded with the following command, the non existent NOT A COMMAND. I am assuming that the code in charge of reading client input first reads the length, then grabs that many bytes and jumps by two characters, regardless of what these characters are.

irb(main):027:0> socket.write("*2\r\n$3\r\nGET\r\n$1\r\nabc*1\r\n$13\r\nNOT A COMMAND\r\n")
=> 35
irb(main):030:0> socket.read_nonblock(1024, exception: false)
=> "$-1\r\n-ERR unknown command `NOT`, with args beginning with: `A`, `COMMAND`, \r\n"

There's a special value for Bulk Strings, the null Bulk String. It is commonly returned when a Bulk String would otherwise be expected, but there was no value to return. This happens in many cases, such as when there are no values for the key passed to the GET command. RESP represents it as a string with a length of -1: $-1\r\n.

Arrays

Arrays can contain values of any types, including other nested arrays. Similarly to Bulk Strings, arrays must first declare their lengths, followed by CRLF, and all items come afterwards, in their regular serialized form. The following is a JSON representation of an arbitrary array:

[ 1, "a-string", [ "another-string-in-a-nested-array" ], "a-string-with\r\n-newlines" ]

The following is the RESP representation of the same array:

*4\r\n:1\r\n$8\r\na-string\r\n*1\r\n$32\r\nanother-string-in-a-nested-array\r\n$24\r\na-string-with\r\n-newlines\r\n

We can include newlines and indentation for the sake of readability

*4\r\n
  :1\r\n
  $8\r\na-string\r\n
  *1\r\n
    $32\r\nanother-string-in-a-nested-array\r\n
  $24\r\na-string-with\r\n-newlines\r\n

RESP has a special notation for the NULL array: *-1\r\n. The existence of two different NULL values, one for Bulk Strings and one for Bulk Arrays is confusing and is one of the many changes in RESP3. RESP3 has a single null value.

Requests & Responses

As we saw in a previous example, requests are sent as arrays of Bulk Strings. The command GET a-key should be sent as *2\r\n$3\r\nGET\r\n$5\r\na-key\r\n, or in plain English: "An array of length 2, where the first string is of length 3 and is GET and the second string is of length 5 and is a-key".

We can illustrate this by sending this string with the TCPSocket class in ruby:

irb(main):001:0> require 'socket'
=> true
irb(main):002:0> socket = TCPSocket.new 'localhost', 6379
irb(main):003:0> socket.write "*2\r\n$3\r\nGET\r\n$5\r\na-key\r\n"
=> 24
irb(main):004:0> socket.read_nonblock 1024
=> "$-1\r\n"

Inline Protocol

RESP's main mode of operation is following a request/response model described above. It also supports a simpler alternative, called "Inline Commands", which is useful for manual tests or interactions with a server. This is similar to how we've used nc in this book so far.

Anything that does not start with a * character — which is the first character of an array, the format Redis expects for a command — is treated as an inline command. Redis will read everything until a newline is detected and attempts to parse that as a command. This is essentially what we've been doing so far when implementing the RedisServer class.

Let's try this quickly with nc:

> nc localhost 6379
# ...
SET 1 2
+OK
GET 1
$1
2

The reason RESP's main mode of operations is more complicated is because inline commands are severely limited. It is impossible to store a key or a value that contains the carriage return and line feed characters since they're use as delimiters even though Redis does support any strings as keys and values as seen in the following example:

> redis-cli
127.0.0.1:6379> SET a-key "foo\nbar"
OK
127.0.0.1:6379> GET a-key
"foo\nbar"

Let's double check with nc to see what Redis stored:

> nc localhost 6379
# ...
GET a-key
$7
foo
bar

We could also use hexdump to triple check:

> echo "GET a-key" | nc localhost 6379 | hexdump -C
# ...
00000000  24 37 0d 0a 66 6f 6f 0a  62 61 72 0d 0a           |$7..foo.bar..|
0000000d

We can see the 0a byte between o/6f & b/62.

Without inline commands sending test commands would be excruciating:

> nc -c localhost 6379
*2
$3
GET
$1
a
$1
1

Note that we're using the -c flags, which tells nc to send CRLF characters when we type the return key, instead of the default of LF. As we've seen above, for RESP arrays, RESP expects CRLF delimiters.

Pub/Sub

Redis supports a Publish/Subscribe messaging paradigm, with the SUBSCRIBE, UNSUBSCRIBE & PUBLISH commands, documented on Pub/Sub page of the official documentation.

These commands have a significant impact of how data flows between clients and servers, and given that we have not yet added support for pub/sub, we will ignore its impact on our implementation of the Redis Protocol for now. Future chapters will add support for pub/sub and will follow the RESP specification.

Pipelining

RESP clients can send multiple requests at once and the RESP server will write multiple responses back, this is called pipelining. The only constraint is that commands must be processed in the same ordered they were received, so that clients can associate the responses back to each request.

The following is an example of sending two commands at once and then reading the two responses, in Ruby:

irb(main):001:0> require 'socket'
=> true
irb(main):002:0> socket = TCPSocket.new 'localhost', 6379
irb(main):003:0> socket.write "SET 1 2\r\nGET 1\r\n"
=> 16
irb(main):004:0> socket.read_nonblock 1024
=> "+OK\r\n$1\r\n2\r\n"

We first wrote the string "SET 1 2\r\nGET 1\r\n", which represents the command SET 1 2 and the command GET in the inline format.

The response we get from the server is a string containing the two responses, fist the Simple String +OK\r\n, followed by the Bulk String $1\r\n2\r\n.

Making our Server speak RESP

As far as I know there is no official test suite that we could run our server against to validate that it correctly follows RESP. What we can do instead is rely on redis-cli as a way to test the RESP implementation of our server. Let's see what happens when we try it with the current server. First let's start the server from Chapter 4:

DEBUG=t ruby -r"./server" -e "RedisServer.new"

and in another shell, let's open redis-cli on port 2000:

> redis-cli -p 2000

You should see the following the server logs:

D, [2020-08-12T16:11:42.461645 #91271] DEBUG -- : Received command: *1
D, [2020-08-12T16:11:42.461688 #91271] DEBUG -- : Response: (error) ERR unknown command `*1`, with args beginning with:
D, [2020-08-12T16:11:42.461925 #91271] DEBUG -- : Received command: $7
D, [2020-08-12T16:11:42.461960 #91271] DEBUG -- : Response: (error) ERR unknown command `$7`, with args beginning with:
D, [2020-08-12T16:11:42.462005 #91271] DEBUG -- : Received command: COMMAND
D, [2020-08-12T16:11:42.462036 #91271] DEBUG -- : Response: (error) ERR unknown command `COMMAND`, with args beginning with:

The server received the string "*1\r\n$7\r\nCOMMAND\r\n", which is the RESP representation of the string "COMMAND" in a single item array, [ "COMMAND" ] in JSON.

The COMMAND command is useful when running Redis in a cluster. Given that we have not yet implementer cluster capabilities, going into details about the COMMAND command is a little bit out of scope. In short the COMMAND command is useful to provide meta information about each command, such as information about the positions of the keys. This is useful because in cluster mode, clients have to route requests to the different nodes in the cluster. It is common for a command to have the key as the second element, the one coming directly after the command itself. This happens to be the case for all the commands we've implemented so far. But some commands have different semantics. For instance MSET can contain multiple keys, so clients need to know where the keys are in the command. While rare, some commands have the first key at a different index, this is the case for the OBJECT command.

Back to redis-cli running against our Redis server, if you then try to send a command, GET 1 for instance, redis-cli will crash after printing the following error:

Error: Protocol error, got "(" as reply type byte

This is because our server writes the string (nil) when it does find an try for the given key. (nil) is what redis-cli displays when it receives a null Bulk String, as we can see with the following example, we first send the GET 1 command with redis-cli and then with nc and observe the response in each case:

❯ nc -c localhost 6379
GET 1
$-1
# ...
> redis-cli
127.0.0.1:6379> GET 1
(nil)

Our server must send the null Bulk String, $-1\r\n, to follow RESP. This is what redis-cli tells us before stopping, it expected a "type byte", one of +, -, :, $ or *, but instead got (.

In order to use redis-cli against our own server, we should implement the COMMAND command, since it sends it directly after starting. We also need to change how we process client input, to parse RESP arrays of Bulk Strings. We also need to support inline commands. Finally, we also need to update the responses we write back, and serialize responses following RESP.

Let's get to it!

Parsing Client Input

Modules & Namespaces

Most of the changes will take place in server.rb. As the codebase started to grow, I thought it would be easier to start using ruby modules, so I nested the Server class under the Redis namespace. This will allow us to create other classes & modules under the Redis namespace as well. All the other classes have been updated to be under the Redis namespace as well, e.g. ExpireHelper is now BYORedis::ExpireHelper. BYO stands for Build Your Own. I'm purposefully not using Redis as it is already used by the popular redis gem. We're not using both at the same time in the same project for now, so it wouldn't really have been a problem. But say that you would like to use the redis gem to communicate with the server we're building, we will prevent any kind of unexpected errors by using different names.

# expire_helper.rb
module BYORedis
  module ExpireHelper

    def self.check_if_expired(data_store, expires, key)
      # ...
    end
  end
end

listing 5.1: Nesting ExpireHelper under the Redis module

Storing partial client buffer

As of the previous chapter we never stored the client input. We would read from the socket when IO.select would tell us there is something to read, read until the end of the line, and process the result as a command.

It turns out that this approach is a bit too aggressive. Clients should be able to send a single command in two parts, there's no reason to treat that as an error.

In order to do this, we are going to create a Client struct to hold the client socket as well a string containing all the pending input we have not process yet:

# server.rb
Client = Struct.new(:socket, :buffer) do
  def initialize(socket)
    self.socket = socket
    self.buffer = ''
  end
end

listing 5.2: The new Client class

We need to adapt process_poll_events to use this new class instead of the raw socket coming as a result of TCPServer#accept:

# server.rb
def process_poll_events(sockets)
  sockets.each do |socket|
    begin
      if socket.is_a?(TCPServer)
        @clients << Client.new(@server.accept)
      elsif socket.is_a?(TCPSocket)
        client = @clients.find { |client| client.socket == socket }
        client_command_with_args = socket.read_nonblock(1024, exception: false)
        if client_command_with_args.nil?
          @clients.delete(client)
          socket.close
        elsif client_command_with_args == :wait_readable
          # ...
        else
          # We now need to parse the input as a RESP array
          # ...
        end
      else
        # ...
      end
    rescue Errno::ECONNRESET
      @clients.delete_if { |client| client.socket == socket }
    end
  end
end

listing 5.3: Updated handling of socket in server.rb

Parsing commands as RESP Arrays

More things need to change in process_poll_events. We first append the result from read_nonblock to client.buffer, which will allow us to continue appending until we accumulate enough to read a whole command. We then delegate the processing of client.buffer to a different method, split_commands:

# server.rb
def process_poll_events(sockets)
  sockets.each do |socket|
    begin
      # ...
      elsif socket.is_a?(TCPSocket)
        # ...
        else
          client.buffer += client_command_with_args
          split_commands(client.buffer) do |command_parts|
            response = handle_client_command(command_parts)
            @logger.debug "Response: #{ response.class } / #{ response.inspect }"
            @logger.debug "Writing: '#{ response.serialize.inspect }'"
            socket.write response.serialize
          end
        end
      else
        # ...
      end
      # ...
    end
  end
end

def split_commands(client_buffer)
  @logger.debug "Full result from read: '#{ client_buffer.inspect }'"

  scanner = StringScanner.new(client_buffer.dup)
  until scanner.eos?
    if scanner.peek(1) == '*'
      yield parse_as_resp_array(scanner)
    else
      yield parse_as_inline_command(scanner)
    end
    client_buffer.slice!(0, scanner.charpos)
  end
end
#...

listing 5.4 Updated handling of client input in server.rb

split_commands is in charge of splitting the client input into multiple commands, which is necessary to support pipelining. As a reminder, since we're adding support pipelining, we have to assume that the content of client.buffer might contain more than one command, and if so, we want to process them all in the order we received them, and write the responses back, in the same order.

It also handles the two different versions of commands, inline, or "regular", as RESP Arrays. We use the StringScanner class, which is really convenient to process data from a string, from left to right. We call String#dup on the argument to StringScanner to make sure that the StringScanner gets its own instance. As we iterate through client.buffer, every time we find a whole command, we want to remove it from the client input. We do this with client_buffer.slice!(0, scanner.charpos). If client_buffer contains two commands, i.e. GET a\r\nGET b\r\n, once we processed GET a, we want to remove the first 7 characters from the string: GET a\r\n, so that we never attempt to process them again. Note that we only do this after yielding, meaning that we only ever treat a command as done after we successfully wrote to the socket.

We first peek at the first character, if it is *, the following should be a RESP array, and we process it as such. Otherwise, we assume that we're dealing with an inline command. Each branch delegates to a method handling the parsing of the string.

The yield approach allows us to process each parsed command one by one, once parsed, we yield it, and it is handled by the handle_client_command method, which has barely changed from the previous chapter.

Let's look at the parse_as_resp_array & parse_as_inline_command methods:

def parse_as_inline_command(client_buffer, scanner)
  command = scanner.scan_until(/(\r\n|\r|\n)+/)
  raise IncompleteCommand if command.nil?

  command.split.map(&:strip)
end

def parse_as_resp_array(scanner)
  unless scanner.getch == '*'
    raise 'Unexpectedly attempted to parse a non array as an array'
  end

  expected_length = scanner.scan_until(/\r\n/)
  raise IncompleteCommand if expected_length.nil?

  expected_length = parse_integer(expected_length, 'invalid multibulk length')
  command_parts = []

  expected_length.times do
    raise IncompleteCommand if scanner.eos?

    parsed_value = parse_as_resp_bulk_string(scanner)
    raise IncompleteCommand if parsed_value.nil?

    command_parts << parsed_value
  end

  command_parts
end

def parse_integer(integer_str, error_message)
  begin
    value = Integer(integer_str)
    if value < 0
      raise ProtocolError, "ERR Protocol error: #{ error_message }"
    else
      value
    end
  rescue ArgumentError
    raise ProtocolError, "ERR Protocol error: #{ error_message }"
  end
end

listing 5.5 Parsing RESP Arrays in server.rb

parse_as_inline_command starts by calling StringScanner#scan_until, with /\r\n/. scan_until keeps iterating through the string, until it encounters something that matches its argument. In our case it will keep going through client_buffer until it finds CRLF, if it doesn't find a match, it returns nil. We're not even trying to process the string in this case, it is incomplete, so we'll leave it in there and eventually reattempt later on, the next time we read from this client.

If the string returned is not nil, it contains the string, and in this case, we do what we used to, we split it on spaces, and return it as an array of string parts, e.g. GET 1\r\n would be returned as [ 'GET', '1' ]

parse_as_resp_array is more complicated. As a sanity check, we test again that the first character is indeed *, getch also moves the internal cursor of StringScanner, moving it to the first character of the expected length. Using scan_until we extract all the characters until the first CRLF characters in the client input.

If nil is returned, this means that we reached the end of the string without encountering CR & LF, and instead of treating this as a client error, we raise an IncompleteCommand error, to give the client a change to write the missing parts of the command later on.

expected_length will contain a string composed of the characters before CRLF & the CRLF characters. For instance, if the scanner was created with the string $3\r\nabc\r\n — The Bulk String representation of the string "3" — expected_length would be equal to "3\r\n". The Ruby String#to_i is not strict enough here. It returns 0 in a lot of cases where we'd want an error instead, such as "abc".to_i == 0. We instead use the Kernel.Integer method, which raises an ArgumentError exception with invalid strings. We catch ArgumentError and raise a ProtocolError instead.

In the next step we iterate as many times as the value of expected_length with expected_length.times. We start each iteration by checking if we reached the end of the string with eos?. If we did, then instead of returning a protocol error, we raise an IncompleteCommand exception. This gives a chance to the client to send the remaining elements of the array later on.

As mentioned above, a request to Redis is always an array of Bulk Strings, so we attempt to parse all the elements as strings, by calling parse_as_bulk_string with the same scanner instance. Before looking at the method, let's see how the two new exceptions IncompleteCommand & ProtocolError are defined and handled:

IncompleteCommand & ProtocolError are custom exceptions defined at the top of the file:

# server.rb
IncompleteCommand = Class.new(StandardError)
ProtocolError = Class.new(StandardError) do
  def serialize
    RESPError.new(message).serialize
  end
end

listing 5.6 The new exceptions in server.rb

RESPError is defined in resp_types.rb:

# resp_types.rb
module BYORedis
  RESPError = Struct.new(:message) do
    def serialize
      "-#{ message }\r\n"
    end
  end
  # ...
end

listing 5.7 The new RESPError class

They are handled in the begin/rescue block in process_poll_events:

# server.rb
begin
  # ...
rescue Errno::ECONNRESET
  @clients.delete_if { |client| client.socket == socket }
rescue IncompleteCommand
  # Not clearing the buffer or anything
  next
rescue ProtocolError => e
  socket.write e.serialize
  socket.close
  @clients.delete(client)
end

listing 5.8 Handling the new exceptions in server.rb

We don't write anything back when encountering an IncompleteCommand exception, we assume that the client has not finished sending the command. On the other hand, for ProtocolError, we write an error back to the client, following the format of a RESP error and we disconnect the client. This is what Redis does too.

Back to parse_as_resp_bulk_string:

# server.rb
def parse_as_resp_bulk_string(scanner)
  type_char = scanner.getch
  unless type_char == '$'
    raise ProtocolError, "ERR Protocol error: expected '$', got '#{ type_char }'"
  end

  expected_length = scanner.scan_until(/\r\n/)
  raise IncompleteCommand if expected_length.nil?

  expected_length = parse_integer(expected_length, 'invalid bulk length')
  bulk_string = scanner.rest.slice(0, expected_length)

  raise IncompleteCommand if bulk_string.nil? || bulk_string.length != expected_length

  scanner.pos += bulk_string.bytesize + 2
  bulk_string
end

listing 5.9 Parsing Bulk Strings

The first step is calling StringScanner#getch, it moves the internal cursor of the scanner by one character and returns it. If the first character is $, we received a Bulk String as expected. Anything else is an error.

Redis accepts empty strings, and while it may be unusual, it is possible for a Redis key to be an empty string, and a value can also be an empty string. If the expected length is negative, then we stop and return a ProtocolError

The next step is extracting the actual string. StringScanner maintains an internal cursor of the progress through the string. At this point this cursor is right after CRLF, where the string content starts. StringScanner#rest returns the string from this cursor until the end, and using slice, we extract only the number of characters indicated by expected_length.

If the result of this operation is nil or shorter than the expected length, we don't want to treat it as an error yet, since it is possible for the clients to write the missing elements of the command, so we raise an IncompleteCommand, in the hope that the client will send the missing parts later on.

The final step is to advance the cursor position in the StringScanner instance. We do this with the StringScanner#pos= method. Notice how we use the bytesize methods and two to it. We use bytesize instead of length to handle characters that span over multiple bytes, such as CJK characters, accentuated characters, emojis and many others. Let's look at the difference in irb:

irb(main):045:1* def print_length_and_bytesize(str)
irb(main):046:1*   puts str.length
irb(main):047:1*   puts str.bytesize
irb(main):048:0> end
=> :print_length_and_bytesize
irb(main):049:0> print_length_and_bytesize('a')
1
1
=> nil
irb(main):050:0> print_length_and_bytesize('é')
1
2
=> nil
irb(main):051:0> print_length_and_bytesize('你')
1
3
=> nil
irb(main):058:0> print_length_and_bytesize('😬')
1
4
=> nil

As we can see, all of these strings return 1 for length, but different values, respectively 2, 3 & 4 for bytesize. Going into details about UTF-8 encoding is out of scope, but the main takeaway from this is that what we consider to be a single character, might span over multiple bytes.

If a client had sent 你 has a Bulk String, we'd expect it to pass the length as 3, and therefore we need to advance the cursor by 3 in the StringScanner instance. We also add two to account for the trailing CRLF characters. Note that, like Redis, we do not actually check that these two characters are indeed CR & LF, we just skip over them.

Updating the command responses

The commands we've implemented so far, GET, SET, TTL & PTTL do not return data that follows the format defined in RESP. GET needs to return Bulk Strings, SET returns the Simple String OK or the null Bulk String if it didn't set the value and the last two, TTL & PTTL, return integers. We will first create new classes to wrap the process of serializing strings and integers to their matching RESP format:

# resp_types.rb
module BYORedis
  # ...
  RESPInteger = Struct.new(:underlying_integer) do
    def serialize
      ":#{ underlying_integer }\r\n"
    end

    def to_i
      underlying_integer.to_i
    end
  end

  RESPSimpleString = Struct.new(:underlying_string) do
    def serialize
      "+#{ underlying_string }\r\n"
    end
  end

  OKSimpleStringInstance = Object.new.tap do |obj|
    OK_SIMPLE_STRING = "+OK\r\n".freeze
    def obj.serialize
      OK_SIMPLE_STRING
    end
  end

  RESPBulkString = Struct.new(:underlying_string) do
    def serialize
      "$#{ underlying_string.bytesize }\r\n#{ underlying_string }\r\n"
    end
  end

  NullBulkStringInstance = Object.new.tap do |obj|
    NULL_BULK_STRING = "$-1\r\n".freeze
    def obj.serialize
      NULL_BULK_STRING
    end
  end

  RESPArray = Struct.new(:underlying_array) do
    def serialize
      serialized_items = underlying_array.map do |item|
        case item
        when RESPSimpleString, RESPBulkString
          item.serialize
        when String
          RESPBulkString.new(item).serialize
        when Integer
          RESPInteger.new(item).serialize
        when Array
          RESPArray.new(item).serialize
        end
      end
      "*#{ underlying_array.length }\r\n#{ serialized_items.join }"
    end
  end
  NullArrayInstance = Object.new.tap do |obj|
    NULL_ARRAY = "*-1\r\n".freeze
    def obj.serialize
      NULL_ARRAY
    end
  end
end

listing 5.10 The new RESP types

RESPArray is not strictly required at the moment since none of the commands we've implemented so far return array responses, but the COMMAND command, which we'll implement below returns an array, so it'll be useful there.

We could have chosen a few different options to represent the null array and the null list, such as adding the logic in serialize methods of RESPArray & RESPBulkString. I instead decided to create two globally available instances that implement the same interface, the serialize method. This allows the code in server.rb to always call serialize on the result it gets from calling the call method. On the other hand, in the *Command classes, it forces us to explicitly handle these null cases, which I find preferable to passing nil values around.

We use the String#freeze method to prevent accidental modifications of the values at runtime. Ruby will throw an exception if you attempt to do so:

irb(main):001:0> require_relative './server'
=> true
irb(main):002:0> BYORedis::NULL_BULK_STRING
=> "$-1\r\n"
irb(main):003:0> BYORedis::NULL_BULK_STRING << "a"
Traceback (most recent call last):
        4: from /Users/pierre/.rbenv/versions/2.7.1/bin/irb:23:in `<main>'
        3: from /Users/pierre/.rbenv/versions/2.7.1/bin/irb:23:in `load'
        2: from /Users/pierre/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/irb-1.2.3/exe/irb:11:in `<top (required)>'
        1: from (irb):3
FrozenError (can't modify frozen String: "$-1\r\n")

That said, do note that "constants" in Ruby aren't really "constants", it is possible to reassign the value at runtime:

irb(main):004:0> BYORedis::NULL_BULK_STRING = "something else"
(irb):4: warning: already initialized constant BYORedis::NULL_BULK_STRING
/Users/pierre/dev/redis-in-ruby/code/chapter-5/resp_types.rb:32: warning: previous definition of NULL_BULK_STRING was here
irb(main):005:0> BYORedis::NULL_BULK_STRING
=> "something else"

While it doesn't prevent all kinds of weird runtime issues, I do like the use of String#freeze to at least be explicit about the nature of the value, signifying that it is not supposed to be modified.

The OK Simple String is so common that I created a constant for it, OKSimpleStringInstance, so that it can be reused instead of having to allocate a new instance every time we need it. Only the SetCommand class uses it for now, but more commands use it, such as LSET, MSET and many others.

Let's start with GET:

# get_command.rb
module BYORedis
  class GetCommand

    # ...

    def call
      if @args.length != 1
        RESPError.new("ERR wrong number of arguments for 'GET' command")
      else
        key = @args[0]
        ExpireHelper.check_if_expired(@data_store, @expires, key)
        value = @data_store[key]
        if value.nil?
          NullBulkStringInstance
        else
          RESPBulkString.new(value)
        end
      end
    end
  end
end

listing 5.11 Updated response in GetCommand

Now that BYORedis::GetCommand has been updated, let's tackle SetCommand:

# set_command.rb
def call
  key, value = @args.shift(2)
  if key.nil? || value.nil?
    return RESPError.new("ERR wrong number of arguments for 'SET' command")
  end

  parse_result = parse_options

  existing_key = @data_store[key]

  if @options['presence'] # ...
    NullBulkStringInstance
  elsif @options['presence'] # ...
    NullBulkStringInstance
  else

    # ...

    OKSimpleStringInstance
  end

rescue ValidationError => e
  RESPError.new(e.message)
rescue SyntaxError => e
  RESPError.new(e.message)
end

listing 5.12 Updated response in SetCommand

The SET command has two possible outputs, either the nil string if the outcome was that nothing was set, as a result of the NX or XX options, or the Simple String OK if the outcome was a successful set. This is where the special case instances NullBulkStringInstance & OKSimpleStringInstance come in handy. By returning them here, the code in server.rb can leverage duck typing and call the serialize method, but under the hood, the same strings will be used, BYORedis::OK_SIMPLE_STRING & BYORedis::NULL_BULK_STRING. This is a very small optimization, but given how common it is to call the SET command, it is interesting to think about things like that to prevent unnecessary work on the server.

And finally we need to update TtlCommand and PttlCommand

# pttl_command.rb
def call
  if @args.length != 1
    RESPError.new("ERR wrong number of arguments for 'PTTL' command")
  else
    key = @args[0]
    ExpireHelper.check_if_expired(@data_store, @expires, key)
    key_exists = @data_store.include? key
    value = if key_exists
              ttl = @expires[key]
              if ttl
                (ttl - (Time.now.to_f * 1000)).round
              else
                -1
              end
            else
              -2
            end
    RESPInteger.new(value)
  end
end

# ttl_command.rb
def call
  if @args.length != 1
    RESPError.new("ERR wrong number of arguments for 'TTL' command")
  else
    pttl_command = PttlCommand.new(@data_store, @expires, @args)
    result = pttl_command.call.to_i
    if result > 0
      RESPInteger.new((result / 1000.0).round)
    else
      RESPInteger.new(result)
    end
  end
end

listing 5.13 Updated response in PttlCommand & TtlCommand

Case insensitivity

It is not explicitly mentioned in the RESP v2 documentation, but Redis treats commands and options as case insensitive. The following examples are all valid: get 1, GeT 1, set key value EX 1 nx.

In order to apply the same handling logic, we changed the keys in the COMMANDS constant to be lower case, and we always lower case the client input when attempting to find a handler for the command:

# server.rb
COMMANDS = {
  'command' => CommandCommand,
  'get' => GetCommand,
  'set' => SetCommand,
  'ttl' => TtlCommand,
  'pttl' => PttlCommand,
}
# ...

def handle_client_command(command_parts)
  @logger.debug "Received command: #{ command_parts }"
  command_str = command_parts[0]
  args = command_parts[1..-1]

  command_class = COMMANDS[command_str.downcase]

  # ...
end

listing 5.14 Updates for case insensitivity in BYORedis::Server

We also need to update the BYORedis::SetCommand class to handle options regardless of the case chosen by clients:

# set_command.rb
# ...
OPTIONS = {
  'ex' => CommandOptionWithValue.new(
    'expire',
    ->(value) { validate_integer(value) * 1000 },
  ),
  'px' => CommandOptionWithValue.new(
    'expire',
    ->(value) { validate_integer(value) },
  ),
  'keepttl' => CommandOption.new('expire'),
  'nx' => CommandOption.new('presence'),
  'xx' => CommandOption.new('presence'),
}
#...
def parse_options
  while @args.any?
    option = @args.shift
    option_detail = OPTIONS[option.downcase]
    # ...
  end
end
#...

listing 5.15 Updates for case insensitivity in SetCommand

The `COMMAND` command

In order to implement COMMAND, we added a describe method to each of the *Command classes, so that the CommandCommand class can iterate over all these classes and call .describe on them, and then serialize the result to a RESP array:

# command_command.rb
module BYORedis
  class CommandCommand

    def initialize(_data_store, _expires, _args)
    end

    def call
      RESPArray.new(Server::COMMANDS.map { |_, command_class| command_class.describe } )
    end

    def self.describe
      [
        'command',
        -1, # arity
        # command flags
        [ 'random', 'loading', 'stale' ].map { |s| RESPSimpleString.new(s) },
        0, # position of first key in argument list
        0, # position of last key in argument list
        0, # step count for locating repeating keys
        # acl categories: https://github.com/antirez/redis/blob/6.0/src/server.c#L161-L166
        [ '@slow', '@connection' ].map { |s| RESPSimpleString.new(s) },
      ]
    end
  end
end

listing 5.16 The new CommandCommand class

# get_command.rb

def self.describe
  [
    'get',
    2, # arity
    # command flags
    [ 'readonly', 'fast' ].map { |s| RESPSimpleString.new(s) },
    1, # position of first key in argument list
    1, # position of last key in argument list
    1, # step count for locating repeating keys
    # acl categories: https://github.com/antirez/redis/blob/6.0/src/server.c#L161-L166
    [ '@read', '@string', '@fast' ].map { |s| RESPSimpleString.new(s) },
  ]
end

# pttl_command.rb

def self.describe
  [
    'pttl',
    2, # arity
    # command flags
    [ 'readonly', 'random', 'fast' ].map { |s| RESPSimpleString.new(s) },
    1, # position of first key in argument list
    1, # position of last key in argument list
    1, # step count for locating repeating keys
    # acl categories: https://github.com/antirez/redis/blob/6.0/src/server.c#L161-L166
    [ '@keyspace', '@read', '@fast' ].map { |s| RESPSimpleString.new(s) },
  ]
end

# set_command.rb

def self.describe
  [
    'set',
    -3, # arity
    # command flags
    [ 'write', 'denyoom' ].map { |s| RESPSimpleString.new(s) },
    1, # position of first key in argument list
    1, # position of last key in argument list
    1, # step count for locating repeating keys
    # acl categories: https://github.com/antirez/redis/blob/6.0/src/server.c#L161-L166
    [ '@write', '@string', '@slow' ].map { |s| RESPSimpleString.new(s) },
  ]
end

# ttl_command.rb

def self.describe
  [
    'ttl',
    2, # arity
    # command flags
    [ 'readonly', 'random', 'fast' ].map { |s| RESPSimpleString.new(s) },
    1, # position of first key in argument list
    1, # position of last key in argument list
    1, # step count for locating repeating keys
    # acl categories: https://github.com/antirez/redis/blob/6.0/src/server.c#L161-L166
    [ '@keyspace', '@read', '@fast' ].map { |s| RESPSimpleString.new(s) },
  ]
end

listing 5.17 Updates for the COMMAND command in SetCommand, GetCommand, TtlCommand & PttlCommand

test.rb & test_helper.rb

Testing the BYORedis::Server class is becoming more and more complicated, in order to keep things clean, I moved a lot of the helper method to the test_helper.rb file, so that test.rb only contains the actual tests.

The assert_command_results helper has been updated to handle the RESP format. For the sake of simplicity, it assumes that the data is not serialized and does that for you. This allows us to write simpler assertions such as:

assert_command_results [
  [ 'SET 1 3 NX EX 1', '+OK' ],
  [ 'GET 1', '3' ],
  [ 'SET 1 3 XX keepttl', '+OK' ],
]

and the assert_command_results will serialize the commands as RESP Arrays for us.

I also added a new assertion helper, assert_multipart_command_results. It allows a little bit more flexibility around expectations for commands sent through multiple write calls. Instead of being a single command like in assert_command_results, the first element of the pair is itself an array of strings, each of them representing a sequence of characters that will be sent to the server. This is handy to test pipelining as well as edge cases with regard to RESP.

# test_helper.rb
# The arguments in an array of array of the form
# [
#   [ [ "COMMAND-PART-I", "COMMAND-PART-II", ... ], "EXPECTED_RESULT" ],
#   ...
# ]
def assert_multipart_command_results(multipart_command_result_pairs)
  with_server do |server_socket|
    multipart_command_result_pairs.each do |command, expected_result|
      command.each do |command_part|
        server_socket.write command_part
        # Sleep for one milliseconds to give a chance to the server to read
        # the first partial command
        sleep 0.001
      end

      response = read_response(server_socket)

      if response.length < expected_result.length
        # If the response we got is shorter, maybe we need to give the server a bit more time
        # to finish processing everything we wrote, so give it another shot
        sleep 0.1
        response += read_response(server_socket)
      end

      assert_response(expected_result, response)
    end
  end
end

def assert_command_results(command_result_pairs)
  with_server do |server_socket|
    command_result_pairs.each do |command, expected_result|
      if command.is_a?(String) && command.start_with?('sleep')
        sleep command.split[1].to_f
        next
      end
      command_string = if command.start_with?('*')
                         command
                       else
                         BYORedis::RESPArray.new(command.split).serialize
                       end
      server_socket.write command_string

      response = read_response(server_socket)

      assert_response(expected_result, response)
    end
  end
end

def assert_response(expected_result, response)
  assertion_match = expected_result&.match(/(\d+)\+\/-(\d+)/)
  if assertion_match
    response_match = response.match(/\A:(\d+)\r\n\z/)
    assert response_match[0]
    assert_in_delta assertion_match[1].to_i, response_match[1].to_i, assertion_match[2].to_i
  else
    if expected_result && !%w(+ - : $ *).include?(expected_result[0])
      # Convert to a Bulk String unless it is a Simple String (starts with a +)
      # or an error (starts with -)
      expected_result = BYORedis::RESPBulkString.new(expected_result).serialize
    end

    if expected_result && !expected_result.end_with?("\r\n")
      expected_result += "\r\n"
    end

    if expected_result.nil?
      assert_nil response
    else
      assert_equal expected_result, response
    end
  end
end

def read_response(server_socket)
  response = ''
  loop do
    select_res = IO.select([ server_socket ], [], [], 0.1)
    last_response = server_socket.read_nonblock(1024, exception: false)
    if last_response == :wait_readable || last_response.nil? || select_res.nil?
      response = nil
      break
    else
      response += last_response
      break if response.length < 1024
    end
  end
  response&.force_encoding('utf-8')
rescue Errno::ECONNRESET
  response&.force_encoding('utf-8')
end

def to_query(*command_parts)
  [ BYORedis::RESPArray.new(command_parts).serialize ]
end

listing 5.18 The new test helpers in test_helper.rb

Conclusion

We can now use redis-cli, with redis-cli -p 2000 to interact with our redis server:

> redis-cli -p 2000
127.0.0.1:2000> COMMAND
1) 1) "command"
   2) (integer) -1
   3) 1) random
      2) loading
      3) stale
   4) (integer) 0
   5) (integer) 0
   6) (integer) 0
   7) 1) @slow
      2) @connection
2) 1) "get"
   2) (integer) 2
   3) 1) readonly
      2) fast
   4) (integer) 1
   5) (integer) 1
   6) (integer) 1
   7) 1) @read
      2) @string
      3) @fast
3) 1) "set"
   2) (integer) -3
   3) 1) write
      2) denyoom
   4) (integer) 1
   5) (integer) 1
   6) (integer) 1
   7) 1) @write
      2) @string
      3) @slow
4) 1) "ttl"
   2) (integer) 2
   3) 1) readonly
      2) random
      3) fast
   4) (integer) 1
   5) (integer) 1
   6) (integer) 1
   7) 1) @keyspace
      2) @read
      3) @fast
5) 1) "pttl"
   2) (integer) 2
   3) 1) readonly
      2) random
      3) fast
   4) (integer) 1
   5) (integer) 1
   6) (integer) 1
   7) 1) @keyspace
      2) @read
      3) @fast
127.0.0.1:2000> GET a-key
(nil)
127.0.0.1:2000> SET name pierre
OK
127.0.0.1:2000> GET name
"pierre"
127.0.0.1:2000> SET last-name J EX 10
OK
127.0.0.1:2000> TTL last-name
(integer) 6
127.0.0.1:2000> PTTL last-name
(integer) 5016
127.0.0.1:2000> PTTL last-name
(integer) 2432
127.0.0.1:2000> DEL name
(error) ERR unknown command `DEL`, with args beginning with: `name`,

All the commands we already implemented work as expected and non implemented commands such as DEL return an unknown command error. So far so good!

In the next chapter we'll write our own Hashing algorithm and ban the use of the Hash class in our code.

Code

As usual, the code is available on GitHub.

Rebuilding Redis in Ruby - Chapter 4 - Adding the missing options to the SET command

Pierre Jambet — Thu, 30 Jul 2020 14:21:23 +0000

What we'll cover

We implemented a simplified version of the SET command in Chapter 2, in this chapter we will complete the command by implementing all its options. Note that we're still not following the Redis Protocol, we will address that in the next chapter.

Planning our changes

The SET commands accepts the following options:

EX seconds -- Set the specified expire time, in seconds.
PX milliseconds -- Set the specified expire time, in milliseconds.
NX -- Only set the key if it does not already exist.
XX -- Only set the key if it already exists.
KEEPTTL -- Retain the Time To Live (TTL) associated with the key

As noted in the documentation, there is some overlap with some of the options above and the following commands: SETNX, SETEX, PSETEX. As of this writing these three commands are not officially deprecated, but the documentation mentions that it might happen soon. Given that we can access the same features through the NX, EX & PX options respectively, we will not implement these three commands.

SET a-key a-value EX 10 is equivalent to SETEX a-key 10 a-value, which we can demonstrate in a redis-cli session:

127.0.0.1:6379> SET a-key a-value EX 10
OK
127.0.0.1:6379> TTL "a-key"
(integer) 8
127.0.0.1:6379> SETEX a-key 10 a-value
OK
127.0.0.1:6379> TTL "a-key"
(integer) 8

TTL returned 8 in both cases, because it took me about 2s to type the TTL command, and by that time about 8s were left, of the initial 10.

The EX & PX options

Both EX & PX options are followed by one argument, an integer for the number of the seconds and milliseconds specifying how long the key will be readable for. Once the duration has elapsed, the key is deleted, and calling GET would return (nil) as if it was never set or explicitly deleted by the DEL command.

When a key is set with either of these two options, or through the SETEX & PSETEX commands, but we are ignoring these for the sake of simplicity, Redis adds the key to different dictionary, internally called db->expires in the C code. This dictionary is dedicated to storing keys with a TTL, the key is the same key and the value is the timestamp of the expiration, in milliseconds.

Redis uses two approaches to delete keys with a TTL. The first one is a lazy approach, when reading a key, it checks if it exists in the expires dictionary, if it does and the value is lower than the current timestamp in milliseconds, it deletes the key and does not proceed with the read.

Notes about "Lazy" & "Eager"

The terms lazy is often used in programming, it describes an operation that is put on the back-burner and delayed until it absolutely has to be performed.

In the context of Redis, it makes sense to describe the eviction strategy described above as lazy since Redis might still store keys that are effectively expired and will only guarantee their deletion until they are accessed past their expiration timestamp.

The opposite approach is "eager", where an operation is performed as soon as possible, whether or not it could be postponed.

The other one is a more proactive approach, Redis periodically scans a subset of the list of keys with a TTL value and deletes the expired one. This action, performed in the serverCron function is part of the event loop. The event loop is defined in ae.c and starts in the aeMain function, it continuously executes the aeProcessEvents function in a while loop, until the stop flag is set to 1, which essentially never happens when the server is running under normal circumstances. The aeStop function is the only function doing this and it is only used in redis-benchmark.c & example-ae.c.

aeProcessEvents is a fairly big function, it would be hard to summarize it all here, but it first uses the aeApiPoll function, which is what we covered in the previous chapter. It processes the events from the poll result, if any and then calls processTimeEvents.

Redis maintains a list of time events, as described in the event loop documentation page, for periodic task. When the server is initialized, one time event is created, for the serverCron function. This function is responsible for a lot of things, this is how it is described in the source code:

This is our timer interrupt, called server.hz times per second.\
Here is where we do a number of things that need to be done asynchronously.\
For instance:

Active expired keys collection (it is also performed in a lazy way on lookup).

Software watchdog.

Update some statistic.

Incremental rehashing of the DBs hash tables.

Triggering BGSAVE / AOF rewrite, and handling of terminated children.

Clients timeout of different kinds.

Replication reconnection.

Many more...

We're only interested in the first item in this list for now, the active expiration of keys. Redis runs the event loop on a single thread. This means that each operation performed in the event loop effectively blocks the ones waiting to be processed. Redis tries to optimize for this by making all the operations performed in the event loop "fast".

This is one of the reasons why the documentation provides the time complexity for each commands. Most commands are O(1), and commands like KEYS, with an O(n) complexity are not recommended in a production environment, it would prevent the server from processing any incoming commands while iterating through all the keys.

If Redis were to scan all the keys in the expires dictionary on every iteration of the event loop it would be an O(n) operation, where n is the number of keys with a TTL value. Put differently, as you add keys with a TTL, you would slow down the process of active expiration. To prevent this, Redis only scans the expires up to certain amount. The activeExpireCycle contains a lot of optimizations that we will not explore for now for the sake of simplicity.

One of these optimizations takes care of maintaining statistics about the server, among those Redis keeps track of an estimate of the number of keys that are expired but not yet deleted. Using this it will attempt to expire more keys to try to keep this number under control and prevent it from growing too fast if keys start expiring faster than the normal rate at which they get deleted.

serverCron is first added to the time events with a time set to 1ms in the future. The return value of functions executed as time events dictates if it is removed from the time event queue or if it is rescheduled in the future. serverCron returns a value based on the frequency set as config. By default 100ms. That means that it won't run more than 10 times per second.

I think that it's worth stopping for a second to recognize the benefits of having two eviction strategies for expired keys. The lazy approach gets the job done as it guarantees that no expired keys will ever be returned, but if a key is set with a TTL and is never read again it would unnecessarily sit in memory, using space. The incremental active approach solves this problem, while still being optimized for speed and does not pause the server to clean all the keys.

Note about "Big O Notation"

The Big O Notation is used to describe the time complexity of operations. In other words, it describes how much slower, or not, an operation would be, as the size of the elements it operates on increases.

The way that I like to think about it is to transcribe the O notation to a function with a single parameter n, that returns the value inside the parentheses after O. So O(n) — which is the complexity of the KEYS command — would become def fn(n); n; end; if written in Ruby, or let fn = (n) => n in javascript. O(1) — which is the complexity of the SET command — would be def fn(n); 1; end;, O(logn) — which is the complexity of the ZADD command — would become def fn(n); Math.log(n); end; and O(n^2) — as far as I know, no Redis command has such complexity — would become def fn(n); n.pow(2); end;.

We can play with these functions to illustrate the complexity of the different commands. SET has a time complexity of O(1), commonly referred to as constant time. Regardless of the number of keys stored in Redis, the operations required to fulfill a SET command are the same, so whether we are operating on an empty Redis server or one with millions of keys, it'll take a similar amount of time. With the function defined above we can see that, if n is the number of keys, fn(n) will always return 1, regardless of n.

On the other hand KEYS has a complexity of O(n), where n is the number of keys stored in Redis.

It's important to note that n is always context dependent and should therefore always be specified, which Redis does on each command page. In comparison, DEL is also documented with having a time complexity of O(n), but, and this is the important part, where n is the number of keys given to the command. Calling DEL a-key has therefore a time complexity of O(1), and runs in constant time.

KEYS iterates through all the items in Redis and return all the keys. With the function defined above, we can see that fn(1) will return 1, fn(10), 10, and so on. What this tells us is that the time required to execute KEYS will grow proportionally to the value of n.

Lastly, it's important to note that this does not necessarily mean that KEYS ran on a server with 100 items will be exactly 100 times slower than running against a server with one key. There are some operations that will have to be performed regardless, such as parsing the command and dispatching to the keysCommand function. These are in the category of "fixed cost", they always have to be performed. If it takes 1ms to run those and then 0.1ms per key — these are only illustrative numbers —, it would take Redis 1.1ms to run KEYS for one key and 10.1ms with 100 keys. It's not exactly 100 times more, but it is in the order of 100 times more.

The NX, XX & KEEPTTL options

These options are easier to implement compared to the previous two given that they are not followed by a value. Additionally, their behavior does not require implementing more components to the server, beyond a few conditions in the method that takes care of storing the key and the value specified by the SET command.

Most of the complexity resides in the validations of the command, to make sure that it has a valid format.

Adding validation

Prior to adding these options, validating the SET command did not require a lot of work. In its simple form, it requires a key and value. If both are present, the command is valid, if one is missing, it is a "wrong number of arguments" error.

This rule still applies but we need to add a few more to support the different combinations of possible options. These are the rules we now need to support:

You can only specify one of the PX or EX options, not both. Note that the redis-cli has a user friendly interface that hints at this constraint by displaying the following key value [EX seconds|PX milliseconds] [NX|XX] [KEEPTTL] when you start typing SET. The | character between EX seconds & PX milliseconds expresses the OR condition.
Following the hints from redis-cli, we can only specify NX or XX, not both.
The redis-cli hint does not make this obvious, but you can only specify KEEPTTL if neither EX or PX or present. The following command SET 1 2 EX 1 KEEPTTL is invalid and returns (error) ERR syntax error

It's also worth mentioning that the order of options does not matter, both commands are equivalent:

SET a-key a-value NX EX 10

SET a-key a-value EX 10 NX

But the following would be invalid, EX must be followed by an integer:

SET a-key a-value EX NX 10

Let's write some code!

We are making the following changes to the server:

Add our own, simplified event loop, including support for time events
Accepts options for the expiration related options, EX & PX
Accepts options for the presence or absence of a key, NX & XX
Delete expired keys on read
Setting a key without KEEPTTL removes any previously set TTL
Implement the TTL & PTTL commands as they are useful to use alongside keys with a TTL

I'm giving you the complete code first and we'll look at the interesting parts one by one afterwards:

require 'socket'
require 'timeout'
require 'logger'
LOG_LEVEL = ENV['DEBUG'] ? Logger::DEBUG : Logger::INFO

require_relative './expire_helper'
require_relative './get_command'
require_relative './set_command'
require_relative './ttl_command'
require_relative './pttl_command'

class RedisServer

  COMMANDS = {
    'GET' => GetCommand,
    'SET' => SetCommand,
    'TTL' => TtlCommand,
    'PTTL' => PttlCommand,
  }

  MAX_EXPIRE_LOOKUPS_PER_CYCLE = 20
  DEFAULT_FREQUENCY = 10 # How many times server_cron runs per second

  TimeEvent = Struct.new(:process_at, :block)

  def initialize
    @logger = Logger.new(STDOUT)
    @logger.level = LOG_LEVEL

    @clients = []
    @data_store = {}
    @expires = {}

    @server = TCPServer.new 2000
    @time_events = []
    @logger.debug "Server started at: #{ Time.now }"
    add_time_event(Time.now.to_f.truncate + 1) do
      server_cron
    end

    start_event_loop
  end

  private

  def add_time_event(process_at, &block)
    @time_events << TimeEvent.new(process_at, block)
  end

  def nearest_time_event
    now = (Time.now.to_f * 1000).truncate
    nearest = nil
    @time_events.each do |time_event|
      if nearest.nil?
        nearest = time_event
      elsif time_event.process_at < nearest.process_at
        nearest = time_event
      else
        next
      end
    end

    nearest
  end

  def select_timeout
    if @time_events.any?
      nearest = nearest_time_event
      now = (Time.now.to_f * 1000).truncate
      if nearest.process_at < now
        0
      else
        (nearest.process_at - now) / 1000.0
      end
    else
      0
    end
  end

  def start_event_loop
    loop do
      timeout = select_timeout
      @logger.debug "select with a timeout of #{ timeout }"
      result = IO.select(@clients + [@server], [], [], timeout)
      sockets = result ? result[0] : []
      process_poll_events(sockets)
      process_time_events
    end
  end

  def process_poll_events(sockets)
    sockets.each do |socket|
      begin
        if socket.is_a?(TCPServer)
          @clients << @server.accept
        elsif socket.is_a?(TCPSocket)
          client_command_with_args = socket.read_nonblock(1024, exception: false)
          if client_command_with_args.nil?
            @clients.delete(socket)
          elsif client_command_with_args == :wait_readable
            # There's nothing to read from the client, we don't have to do anything
            next
          elsif client_command_with_args.strip.empty?
            @logger.debug "Empty request received from #{ client }"
          else
            commands = client_command_with_args.strip.split("\n")
            commands.each do |command|
              response = handle_client_command(command.strip)
              @logger.debug "Response: #{ response }"
              socket.puts response
            end
          end
        else
          raise "Unknown socket type: #{ socket }"
        end
      rescue Errno::ECONNRESET
        @clients.delete(socket)
      end
    end
  end

  def process_time_events
    @time_events.delete_if do |time_event|
      next if time_event.process_at > Time.now.to_f * 1000

      return_value = time_event.block.call

      if return_value.nil?
        true
      else
        time_event.process_at = (Time.now.to_f * 1000).truncate + return_value
        @logger.debug "Rescheduling time event #{ Time.at(time_event.process_at / 1000.0).to_f }"
        false
      end
    end
  end

  def handle_client_command(client_command_with_args)
    @logger.debug "Received command: #{ client_command_with_args }"
    command_parts = client_command_with_args.split
    command_str = command_parts[0]
    args = command_parts[1..-1]

    command_class = COMMANDS[command_str]

    if command_class
      command = command_class.new(@data_store, @expires, args)
      command.call
    else
      formatted_args = args.map { |arg| "`#{ arg }`," }.join(' ')
      "(error) ERR unknown command `#{ command_str }`, with args beginning with: #{ formatted_args }"
    end
  end

  def server_cron
    start_timestamp = Time.now
    keys_fetched = 0

    @expires.each do |key, _|
      if @expires[key] < Time.now.to_f * 1000
        @logger.debug "Evicting #{ key }"
        @expires.delete(key)
        @data_store.delete(key)
      end

      keys_fetched += 1
      if keys_fetched >= MAX_EXPIRE_LOOKUPS_PER_CYCLE
        break
      end
    end

    end_timestamp = Time.now
    @logger.debug do
      sprintf(
        "Processed %i keys in %.3f ms", keys_fetched, (end_timestamp - start_timestamp) * 1000)
    end

    1000 / DEFAULT_FREQUENCY
  end
end

listing 4.1: server.rb

class SetCommand

  ValidationError = Class.new(StandardError)

  CommandOption = Struct.new(:kind)
  CommandOptionWithValue = Struct.new(:kind, :validator)

  OPTIONS = {
    'EX' => CommandOptionWithValue.new(
      'expire',
      ->(value) { validate_integer(value) * 1000 },
    ),
    'PX' => CommandOptionWithValue.new(
      'expire',
      ->(value) { validate_integer(value) },
    ),
    'KEEPTTL' => CommandOption.new('expire'),
    'NX' => CommandOption.new('presence'),
    'XX' => CommandOption.new('presence'),
  }

  ERRORS = {
    'expire' => '(error) ERR value is not an integer or out of range',
  }

  def self.validate_integer(str)
    Integer(str)
  rescue ArgumentError, TypeError
    raise ValidationError, '(error) ERR value is not an integer or out of range'
  end

  def initialize(data_store, expires, args)
    @logger = Logger.new(STDOUT)
    @logger.level = LOG_LEVEL
    @data_store = data_store
    @expires = expires
    @args = args

    @options = {}
  end

  def call
    key, value = @args.shift(2)
    if key.nil? || value.nil?
      return "(error) ERR wrong number of arguments for 'SET' command"
    end

    parse_result = parse_options

    if !parse_result.nil?
      return parse_result
    end

    existing_key = @data_store[key]

    if @options['presence'] == 'NX' && !existing_key.nil?
      '(nil)'
    elsif @options['presence'] == 'XX' && existing_key.nil?
      '(nil)'
    else

      @data_store[key] = value
      expire_option = @options['expire']

      # The implied third branch is if expire_option == 'KEEPTTL', in which case we don't have
      # to do anything
      if expire_option.is_a? Integer
        @expires[key] = (Time.now.to_f * 1000).to_i + expire_option
      elsif expire_option.nil?
        @expires.delete(key)
      end

      'OK'
    end

  rescue ValidationError => e
    e.message
  end

  private

  def parse_options
    while @args.any?
      option = @args.shift
      option_detail = OPTIONS[option]

      if option_detail
        option_values = parse_option_arguments(option, option_detail)
        existing_option = @options[option_detail.kind]

        if existing_option
          return '(error) ERR syntax error'
        else
          @options[option_detail.kind] = option_values
        end
      else
        return '(error) ERR syntax error'
      end
    end
  end

  def parse_option_arguments(option, option_detail)

    case option_detail
    when CommandOptionWithValue
      option_value = @args.shift
      option_detail.validator.call(option_value)
    when CommandOption
      option
    else
      raise "Unknown command option type: #{ option_detail }"
    end
  end
end

listing 4.2: set_command.rb

class GetCommand

  def initialize(data_store, expires, args)
    @logger = Logger.new(STDOUT)
    @logger.level = LOG_LEVEL
    @data_store = data_store
    @expires = expires
    @args = args
  end

  def call
    if @args.length != 1
      "(error) ERR wrong number of arguments for 'GET' command"
    else
      key = @args[0]
      ExpireHelper.check_if_expired(@data_store, @expires, key)
      @data_store.fetch(key, '(nil)')
    end
  end
end

listing 4.3: get_command.rb

class PttlCommand

  def initialize(data_store, expires, args)
    @logger = Logger.new(STDOUT)
    @logger.level = LOG_LEVEL
    @data_store = data_store
    @expires = expires
    @args = args
  end

  def call
    if @args.length != 1
      "(error) ERR wrong number of arguments for 'PTTL' command"
    else
      key = @args[0]
      ExpireHelper.check_if_expired(@data_store, @expires, key)
      key_exists = @data_store.include? key
      if key_exists
        ttl = @expires[key]
        if ttl
          (ttl - (Time.now.to_f * 1000)).round
        else
          -1
        end
      else
        -2
      end
    end
  end
end

listing 4.4: pttl_command.rb

class TtlCommand

  def initialize(data_store, expires, args)
    @data_store = data_store
    @expires = expires
    @args = args
  end

  def call
    if @args.length != 1
      "(error) ERR wrong number of arguments for 'TTL' command"
    else
      pttl_command = PttlCommand.new(@data_store, @expires, @args)
      result = pttl_command.call.to_i
      if result > 0
        (result / 1000.0).round
      else
        result
      end
    end
  end
end

listing 4.5: ttl_command.rb

module ExpireHelper

  def self.check_if_expired(data_store, expires, key)
    expires_entry = expires[key]
    if expires_entry && expires_entry < Time.now.to_f * 1000
      logger.debug "evicting #{ key }"
      expires.delete(key)
      data_store.delete(key)
    end
  end

  def self.logger
    @logger ||= Logger.new(STDOUT).tap do |l|
      l.level = LOG_LEVEL
    end
  end
end

listing 4.6: expire_helper.rb

The changes

Splitting it the logic in multiple files

The server.rb file started getting pretty big so we extracted the logic for GET & SET to different files, and gave them their own classes.

Time events

In order to implement the eviction logic for keys having an expiration, we refactored how we call the IO.select method. Our implementation is loosely based on the one built in Redis, ae. The RedisServer — renamed from BasicServer in the previous chapters — starts the event loop in its constructor. The event loop is a never ending loop that calls select, processes all the incoming events and then process time events, if any need to be processed.

We introduced the TimeEvent class, defined as follows:

TimeEvent = Struct.new(:process_at, :block)

The process_at field is an Integer that represents the timestamp, in milliseconds, for when the event should be processed. The block field is the actual code that will be run. For now, there's only one type of events, server_cron. It is first added to the @time_events list with a process_at value set to 1ms in the future.

Time events can be either one-off, they'll run only once, or repeating, they will be rescheduled at some point in the future after being processed. This behavior is driven by the return value of the block field. If the block returns nil, the time event is removed from the @time_events list, if it returns an integer return_value, the event is rescheduled for return_value milliseconds in the future, by changing the value of process_at. By default the server_cron method is configured with a frequency of 10 Hertz (hz), which means it will run up to 10 times per second, or put differently, every 100ms. This is why the return value of server_cron is 1000 / DEFAULT_FREQUENCY — 1000 is the number of milliseconds, if frequency was 20, it would return 50, as in, it should run every 50ms.

This behavior makes sure that we don't run the server_cron method too often, it effectively gives a higher priority to handling client commands, and new clients connecting.

Select timeout

When we introduced IO.select in Chapter 3, we used it without the timeout argument. This wasn't a problem then because the server had nothing else to do. It would either need to accept a new client, or reply to a client command, and both would be handled through select.

The server needs to do something else beside waiting on select now, run the time events when they need to be processed. In order to do so, Redis uses a timeout with its abstraction over select and other multiplexing libraries, aeApiPoll. Redis can, under some conditions, use no timeout, which we're going to ignore for now, for the sake of simplicity. When using a timeout, Redis makes sure that waiting on the timeout will not delay any future time events that should be processed instead of waiting on select. In order to achieve this, Redis looks at all the time events and finds the nearest one, and sets a timeout equivalent to the time between now and that event. This guarantees that even if there's no activity between now and when the next time event should be processed, redis will stop waiting on aeApiPoll and process the time events.

We're replicating this logic in the select_timeout method. It starts by delegating the task of finding the nearest time event through the nearest_time_event, which iterates through all the time events in the @time_events array and find the one with the smallest value for process_at.

In concrete terms, in RedisServer, server_cron runs every 100ms, so when we call IO.select, the next time event will be at most 100ms in the future. The timeout given to select will be a value between 0 and 100ms.

Parsing options

Probably one of the most complicated changes introduced in this chapter, at least for me as I was implementing it. The logic is in the SetCommand class. We first define all the possible options in the OPTIONS constant. Each option is a key/value pair where the key is the option as expected in the command string and the value is an instance of CommandOption or CommandOptionWithValue. After extracting and validating the first three elements of the string, respectively, the SET string, followed by the key and the value, we split the rest on spaces and process them from left to right, with the shift method. For every option we find, we look up the OPTIONS hash to retrieve the matching CommandOption or CommandOptionWithValue instance. If nil is returned, it means that the given option is invalid, this is a syntax error. Note that once again, for the sake of simplicity, we did not implement case insensitive commands the way Redis does.

If the an option is found, but we had already found one of the same kind, presence or expire, this is also a syntax error. This check allows us to consider the following commands as invalid:

SET key value EX 1 PX 2
SET key value EX 1 KEEPTTL
SET key value NX XX

Finally, we attempt to parse the option argument, if necessary, only EX and PX have an argument, the others one do not, this is why we use two different classes here. parse_option_arguments will return the option itself if we found an option that should not be followed by an argument, that is either NX, XX or KEEPTTL. If we found one of the other two options, option_detail will be an instance of CommandOptionWithValue, we use shift once again to obtain the next element in the command string and feed it to the validator block.

The validator blocks are very similar for the options, they both validate that the string is a valid integer, but the EX validator multiplies the final result by 1000 to convert the value from seconds to milliseconds.

The values are then stored in the @options hash, with either the presence or expire key, based on the kind value. This allows us to read from the @options hash in the call method to apply the logic required to finalize the implementation of these options.

If @options['presence'] is set to NX and there is already a value at the same key, we return nil right away. Similarly if it is set to XX and there is no key, we also return nil.

Finally, we always set the value for the key in the @data_store hash, but the behavior regarding the secondary hash, @expires, is different depending on the value of @options['expire']. If it is set to an integer, we use this integer and add it to the current time, in milliseconds, in the @expires hash. If the value is nil, it means that KEEPTTL was not passed, so we remove any value that may have previously been set by a previous SET command with the same key and value for either PX or EX.

Why not use a regular expression?

Good question! The short answer is that after spending some time trying to use a regular expression, it did not feel easier, as a reference this where I got, just before I gave up:

/^SET \d+ \d+ (?:EX (?<ex>\d+)|PX (?<px>\d+)|KEEPTTL)?(:? ?(?<nx-or-xx>NX|XX))?(?: (?:EX (?<ex>\d+)|PX (?<px>\d+)))?$/

This regexp works for some cases but incorrectly considers the following as valid, Redis cannot process a SET command with KEEPTTL and EX 1:

SET 1 2 KEEPTTL XX EX 1

It might be possible to use a regular expression here, given that the grammar of the SET command does not allow that many permutations but even if it is, I don't think it'll be simpler than the solution we ended up with.

For reference, this is how Redis does it, in a way that it conceptually not that far from how we ended up doing it here. The main difference is that in the Redis source, it is one function, whereas we opted to separate the definition of the options, in the OPTIONS constant, from the actual code that consumes the characters from the string received from the client. I find the separated option a bit more readable and easier to reason about, but the approach used by Redis is definitely more efficient as there are less "things" being allocated, no extra resources to define what the options look like, just strings.

Lazy evictions

The server_cron time event takes care of cleaning up expired key every 100ms, but we also want to implement the "lazy eviction", the same way Redis does. That is, if server_cron hasn't had the chance to evict an expired key yet, and the server receives a GET command for the same key, we want to return nil and evict the key instead of returning it.

This logic is implemented in the ExpireHelper module , in the check_if_expired method. This method checks if there is an entry in the @expires hash, and if there is it compares its value, a timestamp in milliseconds with the current time. If the value in @expires is smaller, the key is expired and it deletes it. This will cause the GetCommand, TtlCommand & PttlCommand classes to return (nil) even if server_cron hasn't had a chance to delete the expired keys.

New commands: TTL & PTTL

We added two new commands, TTL & PTTL. Both return the ttl of the given key as an integer, if it exists, the difference is that TTL returns the value in seconds, whereas PTTL returns it in milliseconds.

Given the similarity of these two commands, we only implemented the logic in the PttlCommand class, and reused from the TtlCommand class where we transform the value in milliseconds to a value in seconds before returning it.

Logger

As the complexity of the codebase grew, it became useful to add logging statements. Such statements could be simple calls to puts, print or p, but it is useful to be able to conditionally turn them on and off based on their severity. Most of the logs we added are only useful when debugging an error and are otherwise really noisy. All these statements are logged with @logger.debug, and the severity of the logger is set based on the DEBUG environment variable. This allows us to enable all the debug logs by adding the DEBUG=t statement before running the server:

DEBUG=true ruby -r"./server" -e "RedisServer.new"

And a few more tests

We changed a lot of code and added more features, this calls for more tests.

We added a special instruction, sleep <duration> to allow us to easily write tests for the SET command with any of the expire based options. For instance, to test that SET key value PX 100 actually works as expected, we want to wait at least 100ms, and assert that GET key returns (nil) instead of value.

We also added a new way to specify assertion, with the syntax [ 'PTTL key', '2000+/-20' ]. This is useful for the PTTL command because it would be impossible to know exactly how long it'll take the computer running the tests to execute the PTTL command after running the SET command. We can however estimate a reasonable range. In this case, we are assuming that the machine running the test will take less than 20ms to run PTTL by leveraging the minitest assertion assert_in_delta.

I also added the option to set the DEBUG environment variable, which you can use when running all the tests or an individual test:

// All tests:
DEBUG=t ruby test.rb // Any values will work, even "false", as long as it's not nil
// Or a specific test
DEBUG=t ruby test.rb --name "RedisServer::SET#test_0005_handles the PX option with a valid argument"

There is now a begin/rescue for Interrupt in the forked process. This is to prevent an annoying stacktrace from being logged when we kill the process with Process.kill('INT', child) after sending all the commands to the server.

require 'minitest/autorun'
require 'timeout'
require 'stringio'
require './server'

describe 'RedisServer' do

  # ...

  def with_server

    child = Process.fork do
      unless !!ENV['DEBUG']
        # We're effectively silencing the server with these two lines
        # stderr would have logged something when it receives SIGINT, with a complete stacktrace
        $stderr = StringIO.new
        # stdout would have logged the "Server started ..." & "New client connected ..." lines
        $stdout = StringIO.new
      end

      begin
        RedisServer.new
      rescue Interrupt => e
        # Expected code path given we call kill with 'INT' below
      end
    end

    yield

  ensure
    if child
      Process.kill('INT', child)
      Process.wait(child)
    end
  end

  def assert_command_results(command_result_pairs)
    with_server do
      command_result_pairs.each do |command, expected_result|
        if command.start_with?('sleep')
          sleep command.split[1].to_f
          next
        end
        begin
          socket = connect_to_server
          socket.puts command
          response = socket.gets
          # Matches "2000+\-10", aka 2000 plus or minus 10
          regexp_match = expected_result.match /(\d+)\+\/-(\d+)/
          if regexp_match
            # The result is a range
            assert_in_delta regexp_match[1].to_i, response.to_i, regexp_match[2].to_i
          else
            assert_equal expected_result + "\n", response
          end
        ensure
          socket.close if socket
        end
      end
    end
  end


  # ...

  describe 'TTL' do
    it 'handles unexpected number of arguments' do
      assert_command_results [
        [ 'TTL', '(error) ERR wrong number of arguments for \'TTL\' command' ],
      ]
    end

    it 'returns the TTL for a key with a TTL' do
      assert_command_results [
        [ 'SET key value EX 2', 'OK'],
        [ 'TTL key', '2' ],
        [ 'sleep 0.5' ],
        [ 'TTL key', '1' ],
      ]
    end

    it 'returns -1 for a key without a TTL' do
      assert_command_results [
        [ 'SET key value', 'OK' ],
        [ 'TTL key', '-1' ],
      ]
    end

    it 'returns -2 if the key does not exist' do
      assert_command_results [
        [ 'TTL key', '-2' ],
      ]
    end
  end

  describe 'PTTL' do
    it 'handles unexpected number of arguments' do
      assert_command_results [
        [ 'PTTL', '(error) ERR wrong number of arguments for \'PTTL\' command' ],
      ]
    end

    it 'returns the TTL in ms for a key with a TTL' do
      assert_command_results [
        [ 'SET key value EX 2', 'OK'],
        [ 'PTTL key', '2000+/-20' ], # Initial 2000ms +/- 20ms
        [ 'sleep 0.5' ],
        [ 'PTTL key', '1500+/-20' ], # Initial 2000ms, minus ~500ms of sleep, +/- 20ms
      ]
    end

    it 'returns -1 for a key without a TTL' do
      assert_command_results [
        [ 'SET key value', 'OK' ],
        [ 'PTTL key', '-1' ],
      ]
    end

    it 'returns -2 if the key does not exist' do
      assert_command_results [
        [ 'PTTL key', '-2' ],
      ]
    end
  end

  # ...

  describe 'SET' do

    # ...

    it 'handles the EX option with a valid argument' do
      assert_command_results [
        [ 'SET 1 3 EX 1', 'OK' ],
        [ 'GET 1', '3' ],
        [ 'sleep 1' ],
        [ 'GET 1', '(nil)' ],
      ]
    end

    it 'rejects the EX option with an invalid argument' do
      assert_command_results [
        [ 'SET 1 3 EX foo', '(error) ERR value is not an integer or out of range']
      ]
    end

    it 'handles the PX option with a valid argument' do
      assert_command_results [
        [ 'SET 1 3 PX 100', 'OK' ],
        [ 'GET 1', '3' ],
        [ 'sleep 0.1' ],
        [ 'GET 1', '(nil)' ],
      ]
    end

    it 'rejects the PX option with an invalid argument' do
      assert_command_results [
        [ 'SET 1 3 PX foo', '(error) ERR value is not an integer or out of range']
      ]
    end

    it 'handles the NX option' do
      assert_command_results [
        [ 'SET 1 2 NX', 'OK' ],
        [ 'SET 1 2 NX', '(nil)' ],
      ]
    end

    it 'handles the XX option' do
      assert_command_results [
        [ 'SET 1 2 XX', '(nil)'],
        [ 'SET 1 2', 'OK'],
        [ 'SET 1 2 XX', 'OK'],
      ]
    end

    it 'removes ttl without KEEPTTL' do
      assert_command_results [
        [ 'SET 1 3 PX 100', 'OK' ],
        [ 'SET 1 2', 'OK' ],
        [ 'sleep 0.1' ],
        [ 'GET 1', '2' ],
      ]
    end

    it 'handles the KEEPTTL option' do
      assert_command_results [
        [ 'SET 1 3 PX 100', 'OK' ],
        [ 'SET 1 2 KEEPTTL', 'OK' ],
        [ 'sleep 0.1' ],
        [ 'GET 1', '(nil)' ],
      ]
    end

    it 'accepts multiple options' do
      assert_command_results [
        [ 'SET 1 3 NX EX 1', 'OK' ],
        [ 'GET 1', '3' ],
        [ 'SET 1 3 XX KEEPTTL', 'OK' ],
      ]
    end

    it 'rejects with more than one expire related option' do
      assert_command_results [
        [ 'SET 1 3 PX 1 EX 2', '(error) ERR syntax error'],
        [ 'SET 1 3 PX 1 KEEPTTL', '(error) ERR syntax error'],
        [ 'SET 1 3 KEEPTTL EX 2', '(error) ERR syntax error'],
      ]
    end

    it 'rejects with both XX & NX' do
      assert_command_results [
        [ 'SET 1 3 NX XX', '(error) ERR syntax error'],
      ]
    end
  end

  # ...
end

Conclusion

The SET commands implemented by RedisServer now behaves the same way it does with Redis. Well, almost. Let's take a look at what happens if we were to use redis-cli against our own server. Let's start by running our server with

ruby -r"./server" -e "RedisServer.new"

and in another shell open redis-cli on port 2000:

redis-cli -p 2000

And type the following:

SET key value EX 200

And boom! It crashes!

Error: Protocol error, got "(" as reply type byte

This is because RedisServer does not implement the Redis Protocol, RESP. This is what the next chapter is all about. At the end of chapter 5 we will be able to use redis-cli against our own server. Exciting!

Code

As usual, the code is available on GitHub.

Appendix A: Links to the Redis source code

If you're interested in digging into the Redis source code but would like some pointers as to where to start, you've come to the right place. The Redis source code is really well architected and overall relatively easy to navigate, so you are more than welcome to start the adventure on your own. That being said, it did take me a while to find the locations of functions I was interested in, such as: "where does redis handle the eviction of expired keys", and a few others.

Before jumping in the code, you might want to read this article that explains some of the main data structures used by Redis: http://blog.wjin.org/posts/redis-internal-data-structure-dictionary.html.

In no particular orders, the following is a list of links to the Redis source code on GitHub, for features related to the implementation of keys with expiration:

Handling of the SET command:

server.c defines all the commands in redisCommand: https://github.com/antirez/redis/blob/6.0.0/src/server.c#L182
t_string.c defines the handler in setCommand: https://github.com/antirez/redis/blob/6.0.0/src/t_string.c#L97-L147
t_string.c defines a more specific handlers after options are parsed where expire values are handled: https://github.com/antirez/redis/blob/6.0.0/src/t_string.c#L71-L79 & https://github.com/antirez/redis/blob/6.0.0/src/t_string.c#L89
db.c defines the setExpire function: https://github.com/antirez/redis/blob/6.0.0/src/db.c#L1190-L1206

Key deletion in `serverCron`

server.c defines the handler for GET: https://github.com/antirez/redis/blob/6.0.0/src/server.c#L187-L189
t_string.c defines the handler for getCommand: https://github.com/antirez/redis/blob/6.0.0/src/t_string.c#L179-L181 & the generic one: https://github.com/antirez/redis/blob/6.0.0/src/t_string.c#L164-L177
db.c defines lookupKeyReadOrReply: https://github.com/antirez/redis/blob/6.0.0/src/db.c#L163-L167
db.c defines lookupKeyRead https://github.com/antirez/redis/blob/6.0.0/src/db.c#L143-L147 as well as lookupKeyReadWithFlags: https://github.com/antirez/redis/blob/6.0.0/src/db.c#L149-L157
db.c defines expireIfNeeded: https://github.com/antirez/redis/blob/6.0.0/src/db.c#L1285-L1326
expire.c defines activeExpireCycleTryExpire which implements the deletion of expired keys: https://github.com/antirez/redis/blob/6.0.0/src/expire.c#L35-L74
expire.c defines activeExpireCycle which implement the sampling of keys and the logic to make sure that there are not too many expired keys in the expires dict: https://github.com/redis/redis/blob/6.0.0/src/expire.c#L123

Appendix B: Playing with RedisServer using `nc`

If you want to manually interact with the server, an easy way is to use nc, the same way we used in Chapter 1. nc has no awareness of the Redis command syntax, so it will not stop you from making typos:

❯ nc localhost 2000
GET 1
(nil)
SET 1 2
OK
GET 1
2
SET 1 2 EX 5
OK
GET 1
2
GET 1
2
GET 1
(nil)
SET 1 2 XX
(nil)
SET 1 2 NX
OK
SET 1 2 XX
OK
DEL 1
(error) ERR unknown command `DEL`, with args beginning with: `1`,
SET 1 2 PX 100
OK
SET 1 2 XX
(nil)

Forem: Pierre Jambet

A toy Redis Server, in Clojure

What we're building

Starting small, a TCP server and nothing else

Keeping the connections open

Making the server stateful

Channels

Putting everything together.

Laying the foundations

SET

DEL

INCR

Atoms

Updating our server

Conclusion

A toy Redis Server, in Go

What we're building

The TCP part

Reading command arguments

Handling commands, a first attempt

GET

SET

INCR

DEL

The full version

A race-condition-free version

The full version

Want more?

Links:

Active Record caches generated queries for find_by queries

What exactly does Active Record cache, and when?

Why does it actually matter?

Final words

Links:

Select Syscall in Rust

Disclaimer

The select syscall

Using the libc crate

Show me some code

Conclusion

Ruby Symbol to Proc explained, the short version

What's with this weird looking syntax with an ampersand and a symbol

It's not just for parameter-less methods

We can pass almost anything after the ampersand

Links:

Conclusion

Appendix, Arguments vs Parameters

Rebuilding Redis in Ruby - Chapter 10 - Adding Sorted Sets Commands

What we'll cover

How Redis does it

Updating our SortedArray class

Final touches

Creating and Updating Sorted Sets

The ZADD command

Adding Members to a ZSet

Counting members in a Sorted Set

Reading from Sorted Sets

By Rank

By Score

A note about blocks, arity and Ruby being weird

Serializing a ZSet

Serializing a List

By Lexicographic Order

Set Operations, Union & Intersection

Set Intersection

The Array#zip method

Finalizing the set intersection operation

Sorted Set Intersection, but store the result this time

Set Union

Sorted Set Union stored instead of returned

Member data retrieval commands

Remove commands

Rank Ranges

Lex Ranges

Score Ranges

Reverse commands

Reverse Range by Score and Lex

ZREVRANGEBYLEX

ZREVRANGEBYSCORE

ZREVRANK

The `select` syscall

Using the `libc` crate

Serializing a `ZSet`

Serializing a `List`

The `Array#zip` method

Creating a Set with `SADD`

Appendix A: A more idiomatic `IntSet`

Our own `Dict` class

`Dict`, `HashTable` & `DictEntry`

No more `Hash` & `{}`

Adding the `DEL` command

The `COMMAND` command