Forem: tarantool

Raft (not)almighty: how to make it more robust

tarantool — Mon, 08 Aug 2022 11:51:00 +0000

Everybody loves Raft. There is a common opinion that the presence of this algorithm in a distributed system implies that this system will be just fine. Namely:

As long as most nodes in the cluster are live and connected to each other, the cluster will be writable (with pauses for leader election).
If the leader is functional and connected to the majority, the cluster will be writable at all times.
If the leader goes down, a new leader will be chosen “quickly” — whatever that means.

In fact, simply following the Raft specification (https://raft.github.io/raft.pdf or its version from the Diego Ongaro's PhD thesis) is not enough to achieve everything listed above. Even etcd got burned on this once, leading to the Cloudflare outage in 2020.

My name is Sergey Petrenko, and I have been working on replication in Tarantool for 4 years now. Today I want to tell you about the weak spots of the Raft algorithm and ways to overcome them. This article is a free paraphrase of my and Boris Stepanenko's speech at Hydra 2022. If you are not familiar with Raft, I suggest you read my article on it.

Let me start a little bit from afar. Suppose you have a system with Raft and want to use it in production. We've all heard — and more than once — that Raft guarantees no more than one leader in each term, and it can survive the loss of just under half of the nodes in the cluster without losing performance. That is, if the leader is live, it will be able to process entries, and if there is no leader, a new one will be chosen. It would seem that apart from these guarantees, we don't need anything else for a prolonged operation of the system. Is this really the case? We're about to find out.

First, let's take a closer look at the claim that the cluster will remain operational as long as most of the servers are live and connected to each other. The misunderstanding of this claim is what eventually resulted in the Cloudflare outage.

Pre-Vote

Let's begin with an example. We'll look at how a Raft cluster will behave in the case of partial loss of connectivity.

Suppose our cluster consists of three nodes: A, B, and C. A is the leader. B and C are its replicas; it's the 2022nd term. What does the loss of the connection between A and C result in? Without a single heartbeat from the leader for election_timeout, server C decides that the leader is missing and starts the election in term 2023.

Server B is the only one to whom server C can send a RequestVote.

Once server B receives a request with a larger term, it will increase its own term as well. However, it won't necessarily vote for C, as B might have more recent data if the leader has written something since the connection between A and C was lost.

In response to the next message from the leader (which is still server A), server B will report that the term has increased, forcing server A to resign. The cluster is left without a leader, so the election starts. Until the leader is chosen, the cluster is not writable.

If there happen to be no new entries since the connection was lost, then server B will vote for server C as soon as it receives its RequestVote.

And if that happens, we're caught in an infinite loop:

Server C can't see the leader.
When election_timeout expires, server C starts the new election.
The increase of the term reaches server A through server B.
Server B votes for server C, so
server A resigns.
A and C change places, and the whole thing repeats from step 1, but in the new term.

A client connected to such a cluster with a blinking leader will most likely not be able to write anything at all. As soon as the acting leader is elected and the client tries to perform a write operation, the leader already resigns without even knowing where it should forward the write request, as it does not see the new leader directly. All of this will keep on happening until, by some miracle, at least one entry can be made, or until the connection between A and C is restored. This is exactly what happened in Cloudflare with etcd. And the external system quite reasonably decided that if it cannot write anything to the etcd cluster, it means that most of the nodes in it have failed and emergency measures are to be applied. Just 6 minutes of etcd being unavailable for writing led to an accident lasting more than 6 hours.

You might feel like some of Raft's warranties had been violated here. Actually, no.

At every moment the leader is connected to the majority (including itself), and the leader itself does not fail. And yet our expectations are shattered. The cluster is not operational.

But the thing is, Raft never promised operability in such a situation. The promise it made was that “a leader will be chosen if most of the nodes are functional and connected to each other”. There's no mention of the leader being permanent at all. And the guarantee that a leader will be chosen is not actually breached. In the example above, it is determined approximately once per election_timeout. It turns out that, in practice, Raft's guarantees are not sufficient.

But that's only half the trouble.

Let's go back to term 2022, and assume that in our cluster of nodes A, B, and C, node C has lost all communication with the others. Server A is the leader, it's still connected to server B and can process write requests. So far, so good.

Server C, as in the example above, will start a new election every election_timeout. Of course, server С can't win any of them. It just won't get a majority of votes. However, its term will grow infinitely, and at the moment of restoring connectivity, server A will again resign as soon as it sees the larger term. The cluster went read-only again for at least one round of elections for no apparent reason.

Raft's solution

The problem of such “disruptive” servers is actually brought up by the author of Raft, Diego Ongaro, in his dissertation. He does, however, mention them in relation to configuration changes.

The solution proposed by him is a new stage of the election called Pre-Vote. The idea is that each server sends out a mock vote request to everyone prior to an election, which is called PreVote. This request contains the same fields a normal RequestVote does, and the voter responds to it using the same rules as for RequestVote. The only difference is that the voter does not raise the term upon receiving this request, and also answers negatively or simply ignores the request if it still sees the leader in the current term.

A candidate raises the term and starts a real election only after receiving confirmation that the majority of voters are ready to vote for it.

Let's check if PreVote really would have saved Cloudflare from that incident. Consider the same situation, where C lost connection with the leader, but not with the other replica. Although C can send to B a PreVote request, B will respond to it negatively, since it still sees the leader. C will not be able to get a majority — 2 answers to its PreVote.

In the case of complete loss and subsequent restoration of connectivity, PreVote will also help. Indeed, server C will not be able to get any response to the PreVote request, and therefore will not start the election.

The only disadvantage of this solution for us is the violation of backward compatibility. Raft has been operating in Tarantool for some time now, and there is no PreVote request in the protocol. It can, of course, be added, but the old servers wouldn't know how to handle new requests. We would have to introduce new logic on the sender side: if the server is old, we don't send it a PreVote request and by default assume it responded positively. We don't like such redundancy, and aim to get rid of the existing extra code for supporting old versions. A solution that extends one of the existing requests would suit us better. Older servers simply ignore new fields in an existing request, which means no additional logic is needed. So that's the way we chose.

Our solution

“In Tarantool, the server knows enough not to start an election when it's not necessary”, we thought, and we did Pre-Vote our way. We made all the replicas tell the others whether they see the current leader or not. Having gathered information about who sees a leader and who does not, you can decide whether to start an election. If anyone sees a leader, or if there is no connection to the majority, the election wouldn't be initiated.

The strength of our solution is in its backward compatibility. We didn't need to introduce a new request and handle sending it to the old servers in a special way if there are servers with different versions of Tarantool in the cluster.

The downside is that we do not compare the leader's logs and the voters' logs, which means we start the election even if the voters' logs contain more recent data. That's not necessarily a bad thing: as long as someone sees a leader, there won't be an election. And when everyone stops seeing it, the election will start anyway. Whether it ends in one or several rounds is not so important, thanks to another one of our modifications, Split-Vote detection. Now, let's talk about it.

Split-Vote detection

I don't have a Cloudflare's or other famous company's scary case for you this time, but hopefully it will be interesting all the same.

As you probably know, not every election round ends with a leader emerging. For example, if several nodes in a cluster at the same time notice the leader missing, they will start the election independently, before they receive RequestVote requests from each other. The votes of the remaining replicas might get divided among several candidates in such a way that no single candidate will win a majority. We'll call this situation a split-vote.

Raft handles this by randomizing the election_timeout. On each server and in each round, a new random value is chosen, slightly different from the configured one. This increases the chances that only one candidate will have time to start the next election, and the others will receive a RequestVote from it before their own election_timeout expires. If Raft hadn't implemented this, the election might never have ended: all servers would have restarted in sync, casting their vote for themselves. But there's still room for optimization: every extra round of elections is an election_timeout of read-only cluster downtime.

Our solution

In fact, rounds that ended up with a draw are a waste of time. If a leader had been elected, it would probably have happened at the beginning of the round (approximately within the packet exchange time for the most distant servers). The election_timeout is much longer than this, according to Raft.

In addition, unlike the canonical Raft implementation, in Tarantool the server sends information about the vote not only to the candidate for whom it voted, but to all cluster members. This means that each of the servers can keep track of how many votes each candidate received. Upon seeing that none of the candidates can win in the current round anymore, the server speeds up the start of a new round. An average of 0.05 * election_timeout elapses from the moment a draw is detected to the start of a new round. It happens because we choose a random delay within election_timeout / 10 to restart the election. We did it for exactly the same purpose Raft did: to avoid a new draw. Thus, at best, we save approximately 0.9 * election_timeout on each round with a draw. This improvement is quite noticeable: electing the leader within two rounds in the original Raft means spending about election_timeout + some fraction of election_timeout time for that. With our implementation, however, two such rounds would take just a fraction of an election_timeout. Which is faster than it takes for the original Raft to realize that a draw has occurred.

In the worst case scenario, when a draw happens 2 or more times in a row, we process this even more rapidly. Every draw round is practically costless for us. Moreover, we do not increase the probability of a second draw in any way: the random delay for restarting the election in this case is generated within the same limits as in the usual situation.

CheckQuorum

Raft ensures that there cannot be two leaders in one term. However, nothing is said about the possibility of two leaders existing at one point in time: in the old term and in the new one. We would sure like to avoid that kind of situation. The reason is that clients connected to the old leader may not be aware that there is a new one. They would simply have no reason to look for the new leader until the old one actually resigned. The old leader will still be writable, and synchronous transactions will be rolled back after not gaining ACKs from most replicas.

For us, the presence of two leaders at one point in time can be critical, since Tarantool supports not only synchronous, but also asynchronous replication. Each leader is writable, and if you write asynchronous transactions on the old leader, the client may miss that the leader has changed at all. This will result in some clients reaching out to the new leader and some to the old one. And we're going to get a split brain. To avoid this, the old leader must resign and go read-only as soon as there is a chance of it being displaced.

The situation with two leaders existing is possible only when the old one has lost communication with most of the servers in the cluster. Indeed, since you have to get a majority to win an election, the only way the old leader will not know about the election is if it has no connection to the majority. If it was connected to at least one server that voted for a new leader, it would immediately resign. And there can be no situation where the old leader is connected to a majority among which no one voted, and at the same time the new leader is connected to a voted majority.

So, we want to make sure that the leader resigns as soon as it loses connection to the majority of the cluster. If the current leader's connectivity is lost, it's quite likely that the majority has already chosen a different one.

There is another reason why CheckQuorum is necessary: without it, the PreVote alone can cause the cluster to be locked: the current leader cannot write anything, and none of the replicas can start an election. Let's look at an example: suppose we had a cluster of five servers where D was the leader. What happened next was an extraordinary chain of events: first, server E crashed, then for some reason the connection between A and D broke, and finally, for some other reason, the connection between C and D broke.

As a result, although server D remains the leader, it can not commit synchronous transactions, as it can not gain three confirmations. Server B does not start the election since it still sees the leader — D. And servers A and C won't start the election because server B has told them that the leader is live.

CheckQuorum helps to save this situation by making a leader which loses connection with the majority resign and allow one of the replicas to take its place.

It remains to be decided at what point the leader should resign. For the blocked cluster example, it doesn't really matter. The main point here is that the leader will resign eventually, allowing its replicas to start the election.

If only synchronized replication is involved, everything will be fine. The speed of resigning is not critical, either. The old leader will not be able to confirm any of the synchronous transactions that came after the connection broke anyway.

If asynchronous transactions are involved, you have to do several things to ensure consistency.

The bare minimum is that the old leader must resign **strictly **before the replicas can elect a new leader. This is necessary to ensure that the cluster does not have two writable nodes at the same time. We will call this mode the strict CheckQuorum.

But that is not enough. We cannot guarantee that no asynchronous transaction will be written after a connection failure. In any case, it will take some time, however short, before a loss of connection is detected. This means that the old leader might write a transaction that won't be on the new one. When the connection is finally restored, the transaction will reach the other nodes, and the consistency of everything the new leader wrote may be compromised. Hence, our ultimate goal.

The ultimate goal: after restoring connectivity, the new leader and all its replicas must not apply the transactions logged by the previous leader. We will cover this in more detail in the next chapter, but for now let's talk about how to achieve the necessary minimum.

Both the leader and the replicas monitor the state of the connection via heartbeats. If no heartbeats were received from the server within 4 * replication_timeout, the connection is considered broken. A replica's heartbeat is a response to the master's heartbeat, and is sent only after receiving the heartbeat from the master.

As you can see, after the last heartbeat exchange, the leader restarts the connectivity timer later than the replica does. In the worst case, the last heartbeat from the replica will reach the leader exactly as the timeout expires. The leader does not know at what point the replica sent its heartbeat, and its timeout might have already elapsed by then.

If the replica in addition manages to hold a quick election, the old leader would resign too late.

So, in the very worst case, the leader will notice the connection failure twice as late as the replica. So, in order to guarantee the resignation of a leader strictly before the majority of nodes starts new election, the timeout on the leader needs to be half as much as the one on the replicas. That's what we are going to do.

Split-Brain detection

Tarantool allows you to configure a quorum. This is handy in a lot of cases; for example, you can urgently unblock a cluster in which most of the nodes have failed. But it is also dangerous: with a quorum lower than N / 2 + 1, there may be two unconnected leaders. Either in the same term, or in different ones. Both leaders can independently confirm synchronous transactions and write asynchronous ones. If you restore connectivity after two leaders have been working in the cluster for some time, the changes of one will overwrite the changes of the other. To prevent this, you need to detect transactions from a competing leader in the replication stream, and, without applying them, terminate the connection to the node that sent them.

PROMOTE entry

The marker of a new leader emergence is the PROMOTE entry. It contains the term in which the leader was elected, the ID of that leader, the ID of the previous leader, and the last LSN the previous leader received. This information is sufficient to construct a linear history of leadership from the very first term to the end. When the cluster is operating correctly, each incoming PROMOTE is matched with information known to the node. The term should be the largest of all PROMOTEs so far, the ID of the previous leader must match the ID in the previous PROMOTE, and the LSN of the previous leader — with the LSN of the last confirmed transaction of the previous leader.

If at least one of the conditions is not met, then a split-brain has occurred.

We also need to detect cases where the old leader kept confirming transactions after the new leader appeared in the cluster. In fact, any transaction coming from a node that did not send the last PROMOTE is an indicator of a split-brain.

This last example also solves our problem with strict CheckQuorum: now any transaction (both synchronous and asynchronous) coming from the old leader results in breaking the connection with it and would not be applied, thereby preserving data consistency on the new leader and its replicas. And so the old leader cannot influence the state of the new leader and its replicas.

Lessons learned

The canonical version of the Raft algorithm does not provide full cluster operability in case of partial loss of connectivity. To deal with this, the following two improvements are used: PreVote and CheckQuorum.

Our variation of the canonical implementation allowed faster elections with draw detection, although at the same it required additional modifications to ensure consistency: strict CheckQuorum and Split-Brain detection.

You can download Tarantool on the official website and get help in our Telegram chat.

How we wrote Tarantool Kubernetes Operator

tarantool — Fri, 29 Jul 2022 14:04:00 +0000

Author: Konstantin Nosorev

Kubernetes is a fast-growing open-source project that allows managing Linux containers as a single system. With Kubernetes, we can easily start complex systems using YAML configurations. Systems are managed via declarative resources. The hierarchical structure of resources allows creating large systems with a minimum of configuration files. That's why more and more people move their infrastructure to Kubernetes, including both stateless and stateful applications. So why deny yourself the convenience of using Tarantool inside Kubernetes?

Hi! My name is Kostya, and today I'll tell you about the problems we encountered while developing Tarantool Kubernetes Operator — its Enterprise version for Kubernetes/Openshift. Welcome, everyone who is interested!

Tarantool is an efficient platform for in-memory computation and building high-loaded applications. It combines a database and an application server. As a database, it has a number of unique characteristics: high efficiency of hardware management, flexible data schema, support for both in-memory and disk storage, and the ability to scale by using the Lua language. As an application server, the platform allows storing code very close to your data, thus achieving the minimum response time and maximum throughput.

The Tarantool ecosystem is constantly growing. Today it already has a lot of connectors for popular programming languages (Golang, Python, Java, etc.), extension modules for building applications with blocks (vshard, queue, etc.), and frameworks that speed up the development process (Cartridge and Luatest).

For now, I'd like to talk about applications developed with the Tarantool Cartridge framework. This framework is designed for developing complex distributed systems. With Tarantool Cartridge, you can focus on writing business logic instead of wasting time on solving problems concerning the infrastructure.

Main capabilities of Tarantool Cartridge:

• Automated orchestration of a Tarantool cluster
• Extending the application functionality with new roles
• Application template for development and deployment
• Built-in automated sharding
• Integration with the Luatest test framework
• Managing a cluster with WebUI and API
• Packaging and deployment tools

Each cluster application built with Cartridge is based on roles — Lua modules that describe application business logic. For example, it could be the modules that deal with storing data, provide the HTTP API or cache data from Oracle. A role is assigned to a replica set — a set of instances unified by replication. The role is then enabled on each replica set individually. Different replica sets can have a different set of roles.

For more information about Cartridge, see the following articles:

• Scaling clusters without any hassle
• Distributed storage in 30 minutes

Cartridge has cluster configuration stored on each cluster node. The configuration describes the topology of the cluster. You can also add some configuration that your role will use to it. Such configuration can be changed in runtime to manage role's behavior.

Working with a framework is fine when you don't have a lot of instances. But if you set up more than 100 instances, you might face some difficulties configuring and updating large clusters. That's where Kubernetes comes in to solve a large part of these problems. But what if we want to use all advantages of Kubernetes to simplify the process of deployment and support of Tarantool Cartridge? The answer then is Tarantool Kubernetes Operator.

A little bit about Kubernetes operators

Kubernetes operator is a program for managing applications inside Kubernetes. Operators are a part of the main reconciliation cycle, which is intended to bring the current cluster state closer to the one described in the resources. Simply put, it is a manager that helps solve some often arising situations automatically. The operator is designed to help people who are unfamiliar with the specifics of an application to deploy and operate this application in a Kubernetes cluster.

How does an operator work?

The operator follows the changes to the resources it is assigned to observe and reacts to these changes. Most often, operators use custom resource definitions (CRD) that describe some resource.

Let's consider the following situation involving Tarantool Kubernetes Operator. During installation with helm, the operator creates two CRDs, Cluster and Role.

Cluster description example:

apiVersion: tarantool.io/v1alpha1
kind: Cluster
metadata:
  name: tarantool-cluster
spec:
  roles:
    - name: router
    - name: storage
...

Role description example:

apiVersion: tarantool.io/v1alpha1
kind: Role
metadata:
  name: router
spec:
  replicasets: 1
  vshard:
    clusterRoles: 
    - failover-coordinator
    - app.roles.router
    replicasetTemplate:
        replicas: 2
        podTemplate:
          spec:
            containers:
              - name: cartridge
                image: "tarantool/tarantool-operator-examples-kv:0.0.4"
...

During live performance, for each replica set the operator creates a Statefulset, since this resource is necessary for Volume and Persistent volume claim (PVC, a template used to create Persistent volume for pods). The resulting hierarchy of Kubernetes resources looks like this:

• Cluster — the main resource including general cluster settings such as Cluster-wide config and Failover settings.
• Role — in this context, it is a Kubernetes resource; it includes a template description for replica sets, information about the assigned Cartridge roles, as well as the number of replica sets with such settings and Tarantool instances in each replica set.

Kubernetes resources hierarchy

The operator is based on Operator SDK (https://sdk.operatorframework.io/) and includes two main controllers: Cluster and Role.

Each controller implements the Reconciler interface and subscribes to changes of specific resources. This is how it looks in code:

func (r *RoleReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&Role{}).
        Watches(&source.Kind{Type: &appsV1.StatefulSet{}}, &handler.EnqueueRequestForOwner{
            IsController: true,
            OwnerType:    &Role{},
        }).
        Watches(&source.Kind{Type: &coreV1.Pod{}}, &handler.EnqueueRequestForOwner{
            IsController: true,
            OwnerType:    &Role{},
        }).
        Complete(r)
}

When resources to which the controller is subscribed change, the Reconcile method is called. The controller compares the resource configuration and the current state of the cluster, then fixes the difference.

Let's take a look at the Cluster controller example:

func (r *ClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    clusterContext := reconcilers.MakeReconciler(ctx, r.Reconciler)
    return clusterContext.RunCluster(ctx,
        reconcilers.GetObjectFromRequest(req),
        reconcilers.CheckDeletion,
        reconcilers.CheckFinalizer,
        reconcilers.SetupRolesOwnershipStep,
        reconcilers.SyncClusterWideServiceStep,
        reconcilers.WaitForRolesPhase(RoleReady),
        reconcilers.GetLeader,
        reconcilers.CreateTopologyClient,
        reconcilers.Bootstrap,
        reconcilers.SetupFailover,
        reconcilers.ApplyCartridgeConfig)
}

When you write a controller, keep in mind that the order of event processing is always random. So you can't expect that when a role resource changes, Reconcile will be called on the role controller first and then on the cluster controller, or vice versa.

Now that you know how the operator works, let's look at the main features of Tarantool Kubernetes Operator Enterprise. Currently, the operator can:

• Deploy a Cartridge cluster
• Change Failover configuration
• Perform a Rolling update
• Scale a cluster both ways: by the number of replica sets and by the number of replicas in each replica set
• Manage application settings
• Change Persistent volume without losing data or downtime, bypassing the Kubernetes restrictions (Kubernetes doesn't allow changing Persistent volume without recreating the resource).

Now let's move on to the difficulties we faced when we were writing the operator.

Divide and conquer

Development of the Enterprise version of the operator started with reevaluating its Community version where three CRDs were used to describe a cluster:

• Role
• ReplicasetTemplate (inherits Statefulset fields)
• Cluster

The first step was changing CRDs:

• Role
• Cluster

Our mistake was creating only one controller responsible for working with a cluster. This led to serious problems when we wanted to extend the operator's functionality. The code describing the Reconcile method began to grow very quickly. Each stage created at least 5 — 10 lines of code.

An example of a method for the cluster controller before its refactoring:

func (r *ClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := ctrlLog.FromContext(ctx)
    log.Info("Reconcile cluster")

    cluster, err := r.GetCluster(ctx, req.NamespacedName.Namespace, req.NamespacedName.Name)
    if err != nil {
        if !apiErrors.IsNotFound(err) {
            log.Error(err, "Unable to retrieve cluster")

            return reconcile.Result(
                ctx,
                reconcile.WithError(
                    errors.Wrap(err, "unable to retrieve cluster for reconcile"),
                    10*time.Second,
                ),
            )
        }

        return reconcile.Result(ctx)
    }
    ...
    return reconcile.Result(
            ctx,
            reconcile.WithClusterPhaseUpdate(r.Status(), cluster, ClusterReady),
    )
}

We managed to solve this problem by dividing the logic into several controllers — Cluster and Role.

Now Cluster deals only with the general cluster settings — Failover and application configuration.

The Role controller is responsible for making Statefulsets. Over them, this controller creates replica sets and settings for specific instances.

But we didn't stop there. The Reconcile methods have similar steps in both controllers: getting the current object, deleting the object, creating an object for working with Tarantool topology, etc. In the end, we came to a rather elegant solution: now the Reconcile method is built with separate steps, and the code looks much more clear and readable.

func (r *ClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    clusterContext := reconcilers.MakeReconciler(ctx, r.Reconciler)
    return clusterContext.RunCluster(ctx,
        reconcilers.GetObjectFromRequest(req),
        reconcilers.CheckDeletion,
        reconcilers.CheckFinalizer,
        reconcilers.SetupRolesOwnershipStep,
        reconcilers.SyncClusterWideServiceStep,
        reconcilers.WaitForRolesPhase(RoleReady),
        reconcilers.GetLeader,
        reconcilers.CreateTopologyClient,
        reconcilers.Bootstrap,
        reconcilers.SetupFailover,
        reconcilers.ApplyCartridgeConfig)
}

Besides, it comes with a pleasant bonus: a common code base for both the Enterprise and Community versions of the operator has become possible. Such modular structure allows developing the operator with modules that can be connected in the version where they are needed.

P.S. Yes, soon we will be reworking the community version of the operator. Then we'll also talk about how the operator works in more detail.

Rolling update

As I mentioned before, replica sets are supplied to Kubernetes through the standard resource Statefulset which already has two strategies for application update:

• OnDelete — pods inside Statefulset won't be updated automatically.
• RollingUpdate — pods are updated individually.

The RollingUpdate strategy isn't suitable for applications where pods don't have equal rights, which is the case with Tarantool. In one replica set instances can execute two roles:

• Master — an instance where data can be read from and written to.
• Replica — an instance with read-only access (ReadOnly mode).

In RollingUpdate Kubernetes doesn't know in which pod the master is currently located. Therefore, it can begin application update from the master, which would lead to partial unavailability for writing. The solution to this problem was writing our own update strategies:

• OnDelete — repeats the same-titled Statefulset strategy.
• ClusterPartitionUpdate — the strategy used for instances with no data. It is similar to the usual update strategy since functionally there is no master (no data).
• SwitchMasterUpdate — the strategy used for instances with data. It works within one replica set using the following algorithm:

Update all replicas
Switch the master to new instances
Update the previous master

You might wonder why a database contains instances with no data. It's important to remember that Tarantool is a database and an application server in one package. The thing is, sharding requires a separate instance (or instances) working as a router. A router is basically a controller telling where to go for necessary data.

Most often, there is no need to unite routers into replica sets, so the ClusterPartitionUpdate works over all replica sets and not inside one specific Statefulset.

Such strategies are easy to implement in code:

• Check the update condition
• Perform some operations, if necessary
• Delete the pod
• Wait until the Statefulset/Deployment controller creates new pods with a new image
• Repeat these steps until all necessary pods are updated

Solving network issues

In development, sometimes the situations arise where the operator requires access to all pods inside the Kubernetes network. This isn't a problem when the operator works normally inside Kubernetes. But what if you want to debug your code outside Kubernetes?

One of possible solutions is raising a VPN inside Kubernetes. That's what we did when we just started developing Tarantool Kubernetes Operator, since we used GraphQL API for Cartridge clusters configuration. But this solution additionally loads the developer's PC.

Another solution doesn't work for every application, but it worked wonderfully for operating with Tarantool: get rid of network requests and switch to using pod exec inside the container with the application. The current version of the operator uses this approach for Tarantool configuration. Tarantool's ecosystem includes the console utility, Tarantoolctl, that allows connecting to an operating instance through a control socket and configure the cluster with Lua code.

This approach helped us solve one more problem. In Cartridge, you can enable authorization. It used to be a problem when you used an HTTP connection. But when we connect through a socket, we already have maximum access rights, so the authorization problem is solved.

Naming the Statefulset when PVC changes

Sometimes when working with Statefulset you might want to change the size of Persistent volume claim. But in Kubernetes the Statefulset's PVC section is unchangeable. Since we are working with a database, the amount of data grows and at some point we'll have to increase the disk volume.

So we added a feature that allows changing the role's PVC. Here, a problem arises with pod names: in Kubernetes, two pods with the same name cannot work simultaneously. Initially, pod names were built by the following rule: <role_name>-<statefulset_ordinal>-<pod_ordinal>. PVC update uses the following algorithm:

• Create a new Statefulset with a new PVC
• Create for it a new replica set with a required weight
• Set the replication weight to 0 for the old replica set
• Wait until the old replica set has no data
• If topology leader is located in the old replica set, change it
• Delete the old replica set and all its instances

You might notice that the old rules of naming Statefulset and pods didn't suit us. We decided to change the naming rules to <role_name>-<statefulset_ordinal>-<hash_of_replicaSetTemplate>-<pod_ordinal>. ReplicasetTemplate uses standard fields of PVC, which has private fields that can be changed in runtime. So, we decided to take a 32-bit hash from the JSON representation of the ReplicasetTemplate object. This solution is not very elegant, but it let us get rid of dynamic fields. Example of a new name — router-0-7dfd9f68f-0.

Testing the Operator

As every software, the operator needs to be tested. In our case, we use two types of tests: Unit and E2E. For testing, usually mock code generation is used (for example, via golang/mock). We didn't like this option, so we decided to use Testify's mock module that allows to mock required function interfaces using the reflection API — the interfaces used to configure Tarantool.

If you are interested, here's an article that compares those libraries, testify/mock and golang/mock: GoMock vs. Testify: Mocking frameworks for Go

To create a fake Kubernetes cluster, we used a library by Kubernetes developers: «sigs.k8s.io/controller-runtime/pkg/client/fake».

Currently, unit tests work by the following schema:

• Create a fake topology and a Kubernetes cluster client
• Call the Reconcile method
• Check that the right topology methods were called, and the resources were changed correctly.

Those tests look like this:

BeforeEach(func() {
   cartridge = helpers.NewCartridge(namespace, clusterName).
      WithRouterRole(2, 1).
      WithStorageRole(2, 3).
      Finalized()

   fakeTopologyService = new(mocks.FakeCartridgeTopology)

   fakeTopologyService.
      On("BootstrapVshard", mock.Anything).
      Return(nil)
   fakeTopologyService.
      On("GetFailoverParams", mock.Anything).
      Return(&topology.FailoverParams{Mode: "disabled"}, nil)
   fakeTopologyService.
      On("GetConfig", mock.Anything).
      Return(map[string]interface{}{}, nil)
})

A test is written by the following schema:

cartridge.WithAllRolesReady().WithAllPodsReady()

fakeClient := cartridge.BuildFakeClient()

resourcesManager := resources.NewManager(fakeClient, scheme.Scheme)
clusterReconciler := &ClusterReconciler{...}
_, err := clusterReconciler.Reconcile(...)
Expect(err).NotTo(HaveOccurred(), "an error during reconcile")

err = fakeClient.Get(ctx, types.NamespacedName{Namespace: namespace, Name: clusterName}, cartridge.Cluster)
Expect(err).NotTo(HaveOccurred(), "cluster gone")

Expect(cartridge.Cluster.Status.Bootstrapped).To(BeTrue(), "cluster not bootstrapped")

As for E2E tests, we used the E2E framework for their implementation. It allowed us to fully check the operator's Helm chart and test it in different Kubernetes versions with KinD. Due to the specifics of tests in Kubernetes, we have to wait until different pods are created. Therefore, the duration of all tests grows very fast. E2E framework helped us solve this problem since it supports parallel start of test cases. It let us shorten the time of tests from 30 to 8 minutes.

Conclusion

That's all I wanted to talk about. Here are a few useful links on this topic.

• You can download Tarantool on the official website
• Get help in our Telegram chat
• Read more about Operator SDK here
• Creating fake Kubernetes topology
• The testify library we used for unit tests
• E2E framework

How we compress data in large projects

tarantool — Wed, 29 Jun 2022 17:37:19 +0000

Hi everyone! My name is Alexander Klenov, and I work in Tarantool. In April, we released Tarantool 2.10 Enterprise Edition–an updated version of the Tarantool in-memory computation platform. The 2.10 version includes several new features.

In this article, I want to talk in detail about one of the features–data compression in RAM. I am going to tell you how to use it, what it can and cannot do, and what aspects to pay attention to.

How to enable data compression

It's very easy to do–just specify in the field's settings that it needs to be compressed: compression = ' < option > '.

There are two compression options available:

• zstd,
• lz4.

For example:

local space = box.schema.space.create(space_name, { if_not_exists = true })
space:format({
    {name = 'uid', type = 'string'},
    {name = 'body', type = 'string', compression = 'zstd'},
})

If your project already has some uploaded data, you'll need to perform a background migration:

box.schema.func.create('noop', {
    is_deterministic = true, body = 'function(t) return t end'
})

space:upgrade{func = 'noop', format = {
    {name = 'uid', type = 'string'},
    {name = 'body', type = 'string', compression = 'zstd'},
}}

See the documentation for more details.

What can be achieved by using data compression?

So, we've set up the feature. Let's see what it can give us. As an example, we'll use real data of a major telecom company. The data includes 100 thousand different JSON documents with a total volume of 316 MB.

We will try out different compression mechanisms. The CPU capacity in our case is 3.6 GHz. To see the difference better, let's add compression using the external ZLIB library.

Testing showed the following results:

*ZLIB external library

See the source code of the test on GitHub.

The key conclusion is that all these three methods work differently. ZSTD compresses efficiently but slowly, although it unpacks documents rather quickly. LZ4 compresses and unpacks quickly, but the compression ratio is lower than in other methods. Compression using ZLIB is comparable to the results of ZSTD and is not too slow, but the decompression speed is pretty low.

Compression in a cluster

In real-live projects, Tarantool instances are divided by roles. For our example, it's important to distinguish two roles:

• Router
• Storage

Compressing cluster data is not a linear task like compressing just one node. There are a couple of ways to do it, depending on where the compression is performed. These ways have their advantages and disadvantages. Let's take a look at three examples.

Option one — quick result.

Here, compression and decompression happens in the storage. This can be implemented with the functionality described in this article. It's useful when you need to free a lot of space quickly with minimal changes.

Sounds good, but this creates additional load to the storage's CPU and slows it down. Moreover, each replica will be repeating this compression. As a result, the performance of the entire cluster decreases. A storage with a built-in packer is more expensive to scale. Instead of one additional packer, you'll need to install a storage with a packer in it.

See an example of how to do this here:

The local variable enable_zlib in the router must be set to false. In the storage, in the 11th line, you must specify the compression type.

This option is good when you need to free space quickly. It works well as a temporary solution, but can also become a permanent one, if the load allows it. It's cheap and fast.

But if you want to keep the cluster performance on the same level, it would take more effort.

Option two — maximum performance

Compression and decompression happens in the router. This method offers better performance, but the built-in compression won't work here — you have to connect an external library and implement additional logic for it. The master and replicas will receive already packed data.

As an example, we'll use the router — storage pair described above. Except here, the enable_zlib variable in the router must be set to true. We don't need to specify the compression type in the storage in this case.

Advantages:

• We won't load the storage.
• We are reducing the traffic between the router and the storage.
• It's easier to scale — we just need to set up the necessary number of routers with a packer until we reach the required level of performance.

Disadvantages:

• We need to implement packer logic on the router.
• A delay on the router due to compression and decompression appears.

If we need more flexibility, we can create a separate role to manage data compression. It would allow configuring the number of routers separately from the number of compression instances. Resources utilization will become more flexible. Let's take a look at the third option.

Option three — lazy compression

We compress the data in a separate instance and decompress it in the router. So, we write the data to the storage as it is, uncompressed. Then we implement a separate packer role that goes through master storages and packs everything that is not packed. If new data is uploaded too fast, and you need to pack more quickly, you can simply add more packer instances. The replicas will receive already packed data.

For better understanding of this concept, see an example here.

It can be tested with the same load test. The test will show the same results as writing without compression, since the compression does not occur during writing but in a separate parallel thread.

Advantages:

• There is no lag on the router for data write.
• Data compression scales with maximum efficiency.
• There are less cluster instances.
• We don't load the storage as in the first case.

Disadvantages:

• Additional traffic in the cluster — between the storage and the packer.
• Additional load on the storage and one more read and write operation.
• We need to implement a separate role — packer.
• We need to implement unpacking logic on the router.

It can be useful if:

• You want to write data quickly — at once, without compression.
• You want to regulate compression speed by increasing or decreasing the number of instances, and use resources more flexibly.

Moving compression to the router

After looking at different options, let's see what happens when we move compression from the storage to the router, since this is the most efficient method. Let's build a small cluster with two instances — a router and a storage. Then, we'll load it while using different compression methods.

Test bench implementation:

• Router implementation.
• Storage implementation.
• Module for write load testing. Supplemental module for generating data.

The maximum test duration is set at 30 seconds. It is not too long to wait for the tests to finish, and on the other hand, it's enough time to test the load. During this time, we generate 100,000 write requests to the router on behalf of 10 virtual users. See the console output here. The results are summarized in the table below.

Based on the key parameters — number of requests per second (RPS), latency (http_req_duration), and total request execution time (iteration_duration) — compression on the router performs much better than compression on the storage.

This is primarily because ZSTD compression takes longer than ZLIB compression. Also, data from the router to the storage is transferring already compressed, so less data is written to the storage. Another advantage is that you don't need to load the storage with compression.

Conclusion

Tarantool keeps growing and gains new features. The compression mechanism built into 2.10EE is useful, but it can't solve everything. The best way to compress data depends on the specifics of your project — hence the recommendations for use.

If you do not have the need or the ability to use an external library, you can use the built-in LZ4 compression tool. This method allows freeing about 40% of the space with almost no RPS drawdown (6351 with LZ4 vs. 7076 without compression). And it can be enabled in just one click, which, in my opinion, is a great advantage.

If you want better, more efficient, or cheaper data compression, you'll need to make additional efforts, which are described above.

You can download Tarantool on the official website and get help in our Telegram chat.

Alternative to MapReduce for search in distributed databases

tarantool — Mon, 25 Apr 2022 16:07:42 +0000

Author: Satbek Turganbayev

Hello, my name is Satbek, and I'm part of the Tarantool team. I will tell you how to implement a search in a sharded cluster, the speed of which does not depend on the number of masters and the amount of stored data. I call this method an index layer:

• I will describe the general process of developing a search algorithm.
• I will give an example of implementation,
• And I will also give some recommendations for its development.

In the article, I consistently implement a simple CRUD service with sharded storage, as well as a data search. This will help you better understand the problems arising with distributed searches and how the index layer solves them.

We will use the Tarantool database (version ≥ 1.10), as well as the Tarantool-Cartridge clustering framework (version 2.7.0).

For a better understanding, you should learn a little about the Tarantool-Cartidge framework, the vshard module, and the Lua language, since the example is written in it.

General description of the application

As an example, let's implement a simple CRUD service with one user table sharded by id. Tarantool uses the term space. You can find the source code for the application here.

The required data will be stored in the user space in the following format:

The users are sharded by id. There is a primary index by id and secondary index by name.

How do we search in such a cluster?

Search by id

-- router side
local bucket_id = vshard.router.bucket_id(user_id)
user = vshard.router.callrw(bucket_id, 'get_user_by_id', {user_id}, {timeout = 1})
-- storage side
local M = {}
function M.get_user_by_id(id)
    local user_t = get_user_tuple_by_id(id)
    if user_t == nil then
        return nil, M.user_not_exists_err:new('id=%s', id)
    end
    return user_tuple_to_output(user_t)
end

To search by id, we use the standard Tarantool approach. We calculate the bucket_id on the router side to find out which storage node our data is on, and then send a request there. As a result, we get what we are looking for in no more than two queries.

Search by name

Next, we implement a search by name. Here, one request is not enough: we need to access each master storage in our cluster, since we don’t know where our data is.

-- router side
local M = {}
function M.find_users_by_name(name)
    local storage_uris = cartridge_rpc.get_candidates('app.roles.storage', {leader_only = true})
    local res_by_uri, err = cartridge_pool.map_call('storage_api.get_users_by_name', {name}, {uri_list = storage_uris, timeout = M.vshard_timeout})

    if err ~= nil then
        return nil, err
    end

    local result = {}
    for _, res in pairs(res_by_uri) do
        for _, user in pairs(res) do
            table.insert(result, user)
        end
    end
    return result
end

-- storage side
local M = {}
function M.get_users_by_name(user_name)
    local yield_every = 100
    local count = 1

    local result = {}
    for _, t in box.space.user.index.name:pairs({user_name}, 'EQ') do
        if count % yield_every == 0 then
            fiber.yield()
        end

        count = count + 1
        table.insert(result, user_tuple_to_output(t))
    end

    return result
end

In this case, the number of requests will be no less than the number of masters in the cluster. However, this approach is acceptable if it is known that the required data lies on several masters and if there are few masters. Otherwise, an additional index for the name field is required to search efficiently. In the context of searching by person's name, the assumptions described above are valid if there is sufficient amount of data.

Widening the search

Now let's say you want to search for a user by phone number as well. To do this, you can use the search by name example we described above: make an additional index for storage and implement MapReduce for search. This algorithm will work and it's relatively easy to implement, especially with the help of the Tarantool/crud module. However, it has a significant drawback: the “phone number” field, as a rule, is only in one user entry. So, the entry with the desired value will be located on only one master. But in the MapReduce algorithm, we will have to visit all the masters in the cluster, and if you have two of them, you will have to make two read requests, if 10 — then 10 requests, etc. And all of these will be network requests, which are very expensive compared to reading by index, and they can fail. This leads to a new problem — redundant network requests when searching for unique fields.

Index layer structure

The difficulty here is that we need to find unique data, but we do not know which master it is stored on. So, we need to store this information. We must use up memory to reduce network requests.

Let's create a special user_search_index space:

This space will be the search layer. When writing data, we will calculate data_hash of the information used for search:

-- one of the many options for doing this
local msgpack = require('msgpack')
local digest = require('digest')

local data = {'phone-number', '202-555-0165'}
local data_hash = digest.md5_hex(msgpack.encode(data))

The user_search_index space is sharded by data_hash. In order to avoid collisions, you need to store data. Here you can find the application module that implements the hash building logic:

local M = {}

local digest = require('digest')
local msgpack = require('msgpack')
M.cmp_data = {}

local Identifier = {}

function Identifier:new(data)
    local obj = {}
    obj.hash = digest.md5_hex(msgpack.encode(data))
    obj.data = data
    return obj
end

function M.phone_number(phone_number)
    local data = {'phone_number', phone_number}
    return Identifier:new(data)
end

function M.cmp_data.phone_number(data_one, data_two)
    return #data_one == 2 and #data_two == 2 and (data_one[1] == data_two[1]) and (data_one[2] == data_two[2] ~= nil)
end

return M

Read

While searching, we:

• calculate the hash of the search data;
• use it to find data in user_search_index;
• send a request to storage and get the data by primary-key.

Example code for searching by phone number:

-- api.lua
local M = {}
function M.find_users_by_phone_number(phone_number)
    local user_ids, err = M.search_index.user_id.get_by_phone_number(phone_number)
    if err ~= nil then
        err = errors.wrap(err)
        return nil, err
    end

    return get_users_by_ids(user_ids)
end

-- search_api.lua
local M = {}
function M.user_id.get_by_phone_number(phone_number)
    local identifier = M.identifier.phone_number(phone_number)

    local bucket_id = M.vshard_router.bucket_id_mpcrc32(identifier.hash)

    local ids, err = M.vshard_router.callrw(bucket_id, 'search_storage_api.user_id.get_by_phone_number',
        {identifier.hash, identifier.data}, {timeout = M.vshard_timeout}
    )
    if err ~= nil then
        err = errors.wrap(err)
        return nil, err
    end
    return ids
end

-- search_storage_api.lua
local M = {}
local function get_users_by_hash(hash, data, cmp_func)
    local result = {}
    for _, t in box.space.user_search_index.index.hash:pairs({hash}, 'EQ') do
        if t.data_hash ~= hash then
            break
        end
        if cmp_func(t.data, data) then
            result[#result + 1] = t.user_id
        end
    end
    if #result == 0 then
        return nil, not_found_err:new()
    end
    return result
end

function M.user_id.get_by_phone_number(hash, data)
    return get_users_by_hash(hash, data, M.cmp_data.phone_number)
end

Here we managed without MapReduce. In the case of unique data, we make one network request to find the primary_key, and then another one to get the desired data.

Indexing arbitrary data

The index layer allows you to move indexing from the database level to the application code level. This opens up great possibilities for indexing arbitrary data. For example, it is quite easy to make a composite index based on name and date of birth.

-- identifier.lua

local M = {}

local digest = require('digest')
local msgpack = require('msgpack')
M.cmp_data = {}

local Identifier = {}

function Identifier:new(data)
    local obj = {}
    obj.hash = digest.md5_hex(msgpack.encode(data))
    obj.data = data
    return obj
end

...

function M.name_birthdate(name, birthdate)
    local data = {'name', name, 'birthdate', birthdate}
    return Identifier:new(data)
end

function M.cmp_data.name_birthdate(data_one, data_two)
    return #data_one == 4 and #data_two == 4 and
            (data_one[1] == data_two[1]) and
            (data_one[2] == data_two[2] ~= nil) and
            (data_one[3] == data_two[3] ~= nil) and
            (data_one[4] == data_two[4] ~= nil)
end

return M

The rest is identical to searching by phone number.

Write

Earlier, I described how to implement search using the index layer. Now let's look at how it is built. When you create or update data, you must also create or update the index layer. The implementation of these processes depends on the specific task, but most likely it will be happening in the background. You can use the Tarantool/queue and Tarantool/sharded-queue modules for this. This example has a naive implementation of building an index layer using Tarantool/queue.

Implementing the index layer complicates data writing and updating, but significantly speeds up reading. In our case, working with the index layer when writing data looks like this:

The index layer is updated in the background after the data itself has been updated. That is, it is possible that in some cases the search will not return anything, but the data will already be saved to the database. Or, in the case of an update, the search for the old data will return new data, since the content has been updated, but the search layer has not. But eventually the data will be consistent. Depending on your case, you might need to implement a different logic: first update the index layer, and then the data.

Conclusion

Index layer:

• Significantly reduces the number of network read requests when searching for unique data.
• Allows not storing derived data for searching them.
• Helps implement quick search for any unique data.

You can download Tarantool on the official website and get help in the Telegram chat.

Scaling clusters without any hassle

tarantool — Thu, 10 Mar 2022 07:11:12 +0000

Anyone who has worked with large clusters knows that data constantly keeps growing. Sooner or later, developers of distributed systems face the challenge of scaling. Nowadays, it is hardly a problem to find space for data storage. But how to deal with application adjustments and configuration? You can avoid adjustments if the system allows for scalability inherently. You can differentiate application nodes by feature and deploy only the ones you need.

Hello, my name is Igor, and I am a part of the Tarantool DB team. We have extensive experience in developing high-performance products, for example, data storage systems for major retailers and mobile network operators. Today, I will talk about our approach to cluster scaling and provide a typical example. This article is targeted at anyone who deals with big data and considers scaling.

It is an extended version of the talk I delivered at SaintHighload++ 2021.

Challenge

When developing data-intensive applications, we pursue three objectives:

Scalability. We build up data and business logic.
Reliability. We need redundant nodes to access the data. We also need a mechanism to keep the cluster always writable, even if some nodes fail.
Supportability. We need tools for management and monitoring. We also want to be able to expand the cluster without changing the application code.

Moreover, we want to use tools that help standardize cluster application development. Then we wouldn't have to write code from scratch every time, and we could build new applications into existing pipelines and share expertise between projects.

Cluster architecture

Any cluster starts with its clients — applications or systems that will interact with the cluster, write data, and read it.

When client requests first get to the cluster, each one of them goes through the router. This node redirects incoming requests to other nodes. It is the router that connects clients with their data.

Next, the router sends client requests to storages, which fall into another category of cluster components. As the name implies, storages store data, write it and send it back upon the router's request.

Each cluster node performs its role, i. e., it follows a certain logic. When scaling a cluster, we can add a new application node and assign one of the available roles to it.В For example, we can add a node with the "storage" role if we need more storage space. Other storages will take care of redistributing data to the new node, and the router will start sending requests there.

The same goes for routers. If we need to increase the number of nodes accepting client requests, we have to add a new node to the cluster with the "router" role and make sure that requests pass through it. Here all routers are equivalent, and they send requests to all storages.

The next important cluster component is the failover coordinator. This cluster node keeps other nodes writable. It also writes the cluster state to an external storage, which is called the state provider. It must be independent of the rest of the cluster and meet reliability requirements.

We can add nodes with new roles to expand the cluster. For example, we need to replicate data from an external storage for cluster operation. In this case, we need to add a node that executes this logic. It will connect to the external storage and send data to the routers to distribute it across the cluster.

We can use Cartridge to create a cluster.

Cartridge

Cartridge is a framework for developing cluster applications. Cartridge manages the cluster, controls sharding, allows implementing business logic and data schemas using roles, and provides failover to keep application instances writable.В Cartridge has many additional tools that you can build into CI/CD pipelines and use to manage the running cluster.В For Cartridge, there is also an Ansible role, an integration testing module, a utility for creating an application from a template and running a cluster locally, a module for collecting metrics, and a Grafana panel.

Let's now look inside Cartridge.

Tarantool

Cartridge is a framework powered by several instances of Tarantool, an in-memory computing platform. It is both an in-memory database and an application server written in Lua. Tarantool is very fast due to storing data in RAM but also reliable. It saves data snapshots to the hard disk and allows you to set up replication and sharding.

Replication

Tarantool instances can be combined into a replica set — a set of nodes connected by asynchronous replication. When configuring a replica set, you can specify which instance will be read-only. We will call the instances that accept writes "masters", and the rest — "replicas".

Here's an example of master replication configuration:

box.cfg{
    listen = 3301,
    read_only = false,
    replication = {
        '127.0.0.1:3301',
        '127.0.0.1:3302',
    },
}

And here's an example of a replica configuration:

box.cfg{
    listen = 3302,
    read_only = true,
    replication = {
        '127.0.0.1:3301',
        '127.0.0.1:3302',
    },
}

Sharding

Tarantool has vshard, a special module for data sharding. It allows dividing data into chunks to be stored on different application nodes.В

There are two entities in vshard: vshard.storage and vshard.router. Storages contain buckets with data. They also rebalance the data if new instances are added. Routers send requests to storage nodes.

In vshard, you need to configure routers and storages on each node. When expanding a vshard-based cluster, you have to either modify the application code or write a vshard wrapper, which will configure instances when new nodes are added.

Here's an example of vshard configuration on Tarantool instances:

local sharding = {
    ['aaaaaaaa-0000-4000-a000-000000000000'] = {
       replicas = {
           ['aaaaaaaa-0000-4000-a000-000000000011'] = {
               name   = 'storage',
               master = true,
               uri    = "sharding:pass@127.0.0.1:30011"
             },
        }
    }
}

vshard.storage.cfg(
    {
        bucket_count = 3000,
        sharding     = sharding,
        ...
    },
    'aaaaaaaa-0000-4000-a000-000000000011'
)

vshard.router.cfg({sharding = sharding,})

Summary

Cartridge is powered by Tarantool functionality. To build a cluster, we take several Tarantool instances, configure replication, add sharding, and automate adding new nodes. In addition, Cartridge provides a few more features that are not available in basic Tarantool:

Failover.
Automatic sharding configuration when new nodes are added.
WebUI for cluster monitoring.

Cartridge instance roles

I have already shown that each instance in a cluster performs a certain role. Cartridge roles are Lua modules with a specific API that describe the application business logic.

What can roles be used for?

A storage for data tables.
Client HTTP API.
External data replication.
Metrics or call chain collection, instance performance monitoring.
Execution of any custom logic written in Lua.

A role is assigned to a replica set and enabled on each replica set instance. A role always knows if it is enabled on the master or the replica. You can control its behavior by changing the configuration without touching the role's code.В

API

return {
    role_name = 'custom-role',

    init = init,
    validate_config = validate_config,
    apply_config = apply_config,
    stop = stop,

   dependencies = {
        'another-role',
    },
}

Each role has a standard API:

The init function performs the initial role configuration.В
validate_config checks if the configuration is valid.В
apply_config applies the configuration.
The stop function disables the role.В

A role has a list of dependencies and will only start after all the dependencies in this list are initialized.

Predefined roles

Cartridge has several predefined roles: some are available out of the box, others are installed with different modules.

Built-in roles:

vshard-router sends requests to data storage nodes.
vshard-storage saves and manages data buckets.
failover-coordinator controls cluster failover.

Other useful roles:

metrics collects application metrics.
crud-storage and crud-router simplify interaction with sharded spaces.

Role dependencies

Role dependencies form the essential mechanism underlying all Cartridge applications. A role will only start when other roles, which it depends on, are successfully initialized. This means that each role can use functions of other roles, as long as they are specified in the dependencies list.В

Let's look at an example. Suppose we have written our own custom-storage role, which describes data tables creation. It will depend on the crud-storage role to use simplified interfaces for receiving data and writing it to our storage. The crud-storage role already depends on the vshard-storage role, which takes care of data sharding. It looks like this in the code:

app/roles/custom-storage.lua

return {
    role_name = 'custom-storage',

    dependencies = {
        'crud-storage',
    },
}

cartridge/roles/crud-storage.lua

return {
    role_name = 'crud-storage',

    dependencies = {
        'vshard-storage',
    },
}

Cartridge guarantees the role startup order. So the custom-storage role can use all the features of vshard-storage and crud-storage because they are initialized before it.

Entry point

The entry point of each Cartridge cluster instance is the init.lua file. To configure a cluster instance, the init file calls the cartridge.cfg function, which expects the list of available roles and default Cartridge and Tarantool settings. The init file can be run from the console with Tarantool, as well as with cartridge-cli or systemd/supervisord.

tarantool ./init.lua

local ok, err = cartridge.cfg({
    roles = {
        'cartridge.roles.vshard-storage',
        'cartridge.roles.vshard-router',
        'cartridge.roles.metrics',
        'app.roles.customвЂ™,
    },
    ... -- cartridge opts 
}, {
    ... -- tarantool opts
})

Cluster scaling with roles

Let us consider a typical scaling scenario. Suppose we have a two-instance cluster: one has the "router" role, and the other has the "storage" role.

We add new replicas when we need extra backup: for example, there is a new redundant data center. Or when we want to increase the readability of replica sets: for example, if most read requests are sent to the replica.

We add new shards when the memory usage on instances is about 60-80%. We can do it earlier if we see it constantly growing on memory usage graphs and predict that the space will run out soon.

We add routers according to business metrics or when we see a high CPU load of about 80-100%. For example, the growing number of cluster users increases its load.

We add new roles when the application business logic changes or extends, or we need additional monitoring. For example, we can add the metrics or the tracing module to the cluster.

Note that you can add new instances to the cluster and enable available roles on existing replica sets — without changing the code. But to add new roles that didn't previously exist in the cluster, you need to add them to init.lua and restart the application.

Also, keep in mind that you can assign multiple roles to any cluster instance. Any instance can be a router, a storage, and a replica, as well as perform any custom logic at the same time.

Cluster configuration

The cluster configuration is the main way to manage a cluster. It stores the topology, vshard settings transferred to vshard.storage.cfg and vshard.router.cfg, and custom role settings. Each application instance stores the complete configuration. Cartridge ensures that all nodes have identical configurations at any given time.

Let us take a look at how the new configuration is propagated. Suppose the configuration changed on one of the nodes. It will propagate to all other instances after that.

Each instance receives the configuration changes and calls its validate_config functions to check if the received configuration is valid. If at least one instance fails to validate the configuration changes, all other instances discard them, and the user who applies the new configuration gets an error.

If the configuration is valid, Cartridge waits for each cluster instance to confirm that the validate_config check was successful, and the configuration can be applied.

Then Cartridge executes all apply_config functions and applies the new configuration on each application instance. It replaces the old configuration on the disk.

If anything goes wrong when applying the configuration, and apply_config returns an error, Cartridge will display the corresponding information in WebUI and suggest reapplying the configuration.

Failover

Replication guarantees that our data doesn't get lost, but we also need to keep the applications writable. For this, Cartridge provides the failover.В Failover works in a cluster that has instances with the failover-coordinator role for managing master rotating. The cluster should also have the state-provider instance, which stores the list of current masters. Two state-provider modes are currently supported: etcd and a standalone Tarantool instance called stateboard.

The failover-coordinator ensures that at least one master is available in the replica set.

If the master crashes or stops responding to requests, the failover-coordinator finds out about it and appoints one of the replicas in the same replica set as the master.

In this case, the information about the current master is saved in the external storage. So even if the old master is back up and running, it will no longer accept write requests. The new master will remain until it crashes or becomes unavailable.

Tools

How do you support Cartridge? We have a large ecosystem of tools:

cartridge-cli serves several purposes: creating an application from a template, running a cluster locally, connecting to instances, and calling administration functions.
For testing, there is cartridge.test-helpers, which allows you to create any type of cluster in integration tests.
The WebUI is used for monitoring: it allows you to find out about some issues in Cartridge and suggests how to fix them; there is a metrics package and a Grafana dashboard.
ansible-cartridge helps with deployment: it allows you to add new instances to the cluster, update existing ones, and much more.

Cartridge in production

Cartridge has proven itself in production. Our customers have been using it for at least three years. Here are some statistics on Cartridge usage:

> 100 data marts in production: cache and master storages.
From a few MB to several TB.
4 — 500 instances per cluster.
Masters and replicas in different data centers.
A single CI/CD pipeline for all applications in a closed network.
Common modules and roles in projects.

Conclusion

We discussed our approach to cluster scaling and the following topics:

Cartridge roles.
Distributed application configuration.
Fault tolerance provided by failover.
Cartridge usage statistics in production.

I hope you found this article useful, and you will try scaling with Cartridge. Please share your experience of scaling other databases in the comments.

What's next?

Our official website
Try Cartridge in the sandbox.
Check out the documentation on the official website.
Ask your questions to the community in the Telegram chat.

Developing an authentication system in Java+Tarantool

tarantool — Mon, 21 Feb 2022 12:01:10 +0000

Author: Alexander Goryakin

My name is Alexander, I am a software engineer in the architecture and pre-sale department at VK. In this article, I'm going to show you how to build an authentication system based on Tarantool and Java. In pre-sales, we often have to implement such systems. There are plenty of authentication methods: by password, biometric data, SMS, etc. To make it simple, I'll show you how to implement password authentication.

This article should be useful for those who want to understand the construction of authentication systems. I will use a simple example to demonstrate all the main parts of such an architecture, how they relate to each other and how they work as a whole.

The authentication system verifies the authenticity of the data entered by the user. We encounter these systems everywhere, from operating systems to various services. There are many types of authentication: by login and password pair, with electronic signature, biometric data, etc. I chose the login-password pair as an example, as it's the most common and quite simple. And it also allows showing the basic features of Cartridge and Cartridge Java, with a fairly small amount of code. But first things first.

Fundamentals of authentication systems

In any authentication system, you can usually identify several elements as follows:

subject that will undergo the procedure;
characteristic of the subject, its distinguishing feature;
host of the authentication system, who is responsible for it and controls its operation;
authentication mechanism, that is, the operating principles of the system;
access control mechanism, which grants certain access rights to a subject.

The authentication mechanism can be provided by the software that verifies the authenticity of the subject characteristics: a web service, an operating system module, etc. Most often, the subject characteristics must be stored somewhere, which means there must be a database, MySQL or PostgreSQL, for example.

If there is no existing software that allows you to implement an authentication mechanism according to certain rules, you have to write it by yourself. Among these cases, I can list authentication by several characteristics, with complicated verification algorithms, etc.

What are Tarantool Cartridge and Cartridge Java?

Tarantool Cartridge is a framework for scaling and managing a cluster of multiple Tarantool instances. Besides creating a cluster, it also allows you to manage that cluster quite effectively, such as expanding it, automatically resharding it and implementing any role-based business logic.

To work with the cluster from an application, you need to use connectors”drivers for interaction with the database and the cluster using the iproto binary protocol. Tarantool currently has connectors for programming languages such as Go, Java, Python, to name a few. Some of them can only work with one instance of Tarantool, while others can work with entire clusters. One of those connectors is Cartridge Java. It allows you to interact with a cluster from a Java application. This brings up a reasonable question: why this particular language?

Why Java?

I work in the architecture and pre-sales department, which means that we make pilot projects for customers from different areas of business. By a pilot project, I mean a prototype of a system, that will later be finalized and handed over to the customer. That is why our customers are mostly people who use programming languages that allow them to create full enterprise solutions. One of those is Java, so we chose Cartridge Java connector for this example.

Why authentication process?

The next question that arises is the choice of a service on which we will demonstrate our technology. So why did we take authentication and not some other service? The answer is quite simple: this is the most common problem that people try to solve not only with Tarantool but with other databases as well.

Users encounter authentication in almost all more or less major applications. Most commonly, databases such as MySQL or PostgreSQL are used to store user profiles. However, using Tarantool here is most appropriate since it can handle tens of thousands of queries per second due to the fact that all data is stored in RAM. And if an instance crashes, it can recover rather quickly via snapshots and write-ahead logs.

Now let's get to the structure of our sample service. It will consist of two parts:

Tarantool Cartridge application, serving as a database;
Java application, providing an API for performing basic operations.

Let's start by looking at the first part of our service.

Tarantool Cartridge application

This application will provide a small cluster of one router, two sets of storage replicas, and one stateboard.

Router is an instance with the router role. It is responsible for routing requests to storage. We're going to expand its functionality a little bit. I'll explain how to do it further below.

Replica set (storage replica set) refers to a group of N instances with the storage role, of which one is the master, and the rest are its replicas. In our case, these are pairs of instances that act as profile storage.

Stateboard is responsible for configuring the failover mechanism of the cluster in case of failure of individual instances.

Creating and configuring an application

Let's create an application by executing

$ cartridge create --name authentication

This will create "authentication" directory, containing everything you need to create a cluster. Let's define a list of instances in the instances.yml file:

---
authentication.router:
  advertise_uri: localhost:3301
  http_port: 8081

authentication.s1-master:
  advertise_uri: localhost:3302
  http_port: 8082

authentication.s1-replica:
  advertise_uri: localhost:3303
  http_port: 8083

authentication.s2-master:
  advertise_uri: localhost:3304
  http_port: 8084

authentication.s2-replica:
  advertise_uri: localhost:3305
  http_port: 8085

authentication-stateboard:
  listen: localhost:4401
  password: passwd

Now we have to configure the roles.

Configuring roles

For our application to work with the Cartridge Java connector, we need to create and configure new roles. You can do this by copying the custom.lua file and renaming the copies into storage.lua and router.lua, placing them into the app/roles directory, and then changing the settings in them. First, change the name of the role”the value in the role_name field”in the return statement. In router.lua the role will be router and in storage.lua it will be storage. Second, specify the corresponding role names in init.lua in the roles section of the cartridge.cfg file.

In order to work with Cartridge Java, we need to install the ddl module by adding 'ddl == 1.3.0-1' to the dependencies section of the file with the .rockspec extension. And add the get_schema function to router.lua after that:

function get_schema()
    for _, instance_uri in pairs(cartridge_rpc.get_candidates('app.roles.storage', { leader_only = true })) do
        local conn = cartridge_pool.connect(instance_uri)
        return conn:call('ddl.get_schema', {})
    end
end

Add the following to the init function:

rawset(_G, 'ddl', { get_schema = get_schema })

In addition, add the following condition to the init function in storage.lua:

 if opts.is_master then
        rawset(_G, 'ddl', { get_schema = require('ddl').get_schema })
 end

It means that we have to execute the rawset function on those storages which are masters. Now let's move on to defining the cluster topology.

Defining a cluster topology and launching the cluster

Let's specify the cluster topology in the replicasets.yml file:

router:
  instances:
  - router
  roles:
  - failover-coordinator
  - router
  all_rw: false
s-1:
  instances:
  - s1-master
  - s1-replica
  roles:
  - storage
  weight: 1
  all_rw: false
  vshard_group: default
s-2:
  instances:
  - s2-master
  - s2-replica
  roles:
  - storage
  weight: 1
  all_rw: false
  vshard_group: default

After establishing the instance configuration and topology, execute the commands to build and run our cluster:

$ cartridge build
$ cartridge start -d

The instances that we defined in instances.yml will be created and launched. Now we can access http://localhost:8081 in a browser to manage our cluster via GUI. All the created instances will be listed there. However, they are not configured or combined into replica sets as we described in replicasets.yml just yet. To avoid configuring instances manually, run the following:

$ cartridge replicasets setup -bootstrap-vshard

If we check the list of our instances now, we'll see that the topology is now set up, that is, the instances have the appropriate roles assigned to them, and they are combined into replica sets:

Furthermore, the initial bootstrapping of the cluster was performed, which resulted in a working sharding. And now we can use our cluster!

Building a data model

Well, actually we can't make use of it just yet, since we don't have a proper data model to describe the user. Let's see, what do we need to describe the user? What kind of information about the user do we want to store? Since our example is quite simple, let's use the following fields as general information about the user:

uuid, user's unique identifier;
login, user's login;
password, the hash sum of the user's password.

These are the main fields that the data model will contain. They are sufficient for most cases when there are few users and the load is pretty low. But what happens when the number of users becomes immense? We would probably want to implement sharding, so we can distribute users to different storages, and those in turn to different servers or even different data centers. Then what field should we use to shard the users? There are two options, UUID and login. In this example, we're going to shard the users by login.

Most often, the sharding key is chosen so that a storage will contain records with the same sharding key, even if they belong to different spaces. But since there is only one space in our case, we can choose any field we like. After that, we have to decide which algorithm to use for sharding. Fortunately, this choice is not necessary because Tarantool Cartridge already has the vshard library, which uses a virtual sharding algorithm. To use this library, we need to add one more field to the data model, bucket_id. This field's value will be calculated based on the login field's value. And now we can describe our space in full:

local user_info = box.schema.create_space('user_info', {
            format = {
                { name = 'bucket_id', type = 'unsigned' },
                { name = 'uuid', type = 'string' },
                { name = 'login', type = 'string' },
                { name = 'password', type = 'string' },
            },
            if_not_exists = true,
        })

To start using the space, we have to create at least one index. Let's create a primary index primary based on the login field:

user_info:create_index('primary', {
            parts = { 'login' },
            if_not_exists = true,
        })

Since we are using vshard, we also need to create a secondary index based on the bucket_id field:

user_info:create_index('bucket_id', {
            parts = { 'bucket_id' },
            if_not_exists = true,
            unique = false
        })

Now let's add a sharding key based on the login field:

utils.register_sharding_key('user_info', {'login'})

Performing migrations

We'll use the migrations module to work with spaces. To do this, add this line to the dependencies section of the file with the .rockspec extension:

'migrations == 0.4.0-1'

To use this module, create a migrations directory in the application's root directory and put a 0001_initial.lua file with the following contents there:

local utils = require('migrator.utils')

return {
    up = function()
        local user_info = box.schema.create_space('user_info', {
            format = {
                { name = 'bucket_id', type = 'unsigned' },
                { name = 'uuid', type = 'string' },
                { name = 'login', type = 'string' },
                { name = 'password', type = 'string' },
            },
            if_not_exists = true,
        })

        user_info:create_index('primary', {
            parts = { 'login' },
            if_not_exists = true,
        })

        user_info:create_index('bucket_id', {
            parts = { 'bucket_id' },
            if_not_exists = true,
            unique = false
        })

        utils.register_sharding_key('user_info', {'login'})

        return true
    end
}

To create our space, we have to send a POST request to http://localhost:8081/migrations/up, such as this:

$ curl –X POST http://localhost:8081/migrations/up

By doing so, we perform the migration. To create new migrations, add new files with names beginning with 0002-…, to the migrations directory and run the same command.

Creating stored procedures

After constructing the data model and building the space for it, we need to create functions through which our Java application will interact with the cluster. Such functions are referred to as stored procedures. They are called on routers and they process the data by invoking certain space methods.

What kind of operations with user profiles do we want to perform? Since we want to use our cluster primarily as profile storage, it's obvious that we should have a function to create profiles. In addition, since this application is an example of authentication, we should be able to get information about the user by their login. And finally, we should have a function to update a user's information, in case a user forgets their password, for instance, and a function to delete a user if they want to delete their account.

Now that we have defined which basic stored procedures we want, it's time to implement them. The entire code for them will be stored in the app/roles/router.lua file. Let's start by implementing the user creation, but first we'll set up some auxiliary constants:

local USER_BUCKET_ID_FIELD = 1
local USER_UUID_FIELD = 2
local USER_LOGIN_FIELD = 3
local USER_PASSWORD_FIELD = 4

As you can see from their names, these constants define the numbers of the corresponding fields in the space. These constants will allow us to use meaningful names when indexing the fields of the tuple in our stored procedures. Now let's move on to creating the first stored procedure. It will be named create_user and will receive UUID, username, and password hash as parameters.

function create_user(uuid, login, password_hash)
    local bucket_id = vshard.router.bucket_id_mpcrc32(login)

    local _, err = vshard.router.callrw(bucket_id, 'box.space.user_info:insert', {
        {bucket_id, uuid, login, password_hash }
    })

    if err ~= nil then
        log.error(err)
        return nil
    end

    return login
end

First, we use vshard.router.bucket_id_mpcrc32 to calculate the bucket_id parameter, which will be used to shard our entries.
Then we call the insert function from the space on the bucket with the calculated bucket_id, and pass a tuple consisting of bucket_id, uuid, login and password_hash fields to this space. This call is performed using the vshard.router.callrw call of the vshard library, which allows write operations to the space and returns the result of the function being called (and an error if it fails).
Finally, we check if our function has been executed successfully. If yes — the data was inserted into the space — we return the user's login. Otherwise, we return nil.

Now let's create the next stored procedure, the one for getting information about the user by their login. This one will be named get_user_by_login. We will apply the following algorithm to it:

Calculate the bucket_id by login.
Call the get function for the calculated bucket via the vshard.router.callbro function.
If a user with the specified login exists, then we return the tuple with information about the user, otherwise return nil.

Implementation:

function get_user_by_login(login)

    local bucket_id = vshard.router.bucket_id_mpcrc32(login)

    local user = vshard.router.callbro(bucket_id, 'box.space.user_info:get', {login})
    return user
end

Besides authentication, it will also be helpful in updating and deleting user information.

Let's consider the case where the user decided to update their information, for example, their password. We're going to write a function named update_user_by_login that will accept the user's login and the new password's hash. Which algorithm should we use for that task? Let's start by trying to get the user's information via the get_user_by_login function we have implemented. If the user doesn't exist, we'll return nil. Otherwise, we'll calculate bucket_id by the user's login and call the update function for our space on the bucket with the calculated id. We'll pass the user's login and the tuple containing information about the field we need to update — the new password hash — to this function. If an error occurred during the update, then we will log it and return nil, otherwise we will return the tuple with the user's information. In Lua, this function will look like this:

function update_user_by_login(login, new_password_hash)
    local user = get_user_by_login(login)

    if user ~= nil then
        local bucket_id = vshard.router.bucket_id_mpcrc32(user[USER_LOGIN_FIELD])

        local user, err = vshard.router.callrw(bucket_id, 'box.space.user_info:update', { user[USER_LOGIN_FIELD], {
            {'=', USER_PASSWORD_FIELD, new_password_hash }}
        })

        if err ~= nil then
            log.error(err)
            return nil
        end

        return user
    end

    return nil
end

And lastly, let's implement the function for deleting a user. It will be named delete_user_by_login. The algorithm will be somewhat similar to the update function, the only difference being that if a user exists in the space, the delete function will be called and the information about the deleted user will be returned, otherwise the function will return nil. This stored procedure's implementation goes as follows:

function delete_user_by_login(login)

    local user = get_user_by_login(login)

    if user ~= nil then

        local bucket_id = vshard.router.bucket_id_mpcrc32(user[USER_LOGIN_FIELD])

        local _, _ = vshard.router.callrw(bucket_id, 'box.space.user_info:delete', {
            {user[USER_LOGIN_FIELD]}
        })

        return user
    end

    return nil

end

What was done

We built an application.
Configured roles for it.
Set up a cluster topology.
Launched the cluster.
Described a data model and created migration logic.
Implemented stored procedures.

Now we can restart the cluster and start filling it with data. In the meantime, we'll move on to developing the Java application.

Java application

The Java application will serve as an API and will provide the business logic for user authentication. Since it's an enterprise application, we will create it using the Spring framework. We are going to use the Apache Maven framework to build it.

Setting up the connector

To set the connector, add the following dependency in the dependencies section of the pom.xml file:

<dependency>
     <groupId>io.tarantool</groupId>
     <artifactId>cartridge-driver</artifactId>
     <version>0.4.2</version>
</dependency>

After that, we must update the dependencies. You can find the latest connector's version here. After installing the connector, we need to import the necessary classes from io.tarantool.driver package.

Connecting to the cluster

After setting up the connector, we need to create a class that will be responsible for its configuration and will connect the application to the Tarantool Cartridge cluster. Let's call this class TarantoolConfig. We will specify that it is a configuration class and that its parameters are defined in the application-tarantool.properties file:

@Configuration
@PropertySource(value="classpath:application-tarantool.properties", encoding = "UTF-8")

The application-tarantool.properties file contains the following lines:

tarantool.nodes=localhost:3301 # node list
tarantool.username=admin # user name
tarantool.password=authentication-cluster-cookie # password

They specify the values of the fields required to connect to the cluster. This is why the constructor of our class takes these parameters as input:

public TarantoolClient tarantoolClient(
            @Value("${tarantool.nodes}") String nodes,
            @Value("${tarantool.username}") String username,
            @Value("${tarantool.password}") String password)

We will use username and password fields to create credentials for authentication:

SimpleTarantoolCredentials credentials = new SimpleTarantoolCredentials(username, password);

Let's create a custom configuration for connecting to the cluster, namely specify the authentication parameters and the request timeout:

TarantoolClientConfig config = new TarantoolClientConfig.Builder()
                .withCredentials(credentials)
                .withRequestTimeout(1000*60)
                .build();

Then we have to pass the list of nodes to the AddressProvider which converts a string into a list of addresses and returns this list:

TarantoolClusterAddressProvider provider = new TarantoolClusterAddressProvider() {
            @Override
            public Collection<TarantoolServerAddress> getAddresses() {
                ArrayList<TarantoolServerAddress> addresses = new ArrayList<>();

                for (String node: nodes.split(",")) {
                    String[] address = node.split(":");
                    addresses.add(new TarantoolServerAddress(address[0], Integer.parseInt(address[1])));
                }

                return addresses;
            }
        };

Finally, let's create a client that will connect to the cluster. We wrap it into a proxy-client and return the result wrapped into a retrying-client, which, if the connection fails, tries to reconnect until it reaches the specified number of attempts:

ClusterTarantoolTupleClient clusterClient = new ClusterTarantoolTupleClient(config, provider);
        ProxyTarantoolTupleClient proxyClient = new ProxyTarantoolTupleClient(clusterClient);

        return new RetryingTarantoolTupleClient(
                proxyClient,
                TarantoolRequestRetryPolicies.byNumberOfAttempts(
                        10, e -> e.getMessage().contains("Unsuccessful attempt")
                ).build());

Full code of the class:

@Configuration
@PropertySource(value="classpath:application-tarantool.properties", encoding = "UTF-8")
public class TarantoolConfig {

    @Bean
    public TarantoolClient tarantoolClient(
            @Value("${tarantool.nodes}") String nodes,
            @Value("${tarantool.username}") String username,
            @Value("${tarantool.password}") String password) {

        SimpleTarantoolCredentials credentials = new SimpleTarantoolCredentials(username, password);

        TarantoolClientConfig config = new TarantoolClientConfig.Builder()
                .withCredentials(credentials)
                .withRequestTimeout(1000*60)
                .build();

        TarantoolClusterAddressProvider provider = new TarantoolClusterAddressProvider() {
            @Override
            public Collection<TarantoolServerAddress> getAddresses() {
                ArrayList<TarantoolServerAddress> addresses = new ArrayList<>();

                for (String node: nodes.split(",")) {
                    String[] address = node.split(":");
                    addresses.add(new TarantoolServerAddress(address[0], Integer.parseInt(address[1])));
                }

                return addresses;
            }
        };

        ClusterTarantoolTupleClient clusterClient = new ClusterTarantoolTupleClient(config, provider);
        ProxyTarantoolTupleClient proxyClient = new ProxyTarantoolTupleClient(clusterClient);

        return new RetryingTarantoolTupleClient(
                proxyClient,
                TarantoolRequestRetryPolicies.byNumberOfAttempts(
                        10, e -> e.getMessage().contains("Unsuccessful attempt")
                ).build());
    }
}

The application will connect to the cluster after the first request was sent to Tarantool on the application's launching. Now let's move on to creating an API and a user data model for our application.

Creating an API and a user data model

We're going to use the OpenAPI specification of version 3.0.3. Let's create three endpoints, each of which will accept and process the corresponding types of requests:

/register
- POST, creating a user.
/login
- POST, user authentication.
/{login}
- GET, obtaining user information;
- PUT, updating user information;
- DELETE, deleting a user.

We will also add descriptions for the methods that handle each request we send and each response the application returns:

authUserRequest
authUserResponse
createUserRequest
createUserResponse
getUserInfoResponse
updateUserRequest

The stored procedures we've implemented in Lua will be called by controllers when processing these methods.

Now we need to generate classes that correspond to the described methods and responses. We'll use the swagger-codegen plugin for that. Add the plugin description to the build section of the pom.xml file:

<plugin>
   <groupId>io.swagger.codegen.v3</groupId>
   <artifactId>swagger-codegen-maven-plugin</artifactId>
   <version>3.0.21</version>
   <executions>
      <execution>
         <id>api</id>
         <goals>
            <goal>generate</goal>
          </goals>
          <configuration>
             <inputSpec>${project.basedir}/src/main/resources/api.yaml</inputSpec>
             <language>java</language>
             <modelPackage>org.tarantool.models.rest</modelPackage>
             <output>${project.basedir}</output>
             <generateApis>false</generateApis>
             <generateSupportingFiles>false</generateSupportingFiles>
             <generateModelDocumentation>false</generateModelDocumentation>
             <generateModelTests>false</generateModelTests>
             <configOptions>
                <dateLibrary>java8</dateLibrary>
                <library>resttemplate</library>
                <useTags>true</useTags>
                <hideGenerationTimestamp>true</hideGenerationTimestamp>
             </configOptions>
         </configuration>
      </execution>
   </executions>
</plugin>

In these lines, we specify the path to the api.yaml file that describes the API, and the path to the directory where the generated Java files are to be placed. After running the build, we will get the generated request and response classes, which we are going to use when creating controllers.

Let's move on to creating a user data model. The corresponding class will be called UserModel and we'll place it in the models directory. In the same directory, in its rest subdirectory, there are also the classes for requests and responses. The model will describe the user and will contain three private fields: uuid, login and password. It will also have getters and setters to access these fields. So, our data model's class goes as follows:

public class UserModel {

    String uuid;
    String login;
    String password;

    public String getUuid() {
        return uuid;
    }

    public void setUuid(String uuid) {
        this.uuid = uuid;
    }

    public String getLogin() {
        return login;
    }

    public void setLogin(String login) {
        this.login = login;
    }

    public String getPassword() {
        return password;
    }

    public void setPassword(String password) {
        this.password = password;
    }
}

Creating services and controllers

In order to work with Tarantool when processing queries, we are going to use services that allow us to hide all the logic by calling methods of a certain class. We're going to use four basic methods:

getUserByLogin to get the user's information by their login;
createUser to create a new user;
updateUser to update the information of a user;
deleteUser to delete a user by their login.

To describe the basic service, let's create an interface that contains the signatures of these four methods, and then inherit the service that will contain our Tarantool logic from it. We'll call it StorageService:

public interface StorageService {

    UserModel getUserByLogin(String login);

    String createUser(CreateUserRequest request);

    boolean updateUser(String login, UpdateUserRequest request);

    boolean deleteUser(String login);
}

Now, let's create the TarantoolStorageService class inherited from this interface. First, we must create a constructor for this class that will take TarantoolClient as input to be able to make queries to Tarantool. Let's save the client in a private variable and add the final modifier to it:

private final TarantoolClient tarantoolClient;

    public TarantoolStorageService(TarantoolClient tarantoolClient) {
        this.tarantoolClient = tarantoolClient;
    }

Now let's override the method of getting the user by login. First, we create a variable userTuple of List<ObjРµct> type initialized by the null value:

List<Object> userTuple = null;

After the initialization, we try to execute tarantoolClient's method call, which will result in Future. Since this method is asynchronous, we call the get method with 0 argument to get the result of its execution. If an exception is thrown during the call method execution, we should catch it and log it to the console.

try {
    userTuple = (List<Object>) tarantoolClient.call("get_user_by_login",login).get().get(0);
} catch (InterruptedException | ExecutionException e) {
    e.printStackTrace();
}

And if the method was executed successfully, we create an object of the UserModel class, fill all the fields and return it. Otherwise, we return null.

if(userTuple != null) {
            UserModel user = new UserModel();
            user.setUuid((String)userTuple.get(1));
            user.setLogin((String)userTuple.get(2));
            user.setPassword((String)userTuple.get(3));

            return user;
        }

        return null;

Full code of the getUserByLogin method:

public UserModel getUserByLogin(String login) {

        List<Object> userTuple = null;

        try {
            userTuple = (List<Object>) tarantoolClient.call("get_user_by_login", login).get().get(0);
        } catch (InterruptedException | ExecutionException e) {
            e.printStackTrace();
        }

        if(userTuple != null) {
            UserModel user = new UserModel();
            user.setUuid((String)userTuple.get(1));
            user.setLogin((String)userTuple.get(2));
            user.setPassword((String)userTuple.get(3));

            return user;
        }

        return null;
    }

We override other methods in the same way, but with some changes. Since the logic is quite similar to the one presented above, I'll just provide the full code of this class:

@Service
public class TarantoolStorageService implements StorageService{

    private final TarantoolClient tarantoolClient;

    public TarantoolStorageService(TarantoolClient tarantoolClient) {
        this.tarantoolClient = tarantoolClient;
    }

    @Override
    public UserModel getUserByLogin(String login) {

        List<Object> userTuple = null;

        try {
            userTuple = (List<Object>) tarantoolClient.call("get_user_by_login", login).get().get(0);
        } catch (InterruptedException | ExecutionException e) {
            e.printStackTrace();
        }

        if(userTuple != null) {
            UserModel user = new UserModel();
            user.setUuid((String)userTuple.get(1));
            user.setLogin((String)userTuple.get(2));
            user.setPassword((String)userTuple.get(3));

            return user;
        }

        return null;
    }

    @Override
    public String createUser(CreateUserRequest request) {

        String uuid = UUID.randomUUID().toString();
        List<Object> userTuple = null;

        try {
            userTuple = (List<Object>) tarantoolClient.call("create_user",
                    uuid,
                    request.getLogin(),
                    DigestUtils.md5DigestAsHex(request.getPassword().getBytes())
            ).get();
        } catch(InterruptedException | ExecutionException e) {
            e.printStackTrace();
        }

        if(userTuple != null) {
            return (String) userTuple.get(0);
        }

        return null;
    }

    @Override
    public boolean updateUser(String login, UpdateUserRequest request) {

        List<Object> userTuple = null;

        try {
            userTuple = (List<Object>) tarantoolClient.call("update_user_by_login",
                    login, DigestUtils.md5DigestAsHex(request.getPassword().getBytes())
            ).get().get(0);
        } catch(InterruptedException | ExecutionException e) {
            e.printStackTrace();
        }

        return userTuple != null;
    }

    @Override
    public boolean deleteUser(String login) {
        List<Object> userTuple = null;

        try {
            userTuple = (List<Object>) tarantoolClient.call("delete_user_by_login",
                    login
            ).get().get(0);
        } catch(InterruptedException | ExecutionException e) {
            e.printStackTrace();
        }

        return userTuple != null;
    }
}

After implementing this auxiliary service, we need to create services that contain user authentication and modification logic. The service for modifying and retrieving information about the user will be called UserService. It is quite straightforward in its implementation, as it's initialized by an object of the StorageService class and simply calls the methods defined in it. So I'll just provide the full code for this class, too:

@Service
public class UserService {
    private final StorageService storageService;

    public UserService(StorageService storageService) {
        this.storageService = storageService;
    }

    public String createUser(CreateUserRequest request) {
        return this.storageService.createUser(request);
    }

    public boolean deleteUser(String login) {
        return this.storageService.deleteUser(login);
    }

    public UserModel getUserByLogin(String login) {
        return this.storageService.getUserByLogin(login);
    }

    public boolean updateUser(String login, UpdateUserRequest request) {
        return this.storageService.updateUser(login, request);
    }
}

The second service, which authenticates the user, we will call AuthenticationService. It will also be initialized with an object of the StorageService class and will contain only one method, authenticate, responsible for user authentication. How exactly is the authentication performed? This method calls the user's information from Tarantool by the user's login. Then it calculates the MD5 hash of the password and compares it with the one received from Tarantool. If the hashes match, the method returns a token, which for simplicity is just the user UUID, otherwise, it returns null. Full code of the AuthenticationService class:

@Service
public class AuthenticationService {

    private final StorageService storageService;

    public AuthenticationService(StorageService storageService) {
        this.storageService = storageService;
    }

    public AuthUserResponse authenticate(String login, String password) {
        UserModel user = storageService.getUserByLogin(login);

        if(user == null) {
            return null;
        }

        String passHash = DigestUtils.md5DigestAsHex(password.getBytes());

        if (user.getPassword().equals(passHash)) {

            AuthUserResponse response = new AuthUserResponse();
            response.setAuthToken(user.getUuid());
            return response;

        } else {
            return null;
        }
    }
}

Now let's create two controllers responsible for authentication of the user and processing their information. The first one will be AuthenticationController, and the second one will be UserController.

Let's start with the AuthenticationController. Each controller is initialized with its own service, so we initialize the first one with an object of the AuthenticationService class. Our controller will also contain a mapping to the /login endpoint. It will parse the request, call the authenticate method of the service, and — based on the result of the call — return either UUID and code 200 or code 403 (Forbidden). Full code for this controller:

@RestController
public class AuthenticationController {
    private final AuthenticationService authenticationService;

    public AuthenticationController(AuthenticationService authenticationService) {
        this.authenticationService = authenticationService;
    }

    @PostMapping(value = "/login", produces={"application/json"})
    public ResponseEntity<AuthUserResponse> authenticate(@RequestBody AuthUserRequest request) {

        String login = request.getLogin();
        String password = request.getPassword();

        AuthUserResponse response = this.authenticationService.authenticate(login, password);

        if(response != null) {

            return ResponseEntity.status(HttpStatus.OK)
                    .cacheControl(CacheControl.noCache())
                    .body(response);
        } else {
            return new ResponseEntity<>(HttpStatus.FORBIDDEN);
        }
    }
}

The second controller, UserController, will be initialized with an object of the UserService class. It will contain mappings to the /register and /{login} endpoints. This controller's full code:

@RestController
public class UserController {

    private final UserService userService;

    public UserController(UserService userService) {
        this.userService = userService;
    }

    @PostMapping(value = "/register", produces={"application/json"})
    public ResponseEntity<CreateUserResponse> createUser(
            @RequestBody CreateUserRequest request) {
        String login = this.userService.createUser(request);

        if(login != null) {

            CreateUserResponse response = new CreateUserResponse();
            response.setLogin(login);

            return ResponseEntity.status(HttpStatus.OK)
                    .cacheControl(CacheControl.noCache())
                    .body(response);
        } else {
            return new ResponseEntity<>(HttpStatus.BAD_REQUEST);
        }
    }

    @GetMapping(value = "/{login}", produces={"application/json"})
    public ResponseEntity<GetUserInfoResponse> getUserInfo(
            @PathVariable("login") String login) {
        UserModel model = this.userService.getUserByLogin(login);
        if(model != null) {
            GetUserInfoResponse response = new GetUserInfoResponse();
            response.setUuid(model.getUuid());
            response.setLogin(model.getLogin());
            response.setPassword(model.getPassword());

            return ResponseEntity.status(HttpStatus.OK)
                    .cacheControl(CacheControl.noCache())
                    .body(response);
        } else {
            return new ResponseEntity<>(HttpStatus.NOT_FOUND);
        }
    }

    @PutMapping(value = "/{login}", produces={"application/json"})
    public ResponseEntity<Void> updateUser(
            @PathVariable("login") String login,
            @RequestBody UpdateUserRequest request) {
        boolean updated = this.userService.updateUser(login, request);

        if(updated) {
            return ResponseEntity.status(HttpStatus.OK)
                    .cacheControl(CacheControl.noCache())
                    .build();
        } else {
            return new ResponseEntity<>(HttpStatus.NOT_FOUND);
        }
    }

    @DeleteMapping(value = "/{login}", produces={"application/json"})
    public ResponseEntity<Void> deleteUser(
            @PathVariable("login") String login) {
        boolean deleted = this.userService.deleteUser(login);

        if(deleted) {
            return ResponseEntity.status(HttpStatus.OK)
                    .cacheControl(CacheControl.noCache())
                    .build();
        } else {
            return new ResponseEntity<>(HttpStatus.NOT_FOUND);
        }
    }

}

This concludes the development of our Java application. All that's left to do now is build it. You can do that by running

$ mvn clean package

After the application is built, you can run it with:

$ java -jar ./target/authentication-example-1.0-SNAPSHOT.jar

Now we have finished developing our service! You can see its full code here.

What was done

Installed the Java connector.
Set up a connection to the cluster.
Developed an API.
Created controllers and services.
Built our application.

What's left to do is to test the service.

Checking if the service works

Let's check how correctly each of the requests is being processed. We'll use Postman for that task. We will use a test user with login1 as their username and password1 as their password.

We start by creating a new user. The request will look like this:

The result is:

Now let's check the authentication:

Check the user's data:

Trying to update the user's password:

Checking if the password was updated:

Deleting the user:

Trying to authenticate again:

Checking the user's data again:

All requests are executed correctly, we receive the expected results.

Conclusion

As an example, we implemented an authentication system consisting of two applications:

A Tarantool Cartridge application that implements the business logic for handling user information and data storage.
A Java application providing an API for authentication.

Tarantool Cartridge is a framework for scaling and managing a cluster of multiple Tarantool instances, and also for developing cluster applications.

We used the Cartridge Java Connector, which replaced the outdated Tarantool Java Connector, to communicate between the applications we wrote. It allows you to work not only with single instances of Tarantool, but also with entire clusters, which makes the connector more versatile and irreplaceable for developing enterprise applications.

Distributed storage in 30 minutes

tarantool — Mon, 14 Feb 2022 13:02:44 +0000

Author: Igor Zolotarev

Hello, my name is Igor, and I am a part of the Tarantool DB team. When developing, I often need rapid prototypes of database applications, for example, to test code or to create an MVP. Of course, I would like such a prototype to require minimal effort to refine, in case it is decided to use it in production.

I don't like wasting my time configuring an SQL database, thinking about how to manage data sharding, or spending even more time studying connector interfaces. I prefer just to write a few lines of code, run it, and have everything working out of the box. To develop distributed applications rapidly, I use Cartridge, a framework for managing cluster applications based on Tarantool, a NoSQL database.

Today I will show how to quickly write a Cartridge-based application, cover it with tests, and run it. The article will be of interest to anyone tired of spending a lot of time prototyping applications, as well as those who want to try a new NoSQL technology.

From this article, you will learn what Cartridge is and what principles to have in mind when writing cluster business logic in it.

We will write a cluster application for storing data about employees of a company. The steps to accomplish it are:

Creating an application from a template with cartridge-cli
Describing your business logic in Lua in terms of Cartridge cluster roles
- Data Storage
- Custom HTTP API
Writing tests
Launching and configuring a small cluster locally
- Downloading configuration
- Configuring failover

Cartridge framework

Cartridge is a framework for developing cluster applications. It manages several instances of the Tarantool NoSQL database and shards data using the vshard module. Tarantool is a persistent in-memory database. It is very fast due to storing data in RAM but also reliable since Tarantool dumps all data to the hard disk and allows you to set up replication. Cartridge takes care of configuring Tarantool nodes and sharding cluster nodes, which leaves a developer with only writing the business logic of the applications and configuring the failover.

Advantages of Cartridge

Sharding and replication out of the box
Built-in failover support
CRUD, a NoSQL cluster query language
Integration testing of the entire cluster
Ansible-based cluster management
Cluster administration utility
Monitoring tools

Creating the first application

To do this, we will need cartridge-cli. It is a utility for working with Cartridge applications. It allows you to create an application from a template, manage a locally running cluster, and connect to Tarantool instances.

Installing Tarantool and cartridge-cli

On Debian or Ubuntu:

curl -L https://tarantool.io/fJPRtan/release/2.8/installer.sh | bash

sudo apt install cartridge-cli

On CentOS, Fedora, or ALT Linux:

curl -L https://tarantool.io/fJPRtan/release/2.8/installer.sh | bash

sudo yum install cartridge-cli

On macOS:

brew install tarantool

brew install cartridge-cli

Let's create a template application named myapp:

cartridge create --name myapp

cd myapp

tree .

Now we get a project structure similar to this:

myapp
├── app
│ └── roles
│ └── custom.lua
├── test
├── init.lua
├── myapp-scm-1.rockspec

the init.lua file is the entry point of the Cartridge application. It defines the cluster's configuration and calls the functions required at the start of each application node.
the app/roles/ directory contains "roles" describing the application's business logic.
the myapp-scm-1.rockspec file specifies the application's dependencies

By now, we already have a working "Hello, world!" application. It can be started with the following commands

cartridge build

cartridge start -d

cartridge replicasets setup --bootstrap-vshard

After that, access to localhost:8081/hello will show "Hello world!".

Let's now create a small template-based application, a sharded storage with an HTTP API for storing and receiving data. To do this, we need to understand how to implement cluster business logic in Cartridge.

Writing business logic in Cartridge

Each cluster application is based on roles, Lua modules describing the application's business logic. For example, they can be modules that store data, provide HTTP API, or cache data from an Oracle database. A role is assigned to a set of instances joined by replication (a replica set), and it is enabled at each instance. Replica sets can have different set of roles.

In Cartridge, on each cluster node, there is a cluster configuration. It describes the cluster's topology and, optionally, the configuration that your role would use. Such configuration can be changed in runtime to affect the role's behavior.

Each role has a structure similar to this:

return {
    role_name = 'your_role_name',

    init = init,
    validate_config = validate_config,
    apply_config = apply_config,
    stop = stop,

    rpc_function = rpc_function,
    dependencies = {
        'another_role_name',
    },
}

Role lifecycle

An instance is starting.
The role named role_name waits for the start of all its dependent roles specified in dependencies.
The validate_config function is called to check whether the role's configuration is valid.
The role initialization function init is called. This function performs the actions that need to be done once, when the role is started for the first time.
The apply_config function is called to apply the configuration (if it is specified). The validate_config and apply_config functions are also called whenever the role's configuration changes.
The role is saved in the registry. From there it will be available to other roles on the same node via cartridge.service_get('your_role_name').
The functions declared in a role will be available from other nodes via cartridge.rpc_call('your_role_name', 'rpc_function').
Before a role is stopped or restarted, the stop function is launched. It terminates the role, for example, removing the fibers created by the role.

Cluster NoSQL queries

There are several ways to write cluster queries in Cartridge:

Calling functions via the vshard API (this is a complicated but flexible way):

vshard.router.callrw(bucket_id, 'app.roles.myrole.my_rpc_func', {...})

Tarantool CRUD
- Simple function calls: crud.insert / get / replace / ...
- Support for calculating bucket_id is limited
- Roles must depend on crud-router / crud-storage

Application structure

Suppose we want a cluster with one router and two groups of storages with two instances each. This topology is typical for both Redis Cluster and MongoDB Cluster. For the stateful failover to save the state of the current masters, the cluster will include stateboard, yet another instance. When increased reliability is required, it is better to use an etcd cluster instead of stateboard.

The router will distribute requests across the cluster and manage the failover.

Writing custom roles

We will need to write two roles: one for data storage, and one for HTTP API.

In the app/roles directory, we create two new files: app/roles/storage.lua and app/roles/api.lua

Data Storage

Let's describe the role for data storage. In the init function, we will create a table and indexes for it, then add crud-storage to its dependencies.

The Lua code in the init function is equivalent to the following pseudo-SQL code:

CREATE TABLE employee(
    bucket_id unsigned,
    employee_id string,
    name string,
    department string,
    position string,
    salary unsigned
);
CREATE UNIQUE INDEX primary ON employee(employee_id);
CREATE INDEX bucket_id ON employee(bucket_id);

Add the following code to the app/roles/storage.lua file:

local function init(opts)
    -- opts has the attribute indicating if the function is called at the master or at the replica
    --  we create tables only at the master instance, they will appear automatically at the replica
    if opts.is_master then
        -- Creating a table with employees
        local employee = box.schema.space.create('employee', {if_not_exists = true})

        -- setting the format
        employee:format({
            {name = 'bucket_id', type = 'unsigned'},
            {name = 'employee_id', type = 'string', comment = 'ID СЃРѕС‚СЂСѓРґРЅРёРєР°'},
            {name = 'name', type = 'string', comment = 'Р¤Р˜Рћ СЃРѕС‚СЂСѓРґРЅРёРєР°'},
            {name = 'department', type = 'string', comment = 'РћС‚РґРµР»'},
            {name = 'position', type = 'string', comment = 'Р”РѕР»Р¶РЅРѕСЃС‚СЊ'},
            {name = 'salary', type = 'unsigned', comment = 'Р—Р°СЂРїР»Р°С‚Р°'}
        })

        -- Create the primary index
        employee:create_index('primary', {parts = {{field = 'employee_id'}},
            if_not_exists = true })

        -- Indexing by bucket_id, it is necessary for sharding
        employee:create_index('bucket_id', {parts = {{field = 'bucket_id'}},
            unique = false,
            if_not_exists = true })
    end

    return true
end

return {
    init = init,
    -- <<< remembering the crud-storage dependency
    dependencies = {'cartridge.roles.crud-storage'},
}

We will not need the rest of the functions from the role's API, since our role has no configuration and it does not allocate resources to be cleaned after the role's work is complete.

HTTP API

We will need the second role to fill the tables with data and retrieve this data on request. The role will access the Cartridge's built-in HTTP server. It depends on crud-router.

Let's define a function to handle POST requests. The request body will contain the object to be saved to the database.

local function post_employee(request)
    -- getting an object from the request body
    local employee = request:json()

    -- writing it to the database
    local _, err = crud.insert_object('employee', employee)

    -- if an error occurs, writing it to the log and returning 500
    if err ~= nil then
        log.error(err)
        return {status = 500}
    end
    return {status = 200}
end

The GET method will take the employees' salary values as a parameter. The expected response is a JSON with a list of employees whose salary is higher than the one specified in the request.

SELECT employee_id, name, department, position, salary
FROM employee
WHERE salary >= @salary

local function get_employees_by_salary(request)
    -- get the salary parameter from the query
    local salary = tonumber(request:query_param('salary') or 0)

    -- selecting the employee data
    local employees, err = crud.select('employee', {{'>=', 'salary', salary}})

    -- if an error occurs, writing it to the log and returning 500
    if err ~= nil then
        log.error(err)
        return { status = 500 }
    end

    -- the employees table stores the list of rows that meet the condition and the space format
    -- the unflatten_rows function converts a table row to a key-value table
    employees = crud.unflatten_rows(employees.rows, employees.metadata)

    employees = fun.iter(employees):map(function(x) 
        return {
            employee_id = x.employee_id, 
            name = x.name, 
            department = x.department, 
            position = x.position, 
            salary = x.salary,
        }
    end):totable()
    return request:render({json = employees})
end

Now let's write the init function for the role. Here we will turn to the Cartridge's registry to get an HTTP server and use it to assign the HTTP endpoints for the application.

local function init()
    -- getting an HTTP-server from the Cartridge's registry
    local httpd = assert(cartridge.service_get('httpd'), "Failed to get httpd serivce")

    -- setting the routes
    httpd:route({method = 'GET', path = '/employees'}, get_employees_by_salary)
    httpd:route({method = 'POST', path = '/employee'}, post_employee)

    return true
end

Putting everything together:

app/roles/api.lua

local cartridge = require('cartridge')
local crud = require('crud')
local log = require('log')
local fun = require('fun')

-- the 'GET /employees' method returns a list of employees with salaries greater than the one specified in the request
local function get_employees_by_salary(request)
    -- getting the salary parameter from the query
    local salary = tonumber(request:query_param('salary') or 0)

    -- selecting the employee data
    local employees, err = crud.select('employee', {{'>=', 'salary', salary}})

    -- if an error occurs, writing it to the log and returning 500
    if err ~= nil then
        log.error(err)
        return { status = 500 }
    end

    -- the employees table stores the list of rows that meet the condition and the space format
    -- the unflatten_rows function converts a table row to a key-value table
    employees = crud.unflatten_rows(employees.rows, employees.metadata)
    employees = fun.iter(employees):map(function(x) 
        return {
            employee_id = x.employee_id, 
            name = x.name, 
            department = x.department, 
            position = x.position, 
            salary = x.salary,
        }
    end):totable()
    return request:render({json = employees})
end

local function post_employee(request)
    -- getting an object from the request body
    local employee = request:json()

    -- writing it to the database
    local _, err = crud.insert_object('employee', employee)

    -- if an error occurs, writing it to the log and returning 500
    if err ~= nil then
        log.error(err)
        return {status = 500}
    end
    return {status = 200}
end

local function init()
    -- getting an HTTP-server from the Cartridge's registry
    local httpd = assert(cartridge.service_get('httpd'), "Failed to get httpd service")

    -- setting the routes
    httpd:route({method = 'GET', path = '/employees'}, get_employees_by_salary)
    httpd:route({method = 'POST', path = '/employee'}, post_employee)

    return true
end

return {
    init = init,
    -- addind the crud-storage dependency
    dependencies = {'cartridge.roles.crud-router'},
}

init.lua

Let's describe the init.lua file. It is the entry point of a Cartridge application. To configure a cluster instance, the function cartridge.cfg() should be called in the init file of the cartridge.

cartridge.cfg(<opts>, <box_opts>)

<opts>, the default cluster parameters
- the list of available roles (all roles must be specified, even the permanent ones, to have them appear in the cluster)
- sharding parameters
- WebUI configuration
- etc
<box_opts>, the Tarantool default parameters (passed to the instance's box.cfg{})

#!/usr/bin/env tarantool

require('strict').on()

-- specifying the path to search for modules
if package.setsearchroot ~= nil then
    package.setsearchroot()
end

-- configuring Cartridge
local cartridge = require('cartridge')

local ok, err = cartridge.cfg({
    roles = {
        'cartridge.roles.vshard-storage',
        'cartridge.roles.vshard-router',
        'cartridge.roles.metrics',
        -- <<< Adding crud roles
        'cartridge.roles.crud-storage',
        'cartridge.roles.crud-router',
        -- <<< Adding custom roles
        'app.roles.storage',
        'app.roles.api',
    },
    cluster_cookie = 'myapp-cluster-cookie',
})

assert(ok, tostring(err))

The final step is to describe the dependencies of the application in the myapp-scm-1.rockspec file.

package = 'myapp'
version = 'scm-1'
source  = {
    url = '/dev/null',
}
-- Adding the dependencies
dependencies = {
    'tarantool',
    'lua >= 5.1',
    'checks == 3.1.0-1',
    'cartridge == 2.7.3-1',
    'metrics == 0.11.0-1',
    'crud == 0.8.0-1',
}
build = {
    type = 'none';
}

The application's code is ready to run, but let's write some tests to make sure it works as expected.

Writing tests

Every application needs testing. The usual luatest is sufficient for unit tests. But to write an integration test, you may want to use the cartridge.test-helpers module. It is shipped with Cartridge and can be used to run a cluster of any structure for the tests.

local cartridge_helpers = require('cartridge.test-helpers')
-- creating a test cluster
local cluster = cartridge_helpers.Cluster:new({
    server_command = './init.lua', -- test application entrypoint
    datadir = './tmp', -- directory for xlog, snap, and other files
    use_vshard = true, -- enable cluster sharding
    -- list of replica sets:
    replicasets = {
        {
            alias = 'api',
            uuid = cartridge_helpers.uuid('a'),
            roles = {'app.roles.custom'}, -- list of roles assigned to the replicaset
            -- list of instances in the replicaset:
            servers = {
                { instance_uuid = cartridge_helpers.uuid('a', 1), alias = 'api' },
                ...
            },
        },
        ...
    }
})

Let's write an auxiliary module to use in the integration tests. In this module, a test cluster with two replica sets is created. Each replica set contains one instance:

The auxiliary module code:

test/helper.lua

local fio = require('fio')
local t = require('luatest')
local cartridge_helpers = require('cartridge.test-helpers')

local helper = {}

helper.root = fio.dirname(fio.abspath(package.search('init')))
helper.datadir = fio.pathjoin(helper.root, 'tmp', 'db_test')
helper.server_command = fio.pathjoin(helper.root, 'init.lua')

helper.cluster = cartridge_helpers.Cluster:new({
    server_command = helper.server_command,
    datadir = helper.datadir,
    use_vshard = true,
    replicasets = {
        {
            alias = 'api',
            uuid = cartridge_helpers.uuid('a'),
            roles = {'app.roles.api'},
            servers = {
                { instance_uuid = cartridge_helpers.uuid('a', 1), alias = 'api' },
            },
        },
        {
            alias = 'storage',
            uuid = cartridge_helpers.uuid('b'),
            roles = {'app.roles.storage'},
            servers = {
                { instance_uuid = cartridge_helpers.uuid('b', 1), alias = 'storage' },
            },
        },
    }
})

function helper.truncate_space_on_cluster(cluster, space_name)
    assert(cluster ~= nil)
    for _, server in ipairs(cluster.servers) do
        server.net_box:eval([[
            local space_name = ...
            local space = box.space[space_name]
            if space ~= nil and not box.cfg.read_only then
                space:truncate()
            end
        ]], {space_name})
    end
end

function helper.stop_cluster(cluster)
    assert(cluster ~= nil)
    cluster:stop()
    fio.rmtree(cluster.datadir)
end

t.before_suite(function()
    fio.rmtree(helper.datadir)
    fio.mktree(helper.datadir)
    box.cfg({work_dir = helper.datadir})
end)

return helper

The integration test code:

test/integration/api_test.lua

local t = require('luatest')
local g = t.group('integration_api')

local helper = require('test.helper')
local cluster = helper.cluster

g.before_all = function()
    g.cluster = helper.cluster
    g.cluster:start()
end

g.after_all = function()
    helper.stop_cluster(g.cluster)
end

g.before_each = function()
    helper.truncate_space_on_cluster(g.cluster, 'employee')
end

g.test_get_employee = function()
    local server = cluster.main_server

    -- filling the storage with data via HTTP API:
    local response = server:http_request('post', '/employee',
        {json = {name = 'John Doe', department = 'Delivery', position = 'Developer',
        salary = 10000, employee_id = 'john_doe'}})
    t.assert_equals(response.status, 200)

    response = server:http_request('post', '/employee',
        {json = {name = 'Jane Doe', department = 'Delivery', position = 'Developer',
        salary = 20000, employee_id = 'jane_doe'}})
    t.assert_equals(response.status, 200)

    -- Making a GET request and checking if the output data is correct
    response = server:http_request('get', '/employees?salary=15000.0')
    t.assert_equals(response.status, 200)
    t.assert_equals(response.json[1], {name = 'Jane Doe', department = 'Delivery', employee_id = 'jane_doe',
        position = 'Developer', salary = 20000
    })
end

Running the tests

If you had launched the application before

Stopping the application:

cartridge stop

Removing the directory containing the data:

rm -rf tmp/

Building the application and setting the dependencies:

cartridge build

./deps.sh

Running the linter:

.rocks/bin/luacheck .

Running the tests to record the coverage:

.rocks/bin/luatest --coverage

Generating the coverage reports and looking at the result:

.rocks/bin/luacov .
grep -A999 '^Summary' tmp/luacov.report.out

Running locally

To run applications locally, you can use cartridge-cli, but the roles we have written should be added to replicasets.yml:

router:
  instances:
  - router
  roles:
  - failover-coordinator
  - app.roles.api
  all_rw: false
s-1:
  instances:
  - s1-master
  - s1-replica
  roles:
  - app.roles.storage
  weight: 1
  all_rw: false
  vshard_group: default
s-2:
  instances:
  - s2-master
  - s2-replica
  roles:
  - app.roles.storage
  weight: 1
  all_rw: false
  vshard_group: default

To see the parameters of the configured instances, take a look at the instances.yml file.

Running the cluster locally:

cartridge build
cartridge start -d
cartridge replicasets setup --bootstrap-vshard

Now we can enter WebUI to load the roles' configuration and to configure the failover. To configure a stateful failover, do the following:

click the Failover button
choose stateful
specify the address and the password:
- localhost:4401
- passwd

Let's see how it works. Now the leader in the s-1 replica set is s1-master.

Let's stop it:

cartridge stop s1-master

Now s1-replica becomes the leader:

Let's restore s1-master:

cartridge start -d s1-master

s1-master is up again, but s1-replica is still the leader because of the stateful failover:

Let's load the configuration for the cartridge.roles.metrics role. To do this, switch to the Code tab and create the metrics.yml file with the following contents:

export:
  - path: '/metrics'
    format: prometheus
  - path: '/health'
    format: health

After we click the Apply button, the metrics will become available at each node of the application at the localhost:8081/metrics endpoint. The health-check page at the localhost:8081/health address will also appear.

This completes the basic setup of a small application: the cluster is ready to run and now we can write an application to communicate with the cluster using the HTTP API or via a connector. We can also expand the functionality of the cluster.

Conclusion

Many developers hate wasting time configuring a database. We prefer simply writing code and leaving cluster management to a framework. To solve this problem, I use Cartridge, a framework that manages a cluster containing several instances of a Tarantool database.

Now you know:

how to build a reliable cluster application based on Cartridge and Tarantool,
how to write the code for a small application to store information about employees,
how to add tests,
how to configure a cluster.

I hope my story was helpful and you will start using Cartridge to create applications. I would be glad to hear feedback on whether you managed to write a Cartridge application quickly and easily as well as questions about its use.

What's next?

Check out the documentation on the official website
Try Cartridge in the sandbox
Ask your questions to the community in the Telegram chat

Ten-year experience in DBMS testing

tarantool — Fri, 04 Feb 2022 15:35:48 +0000

Author: Sergey Bronnikov

Hi, my name is Sergey Bronnikov, and I work on the Tarantool database. Once I joined, I started taking notes regarding Tarantool development. Now I've decided to rewrite these notes as an article. It might be of interest to C/C++ testers or Tarantool users who want to know how much effort we put into preventing potential issues in new versions.

How SQLite Is Tested by Richard Hipp is a similar article, which is quite popular. But SQLite specifics make it hard to reuse its tools in other projects. This stems from the commitment of the SQLite development team to maintain the library until at least 2050. Hence, they write all the tools from scratch to reduce external dependencies (e.g., the test runner, the mutation testing tool, Fossil SCM). There are no such requirements for us, so we are not limited in our choice of tools and can use anything beneficial. And if any tool appeals to you, you can easily bring it into your C/C++ project. If that's your cup of tea, you should read the entire article.

As you know, testing is part of development. In this article, I will talk about our approach to Tarantool development, which helps us catch the vast majority of bugs before the final release. For us, testing is inseparable from the development itself, and everyone in the team is responsible for quality. I couldn't fit everything into a single article, so I have provided links to other supporting articles at the very end.

The Tarantool's core consists of the code written entirely by us, external components, and libraries. By the way, some components and libraries were also written by us. This is important because we test most of the third-party components only indirectly during integration testing.

In most cases, external components have good quality, but there was an exception — the libcurl library. Sometimes it could cause memory corruptions. Therefore, libcurl became a Git module in our repository rather than a runtime dependency.

LuaJIT provides Lua language support, including both the language execution environment and the JIT tracer compiler. Our LuaJIT has long differed from the vanilla version in a set of patches adding features, such as the profiler, and new tests. That is why we test our fork thoroughly to prevent regression. LuaJIT source code is open and distributed under a free license, but it does not include regression tests. Therefore, we have assembled our regression test suite from PUC-Rio Lua tests, test suite by François Perrad, tests for other LuaJIT forks, and of course, our own tests.

Other external libraries are the following:

MsgPuck to serialize MessagePack data.
libcoro to implement fibers.
libev to provide asynchronous I/O.
c-ares to resolve DNS names asynchronously.
libcurl to work with the HTTP protocol.
icu4c to support Unicode.
OpenSSL, libunwind and zstd to compress data.
small — our set of specialized memory allocators.
lua-cjson to work with JSON, lua-yaml, luarocks, xxHash, PMurHash, etc.

The bulk of the project is written in C, smaller parts in C++ (a total of 36 KLOC) and an even smaller part in Lua (14 KLOC).

Detailed cloc statistics is provided below:

767 text files.
758 unique files.
82 files ignored.
github.com/AlDanial/cloc v 1.82 T=0.78 s (881.9 files/s, 407614.4 lines/s)

-------------------------------------------------------------------------------
Language      files   blank   comment   code
-------------------------------------------------------------------------------
C             274     12649   40673     123470
C/C++ Header  287     7467    36555     40328
C++           38      2627    6923      24269
Lua           41      1799    2059      14384
yacc          1       191     342       1359
CMake         33      192     213       1213
...
-------------------------------------------------------------------------------
SUM:          688     25079   86968     205933
-------------------------------------------------------------------------------

The other languages are related to the project infrastructure or tests: CMake, Make, Python (we don't use Python anymore to write tests, but some older tests were written in it).

Detailed cloc output regarding languages used in tests is provided below:

2076 text files.
2006 unique files.
851 files ignored.

github.com/AlDanial/cloc v 1.82 T=2.70 s (455.5 files/s, 116365.0 lines/s)

-------------------------------------------------------------------------------
Language      files   blank   comment   code
-------------------------------------------------------------------------------
Lua           996     31528   46858     194972
C             89      2536    2520      14937
C++           21      698     355       4990
Python        57      1131    1209      4500
C/C++ Header  11      346     629       1939
SQL           4       161     120       1174
...
-------------------------------------------------------------------------------
SUM:         1231     37120   51998     225336
-------------------------------------------------------------------------------

Such distribution of used programming languages causes testing to focus mainly on identifying problems related to manual memory management (stack overflow, heap buffer overflow, use-after-free, etc.). Our continuous integration system handles it pretty well. I will tell you more about our CI system in the next section.

Continuous integration

We have several branches of Tarantool in development: the main branch (master) and one branch for each version (1.10.x, 2.1.x, 2.2.x, etc.). Minor versions introduce new functions, and the bugs are fixed in all the branches. When merging the branch, we run the entire cycle of regression tests with different compilers and build options. We also build packages for various platforms and do so much more — see details below. Everything is done automatically in a single pipeline. Patches only make it to the main branch after passing the entire pipeline. For now, we apply patches to the main branch manually, but we're aiming for automation.

Currently, we have about 870 integration tests, and they run for 10 minutes on five parallel threads. It may not seem much, but CI testing involves different OS families and versions, architectures, various compilers with options, so the total testing time can be up to half an hour.

We run tests for a large number of operating systems:

Six Ubuntu versions
Three Debian versions
Five Fedora versions
Two CentOS versions
Two OpenSUSE and FreeBSD versions
Two macOS versions

Some configurations also depend on compiler versions and options. Some platforms are supported formally (e.g., macOS) — they are used mainly by developers. Others, such as FreeBSD, are actively tested, but I haven't heard of any cases of using Tarantool FreeBSD port in production. Others, such as Linux, are widely used in production by Tarantool customers and users.

Therefore, the last-mentioned platforms are given more attention in development. Running tests on different operating systems affects the project quality. Different OS families have different memory allocators and may have diverse libc implementations — such variations also allow us to find bugs.

The primary architecture is amd64; we recently added support for ARM64, and it is also represented in CI. Running tests on processors with different architecture makes the code more portable by separating platform-dependent and platform-independent code. It helps detect bugs related to different byte order (big-endian vs. little-endian), instruction execution speed, different mathematical function results, or to such rarities as negative zero. This type of testing makes it easier to port the code to the new architecture, if necessary. LuaJIT is the most platform-dependent since it uses the assembler a lot and generates machine code from Lua code.

Back when there were not so many cloud CI systems, we used Jenkins just like many other projects. Then Travis CI appeared, which is integrated with GitHub, and we migrated. Then our testing matrix grew a lot, and the free version of Travis CI didn't allow us to connect our servers, so we migrated to Gitlab CI. Due to integration issues with GitHub pull requests, we gradually migrated to GitHub Actions as soon as it appeared. Now we use it in all the projects (and we have several hundred of them in our GitHub organization).

We use Github as our platform for the whole development cycle: task scheduling, code repository, testing new changes. All the testing is done there. For this purpose, we use both our physical servers or virtual machines in VK Cloud Solutions, as well as virtual machines provided by Github Actions. GitHub is not flawless: sometimes it's unavailable, sometimes it glitches, but it is good value for money.

Portable code has to be tested on different operating systems and architectures.

Code review

Like all civilized projects with a good development culture, we submit all the patches for a thorough review by two other developers. The code review procedure is described in this open document. It contains style and self-check guidelines to follow before submitting the patch for review. I won't narrate the whole document; I will just list the points related to testing:

Every bug-fixing patch should have a test to reproduce the issue.
Every feature-introducing patch should have one, or better yet, many tests covering the feature.
The test can't pass without the patch.
The test shouldn't be flaky — it has to produce the same result each time it is run.
The test shouldn't be slow to keep the short test duration. Long tests are run with a different test runner option.

Code review allows us to check changes with another pair of eyes.

Static and dynamic analysis

We use static analysis to maintain the general programming style and search for errors. The code style should follow the style guides for Lua, Python, and C. The C style guide is similar to the Linux kernel coding style in a lot of ways, and the Lua style guide follows the default style in luacheck, except for some warnings, which we usually turn off. This allows us to maintain the unified code style and improves readability.

In CMake build files, we use compiler flags that enable extra checks at build time, and we run "make clean" when there are no raw warnings. Besides static analysis in the compilers, we use Coverity static analysis. We used PVS-Studio once, and it detected several non-critical errors in Tarantool itself and in tarantool-c connector. Sometimes, we used cppcheck, not that it found many bugs.

Tarantool codebase contains a lot of Lua code, and we decided to fix all the warnings cppcheck found. Most of them were related to programming style violations, and it found only four source code errors and a single error in the test code. So if you're writing in Lua, don't disregard luacheck and use it from the beginning.

All new changes are tested on builds with dynamic parsers to detect memory problems in C/C++ (AddressSanitizer) and undefined behavior in C/C++ (UndefinedBehaviorSanitizer). Since these parsers can affect application performance, the flags that enable them are disabled by default. AddressSanitizer proved itself well in CI, but it still has a considerable overhead in canary builds. I tried using the Firefox Nightly build when Mozilla introduced ASan, and it wasn't comfortable to use (let alone a DBMS with high-speed requirements). But GWP-ASAN has a smaller overhead, and we are thinking about using it in packages with nightly builds.

If the sanitizers detect issues at the code level, the asserts in our code reveal problems related to invariant violations. Technically, they are macros, a part of the standard C library. assert() checks the passed expression and terminates if the result is zero. There are about 5,000 of these checks, and they are only enabled in debug builds and disabled in release builds.

The build system also supports Valgrind, but its code execution is much slower than with sanitizers, so this build is not tested in CI.

Functional regression tests

Since the Lua interpreter is built into Tarantool, and the DBMS interface is implemented using Lua API, using Lua for tests seems quite reasonable. Most of our regression tests are written in Lua using the built-in Tarantool modules. One of them is the TAP module for testing Lua code. It implements a set of primitives to check the code and structure tests. Conveniently, there is a certain minimum — enough to test Lua applications. Many modules and applications, which we make, only use this module for testing. As the name suggests, it allows you to output the results in the TAP (Test Anything Protocol) format; this is probably the oldest format for test reporting. Some of the tests are parameterized (e.g., performed with two engines), and the number of tests grows one and a half times if we count all of them in different configurations.

Most Tarantool functions are available using the Lua API, and others can be accessed using the FFI. The FFI is convenient when a C function should not be part of the Lua API, but it is needed for a test. The main thing is that it is not declared static. Here is an example of using C code in Lua with the FFI (isn't it concise?):

local ffi = require "ffi"

ffi.cdef [[
int printf(const char *fmt, ...);
]]
ffi.C.printf("Hello %s!", "world")

Some parts of Tarantool, such as raft, http_parser, csv, msgpuck, swim, uuid, vclock and other self-contained libraries, have modular tests. To write them, we use a header-only C library in the TAP-test style.

We use our own tool to run tests, test-run.py. Nowadays it may not seem reasonable to write a test runner from scratch, but it already exists, and we support it. There are different types of tests in the project — unit tests are written in C and run as binaries. Like for TAP tests, test-run.py analyzes their output in the TAP format in terms of test script success:

TAP version 13
1..20                                                                      
ok 1 — trigger is fired                                                    
ok 2 — is not deleted                                                      
ok 3 — ctx.member is set    
ok 4 — ctx.events is set                                                   
ok 5 — self payload is updated                                             
ok 6 — self is set as a member
ok 7 — both version and payload events are presented
ok 8 — suspicion fired a trigger                                           
ok 9 — status suspected       
ok 10 — death fired a trigger
ok 11 — status dead                                                                                                                                       
ok 12 — drop fired a trigger                                               
ok 13 — status dropped                                                     
ok 14 — dropped member is not presented in the member table                
ok 15 — but is in the event context      
ok 16 — yielding trigger is fired
ok 17 — non-yielding still is not
ok 18 — trigger is not deleted until all currently sleeping triggers are finished

Some tests compare the actual test output with the reference output: the new test output is saved to a file and compared with the actual one when running further tests. This approach is quite popular in SQL tests (both in MySQL and in PostgreSQL): you write the necessary SQL structures, run the script, make sure that the output is correct, and save it to a file. You just have to make sure that the input is always deterministic. Otherwise, you'll just end up with more flaky tests. The output may depend on the operating system locale (NO_LOCALE=1 will help), on error messages, on the time and date in the output, etc.

We use this approach in tests to support SQL or replication because it's convenient for code debugging: you can paste tests directly into the console and switch between instances. You can experiment interactively and then use this code as a snippet for the ticket or make a test out of it.

test-run.py allows us to run all types of tests in the same way as generating the whole report.

For testing Lua projects, we have a different framework, luatest. This is originally a fork of another good framework, luaunit. Project forking provided tighter integration with Tarantool (e.g., we added specific fixtures). It also allowed us to implement many new features regardless of the luaunit development: integration with luacov, XFail status support, etc.

The history of SQL tests in Tarantool is fascinating. We used VDBE to adopt a part of SQLite code, namely the SQL query parser and the bytecode compiler. One of the main reasons was that SQLite code has almost 100% test coverage. However, the tests were written in TCL, and we don't use it at all. So we had to write a TCL-Lua convertor to port tests written in TCL, and imported them into the code base after optimizing the resulting code. We still use these tests and add new ones when necessary.

Fault tolerance is one of the server software requirements. Hence, many of our tests have error injections at the Tarantool code level. For this purpose, the source code has a set of macros and the Lua API interface to enable them. For example, we want to add an error that will emulate a delay when writing to WAL. We add a string to the ERRINJ_LIST array in src/lib/core/errinj.h:

--- a/src/lib/core/errinj.h
+++ b/src/lib/core/errinj.h
@@ -151,7 +151,6 @@ struct errinj {
_(ERRINJ_VY_TASK_COMPLETE, ERRINJ_BOOL, {.bparam = false}) \
_(ERRINJ_VY_WRITE_ITERATOR_START_FAIL, ERRINJ_BOOL, {.bparam = false})\
_(ERRINJ_WAL_BREAK_LSN, ERRINJ_INT, {.iparam = -1}) \
+ _(ERRINJ_WAL_DELAY, ERRINJ_BOOL, {.bparam = false}) \
_(ERRINJ_WAL_DELAY_COUNTDOWN, ERRINJ_INT, {.iparam = -1}) \
_(ERRINJ_WAL_FALLOCATE, ERRINJ_INT, {.iparam = 0}) \
_(ERRINJ_WAL_IO, ERRINJ_BOOL, {.bparam = false}) \

Then we include this error into the code responsible for writing to WAL:

--- a/src/box/wal.c
+++ b/src/box/wal.c
@@ -670,7 +670,6 @@ wal_begin_checkpoint_f(struct cbus_call_msg *data)
}
vclock_copy(&msg->vclock, &writer->vclock);
msg->wal_size = writer->checkpoint_wal_size;
+ ERROR_INJECT_SLEEP(ERRINJ_WAL_DELAY);
return 0;
}

After that, we can enable delayed logging in the debug build using the following Lua function:

$ tarantool
Tarantool 2.8.0-104-ga801f9f35
type 'help' for interactive help
tarantool> box.error.injection.get('ERRINJ_WAL_DELAY')

---
- false
...
tarantool> box.error.injection.set('ERRINJ_WAL_DELAY', true)

---
- true
...

We have added a total of 90 errors to different parts of Tarantool, and at least one functional test corresponds to each error.

Ecosystem integration testing

The Tarantool ecosystem consists of a large number of connectors for different programming languages and auxiliary libraries to implement popular architectural patterns (e.g., cache or persistent queue). There are also products written in Lua using Tarantool: Tarantool DataGrid and Tarantool Cartridge. We test backward compatibility by running extra tests on pre-release versions of Tarantool, including these modules and products.

Randomized testing

I want to pay special attention to some tests because they differ from standard tests in that their data is generated automatically and randomly. There are no such tests in the standard regression set — they are run separately.

The Tarantool's core is written mostly in C, and even careful development doesn't help avoid memory management issues, such as use-after-free, heap buffer overflow, NULL pointer dereference. Such issues are utterly undesirable for server software. Fortunately, the recent development of dynamic analysis and fuzz-testing technologies makes it possible to reduce the number of these issues.

I have already mentioned that Tarantool uses third-party libraries. Many of them already use fuzz testing: the curl, c-ares, zstd, and OpenSSL projects are regularly tested in the OSS-Fuzz infrastructure. Tarantool code has many parts where the code is used for parsing (e.g., SQL or HTTP query parsing) or MsgPack decoding. This code may be vulnerable to bugs related to memory management. The good news is that fuzz testing quickly detects such issues. Tarantool also has integration with OSS-Fuzz, but there are not many tests yet, and we found a single bug in the http_parser library. The number of such tests might eventually grow, and we have detailed instructions for those who want to add a new one.

In 2020, we added support for synchronous replication and MVCC. We had to test this functionality, so we decided to write some tests powered by Jepsen framework. We check consistency by analyzing the transaction history. But the story about testing with Jepsen is big enough for a separate article, so we'll talk about it next time.

Load and performance testing

One of the reasons people tend to choose Tarantool is its high performance. It would be strange not to test this feature. We have an informal test of inserting 1 million tuples per second on common hardware. Anyone can run it on their machine and get 1 Mops on Tarantool. This snippet in Lua might be a good benchmark to run with synchronous replication:

sergeyb@pony:~/sources$ tarantool relay-1mops.lua 2
making 1000000 operations, 10 operations per txn using 50 fibers
starting 1 replicas
master done 1000009 ops in time: 1.156930, cpu: 2.701883
master speed    864363  ops/sec
replicas done 1000009 ops in time: 3.263066, cpu: 4.839174
replicas speed  306463  ops/sec
sergeyb@pony:~/sources$

For performance testing, we also run common benchmarks: the popular YCSB (Yahoo! Cloud Serving Benchmark), NoSQLBench, LinkBench, SysBench, TPC-H, and TPC-C. We also run C Bench, our own Tarantool API benchmark. Its primitive operations are written in C, and scripts are described in Lua.

Metrics

We collect information to evaluate regression test code coverage. For now, we have covered 83% of all the lines and 51% of all code branches, which is not bad. We use Coveralls to visualize the covered areas. There is nothing new about collecting information on C/C++ code coverage: code instrumentation with the -coverage option, testing, and report generation using gcov and lcov. But when it comes to Lua, the situation is slightly worse: there is a primitive profiler, and luacov provides information only about line coverage. It's a little frustrating.

Release checklist

Each new release involves a bunch of different tasks handled by different teams. These tasks include release tagging in the repository, publishing packages and builds, publishing documentation to the website, checking functional and performance testing results, checking for open bugs, triaging for the next milestone, etc. A version release can easily become chaotic, or some steps may be forgotten. To prevent this from happening, we have described the release process in the form of a checklist, and we follow it before releasing a new version.

Conclusion

As the saying goes, there is always room for improvement. Over time, processes and technologies to detect bugs improve, the bugs become more complicated, and the more complex testing and QA system, the fewer bugs reach users.

Useful links

Get Tarantool on our website
Get help in our telegram channel
Video recording and slides of Roberto Ierusalimschy's lecture on testing Lua interpreter (I highly recommend it!).
How SQLite Is Tested is a popular article on SQLite testing by by Richard Hipp.
The Untold Story of SQLite With Richard Hipp is Richard Hipp's interview where he shares his thoughts on SQLite testing, among others.

Tarantool Running on Apple M1: First Results

tarantool — Fri, 10 Dec 2021 07:56:57 +0000

Apple's M1 chips are not the news any more. Many people know that these chips are fast and that MacOS applications have to be adapted for the new architecture. Tarantool development team decided to take on the same challenge.

My name is Alexey Koryakin, and I am CTO at Tarantool, a part of the VK ecosystem. I will explain why we needed this even though macOS is not used for production servers — I will show how we solved the task. and I will share you the benchmark results.

The Task and the Solution

Tarantool is a high-performance in-memory computing platform that consists of the database and the application server. Many developers install Tarantool on their office computers and write code there. In many cases, it's more convenient than a separate server.

Some developers from our team also prefer installing Tarantool locally. Among them is our product manager, who bought a new M1-based MacBook Air at the beginning of the year. So one time he asked our technical team: "Why doesn't Tarantool run natively on the M1 chip? I have recently bought a new MacBook Air, and I can launch Tarantool only via Rosetta. The native support of Apple M1 chips could become a cool feature for our community so that people switching to new Macs could efficiently develop Tarantool-based systems".

The technical team thought and decided:

Tarantool is known to be very fast. M1 is known to be very fast. And we want to find out how much faster Tarantool could work on M1.
Apple is actively upgrading its Mac product line, migrating them to M1 chips (and now into M1 Max). Developers and other IT specialists across the world are actively switching to the new platform. The existing x86_64 software is launched via the Rosetta translation environment that impairs the full power of the software (including Tarantool) run on new Apple chips. This must be changed.

This is how we've ended up with a new task — support of M1 chips

Around the same time, we were working on the ARM64 support for Linux. Since M1 is essentially ARM64 with some specific features, we assumed that implementing M1 support would be easy. This proved to be not far from the truth: we have finalized most tasks of M1 support with ARM64 support for Linux. The main issues were related to specific features of RISC instructions of ARM architecture. For example, direct transfer of control from one section of the machine code to another one is possible only under 2 MB offsets. Apple ABI that differs from ARM Linux ABI became the distinguishing feature of M1 support. We had to tune Tarantool code specifically for new chips, prying into public Apple documentation.

All in all, support of ARM64 and M1 in particular is a relatively simple engineering task. There were some issues, but we were able to resolve them without reading endless specifications and all-nighters. It took us about 4 months, from May until August, to complete the entire endeavor.

Performance Benchmark

M1 is famous for its outstanding performance. Even the code run through Rosetta works fast. We didn't forget the performance either and decided to check how much faster Tarantool could run.

We compared macOS running on available commodity devices with different hardware. We didn't aim to compare different operating systems or chase after the server CPUs like Xeon. We just wanted to know how much faster Tarantool would run for the developer that switched to a new MacBook, and to assess the perspectives of a new Mac platform. For the test we used several computers owned by the team or available in retail

● MacBook Pro 16,2 (2020),
● Mac mini 8,1 (2018),
● MacBook Air 10,1 (2020), Apple M1.

We wrote a simple benchmark test that does three things:

This is Lua code, which means an application server is involved.
This code writes to the database, which means the transaction engine of the database is involved.
This code runs on М1 and doesn't crash :)

The benchmark runs in one system thread that starts 50 fibers, each of them inserting 100 operations per one transaction. This scenario provides a greater workload for the CPU rather than for memory or storage. And this is exactly what we need to test the CPU. If the transaction consisted of one or three–four updates, then the workload would rather shift from CPU to RAM or storage.

We didd multiple tests inserting from 1 up to 20 million records. We repeated every test 15 times and then calculated the median value. These are complete results to examine all tests in detail. And here, for illustrative purposes, we show the median value on the diagram.

We see that Tarantool performance on M1 chip is twice better than that of the notebook of the same year based on a different processor.

Then we have tested M1 for the tasks fulfilled via the Rosetta translator. The performance turned out to be approximately the same as that of the Mac mini 2018, but significantly higher than that of MacBook Pro 2020.

Wall Clock Benchmark. The same devices plus launch via Rosetta

We understand that this doesn't mean double performance of all applications. They all vary in their code, tasks, and conditions. But in any way, you can expect that your local Tarantool installation will work faster.

To Sum Up...

Starting from 2.10.0-beta Tarantool can natively run on M1 chips. So far this is preliminary support: something may crash or run unstable. We have resolved almost all bugs we knew about, with a few minor ones left. For example, there are some issues with the JIT compiler. But this didn't prevent the team product manager from installing Tarantool on his new MacBook Air and running it every day.

Later we will resolve other known bugs, and those that the developers will report to us. If you have a new M1 Mac, try the latest version of Tarantool. If something crashes, please write the bug report, and we'll help.

Try Tarantool cluster at https://try.tarantool.io. Download Tarantool at the official website, and get help in the Telegram chat.

How to write three times fewer lines of code when doing load testing

tarantool — Wed, 17 Nov 2021 13:34:52 +0000

The key concept of load testing is automating everything that can be automated. Take a tool, write a configuration and a test scenario, then run a simulation of an actual load. The less code the better.

Automating load testing is not as difficult as it may seem at first glance. All it takes is the right tool.

In this article, I will show how I reduced the code of my testing utility threefold without any performance losses. I'm also going to explain why Yandex.Tank combined with Pandora didn't work for me.

What is load testing

My name is Sergey, and I'm a developer on the architecture team at Tarantool. Tarantool is an in-memory computing platform designed to handle exceptionally high loads, up to hundreds of thousands of RPS. That makes load testing essential for us, so I perform it every day. I am sure that almost everybody knows precisely why load testing matters, but let's review the basics just in case. The results of load testing show how your system behaves in different scenarios:

What parts of the system are idle in what cases?
What is the approximate request response time?
At what load does the system become unstable?
What part of the system causes malfunctions?
What part of it puts a limit on the overall performance?

Why we need special tools for load testing

When developing an application on Tarantool, we often have to test the performance of a stored procedure. The application accesses the procedure over the iproto binary protocol. Not every language can be used to test over iproto. There are Tarantool connectors for a number of languages, and you have to write your tests in one of them.

Most testing tools only support HTTP, which is not an option for us. Sure, we could add some controls and make the best of it, but that wouldn't help the end user. Since we pass the stored procedures to the client side, testing via HTTP is unreliable.

Common load testing tools

At first, we considered a popular tool called JMeter. However, we were not impressed by its performance. It's written in Java and therefore is memory-hungry and slow. Besides, we used it to test via HTTP, which meant indirect testing performed through special controls. Then we tried writing custom Go utilities for each project, which was a road to nowhere, it's no use writing code over and over when it's thrown away right after the testing is complete. That's no systematic approach. Let me reiterate that we want to automate as much as we can in load testing. That's how we got to Yandex.Tank and Pandora, as this combination seemed like a perfect tool satisfying all the requirements:

It can easily be adapted to any project.
It's fast, since Pandora is written in Go.
Our team has a lot of experience with Go, so working out the scenarios won't be a problem.

But there were also disadvantages.

Why we stopped using Yandex.Tank

Our time with Yandex.Tank was brief, and here are a few key reasons we gave up on it.

Lots of utility code. The Pandora wrapper that allows you to work with Tarantool contains ~150 lines of code, most of which don't bear any testing logic.

Constant source code recompilation. We encountered this problem when we had to keep loading the system while simultaneously generating various amounts of data. We couldn't find a convenient external way to control data generation parameters, and pre-generation wasn't an option. So we changed the data and compiled a new source every time. Such manipulations could spawn up to 20 loader binaries per test scenario.

Scarce data when using standalone Pandora. Yandex.Tank is a wrapper that provides a pretty neat metrics visualization. Pandora is the engine that generates the load. Effectively, we were using two different tools, which was not always convenient (thankfully, we have Docker).

Configuration file options are not very intuitive. JSON and YAML configurations are a sensitive topic per se. But it becomes really unpleasant when it isn't clear how an option works depending on the values. For us, startup was such an option. It produced the same results on entirely different values, making it difficult to assess the system's actual performance.

All that created the following situation in one of our projects:

huge piles of source code
unclear metrics
overly complicated configuration.

What led us to k6

k6 is a load testing tool written in Go, just like Pandora. Therefore, performance is nothing to worry about. What's appealing about k6 is its modularity, which helps avoid constant source code recompilation. With k6, we write modules to access the Tarantool interface and do other things like generating data. Since modules are independent of one another, it's not necessary to recompile every one of them. Instead, you can customize data generation parameters within a scenario written in... JavaScript! Yep, that's right. No more JSON or YAML configurations, k6 testing scenarios are code! The scenario can be divided into stages, each of which models a different type of load. If you alter the scenario, there's no need to recompile the k6 binary, as they don't depend on one another. That makes two fully independent components written in programming languages. You can finally forget about configurations and just write your code.

Our application

This testing application in Lua stores information about car models. I use this application to test database writes and reads. The application has two main components, API and Storage. The API component gives the user HTTP controls for reading and writing, while Storage is responsible for the application's interaction with the database. Here is the interaction scenario: the user sends a request, and the controls call the database functions necessary to process that request. Check out the application on GitHub.

Getting k6 to work with the application

To create a k6 Tarantool interaction module, we first need to write a Go module using the xk6 framework. This framework provides tools for writing custom k6 modules. First, register the module so that k6 can work with it. We also need to define a new type and its receiver functions, that is, methods to call from the JavaScript scenario:

package tarantool

import (
    "github.com/tarantool/go-tarantool"
    "go.k6.io/k6/js/modules"
)

func init() {
    modules.Register("k6/x/tarantool", new(Tarantool))
}

// Tarantool is the k6 Tarantool extension
type Tarantool struct{}

We can already use this module, but it doesn't do much yet. Let's program it to connect to a Tarantool instance and to invoke the Call function provided by the Go connector:

// Connect creates a new Tarantool connection
func (Tarantool) Connect(addr string, opts tarantool.Opts) (*tarantool.Connection, error) {
    if addr == "" {
        addr = "localhost:3301"
    }
    conn, err := tarantool.Connect(addr, opts)
    if err != nil {
        return nil, err
    }
    return conn, nil
}

// Call invokes a registered Tarantool function
func (Tarantool) Call(conn *tarantool.Connection, fnName string, args interface{}) (*tarantool.Response, error) {
    resp, err := conn.Call(fnName, args)
    if err != nil {
        return nil, err
    }
    return resp, err
}

The full code of the module can be found in this GitHub repo.

This code is already far more compact than what Pandora requires for working with Tarantool. The Pandora version had about 150 lines of code, and now we have 30. However, we haven't implemented any logic yet. Spoiler alert: we're going to end up with ~50 lines of code. k6 will take care of everything else.

Interacting with the module from a scenario

First, we'll import that custom module into our scenario:

import tarantool from "k6/x/tarantool";

Now let's create a connection:

const conn = tarantool.connect("localhost:3301");

connect is the receiver function we've declared in our module. If you want to pass an object that stores connection options, provide it as a second parameter in a simple JSON object. All that's left is to declare testing stages and launch the test:

export const setup = () => {
  tarantool.insert(conn, "cars", [1, "cadillac"]);
};

export default () => {
  console.log(tarantool.call(conn, "box.space.cars:select", [1]));
};

export const teardown = () => {
  tarantool.delete(conn, "cars", "pk", [1]);
};

There are three testing stages in this example:

setup is performed before the test. Here is where you prepare the data or display an information message.
default, which is the main test scenario.
teardown is performed after the test is completed. Here you can erase the test data or display another information message.

After the test is launched and finished, you will see an output like this:

Here is what you can learn from this output:

What scenario is running.
Whether the data is being written to the console or aggregated via InfluxDB.
Scenario parameters.
Scenario console.log output.
Execution process.
Metrics.

The most interesting metrics here are iteration_duration, representing latency, and iterations, representing the total number of iterations performed and their average number per second — the desired RPS.

How about something more substantial?

Let's create a test bench consisting of three nodes, with two of them combined in a cluster. The third node will host k6's load system and a Docker container with Influx and Grafana. This is where we'll send the metrics and visualize them.

Each cluster node will look like this:

We don't place the storage and its replicas in the same nodes: If the first storage is in the first node, its replica is in the second node. Our spaceв (basically a table in Tarantool) will have three fields: id, bucket_id, and model. We'll create a primary key based on id and another index based on bucket_id:


local car = box.schema.space.create(
        'car',
        {
            format = {
                {'car_id', 'string'},
                {'bucket_id', 'unsigned'},
                {'model', 'string'},
            },
            if_not_exists = true,
        }
    )

    car:create_index('pk', {
        parts = {'car_id'},
        if_not_exists = true,
    })

    car:create_index('bucket_id', {
        parts = {'bucket_id'},
        unique = false,
        if_not_exists = true,
    })

Let's test the creation of car objects. To do so, we're going to write a k6 module for generating data. Earlier, I mentioned 30 lines of utility code, and here are the remaining 20 lines of test logic:


var bufferData = make(chan map[string]interface{}, 10000)

func (Datagen) GetData() map[string]interface{} {
    return <-bufferData
}

func (Datagen) GenerateData() {
    go func() {
        for {
            data := generateData()
            bufferData <- data
        }
    }()
}

func generateData() map[string]interface{} {
    data := map[string]interface{}{
        "car_id": uniuri.NewLen(5),
        "model":  uniuri.NewLen(5),
    }

    return data
}

I left out the part with the initialization function and the definition of the type used to call other functions. Now let's create receiver functions that we'll invoke from our JavaScript scenario. Interestingly, we can work with channels without losing any data. Suppose you have a function that writes to bufferData and another that reads from that channel. If you invoke the second function in the read scenario, no data will be lost.

generateData is a function that generates the car model and its id. This is an internal function not extended to our module. generateData launches a goroutine so that we always have enough data generated for insertion. The test scenario for this bench looks like this:


import datagen from "k6/x/datagen";
import tarantool from "k6/x/tarantool";

const conn1 = tarantool.connect("172.19.0.2:3301");
const conn2 = tarantool.connect("172.19.0.3:3301");

const baseScenario = {
  executor: "constant-arrival-rate",
  rate: 10000,
  timeUnit: "1s",
  duration: "1m",
  preAllocatedVUs: 100,
  maxVUs: 100,
};

export let options = {
  scenarios: {
    conn1test: Object.assign({ exec: "conn1test" }, baseScenario),
    conn2test: Object.assign({ exec: "conn2test" }, baseScenario),
  },
};

export const setup = () => {
  console.log("Run data generation in the background");
  datagen.generateData();
};

export const conn1test = () => {
  tarantool.call(conn1, "api_car_add", [datagen.getData()]);
};

export const conn2test = () => {
  tarantool.call(conn2, "api_car_add", [datagen.getData()]);
};

export const teardown = () => {
  console.log("Testing complete");
};

It got a little bigger. There's a new options variable that allows us to configure testing behavior. I created two scenarios and a dedicated function for each one. As the cluster consists of two nodes, we need to test simultaneous connection to these nodes. If you do that with a single function, which was the default earlier, you can't expect the cluster to be fully loaded. Every time unit, you send a request to the first router while the second one is idle, then you send a request to the second one while the first one is idle. Thus, performance goes down. However, it can be prevented, and we'll get back to it soon.

Now let's take a look at our testing scenarios. Under executor, we specify what type of testing we want to launch. If this value is set to constant-arrival-rate, the scenario will simulate a constant load. Suppose we want to produce 10,000 RPS for 100 virtual users during one minute. Let's use the database, not the console, to output the results, so that the information is then displayed on the dashboard:

With the objective of 10,000 RPS, we got only 8,600 RPS, which is not so bad. There was likely just not enough computing power on the client machine where the loader was located. I performed this test on my MacBook Pro (Mid 2020). Here is the data on latency and virtual users:

What about flexibility?

As far as flexibility is concerned, everything is perfect. Scenarios can be modified to check metrics, collect metrics, and more. In addition, you can optimize scenarios in one of the ways described below:

n connections — n scenarios

It is the basic scenario that we've discussed above:

const conn1 = tarantool.connect("172.19.0.2:3301");
const conn2 = tarantool.connect("172.19.0.3:3301");

const baseScenario = {
  executor: "constant-arrival-rate",
  rate: 10000,
  timeUnit: "1s",
  duration: "1m",
  preAllocatedVUs: 100,
  maxVUs: 100,
};

export let options = {
  scenarios: {
    conn1test: Object.assign({ exec: "conn1test" }, baseScenario),
    conn2test: Object.assign({ exec: "conn2test" }, baseScenario),
  },
};

n connections — 1 scenario

In this scenario, the connection to be tested is selected randomly at each iteration. The test unit is 1 second, which means that once per second, we randomly choose one connection among those declared:


const conn1 = tarantool.connect("172.19.0.2:3301");
const conn2 = tarantool.connect("172.19.0.3:3301");

const conns = [conn1, conn2];

const getRandomConn = () => conns[Math.floor(Math.random() * conns.length)];

export let options = {
  scenarios: {
    conntest: {
      executor: "constant-arrival-rate",
      rate: 10000,
      timeUnit: "1s",
      duration: "1m",
      preAllocatedVUs: 100,
      maxVUs: 100,
    },
  },
};

This scenario can be reduced to a single connection. To do so, we need to set up a TCP balancer (nginx, envoy, haproxy), but that's a story for another day.

n connections — n scenarios + restrictions and checks

You can use restrictions to control the obtained metrics. If the 95 percentile latency is greater than 100 ms, the test will be considered unsuccessful. You can set several restrictions for one parameter. You can also add checks, for example, to see what percentage of requests reached the server. The percentage rate is expressed as a number between 0 and 1:


const conn1 = tarantool.connect("172.19.0.2:3301");
const conn2 = tarantool.connect("172.19.0.3:3301");

const baseScenario = {
  executor: "constant-arrival-rate",
  rate: 10000,
  timeUnit: "1s",
  duration: "10s",
  preAllocatedVUs: 100,
  maxVUs: 100,
};

export let options = {
  scenarios: {
    conn1test: Object.assign({ exec: "conn1test" }, baseScenario),
    conn2test: Object.assign({ exec: "conn2test" }, baseScenario),
  },
  thresholds: {
    iteration_duration: ["p(95) < 100", "p(90) < 75"],
    checks: ["rate = 1"],
  },
};

n connections — n scenarios + restrictions and checks + sequential launch

The sequential launch scenario is the most sophisticated among those described in this article. Suppose you want to check n stored procedures without loading the system at that exact time. In this case, you might want to specify the time to start the tests, and you can do so in the second scenario. Keep in mind, however, that your first scenario may still be running at that moment. You can set the time limit for its execution via the gracefulStop parameter. If you set gracefulStop to 0 seconds, the first scenario will definitely be stopped by the time the second one starts:


const conn1 = tarantool.connect("172.19.0.2:3301");
const conn2 = tarantool.connect("172.19.0.3:3301");

const baseScenario = {
  executor: "constant-arrival-rate",
  rate: 10000,
  timeUnit: "1s",
  duration: "10s",
  gracefulStop: "0s",
  preAllocatedVUs: 100,
  maxVUs: 100,
};

export let options = {
  scenarios: {
    conn1test: Object.assign({ exec: "conn1test" }, baseScenario),
    conn2test: Object.assign({ exec: "conn2test", startTime: "10s" }, baseScenario),
  },
  thresholds: {
    iteration_duration: ["p(95) < 100", "p(90) < 75"],
    checks: ["rate = 1"],
  },
};

Performance in comparison to Yandex.Tank + Pandora

We compared both tools on the application described above. Yandex.Tank loaded the router CPU by 53% and the storage CPU by 32%, yielding 9,616 RPS. As for k6, it loaded the router CPU by 54% and the storage CPU by 40%, producing 9,854 RPS. These are the average data from 10 test runs.

Why is that so? Both Pandora and k6 are written in Go. However, despite these similar fundamentals, k6 allows you to test applications in a more programming-like manner.

Conclusion

k6 is a simple tool. Once you've learned how to use it, you can reconfigure it for any project and spend fewer resources. Start by creating a core module, and then attach logic to it. There's no need to rewrite tests from scratch because you can use modules from other projects.

k6 is also a lean tool for load testing. My test logic with the wrapper fit within just 50 lines of code. You can write custom modules to suit your business logic, scenarios, and client requirements.

k6 is about programming, not configuration files. You can try k6 out here and play around with the sample application here.

Get Tarantool on our website and feel free to ask questions in our Telegram chat.

How to Start a Digital Business on an Old IT Infrastructure

tarantool — Mon, 18 Oct 2021 12:45:51 +0000

Most enterprises base their IT infrastructure on systems developed 10 or even 15 years ago. Ilya Letunov, head of the Tarantool platform and VK Cloud Solutions, explains how it affects the technological development of the business and how to launch new digital services using an outdated infrastructure.

For a long time, the market had been developing in highly comfortable conditions. There was a practical product, a monetization model, and a client that we saw and understood. But the year 2021 undermined the existing models of consumer behavior. Now customers choose digital services and expect the best possible experience with them.

The IT infrastructure of many companies was not ready for the traffic surge that followed the transition to remote work. Businesses need to process hundreds of thousands of orders at once, ensure fast and accurate logistics and delivery, store personalized customer profiles, and rely on the omnichannel strategy. Previously, all this gave the business a competitive edge, but now it is a necessity.

Another change I see is the growth of business data. According to IDC estimates, the total global volume of data generated by companies in 2020 was 59 zettabytes. By 2025, it will be as much as 173 zettabytes, which is three times more. Even the IT systems of the market leaders are no longer able to cope with such loads.

The base layer of corporate IT systems is really outdated and not designed for modern workloads. The number of requests to IT systems is growing, but the speed of their processing is not.

Most top managers see the negative effect right away, they don't even have to enter the technical maze.

Due to a low data transfer rate, an online service customer can put an out-of-stock item in the shopping cart. The result is disappointing: no purchase, no consumer satisfaction, and no company profit. On the other hand, rapid information transfer between the user-side service and the infrastructure improves customer experience, increases loyalty, and, most importantly, directly affects sales growth.

There is a contradiction here. Outdated and legacy IT systems prevent the development of digital services. But they cannot be completely thrown away. In a large business, this can lead to a restructuring of such a scale that it would be easier to create a new company from scratch.

What do we do with this?

A Three-Speed IT Infrastructure

You can speed up your legacy systems without getting rid of them, using the concept of three-speed IT. This method is gaining popularity around the world and allows accelerating digital services without global IT restructuring. The acceleration is achieved by developing an intermediate layer of IT services. It connects the top level of user and business applications with the company's critical root systems (OSS, BSS, billing, BI, ERP, CRM).

Usually, those critical systems were developed 10-15 years ago, and their code is outdated. Making changes to them requires a lot of effort, time, and money.

The top IT layer includes mobile applications, personal accounts, online stores, call center integration, real-time analytics, etc. These are all the trendy IT solutions that we have witnessed developing in recent years. The update speed requirements at the top IT level are measured in days and if a new service is launched, even in hours.

The intermediate layer creates a balance between the slow legacy systems and the speed-demanding user services. Hence the name вЂ” three-speed IT.

Most digital transformation projects are stalled by fear of tampering with what already works. Top managers have a point here: with the current speed of change and the fiercest competition, business processes cannot be stopped even for a minute. They must be improved and accelerated on the fly. Additionally, there is always the risk of implementing an expensive ITВ solution that will be quickly replaced by a new one, X times more efficient.

The three IT layers concept removes the risks and offers evolutionary development instead of erratic veering between new solutions. Thus, it gives more room to maneuver and experiment.

I will now explain how three-speed IT works. My first example will be the retail business. Retail is where major players have been actively launching new sales channels and testing digital business models for several years.

Example 1. Retail

In terms of IT infrastructure, many large retailers have been focused on offline customers for a long time. IT had only a supporting role. But as soon as online commerce and express delivery services began to grow rapidly, retailers had to master new sales channels and monetization models. This determined the vector of IT development in traditional offline retail.

In the late summer of 2020, Magnit decided to launch its express delivery of groceries through the Delivery Club service. It means "delivery to your door in 30 minutes or less" for most grocery store items. The difficulty was that Magnit's corporate IT systems, containing the data on what's in stock, technically had no integration point with the express delivery service.

The company faced the choice: either completely rebuild its IT infrastructure towards the new direction or create an intermediate layer connecting the upper, user-facing IT level with the lower, business-critical one.

As a result, the company created an integration layer. It helped accelerate and organize two streams of information вЂ” one regarding products from the central storage and the other about the stock balances of each store.

With traditional methods, integration usually takes up to six months, but the new IT tool reduced the time approximately threefold. The intermediate IT layer allowed launching a new business model while eliminating the costs of rebuilding, adapting, and configuring the infrastructure.

Example 2. Banks

The banking industry is highly developed technologically, especially in Russia. Nevertheless, banks have faced and continue to deal with a serious challenge вЂ” the many times increased burden on their IT systems. The growing popularity of mobile applications and remote banking services contributes to the situation.

An increase in customer service requests prompted Gazprombank to look for an accelerator for its IT systems. The bank implemented an intermediate IT layer, which connected critical systems with business applications operated by contact center employees, in-office workers, and remote staff. The new solution made it possible to load client data into RAM and have them instantly displayed in business applications. As a result, the rate of data transfer from the lower layer to the functional services increased by 50 times at once.

Example 3. Telecommunications

Telecommunications is a spectacular example of a dynamic competitive environment. At MegaFon, with 80 million active users, new projects required improving the digital ecosystem and creating a technological base. You can imagine how large the IT landscape of a major telecom operator is.

Therefore, MegaFon took the evolutionary path of IT infrastructure development, where new systems did not replace but supplemented the existing ones. Such a move alleviated the burden on the underlying platforms, increased the availability of services, and reduced time to market.

After transferring the load to a fast data storage "add-on," MegaFon launched more than 25 services in just a few months. Those services now work for 80 million subscribers in 10 time zones without delays or downtime.

The Future of the Three-Speed Model

I believe that the speed of change and adaptation to new conditions are crucial factors for a modern enterprise to achieve success. The technical limitations of legacy IT systems devalue promising ideas and business models. At the same time, the costs of rebuilding the IT infrastructure and the losses due to interrupted business processes are many times greater than the potential benefits of a full revamp.

The development of the IT infrastructure should be approached carefully, iteratively. In this case, it will bring additional value to the company instead of becoming an item of expenditure. The concept of three-speed IT creates an environment for a smooth evolutionary development of the infrastructure. This is probably one of the most effective ways for a business to go digital today.

Links

Get Tarantool on our website
Get help in our telegram channel

Advanced MessagePack capabilities

tarantool — Wed, 13 Oct 2021 16:36:52 +0000

Photo by Peretz Partensky / CC BY-SA 2.0

MessagePack is a binary format for data serialization. It is positioned by the authors as a more efficient alternative to JSON. Due to its speed and compactness, it's often used as a format for data exchange in high-performance systems. The other reason this format became popular is that it's very easy to implement. Your favorite programming language most likely already has several libraries designed to work with it.

In this article, I'm not going to tell you how MessagePack works or compare it to its counterparts: there are plenty of materials on this topic on the Internet. What's really missing is information about MessagePack's extended type system. I'll try to explain and show you by examples what it is and how to make serialization even more efficient using extension types.

The Extension type

The MessagePack specification defines 9 basic types:

Nil
Boolean
Integer
Float
String
Binary
Array
Map
Extension.

The last type, Extension, is a container designed for storing extension types. Let's look closely at how it works. It will help us with writing our own types. Here is how the container is structured:

Header is the container's header (1 to 5 bytes). It contains the payload size, i.e., the length of the Data field. To learn more about how the header is formed, take a look at the specification.

Type is the ID of the stored type, an 8-bit signed integer. Negative values are reserved for official types. User types' IDs can take any value in the range from 0 to 127.

Data is an arbitrary byte string up to 4 GiB long, which contains the actual data. The format of official types is described in the specification, while the format of user types may depend entirely on the developer's imagination.

The list of official types currently includes only Timestamp with the ID of -1. Occasionally there are proposals to add new types (such as UUIDs, multidimensional arrays, or geo-coordinates), but since the discussions are not very active, I would not expect anything new to be added in the near future.

Hello, World!

Photo by Brett Ohland / CC BY-NC-SA 2.0

That's enough theory, let's start coding! For these examples, we'll use the msgpack.php MessagePack library since it provides a convenient API to handle extension types. I hope you'll find these code examples easy to understand even if you use other libraries.

Since I mentioned UUID, let's implement support for this data type as an example. To do so, we'll need to write an extension — a class to serialize and deserialize UUID values. We will use the symfony/uid library to make handling such values easier.

This example can be adapted for any UUID library, be it the popular ramsey/uuid, PECL uuid module, or a user implementation.

Let's name our class UuidExtension. The class must implement the Extension interface:

use MessagePack\BufferUnpacker;
use MessagePack\Extension;
use MessagePack\Packer;
use Symfony\Component\Uid\Uuid;

final class UuidExtension implements Extension
{
    public function getType(): int
    {
        // TODO
    }

    public function pack(Packer $packer, mixed $value): ?string
    {
        // TODO
    }

    public function unpackExt(BufferUnpacker $unpacker, int $extLength): Uuid
    {
        // TODO
    }
}

We determined earlier what the type (ID) of the extension is, so we can easily implement the getType() method. In the simplest case, this method could return a fixed constant, globally defined for the whole project. However, to make the class more versatile, we'll let it define the type when initializing the extension. Let's add a constructor with one integer argument, $type:

/** @readonly */
private int $type;

public function __construct(int $type)
{
    if ($type < 0 || $type > 127) {
        throw new \OutOfRangeException(
            "Extension type is expected to be between 0 and 127, $type given"
        );
    }

    $this->type = $type;
}

public function getType(): int
{
    return $this->type;
}

Now let's implement the pack() method. From the method's signature, we can see that it takes two parameters: a Packer class instance and a $value of any type. The method must return either a serialized value (wrapped into the Extension container) or null if the extension does not support the value type:

public function pack(Packer $packer, mixed $value): ?string
{
    if (!$value instanceof Uuid) {
        return null;
    }

    return $packer->packExt($this->type, $value->toBinary());
}

The reverse operation isn't much harder to implement. The unpackExt() method takes a BufferUnpacker instance and the length of the serialized data (the size of the Data field from the schema above). Since we've saved the binary representation of a UUID object in this field, all we need to do is read this data and build a Uuid object:

public function unpackExt(BufferUnpacker $unpacker, int $extLength): Uuid
{
    return Uuid::fromString($unpacker->read($extLength));
}

Our extension is ready! The last step is to register a class object with a specific ID. Let the ID be 0:

$uuidExt = new UuidExtension(0);
$packer = $packer->extendWith($uuidExt);
$unpacker = $unpacker->extendWith($uuidExt);

Let's make sure everything works correctly:

$uuid = new Uuid('7e3b84a4-0819-473a-9625-5d57ad1c9604');

$packed = $packer->pack($uuid);
$unpacked = $unpacker->reset($packed)->unpack();

assert($uuid->equals($unpacked));

That was an example of a simple UUID extension. Similarly, you can add support for any other type used in your application: DateTime, Decimal, Money. Or you can write a versatile extension that allows serializing any object (as it was done in KPHP).

However, this is not the only use for extensions. I'll now show you some interesting examples that demonstrate other advantages of using extension types.

"Lorem ipsum" or compressing the incompressible

Photo by dog97209 / CC BY-NC-ND 2.0

If you've ever inquired about MessagePack before, you probably know the phrase from its official website, msgpack.org: "It's like JSON, but fast and small."

In fact, if you compare how much space the same data occupies in JSON and MessagePack, you'll see why the latter is a much more compact format. For example, the number 100 takes 3 bytes in JSON and only 1 in MessagePack. The difference becomes more significant as the number's order of magnitude grows. For the maximum value of int64 (9223372036854775807), the size of the stored data differs by as much as 10 bytes (19 against 9)!

The same is true for boolean values — 4 or 5 bytes in JSON against 1 byte in MessagePack. It is also true for arrays because many syntactic symbols, such as commas separating the elements, semicolons separating the key-value pairs, and brackets indicating the array boundaries, don't exist in binary format. Obviously, the larger the array is, the more syntactic litter accumulates along with the payload.

With string values, however, things are not so straightforward. If your strings do not consist entirely of quotes, line feeds, and other special characters that require escaping, then you won't notice a big difference between their sizes in JSON and in MessagePack. For example, "foobar" has a length of 8 bytes in JSON and 7 in MessagePack. Note that the above only applies to UTF-8 strings. For binary strings, JSON's disadvantage against MessagePack is obvious.

Knowing this peculiarity of MessagePack, you can have a good laugh reading articles that compare the two formats in terms of data compression efficiency while using mainly string data for the tests. Apparently, any conclusions based on the results of such tests would make no practical sense. So take those articles skeptically and run comparative tests on your own data.

At some point, there were discussions about whether to add string compression (individual or in frames) to the specification to make string serialization more compact. However, the idea was rejected, and the implementation of this feature was left to users. So let's try it.

Let's create an extension that will compress long strings. We will use whatever compression tool is at hand, for example, zlib.

Choose the data compression algorithm based on the specifics of your data. For example, if you are working with lots of short strings, take a look at SMAZ.

Let's start with the constructor for our new class, TextExtension. The first argument is the extension ID, and as a second optional argument, we'll add minimum string length. Strings shorter than this value will be serialized in a standard way, without compression. In this way, we will avoid cases where the compressed string ends up longer than the initial one:

final class TextExtension implements Extension
{
    /** @readonly */
    private int $type;

    /** @var positive-int */
    private int $minLength;

    public function __construct(int $type, int $minLength = 100)
    {
        ...

        $this->type = $type;
        $this->minLength = $minLength;
    }

    ...
}

To implement the pack() method, we might write something like this:

public function pack(Packer $packer, mixed $value): ?string
{
    if (!is_string($value)) {
        return null;
    }

    if (strlen($value) < $this->minLength) {
        return $packer->packStr($value);
    }

    // compress and pack
    ...
}

However, this wouldn't work. String is one of the basic types, so the packer will serialize it before our extension is called. This is done in the msgpack.php library for performance reasons. Otherwise, before serializing each value, the packer would need to scan the available extensions, considerably slowing down the process.

Therefore, we need to tell the packer not to serialize certain strings as, you know, strings but to use an extension. As you might guess from the UUID example, it can be done via a ValueObject. Let's call it Text, similar to the extension class:

/**
 * @psalm-immutable
 */
final class Text
{
    public function __construct(
        public string $str
    ) {}

    public function __toString(): string
    {
        return $this->str;
    }
}

So instead of

$packed = $packer->pack('a very long string');

we'll use a Text object to mark long strings:

$packed = $packer->pack(new Text('a very long string'));

Let's update the pack() method:

public function pack(Packer $packer, mixed $value): ?string
{
    if (!$value instanceof Text) {
        return null;
    }

    $length = strlen($value->str);
    if ($length < $this->minLength) {
        return $packer->packStr($value->str);
    }

    // compress and pack
    ...
}

Now we just need to compress the string and put the result in an Extension. Note that the minimum length limit does not guarantee that the string will take less space after compression. For this reason, you might want to compare the lengths of the compressed string and the original and choose whichever is more compact:

$context = deflate_init(ZLIB_ENCODING_GZIP);
$compressed = deflate_add($context, $value->str, ZLIB_FINISH);

return isset($compressed[$length - 1])
    ? $packer->packStr($value->str)
    : $packer->packExt($this->type, $compressed);

Deserialization:

public function unpackExt(BufferUnpacker $unpacker, int $extLength): string
{
    $compressed = $unpacker->read($extLength);
    $context = inflate_init(ZLIB_ENCODING_GZIP);

    return inflate_add($context, $compressed, ZLIB_FINISH);
}

Let's see the result:

$longString = <<<STR
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed 
do eiusmod tempor incididunt ut labore et dolore magna aliqua. 
Ut enim ad minim veniam, quis nostrud exercitation ullamco 
laboris nisi ut aliquip ex ea commodo consequat. Duis aute 
irure dolor in reprehenderit in voluptate velit esse cillum 
dolore eu fugiat nulla pariatur. Excepteur sint occaecat 
cupidatat non proident, sunt in culpa qui officia deserunt 
mollit anim id est laborum.
STR;

$packedString = $packer->pack($longString);
// 448 bytes

$packedCompressedString = $packer->pack(new Text($longString));
// 291 bytes

In this example, we saved 157 bytes, or 35% of what would be the standard serialization result, on just one string!

From "schema-less" to "schema-mixed"

Photo by Adventures with E&L / CC BY-NC-ND 2.0

Compressing long strings is not the only way to save space. MessagePack is a schemaless, or schema-on-read, format that has its advantages and disadvantages. One of the disadvantages in comparison with schema-full (schema-on-write) formats is highly ineffective serialization of repeated data structures. An example of such data is a selection from a database, where all elements of the resulting array have the same structure:

$userProfiles = [
    [
        'id' => 1,
        'first_name' => 'First name 1',
        'last_name' => 'Last name 1',
    ],
    [
        'id' => 2,
        'first_name' => 'First name 2',
        'last_name' => 'Last name 2',
    ],
    ...
    [
        'id' => 100,
        'first_name' => 'First name 100',
        'last_name' => 'Last name 100',
    ],
];

If you serialize this array with MessagePack, the repeated keys of each element in the array will take a substantial part of the total data size. But what if we could save the keys of such structured arrays just once? It would significantly cut down the size and also speed up serialization since the packer would have fewer operations to perform.

Like before, we are going to use extension types for that. Our type will be a value object wrapped around an arbitrary structured array:

/**
 * @psalm-immutable
 */
final class StructList
{
    public function __construct(
        public array $list,
    ) {}
}

If your project includes a library for database handling, there is probably a special class in that library to store table selection results. You can use this class as a type instead of/along with StructList.

Here is how we are going to serialize such arrays. First, we'll check the array size. Of course, if the array is empty or has only one element, there is no reason to store keys separately from values. We'll serialize arrays like these in a standard way.

In other cases, we'll first save a list of keys and then a list of values. We won't be storing an associative array list, which is the standard MessagePack option. Instead, we'll write data in a more compact form:

Implementation:

final class StructListExtension implements Extension
{
    ...

    public function pack(Packer $packer, mixed $value): ?string
    {
        if (!$value instanceof StructList) {
            return null;
        }

        $size = count($value->list);
        if ($size < 2) {
            return $packer->packArray($value->list);
        }

        $keys = array_keys(reset($value->list));

        $values = '';
        foreach ($value->list as $item) {
            foreach ($keys as $key) {
                $values .= $packer->pack($item[$key]);
            }
        }

        return $packer->packExt($this->type,
            $packer->packArray($keys).
            $packer->packArrayHeader($size).
            $values
        );
    }

    ...
}

To deserialize, we need to unpack the keys array and then use it to restore the initial array:

public function unpackExt(BufferUnpacker $unpacker, int $extLength): array
{
    $keys = $unpacker->unpackArray();
    $size = $unpacker->unpackArrayHeader();

    $list = [];
    for ($i = 0; $i < $size; ++$i) {
        foreach ($keys as $key) {
            $list[$i][$key] = $unpacker->unpack();
        }
    }

    return $list;
}

That's it! Now, if we serialize $profiles from the example above as a normal array and as a structured StructList, we'll see a great difference in size — the latter will be 47% more compact.

$packedList = $packer->pack($profiles);
// 5287 bytes

$packedStructList = $packer->pack(new StructList($profiles));
// 2816 bytes

We could go further and create a specialized Profiles type to store information about the array structure in the extension code. This way, we wouldn't need to save the keys array. However, in this case, we would lose in versatility.

Conclusion

We've taken a look at just a few examples of using extension types in MessagePack. To see more examples, check the msgpack.php library. For the implementations of all extension types supported by the Tarantool protocol, see the tarantool/client library.

I hope this article gave you a sense of what extension types are and how they can be useful. If you're already using MessagePack but haven't known about the feature, this information might inspire you to reconsider your current methods of working with the format and start using custom types.

If you're just wondering which serialization format to choose for your next project, the article might help you make a reasonable choice, adding a point in favor of MessagePack :)

Links

Get Tarantool on our website
Get help in our telegram channel

Forem: tarantool

Raft (not)almighty: how to make it more robust

Pre-Vote

Raft's solution

Our solution

Split-Vote detection

Our solution

CheckQuorum

Split-Brain detection

PROMOTE entry

Lessons learned

How we wrote Tarantool Kubernetes Operator

A little bit about Kubernetes operators

How does an operator work?

Divide and conquer

Rolling update

Solving network issues

Naming the Statefulset when PVC changes

Testing the Operator

Conclusion

How we compress data in large projects

How to enable data compression

What can be achieved by using data compression?

Compression in a cluster

Conclusion

Alternative to MapReduce for search in distributed databases

General description of the application

Search by id

Search by name

Widening the search

Index layer structure

Read

Indexing arbitrary data

Write

Conclusion

Scaling clusters without any hassle

Challenge

Cluster architecture

Cartridge

Tarantool

Replication

Sharding

Summary

Cartridge instance roles

API

Predefined roles

Role dependencies

Entry point

Cluster scaling with roles

Cluster configuration

Failover

Tools

Cartridge in production

Conclusion

What's next?

Developing an authentication system in Java+Tarantool

Fundamentals of authentication systems

What are Tarantool Cartridge and Cartridge Java?

Why Java?

Why authentication process?

Tarantool Cartridge application

Creating and configuring an application

Configuring roles

Defining a cluster topology and launching the cluster

Building a data model

Performing migrations

Creating stored procedures

What was done

Java application

Setting up the connector

Connecting to the cluster

Creating an API and a user data model

Creating services and controllers

What was done

Checking if the service works

Conclusion

Links

Distributed storage in 30 minutes

Contents

Cartridge framework