Forem: Przemyslaw Hugh Kaznowski

Clustered SurrealDB for 1.0.0-beta9

Przemyslaw Hugh Kaznowski — Wed, 26 Apr 2023 12:36:38 +0000

In this post, I will show you how to set up a distributed SurrealDB cluster that shares a distributed TiKV cluster.
This architecture allows you to scale your operations to improve writes and reads and seamlessly continue operations during failures.

Introduction and architecture overview

Users of SurrealDB can pick which Key-Value storage engines they want to use.
That means that for single deployment, you can use RocksDB or in-memory storage; for distributed storage, you can use TiKV and FoundationDB.

We will deploy a cluster of TiKV that includes 3 TiKV instances (the KV engine) and 3 PD instances (placement driver, a resource tracking service).
In addition to the above configuration, we will deploy three nodes of SurrealDB that will point to their respective KV engines.
Typically you would want the SurrealDB instances not tied to individual TiKV instances, but that would require a load balancer - something beyond the scope of this article.

Setting up the environment

Because we need access to 6 machines, we will simplify this setup using LXC - a lightweight Linux container system that makes nodes seem like fully-fledged computers.

An important note: LXC does not play nice with docker.
There are ways around that, but I removed docker from my host machine for this example.
It's a VM; usually, you wouldn't host this environment this way anyway.

Let's start by running and configuring our first LXC container for usage.

hugh@hugh-VirtualBox:~$ lxc launch ubuntu: lxc-node-tikv-1

Creating lxc-node-tikv-1
Starting lxc-node-tikv-1

hugh@hugh-VirtualBox:~$ lxc exec lxc-node-tikv-1 bash
root@lxc-node-tikv-1:~# apt install openssh-server
...
root@lxc-node-tikv-1:~# curl --proto '=https' --tlsv1.2 -sSf https://tiup-mirrors.pingcap.com/install.sh | sh

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 7088k  100 7088k    0     0  2571k      0  0:00:02  0:00:02 --:--:-- 2570k
WARN: adding root certificate via internet: https://tiup-mirrors.pingcap.com/root.json
You can revoke this by remove /root/.tiup/bin/7b8e153f2e2d0928.root.json
Successfully set mirror to https://tiup-mirrors.pingcap.com
Detected shell:
Shell profile:  /root/.profile
/root/.profile has been modified to add tiup to PATH
open a new terminal or source /root/.profile to use it
Installed path: /root/.tiup/bin/tiup
===============================================
Have a try:     tiup playground
===============================================

root@lxc-node-tikv-1:~# sudo useradd -m hugh
root@lxc-node-tikv-1:~# sudo adduser hugh sudo

Adding user `hugh' to group `sudo' ...
Adding user hugh to group sudo
Done.

root@lxc-node-tikv-1:~# sudo passwd hugh

New password:
Retype new password:
passwd: password updated successfully

root@lxc-node-tikv-1:~# sudo visudo # Over here I replaced the sudo entry line with "%sudo ALL=(ALL) NOPASSWD:ALL", so added NOPASSWD
root@lxc-node-tikv-1:~# vim /etc/ssh/sshd_config
# Change the following lines
# PasswordAuthentication yes
root@lxc-node-tikv-1:~# source .profile
root@lxc-node-tikv-1:~# tiup cluster

tiup is checking updates for component cluster ...timeout(2s)!
The component `cluster` version  is not installed; downloading from repository.
download https://tiup-mirrors.pingcap.com/cluster-v1.11.3-linux-amd64.tar.gz 8.44 MiB / 8.44 MiB 100.00% 396.61 MiB/s
Starting component `cluster`: /root/.tiup/components/cluster/v1.11.3/tiup-cluster
Deploy a TiDB cluster for production
...

root@lxc-node-tikv-1:~# tiup update --self && tiup update cluster
download https://tiup-mirrors.pingcap.com/tiup-v1.11.3-linux-amd64.tar.gz 6.92 MiB / 6.92 MiB 100.00% 171.31 MiB/s
Updated successfully!
component cluster version v1.11.3 is already installed
Updated successfully!

root@lxc-node-tikv-1:~# tiup cluster template > topology.yaml

tiup is checking updates for component cluster ...
Starting component `cluster`: /root/.tiup/components/cluster/v1.11.3/tiup-cluster template

Great!
We now have our initial node ready.
We need to modify our topology file to reflect the actual topology we will have.
Here is the sample I have from editing the topology.yaml we just exported.

# # Global variables are applied to all deployments and used as the default value of
# # the deployments if a specific deployment value is missing.
global:
  # # The user who runs the tidb cluster.
  user: "hugh"
  # # group is used to specify the group name the user belong to if it's not the same as user.
  # group: "tidb"
  # # SSH port of servers in the managed cluster.
  ssh_port: 22
  # # Storage directory for cluster deployment files, startup scripts, and configuration files.
  deploy_dir: "/tidb-deploy"
  # # TiDB Cluster data storage directory
  data_dir: "/tidb-data"
  arch: "amd64"

# # Monitored variables are applied to all the machines.
monitored:
  # # The communication port for reporting system information of each node in the TiDB cluster.
  node_exporter_port: 9100
  # # Blackbox_exporter communication port, used for TiDB cluster port monitoring.
  blackbox_exporter_port: 9115

# # Server configs are used to specify the configuration of PD Servers.
pd_servers:
  # # The ip address of the PD Server.
  - host: lxc-node-pd-1
  - host: lxc-node-pd-2
  - host: lxc-node-pd-3

# # Server configs are used to specify the configuration of TiKV Servers.
tikv_servers:
  # # The ip address of the TiKV Server.
  - host: lxc-node-tikv-1
  - host: lxc-node-tikv-2
  - host: lxc-node-tikv-3

That is actually my entire topology.yaml file.
I removed TiDB and all the monitoring - we aren't using that for this example.

We will create a snapshot from the image to simplify our setup and start the installation.
We will then create instances that automatically have SSH, the hugh account with a known password, and a sudo group permission without password authentication.
Don't do this in production - this is a highly insecure setup for many reasons.

root@lxc-node-tikv-1:~# shutdown -r 0
hugh@hugh-VirtualBox:~$ lxc snapshot lxc-node-tikv-1 base-installation-tikv
hugh@hugh-VirtualBox:~$ lxc publish lxc-node-tikv-1/base-installation-tikv --alias base-installation-tikv

Instance published with fingerprint: b8841a679a59f98f3c23ba6c8795c84942f19170b4a8c41eb102130467c4cca6

hugh@hugh-VirtualBox:~$ printf "lxc-node-tikv-2\n lxc-node-tikv-3\n lxc-node-pd-1\n lxc-node-pd-2\n lxc-node-pd-3" | xargs -I % lxc launch base-installation-tikv %

Creating lxc-node-tikv-2
Starting lxc-node-tikv-2
Creating lxc-node-tikv-3
Starting lxc-node-tikv-3
Creating lxc-node-pd-1
Starting lxc-node-pd-1
Creating lxc-node-pd-2
Starting lxc-node-pd-2
Creating lxc-node-pd-3
Starting lxc-node-pd-3

We can now start our cluster from the first node we configured.
TiUp will connect to all the other nodes via SSH and password authentication and install the services that way.

hugh@hugh-VirtualBox:~$ lxc exec lxc-node-tikv-1 bash
root@lxc-node-tikv-1:~# source .profile
root@lxc-node-tikv-1:~# tiup cluster deploy tikv-test v6.6.0 ./topology.yaml --user hugh -p

tiup is checking updates for component cluster ...
Starting component `cluster`: /root/.tiup/components/cluster/v1.11.3/tiup-cluster deploy tikv-test v6.6.0 ./topology.yaml --user hugh -p
Input SSH password:





+ Detect CPU Arch Name
+ Detect CPU Arch Name
  - Detecting node lxc-node-pd-1 Arch info ... Done
  - Detecting node lxc-node-pd-2 Arch info ... Done
  - Detecting node lxc-node-pd-3 Arch info ... Done
  - Detecting node lxc-node-tikv-1 Arch info ... Done
  - Detecting node lxc-node-tikv-2 Arch info ... Done
  - Detecting node lxc-node-tikv-3 Arch info ... Done





+ Detect CPU OS Name
+ Detect CPU OS Name
  - Detecting node lxc-node-pd-1 OS info ... Done
  - Detecting node lxc-node-pd-2 OS info ... Done
  - Detecting node lxc-node-pd-3 OS info ... Done
  - Detecting node lxc-node-tikv-1 OS info ... Done
  - Detecting node lxc-node-tikv-2 OS info ... Done
  - Detecting node lxc-node-tikv-3 OS info ... Done
Please confirm your topology:
Cluster type:    tidb
Cluster name:    tikv-test
Cluster version: v6.6.0
Role  Host             Ports        OS/Arch       Directories
----  ----             -----        -------       -----------
pd    lxc-node-pd-1    2379/2380    linux/x86_64  /tidb-deploy/pd-2379,/tidb-data/pd-2379
pd    lxc-node-pd-2    2379/2380    linux/x86_64  /tidb-deploy/pd-2379,/tidb-data/pd-2379
pd    lxc-node-pd-3    2379/2380    linux/x86_64  /tidb-deploy/pd-2379,/tidb-data/pd-2379
tikv  lxc-node-tikv-1  20160/20180  linux/x86_64  /tidb-deploy/tikv-20160,/tidb-data/tikv-20160
tikv  lxc-node-tikv-2  20160/20180  linux/x86_64  /tidb-deploy/tikv-20160,/tidb-data/tikv-20160
tikv  lxc-node-tikv-3  20160/20180  linux/x86_64  /tidb-deploy/tikv-20160,/tidb-data/tikv-20160
Attention:
    1. If the topology is not what you expected, check your yaml file.
    2. Please confirm there is no port/directory conflicts in same host.
Do you want to continue? [y/N]: (default=N) y
...
Cluster `tikv-test` deployed successfully, you can start it with command: `tiup cluster start tikv-test --init`

root@lxc-node-tikv-1:~# tiup cluster start tikv-test --init

tiup is checking updates for component cluster ...
Starting component `cluster`: /root/.tiup/components/cluster/v1.11.3/tiup-cluster start tikv-test --init
Starting cluster tikv-test...
+ [ Serial ] - SSHKeySet: privateKey=/root/.tiup/storage/cluster/clusters/tikv-test/ssh/id_rsa, publicKey=/root/.tiup/storage/cluster/clusters/tikv-test/ssh/id_rsa.pub
+ [Parallel] - UserSSH: user=hugh, host=lxc-node-tikv-2
+ [Parallel] - UserSSH: user=hugh, host=lxc-node-pd-3
+ [Parallel] - UserSSH: user=hugh, host=lxc-node-pd-2
+ [Parallel] - UserSSH: user=hugh, host=lxc-node-tikv-3
+ [Parallel] - UserSSH: user=hugh, host=lxc-node-tikv-1
+ [Parallel] - UserSSH: user=hugh, host=lxc-node-pd-1
+ [ Serial ] - StartCluster
...
Started cluster `tikv-test` successfully
The root password of TiDB database has been changed.
The new password is: 'JuEzYp59+8@$20T_3K'.
Copy and record it to somewhere safe, it is only displayed once, and will not be stored.
The generated password can NOT be get and shown again.

At this point, you should have a running TiKV cluster.
All that remains is to put SurrealDB instances on the PD nodes.
I will demonstrate this only for a single PD node, as the rest are identical.

root@lxc-node-pd-1:~# curl --proto '=https' --tlsv1.2 -sSf https://install.surrealdb.com | sh -s -- --nightly


 .d8888b.                                             888 8888888b.  888888b.
d88P  Y88b                                            888 888  'Y88b 888  '88b
Y88b.                                                 888 888    888 888  .88P
 'Y888b.   888  888 888d888 888d888  .d88b.   8888b.  888 888    888 8888888K.
    'Y88b. 888  888 888P'   888P'   d8P  Y8b     '88b 888 888    888 888  'Y88b
      '888 888  888 888     888     88888888 .d888888 888 888    888 888    888
Y88b  d88P Y88b 888 888     888     Y8b.     888  888 888 888  .d88P 888   d88P
 'Y8888P'   'Y88888 888     888      'Y8888  'Y888888 888 8888888P'  8888888P'

Fetching the latest database version...
Fetching the host system architecture...
Installing surreal-nightly for linux-amd64...


SurrealDB successfully installed in:
  /root/.surrealdb/surreal

To ensure that surreal is in your $PATH run:
  PATH=/root/.surrealdb:$PATH
Or to move the binary to --nightly run:
  sudo mv /root/.surrealdb/surreal --nightly

To see the command-line options run:
  surreal help
To start an in-memory database server run:
  surreal start --log debug --user root --pass root memory
For help with getting started visit:
  https://surrealdb.com/docs

root@lxc-node-pd-1:~# PATH=/root/.surrealdb:$PATH
root@lxc-node-pd-1:~# surreal sql --ns testns --db testdb -u root -p root --conn tikv://lxc-node-pd-1:2379

testns/testdb> create person:hugh content {name:'test'}
[{"id":"person:hugh","name":"test"}]
testns/testdb> select * from person
[{"id":"person:hugh","name":"test"}]

Under a real scenario, you would have the SurrealDB nodes separate from the PD nodes, and the connections would be load balanced across the entire PD node pool.

Takeaways

As you can see, it is possible to set up SurrealDB in a cluster so that writes and reads can scale.
Failure of a single node would have minimal disruption to the rest of the work while keeping your data intact.
Backups can be performed against the TiKV cluster to ensure you can recover in the event of serious failures.

Hopefully, you found this guide helpful, and I look forward to hearing what you get up to with it!

Introduction to SurrealDB

Przemyslaw Hugh Kaznowski — Wed, 08 Feb 2023 16:18:54 +0000

Read the original article on the SurrealDB Blog

What is the product?

If you are reading this, you may be wondering how to get started with this fantastic product you just discovered, SurrealDB. It's a database that does many routine things, so you can focus on what matters to you - processing your data.

In this blog post, I will describe how to get set up and use the client while explaining some of the elementary concepts behind SurrealDB.

Quick Install of SurrealDB

Initially, I wanted to do this tutorial using a fresh install of Alpine Linux 1.13.1. This is tricky because Alpine uses musl instead of GNU libc bindings. So this tutorial is using Ubuntu Desktop 22.04.01 instead. Which distribution exactly doesn't matter - I am doing this to demonstrate that there are no dependencies.

We can then follow the instructions to install SurrealDB.

curl -sSf https://install.surrealdb.com | sh

And finally, we can start the server and client. I am running the database in the top panel of tmux and the client in the bottom panel.

With that setup, we are ready to go!

Namespaces and Databases

If you are anything like me, your first use of the database looked like this.

INSERT INTO person [{name:'Hugh'}, {name:'Rushmore'}];

[{"time":"7.608µs","status":"ERR","detail":"Specify a namespace to use"}]

We have yet to declare which namespace and database we are using. You could solve this problem by inlining your statements.

USE NAMESPACE test; USE DATABASE testdb; INSERT INTO person [{name:'Hugh'}, {name:'Rushmore'}];

[{"time":"5.448µs","status":"OK","result":null},{"time":"1.855µs","status":"OK","result":null},{"time":"31.966464ms","status":"OK","result":[{"id":"person:7fsqx0q0iyeoltyjsr7c","name":"Hugh"},{"id":"person:2fdjz6j6luih4bn44u9c","name":"Rushmore"}]}]

This is quite inconvenient to do for every query, though. Instead, we are going to reconnect while specifying our database and namespace.

hugh@hugh-VirtualBox:~$ surreal sql --conn http://0.0.0.0:8000 -u root -p root --ns testns --db testdb

INSERT INTO person [{name:'Hugh'}, {name:'Rushmore'}];

[{"time":"109.351µs","status":"OK","result":[{"id":"person:adfy4qj5254b8l4po7bp","name":"Hugh"},{"id":"person:g5wkfgggpfayj6j453hg","name":"Rushmore"}]}]

Much more convenient!

So what are these namespaces and databases?
In the simplest terms, they are ways of making sure that you do not get name collisions (who here has had a table called "requests" or "versions"?). But really, there is more to this separation.

At the highest level, we have namespaces. Namespaces are a way to separate areas of security concern. They are a way of giving blanket access to sets of databases. The databases will not be shared by 2 namespaces, so users with full access to separate namespaces will need to be provided special permissions for namespaces they do not have access to. This is a fundamental concept for multi-tenancy.

So if we have this separation at the top level with namespaces, what is the purpose of having databases? Again, this is tied to uniqueness collision and security. You can think of a database (storage, not the DBMS system called surreal) as a space where you want to enforce name uniqueness. So by having separate databases, you can avoid this uniqueness, as with our example above with tables called "requests" and "versions".

This may sound inconvenient, but the intention is to have a single SurrealDB cluster. If you want to develop a new app or service, create a new database (or namespace, if it's for a new client or project).

Table vs Document

In the above example, we have already inserted 2 entries into a table called person. We can confirm that by doing a very familiar select query.

SELECT * FROM person;

[{"time":"1.151239ms","status":"OK","result":[{"id":"person:adfy4qj5254b8l4po7bp","name":"Hugh"},{"id":"person:g5wkfgggpfayj6j453hg","name":"Rushmore"}]}]

Brill! We can see that the "id" fields were populated, and the ID includes the table name. But what if we want to use a custom ID that may be tied with a request ID, legacy system, username or other variables?

INSERT INTO person {id: 'tobie', name: 'Tobie'};

[{"time":"90.142µs","status":"OK","result":[{"id":"person:tobie","name":"Tobie"}]}]

SELECT * FROM person;

[{"time":"63.091µs","status":"OK","result":[{"id":"person:adfy4qj5254b8l4po7bp","name":"Hugh"},{"id":"person:g5wkfgggpfayj6j453hg","name":"Rushmore"},{"id":"person:tobie","name":"Tobie"}]}]

Graph vs Link

So we have created a table (person) of documents (Hugh, Rushmore, Tobie). How are the documents different to relational tables?

INSERT INTO person {name: 'complex', shoes: ['red', 'green', {colour: 'blue', favourite: true}]};

[{"time":"110.782µs","status":"OK","result":[{"id":"person:1vjjz8xi2pbau1zh07ob","name":"complex","shoes":["red","green",{"colour":"blue","favourite":true}]}]}]

Ok, so quite different! We have created a new column called "shoes" on-the-fly. The column contains an array for this entry. And not all the elements of the array are of the same type! We have "red" and "green" as string elements of the shoes column, but then an entire object for the "blue" colour. This is so different that calling these "columns" is a bit of a stretch. That is why we don't refer to entries in a document as columns, only as records.

SELECT id FROM person;

[{"time":"1.21102ms","status":"OK","result":[{"id":"person:1vjjz8xi2pbau1zh07ob"},{"id":"person:adfy4qj5254b8l4po7bp"},{"id":"person:g5wkfgggpfayj6j453hg"},{"id":"person:tobie"}]}]

SELECT shoes FROM person;

[{"time":"125.074µs","status":"OK","result":[{"shoes":["red","green",{"colour":"blue","favourite":true}]},{"shoes":null},{"shoes":null},{"shoes":null}]}]

SELECT shoes.colour FROM person;

[{"time":"89.063µs","status":"OK","result":[{"shoes":{"colour":[null,null,"blue"]}},{"shoes":{"colour":null}},{"shoes":{"colour":null}},{"shoes":{"colour":null}}]}]

You may wonder why there is an unusual [null,null,'blue'] in the results - those are the 3 values for the object: ID, favourite, and colour.

We can do some cool stuff with these nested records. We can create Record Links. Record Links are how you would use Foreign Keys in a relational database to point to either same-table entries, or even other-table entries? Unlike nested document records, these links point to entries in a table that may not belong to the linking document. That means all documents can include all other document entries if they are declared to do so. And they do not need to be updated - they are pointers. You can put the C++ book down now; you don't need to know this to use them ;).

UPDATE person:tobie SET friends=[person:g5wkfgggpfayj6j453hg, person:adfy4qj5254b8l4po7bp];

[{"time":"90.815µs","status":"OK","result":[{"friends":["person:g5wkfgggpfayj6j453hg","person:adfy4qj5254b8l4po7bp"],"id":"person:tobie","name":"Tobie"}]}]

SELECT friends.name FROM person:tobie;

[{"time":"185.621µs","status":"OK","result":[{"friends":{"name":["Rushmore","Hugh"]}}]}]

Pretty neat.

Are these graphs, though? Well, not quite. These are just composable documents. We traverse one-way instead of bi-directionally. How would we link documents together as a graph?

RELATE person:tobie->works_with->person:g5wkfgggpfayj6j453hg SET last_updated = time::now();
RELATE person:tobie->works_with->person:adfy4qj5254b8l4po7bp SET last_updated = time::now();

[{"time":"38.104611ms","status":"OK","result":[{"id":"works_with:fl9wgahk9pl4eqj33es7","in":"person:tobie","last_updated":"2023-02-06T18:33:41.131956605Z","out":"person:g5wkfgggpfayj6j453hg"}]},{"time":"88.334µs","status":"OK","result":[{"id":"works_with:zccalviq0oj6o36wylgy","in":"person:tobie","last_updated":"2023-02-06T18:33:41.156103917Z","out":"person:adfy4qj5254b8l4po7bp"}]}]

SELECT * FROM person WHERE ->works_with->person;

[{"time":"1.31216ms","status":"OK","result":[{"friends":["person:g5wkfgggpfayj6j453hg","person:adfy4qj5254b8l4po7bp"],"id":"person:tobie","name":"Tobie"}]}]

We have identified the mega node!

Take aways

In this tutorial, we have installed SurrealDB, launched it, connected to it, created data and explained how it fits together. Hopefully, now you are comfortable playing with it and can test it with your applications or dataset.