How to use Reduct Storage with TypeScript

Jesse Langford — Wed, 29 Jun 2022 20:21:16 +0000

Reduct Storage is an open-source project using time series storage for quick access to data. It's optimized for small objects and asynchronous read\write operations. It's also 100% open source. You can read more about on the homepage.

In this post, I'll be going through how to install and use Reduct Storage in a Node/Typescript project. The examples shown below exist inside a simple Node project with express as its main vehicle for interaction. Each feature covered in Reduct will have a corresponding endpoint to interact with.

You can follow along inside the example app

Getting Started: Docker

The first thing we need to do is get our test instance of the database set up. Reduct provides a docker container you can use to test against. Run the command below to get it up and running:
docker run -p 8383:8383 -v ${PWD}/data:/data reductstorage/engine:latest
With that running, you can access the Reduct API from http://127.0.0.1:8383
You can find environment variables for running the container here

Installation

To start using Reduct, install the javascript SDK with node or yarn

yarn add reduct-js

npm i reduct-js

Connecting with Database Client

The first thing to do once the SDK is installed is to make a helper function to connect to the database.
For this demo, I'm only going to be using one bucket to write and read data from.

import { Bucket, Client } from "reduct-js";

export const getBucket = async (): Promise<Bucket> => {
    const client = new Client(process.env.CLIENT_URL);
    return await client.getOrCreateBucket(process.env.BUCKET);
};

In the above function, I am creating a new database client using the URL from my environment. Next, I am returning a bucket specified by a string variable. For this demo, I have simply named my bucket "bucket". This function will be used whenever I need to make an action against my database.

For clarity, here is the .env file:

PORT=3000
CLIENT_URL=http://127.0.0.1:8383
BUCKET=bucket

Writing Data

Once the bucket connection is set up, I can start writing data to my database.

export const writeData = async (data: string): Promise<number> => {
    const timestamp = Date.now() * 1000;
    const bucket = await getBucket();
    await bucket.write("entry-1", data, BigInt(timestamp));
    return timestamp;
};

In the function above, I first create a timestamp to be saved along with my data. Next, I use the function we created earlier to retrieve a bucket from my database. Once I have the bucket and a timestamp, I can use the write method on my bucket to write a new entry. The write function takes three arguments.

Entry - The name of the entry
Data - Data as a string
Timestamp- Timestamp for the record

As an added help, I am returning the timestamp from the function in case you want to use it later to retrieve data.

Now we can add an endpoint for writing data that looks like this:

app.post("/", async (req, res, next) => {
  try {
    const { data } = req.body;
    const timestamp = await writeData(data);
    res.json({ timestamp });
  } catch (err) {
    next(err);
  }
});

Using a tool like Postman, you can write a string of text using my endpoint. If it succeeds, you'll get back a timestamp.

![Create Entry Request (https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2r0kvmxe8aq6efpqqxe5.png)

Reading Data

Reading data from Reduct is similarly straightforward.

export const readData = async (timestamp?: bigint): Promise<string> => {
  const bucket = await getBucket();
  const data = await bucket.read("entry-1", timestamp);
  return data.toString();
};

Above is a simple function for reading an entry in a bucket. Like the write data function earlier, we need to get the bucket first. Once we have that, we can call the read function. The read function takes two arguments

Entry - The name of the bucket entry
Timestamp - optional timestamp for a value

If a timestamp is not provided to the read function, the latest record will be returned.

Next, let's incorporate this into our express server.

app.get("/", async (req, res, next) => {
  try {
    const query = req.query.ts;
    const timestamp = isNaN(+query) ? undefined : BigInt(+query);
    const data = await readData(timestamp);
    res.json({ data });
  } catch (err) {
    next(err);
  }
});

Above is a simple get request that allows for a timestamp to be included in the URL as a query parameter. You can choose to provide a timestamp to get a particular record or leave it blank and have the most recent one returned.

Query by Time Interval

Finally, let's look at querying by time interval. Because Reduct relies on timestamps for queries, we have an easy way to get a list of records by providing a start and stop time interval.

export const listData = async (start: bigint, stop: bigint): Promise<any> => {
  const bucket = await getBucket();
  const data = await bucket.list("entry-1", start, stop);
  // optional step for formatting bigint data
  const entries = [];
  for (const entry of data) {
    entries.push({
      size: entry.size.toString(),
      timestamp: entry.timestamp.toString(),
    });
  }
  return entries;
};

In the function above we get the bucket first as usual, then we use the list function to get an array of records from an entry. The list function takes three arguments:

Entry - The name of the bucket entry
Start - Start point of the time period
Stop - Stop point of the time period

The data returned is an array of objects with a timestamp and a size. Both values are bigint types. You should make sure your server can properly serialize this data before returning it in a response. I have done a quick and dirty for loop that transforms each object as an example.

Once we have this function, we can create a new endpoint similar to our read endpoint.

app.get("/interval", async (req, res, next) => {
  try {
    const startQuery = req.query.start;
    const stopQuery = req.query.stop;
    const start = isNaN(+startQuery) ? undefined : BigInt(+startQuery);
    const stop = isNaN(+stopQuery) ? undefined : BigInt(+stopQuery);
    const data = await listData(start, stop);
    res.json({ data });
  } catch (err) {
    next(err);
  }
});

This endpoint copies the same query string process as the read endpoint earlier. It's overly simplified for the purpose of the demo, but should give you an idea of how it can work.

With this, we can now query using a start and stop timestamp

You can then use the timestamp to retrieve the record data with the read function we created earlier.

Wrapping Up

Hopefully, you have a better idea of how to get started using Reduct storage. The API is easy to understand and light on complex terminology. I highly recommend going through the rest of the documentation to get a more complete picture of all the features this project has to offer.

Forem: Jesse Langford