Forem: Sascha

How to build a simple Slack bot

Sascha — Mon, 16 Nov 2020 13:26:57 +0000

Slack is a wonderfully simple communication tool. Everybody is within reach of your fingertips. You can grab anybodies attention with a few key strokes. Distract them with a question whenever you are too bored to google the answer yourself ;-)

It does not take many workspaces you are coerced into joining before you turn off notification for most of the channels you are part of. However, some people have a very high signal to noise ratio and you would not mind being notified of their messages.

Fortunately, this conundrum can be easily be solved with a simple bot. So let's learn how to create such a Slack bot.

If you don't care about a step by step guide you can also just check out the final code.

Building the Slack bot

We will build our bot in Node.js, so you need to have node and npm installed. If you want to deploy your app to Heroku, you will also need a Heroku account, as well having their CLI installed. To run your app locally, you also need to install and run a RethinkDB instance.

To create the application, run the following in a terminal.

$ mkdir stalker-bot && cd stalker-bot
$ npm init -y
$ npm install @slack/events-api @slack/web-api rethinkdb

This will initialize a Node.js app and install all required dependencies.

Listening to Slack events

We will create a Node.js server to listen to Slack events. Create an index.js file and add the following server code.

// index.js

// Initialize Slack event listener
const { createEventAdapter } = require("@slack/events-api");
const slackSigningSecret = process.env.SLACK_SIGNING_SECRET;
const slackEvents = createEventAdapter(slackSigningSecret);

const { handleCommand, handleMessage } = require("./handler.js");

// Listen to message event (message.im, message.channel)
slackEvents.on("message", (event) => {
  // Ignore messages from bots
  if (event.bot_id != null) {
    return;
  }
  if (event.channel_type == "im") {
    handleCommand(event);
  } else if (event.channel_type == "channel") {
    handleMessage(event);
  }
});

// Catch and log errors
slackEvents.on("error", (error) => {
  console.log(error);
});

// Run server
const port = process.env.PORT || 5000;
(async () => {
  const server = await slackEvents.start(port);
  console.log(`Listening for events on ${server.address().port}`);
})();

We first configure the slack libraries, namely the event listener server and the web client. We then listen to message events. Direct messages are interpreted as commands and messages in channels are listened to in case we need to notify a stalker.

Bot commands

We can chat directly with the bot to issue commands. The stalker bot knows about three commands:

subscribe to a user in a channel
unsubscribe from a user in a channel
list all current subscriptions

To save all subscriptions, we will use my favorite document database as of late, RethinkDB. It is similar to MongoDB but additionally has reactivity built into it and it is still open source. We will need two tables, one to save all users and one to save the subscriptions they have. We will deal with managing database connections and running migrations later.

Create a handler.js file and start with the following code. We first configure the Slack web client in order to be able to respond to events and add some database boilerplate before we handle the actual commands.

// handler.js

// Initialize Slack client
const { WebClient } = require("@slack/web-api");
const slackToken = process.env.SLACK_TOKEN;
const slackWeb = new WebClient(slackToken);

// Lazy RethinkDB connection
const r = require("rethinkdb");
const { getRethinkDB } = require("./reql.js");

// Tables
const subTable = "subscriptions";
const userTable = "users";

// matches commands of type "(un)subscribe to/from <@U01C9PRR6TA> in <#C01BHNSMGKT|general>"
const regexUserChannel = /\<\@(?<user_id>\w+)\>.+\<\#(?<channel_id>\w+)\|(?<channel_label>\w+)\>/;

// Handle commands send directly to the bot
exports.handleCommand = async function (event) {
  // Note: since unsubscribe contains subscribe it must come first
  if (event.text.includes("unsubscribe")) {
    unsubscribe(event);
  } else if (event.text.includes("subscribe")) {
    subscribe(event);
  } else if (event.text.includes("list")) {
    list(event);
  } else {
    slackWeb.chat
      .postMessage({
        text:
          "I don't understand. Available commands:\n* subscribe to @user in #channel\n* unsubscribe from @user in #channel\n* list subscriptions",
        channel: event.channel,
      })
      .catch((err) => {
        console.log("Error helping with unknown cmd:", err);
      });
  }
};

// ...

When handling commands we basically search for one of the three commands in the message. We also use a regular expression to be able to extract the user and the channel from the (un)subscribe commands.

Subscribe to a user

To subscribe to a user in a channel we first need to parse said user and channel from the subscription command. The parsed user and channel are saved in a subscription object which can have listeners. The listener, i.e., the command issuer is saved in the user table.

// handler.js
// ...

let subscribe = async function (event) {
  // Try to understand the subscription command
  const match = event.text.match(regexUserChannel);
  if (!match) {
    slackWeb.chat
      .postMessage({
        text:
          'Who do you want to subscribe to? Use "subscribe to @user in #channel".',
        channel: event.channel,
      })
      .catch((err) => {
        console.log("Error helping with sub cmd:", err);
      });

    return;
  }
  let listener = { id: event.user, im: event.channel };
  let user = match.groups.user_id;
  let channel = match.groups.channel_id;

  const conn = await getRethinkDB();
  const subIndex = channel + "-" + user;

  // Create user
  let lis = await r.table(userTable).get(listener.id).run(conn);
  if (lis == null) {
    await r.table(userTable).insert(listener).run(conn);
  }

  let sub = await r.table(subTable).get(subIndex).run(conn);
  if (sub != null) {
    // Subscription exists -> add listener
    sub.listeners.push(listener.id);
    await r
      .table(subTable)
      .get(subIndex)
      .update({ listeners: sub.listeners })
      .run(conn);
    return;
  }

  // Create subscription (incl. listener)
  sub = {
    id: subIndex,
    channel: channel,
    user: user,
    listeners: [listener.id],
  };
  await r.table(subTable).insert(sub).run(conn);

  // Join channel (if already joined we will get a warning)
  slackWeb.conversations
    .join({
      channel: channel,
    })
    .catch((err) => {
      console.log("Error joining conversation:", err);
    });
};

// ...

When a subscription is created, the bot also needs to join the respective channel in order to be able to listen to messages from the desired user.

Unsubscribe from a user

To unsubscribe from a user in a channel we also need to parse the command first and then revert the actions done in the subscription command. We remove the listener, i.e., the command issuer from the subscription or delete the subscription if there are no listeners.

// handler.js
// ...

let unsubscribe = async function (event) {
  const match = event.text.match(regexUserChannel);
  if (!match) {
    slackWeb.chat
      .postMessage({
        text:
          'Who do you want to unsubscribe from? Use "unsubscribe from @user in #channel".',
        channel: event.channel,
      })
      .catch((err) => {
        console.log("Error helping with unsub cmd:", err);
      });
    return;
  }
  let listener = { id: event.user, im: event.channel };
  let user = match.groups.user_id;
  let channel = match.groups.channel_id;

  const conn = await getRethinkDB();
  const subIndex = channel + "-" + user;

  let sub = await r.table(subTable).get(subIndex).run(conn);
  if (sub == null) {
    // No subscription --> do nothing
    return;
  }
  const lisIndex = sub.listeners.indexOf(listener.id);
  if (lisIndex < 0) {
    // Not listening --> do nothing
    return;
  }

  // Remove listener
  sub.listeners.splice(lisIndex, 1);
  if (sub.listeners.length > 0) {
    // There are still other listeners
    await r
      .table(subTable)
      .get(subIndex)
      .update({ listeners: sub.listeners })
      .run(conn);
    return;
  }

  // No more listeners -> remove subscription
  await r.table(subTable).get(subIndex).delete().run(conn);

  let chanSubs_cursor = await r
    .table(subTable)
    .getAll(channel, { index: "channel" })
    .run(conn);
  let chanSubs = await chanSubs_cursor.toArray();
  if (chanSubs.length > 0) {
    // There are still subscriptions
    return;
  }

  // No more subscriptions -> leave channel
  slackWeb.conversations
    .leave({
      channel: channel,
    })
    .catch((err) => {
      console.log("Error leaving conversation:", err);
    });
};

// ...

When there are no more subscriptions to a channel we also have the bot leave it. This will lessen the messages the bot has to react through.

List subscriptions

Listing the subscriptions is a convenience command to see what users we are currently stalking.

// handler.js
// ...

let list = async function (event) {
  const conn = await getRethinkDB();
  let subs_cursor = await r
    .table(subTable)
    .getAll(event.user, { index: "listeners" })
    .run(conn);
  let subs = await subs_cursor.toArray();
  let subList = subs.map(
    (sub) => "* <@" + sub.user + "> in <#" + sub.channel + ">",
  );
  // Respond with subs list
  slackWeb.chat
    .postMessage({
      text: "You are currently subscribed to:\n" + subList.join("\n"),
      channel: event.channel,
    })
    .catch((err) => {
      console.log("Error with list cmd:", err);
    });
};

// ...

Now that we have implemented all commands, lets do the actual stalking.

Do the actual stalking

When we subscribe to a user in a channel, the bot joins said channel. It handles each message and reacts accordingly if the message author is of interest. If there is a listener for said author, the bot sends a direct message to the listener.

// handler.js
// ...

// Handle message overheard in channels
exports.handleMessage = async function (event) {
  const conn = await getRethinkDB();
  const subIndex = event.channel + "-" + event.user;
  let sub = await r.table(subTable).get(subIndex).run(conn);
  if (sub == null) {
    // No subscription, ignore
    return;
  }

  let lis_cursor = await r
    .table(userTable)
    .getAll(r.args(sub.listeners))
    .run(conn);
  lis_cursor.each((err, lis) => {
    // Send IM to lisener
    slackWeb.chat
      .postMessage({
        text:
          "<@" +
          sub.user +
          "> wrote a message in <#" +
          sub.channel +
          ">: " +
          event.text,
        channel: lis.im,
      })
      .catch((err) => {
        console.log("Error notifying about subscribed message:", err);
      });
  });
};

Note: In order for our bot to serve its purpose, we obviously cannot disable notifications for direct messages.

Database management

Until now we have conveniently just gotten a database connection and assumed the required tables already exist. Now, the time has come to manage the actual RethinkDB connection and take care of the required migrations.

RethinkDB connection

We manage our RethinkDB connection lazily, i.e., we only create the (re-)connection when it is actually needed. The connection parameters are parsed from environment variables, or the defaults are used.

// reql.js

const r = require("rethinkdb");

let rdbConn = null;
const rdbConnect = async function () {
  try {
    const conn = await r.connect({
      host: process.env.RETHINKDB_HOST || "localhost",
      port: process.env.RETHINKDB_PORT || 28015,
      username: process.env.RETHINKDB_USERNAME || "admin",
      password: process.env.RETHINKDB_PASSWORD || "",
      db: process.env.RETHINKDB_NAME || "test",
    });

    // Handle close
    conn.on("close", function (e) {
      console.log("RDB connection closed: ", e);
      rdbConn = null;
    });
    // Handle error
    conn.on("error", function (e) {
      console.log("RDB connection error occurred: ", e);
      conn.close();
    });
    // Handle timeout
    conn.on("timeout", function (e) {
      console.log("RDB connection timed out: ", e);
      conn.close();
    });

    console.log("Connected to RethinkDB");
    rdbConn = conn;
    return conn;
  } catch (err) {
    throw err;
  }
};
exports.getRethinkDB = async function () {
  if (rdbConn != null) {
    return rdbConn;
  }
  return await rdbConnect();
};

On Heroku, the RethinkDB Cloud add-on will set the environment variables. For a locally running instance of RethinkDB, the defaults should work.

Migration

The app does not work without a users and subscriptions tables. We thus need a database migration that adds these tables.

// migrate.js

var r = require("rethinkdb");

// Tables
const subTable = "subscriptions";
const userTable = "users";

r.connect(
  {
    host: process.env.RETHINKDB_HOST || "localhost",
    port: process.env.RETHINKDB_PORT || 28015,
    username: process.env.RETHINKDB_USERNAME || "admin",
    password: process.env.RETHINKDB_PASSWORD || "",
    db: process.env.RETHINKDB_NAME || "test",
  },
  async function (err, conn) {
    if (err) throw err;
    console.log("Get table list");
    let cursor = await r.tableList().run(conn);
    let tables = await cursor.toArray();

    // Check if user table exists
    if (!tables.includes(userTable)) {
      // Table missing --> create
      console.log("Creating user table");
      await r.tableCreate(userTable).run(conn);
      console.log("Creating user table -- done");
    }

    // Check if sub table exists
    if (!tables.includes(subTable)) {
      // Table missing --> create
      console.log("Creating sub table");
      await r.tableCreate(subTable).run(conn);
      console.log("Creating sub table -- done");
      // Create index
      await r.table(subTable).indexCreate("channel").run(conn);
      console.log("Creating channel secondary index -- done");
      await r
        .table(subTable)
        .indexCreate("listeners", { multi: true })
        .run(conn);
      console.log("Creating listeners secondary multi index -- done");
    }

    await conn.close();
  },
);

This migration checks if the required tables exists, and if it is missing, it creates them. It also creates the necessary secondary indices, one to find subscriptions by channel and one to find it by listeners.

Create a Heroku app

This step is optional. You can also run the app locally and use ngrok to receive the Slack events.

In order to deploy the application to Heroku we need to create a Heroku app:

$ git init
$ heroku create
Creating app... done, ⬢ fast-inlet-79371
https://fast-inlet-79371.herokuapp.com/ | https://git.heroku.com/fast-inlet-79371.git

Creating a Heroku app will give you back the URL with a random name. Take note of this URL as it is required for the Slack callback URL later on.

We will also need a RethinkDB instance to store and subscribe to the chat messages sent between users. You can do this via the RethinkDB Cloud add-on as follows:

$ heroku addons:create rethinkdb

The RethinkDB Cloud add-on is currently in alpha. Request an invite for your Heroku account email.

Deploy the application to Heroku

To deploy our slack bot to Heroku we need to create a Procfile. This file basically tells Heroku what processes to run.

// Procfile

release: node migrate.js
web: node index.js

The release and web processes are recognized by Heroku as the command to run upon release and the main web app respectively.

Deploy the app to Heroku with

$ echo node_modules > .gitignore
$ git add .
$ git commit -m 'A stalker bot'
$ git push heroku master

The app will not work yet because it is missing two environment variables, namely SLACK_SIGNING_SECRET and SLACK_TOKEN. We will get them when we create the actual Slack application.

Create the Slack application

To create a Slack app go to api.slack.com/apps (if you are not signed in, sign in and then come back to this URL). Click on "Create App" and fill in a name and a workspace to associate the app with.

Permissions

First we need to declare all permissions we need for our app. This can be done in the "OAuth & Permissions" tab. Scroll down to the "Scopes" card and add the following "Bot Token Scopes":

channels:history
channels:join
chat:write
im:history

The channels:history and im:history permission allows the bot to read messages in channels it belongs to as well as direct messages. The channels:join permission allows the bot to join new channels. Finally, the chat:write permission allows the bot to write direct messages (e.g., to you).

Set environment variables

We need two Slack keys in our bot. A signing secret to verify the message events we get from Slack and a token to authenticate our actions as a bot. The signing secret can be found in the "App Credentials" card in the "Basic Information" tab. The OAuth token is shown in the "OAuth & Permissions" tab. Add both keys to your Heroku app with

$ heroku config:set SLACK_SIGNING_SECRET=...
$ heroku config:set SLACK_TOKEN=xoxb-...

This will automatically restart the Heroku app and allow for the event subscription we add next to verify your correctly running endpoint.

Event Subscription

Our app only works if we can react to events that happen in the Slack workplace. Go to the "Event Subscriptions" tab and enable events. For the request URL put in the app URL you got from Heroku and add the events route, e.g., https://fast-inlet-79371.herokuapp.com/events. Then subscribe to the following bot events:

message.channels
message.im

You will see that these two events require the channels:history and im:history permissions which we added in the previous step. Save the changes for them to take effect.

Install app

Now we are ready to install the app in our workspace. Go to the "Basic Information" tab and click on "Install App to Workspace". This will put you in the role of the app user and ask you to grant it the permissions the app requires.

Test it out

Go to your workspace and add the Stalker bot to your Apps. Test it out and subscribe to your favorite person in a busy channel full of noise. Each time the stalked person writes you will get a direct message to notify you.

Get Historical Stats for your Dev.to Articles

Sascha — Thu, 10 Sep 2020 15:51:36 +0000

Recently, I posted my first article on Dev.to. As expected there was an initial wave of some views which eventually stopped. Two weeks later I posted a second article. It was related to the first article and had a link to it. As expected, the views of my first post went up again. Since I love data, I wanted to see how many views belonged to the first wave and how the views on my articles change based on the what else I publish or other events that might happen. Unfortunately, Dev.to only shows me the final view count. To still my data hunger, I created a little app...

Feel free to visit the running app for my stats, or check out the code repository to dive into the code immediately.

TL;DR (for the efficient and impatient)

You want to deploy the app for your own Dev.to stats? You don't have time to read step by step tutorials? Well, here you go...

Deploying the app to Heroku only takes a minute but before you can do this, you need to do the following:

If you don't have one already, create a Heroku account and install their CLI
Go to RethinkDB Cloud and request free alpha access to the RethinkDB Cloud add-on.
Get an API key from Dev.to (Settings → Account → DEV API Keys)

Now run:

$ git clone git@github.com:mostlytyped/dev.to-stats.git
$ cd dev.to-stats/
$ heroku create
$ heroku addons:create rethinkdb
$ heroku config:set API_KEY=<YOUR_DEV_TO_API_KEY>
$ git push heroku master
$ heroku open

Note: the collection worker needs to be enabled manually on the apps resources dashboard. Also, free Heroku apps are put to sleep when they are inactive so the stats collector will not run regularly unless you pay for a dyno. Alternatively, you can visit your site once a day. For now, this is what I do ;-)

That's it, you are done. Enjoy your historical stats.

A step by step tutorial

Going fast is great but how about learning new skills and tools? In this section you will learn how to create the app from the ground up. In particular, you will...

... learn how to use RethinkDB, the totally awesome document database. It is like MongoDB but has reactivity built. This allows you to subscribe to queries. Oh, and it is still open source!
... create an embedded Vue.js app. That is a Vue app that you don't have to compile. I bet you haven't done that before.
... use Chart.js to plot the stats. It is always useful to have a plotting library in your tool kit.

Application setup

We will build a Node.js app, so you need to have node and npm installed. If you want to deploy your app to Heroku, you will also need a Heroku account, as well having their CLI installed. To run your app locally, you need to install and run a RethinkDB instance.

To create the application, run the following in a terminal.

$ mkdir devto-stats && cd devto-stats
$ npm init -y
$ npm install rethinkdb express morgan axios

This will initialize a Node.js app and install all required dependencies.

Prepare a Heroku app

In order to deploy the application to Heroku we need to create a Heroku app:

$ git init
$ heroku create

We will also need a RethinkDB instance to store articles and their daily stats. You can do this via the RethinkDB Cloud add-on as follows:

$ heroku addons:create rethinkdb

The RethinkDB Cloud add-on is currently in alpha. Request an invite for your Heroku account email.

Get the Dev.to API key

To access the stats of your articles you need an API key from Dev.to. You can get one under Settings → Account → DEV API Keys. Add the key to your Heroku app:

$ heroku config:set API_KEY=<YOUR_DEV_TO_API_KEY>

Collect the stats

To collect the stats we basically need to do two things on repeat: (i) get the stats for you articles from Dev.to and (ii) save the stats to RethinkDB. We need to run the stats collection at least every 24h to make sure we get stats once per day (Dev.to only updates the stats once daily).

// collect.js

const axios = require("axios");
const r = require("rethinkdb");
const { getRethinkDB } = require("./reql.js");

// Get articles from Dev.to
// ...

// Save article stats to RethinkDB
// ...

// Run once immediately
saveStats();
// Interval should be less than 24h. Running more than once a day
// is not a problem but a missed day cannot be recovered.
const interval = 6 * 60 * 60 * 1000; // 6h
setInterval(saveStats, interval);

To get the stats we run a simple axios request. Since the articles
are paged we query new pages until we get one that is not
full. The API_KEY environment variable contains your Dev.to
API key.

// collect.js
// ...

// Get articles from Dev.to
const getArticles = async function () {
  let articles = [];
  let page = 1;
  while (true) {
    let articles_page = await axios.get(
      "https://dev.to/api/articles/me?page=" + page,
      {
        headers: {
          "api-key": process.env.API_KEY,
        },
      },
    );
    articles.push(...articles_page.data);

    // If a page is not full we are done
    if (articles_page.data.length < 30) {
      break;
    }
  }
  return articles;
};

// ...

When saving the stats for the day we first need to check if the article already exists in our database. If not we add it. Then we save the stats as long as we have not done so already today.

// collect.js
// ...

// Save article stats to RethinkDB
const saveStats = async function () {
  const now = new Date();
  let day = ("0" + now.getDate()).slice(-2);
  let month = ("0" + (now.getMonth() + 1)).slice(-2);
  let year = now.getFullYear();
  const today = year + "-" + month + "-" + day;
  console.log("Collect stats:", today);

  // Get all articles
  const articles = await getArticles();

  // Save stats
  let conn = await getRethinkDB();
  articles.forEach(async (article) => {
    let db_article = await r.table("articles").get(article.id).run(conn);
    if (!db_article) {
      // New article -> save
      await r
        .table("articles")
        .insert({
          id: article.id,
          title: article.title,
          url: article.url,
          latest_stats: today,
        })
        .run(conn);
      // Save stats
      await r
        .table("stats")
        .insert({
          article_id: article.id,
          date: today,
          comments: article.comments_count,
          reactions: article.public_reactions_count,
          views: article.page_views_count,
        })
        .run(conn);
    } else if (db_article.latest_stats < today) {
      // Existing article -> update
      await r
        .table("articles")
        .get(article.id)
        .update({ latest_stats: today })
        .run(conn);
      // Save stats
      await r
        .table("stats")
        .insert({
          article_id: article.id,
          date: today,
          comments: article.comments_count,
          reactions: article.public_reactions_count,
          views: article.page_views_count,
        })
        .run(conn);
    } else {
      console.log("Already got stats today for article " + article.id);
    }
  });
};

// ...

As you might have noticed get the RethinkDB connection from reql.js. Let's implement this now.

Handling the RethinkDB connection

Connecting to RethinkDB is straight-forward. We only add a bit of logic to handle disconnections gracefully. The RETHINKDB_* environment variables will be set automatically by the RethinkDB Cloud add-on. The defaults work for a locally running RethinkDB instance.

// reql.js

const r = require("rethinkdb");

let rdbConn = null;
const rdbConnect = async function () {
  try {
    const conn = await r.connect({
      host: process.env.RETHINKDB_HOST || "localhost",
      port: process.env.RETHINKDB_PORT || 28015,
      username: process.env.RETHINKDB_USERNAME || "admin",
      password: process.env.RETHINKDB_PASSWORD || "",
      db: process.env.RETHINKDB_NAME || "test",
    });

    // Handle close
    conn.on("close", function (e) {
      console.log("RDB connection closed: ", e);
      rdbConn = null;
    });
    // Handle error
    conn.on("error", function (e) {
      console.log("RDB connection error occurred: ", e);
      conn.close();
    });
    // Handle timeout
    conn.on("timeout", function (e) {
      console.log("RDB connection timed out: ", e);
      conn.close();
    });

    console.log("Connected to RethinkDB");
    rdbConn = conn;
    return conn;
  } catch (err) {
    throw err;
  }
};
exports.getRethinkDB = async function () {
  if (rdbConn != null) {
    return rdbConn;
  }
  return await rdbConnect();
};

Building the server

The server is a simple Express.js app that serves a static frontend from the public directory. The server listens to get requests for one route (/article_stats) and returns an array of articles and their stats.

// index.js

// Express app
const express = require("express");
const app = express();

// Logging middleware
const morgan = require("morgan");
app.use(morgan("combined"));

// Serve frontend
app.use(express.static("public"));

// Lazy RethinkDB connection
const r = require("rethinkdb");
const { getRethinkDB } = require("./reql.js");

// Route to get stats
app.get("/article_stats", async (req, res) => {
  const conn = await getRethinkDB();
  let article_cursor = await r.table("articles").run(conn);
  let articles = await article_cursor.toArray();
  let article_stats = await Promise.all(
    articles.map(async (article) => {
      let stats_cursor = await r
        .table("stats")
        .filter({ article_id: article.id })
        .orderBy(r.asc("date"))
        .run(conn);
      let stats = await stats_cursor.toArray();
      article.stats = stats;
      return article;
    }),
  );
  res.json(article_stats);
});

// Start server
const listenPort = process.env.PORT || "3000";
app.listen(listenPort, () => {
  console.log(`Listening on ${listenPort}`);
});

Building the frontend

For our frontend we will use an embedded Vue.js app. This makes the frontend simple, but gives us access to all of Vue's awesome features. The frontend consists of a layout file as well as JavaScript and CSS assets.

HTML layout

The layout file only serves as a mount point for the Vue app in addition to importing the dependencies and assets.

<!-- public/index.html -->

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Dev.to Historical Stats</title>
    <link href="/css/main.css" rel="stylesheet" />
  </head>

  <body>
    <div id="app"></div>
    <script src="https://cdn.jsdelivr.net/npm/vue/dist/vue.js"></script>
    <script src="https://cdn.jsdelivr.net/npm/chart.js@2.8.0"></script>
    <script src="/js/app.js" language="javascript"></script>
  </body>
</html>

Style sheet

The CSS asset mostly contains the styling for the article table. It is nothing fancy.

/* public/css/main.css */

#app {
  margin: auto;
  width: 80%;
}

#articles {
  font-family: "Trebuchet MS", Arial, Helvetica, sans-serif;
  border-collapse: collapse;
  width: 100%;
}

#articles td,
#articles th {
  border: 1px solid #ddd;
  padding: 8px;
}

#articles tr:nth-child(even) {
  background-color: #f2f2f2;
}

#articles tr:hover {
  background-color: #ddd;
}

#articles th {
  padding-top: 12px;
  padding-bottom: 12px;
  text-align: left;
  background-color: #5c9ead;
  color: white;
}

The Vue app

The JavaScript asset app.js contains the actual Vue app. It consists of a simple component with a Chart.js canvas and an article table in the template. When the component gets created we will get the stats data from the server and create the actual Chart.js chart.

// public/js/app.js

// Vue App
const App = Vue.component("app", {
  data() {
    return {
      articleStats: [],
      chart: {},
    };
  },
  async created() {
    /* Get stats data */
    // ...
    /* Create Chart.js plot from data */
    // ...
  },

  template: `{% raw %}
  <div id="app">
    <div>
        <canvas id="chart"></canvas>
    </div>
    <table id="articles">
      <tr>
        <th></th>
        <th>Article</th>
        <th>Views</th>
        <th>Reactions</th>
      </tr>
      <tr v-for="article in articleStats">
        <td :style="{'background-color': article.color, width: '10px'}"></td>
        <td><a :href=article.url class="title">{{ article.title }}</a></td>
        <td>{{ article.stats[article.stats.length - 1].views }}</td>
        <td>{{ article.stats[article.stats.length - 1].reactions }}</td>
      </tr>
    </table>
  </div>{% endraw %}
      `,
});

// Mount Vue app
var app = new Vue({
  render: (h) => h(App),
}).$mount("#app");

We fetch the article stats from the /article_stats route on the server. In addition we add a random color to each article that we will use for the line in the chart.

// public/js/app.js
// ...

/* Get stats data */

// Fetch article stats from server
const url = new URL(
  document.location.protocol + "//" + document.location.host + "/article_stats",
);
const articleStatsResp = await fetch(url);
let articleStats = await articleStatsResp.json();

// Assign random color to article
const randomColor = function () {
  var r = Math.floor(Math.random() * 255);
  var g = Math.floor(Math.random() * 255);
  var b = Math.floor(Math.random() * 255);
  return "rgb(" + r + "," + g + "," + b + ")";
};
articleStats.forEach((article) => {
  article.color = randomColor();
});
this.articleStats = articleStats;

// ...

Now we need to transform the stats into a Chart.js config object. We will do so in three steps:

We need the x-axis labels. For this we will use the date fields from the longest stats array of all articles (oldest article).
Then we transform the article stats into datasets Chart.js can plot. Most importantly we need to prepend 0 values to the stats array of newer articles to make sure they are all the same length.
Create a Chart.js config object with all the display options we want.

Once we have the Chart.js config object we create a new chart and mount it in the designated HTML canvas element.

// public/js/app.js
// ...

/* Create Chart.js plot from data */

// Get x-Axis labels
let labels = [];
let minDate = "9"; // This will work for the next ~8000 years
this.articleStats.forEach((article) => {
  if (article.stats[0].date < minDate) {
    minDate = article.stats[0].date;
    labels = article.stats.map((stat) => {
      return stat.date;
    });
  }
});

// Transform article stats into Chart.js datasets
let datasets = this.articleStats.map((article) => {
  let data = [];
  // Fill with 0 until first view
  for (let date of labels) {
    if (date >= article.stats[0].date) {
      break;
    }
    data.push(0);
  }
  // Append views
  data.push(
    ...article.stats.map((stat) => {
      return stat.views;
    }),
  );
  // Return data set for this article
  return {
    label: article.title,
    data: data,
    fill: false,
    borderColor: article.color,
    backgroundColor: article.color,
  };
});

// Chart.js config
let chartConfig = {
  type: "line",
  data: {
    datasets: datasets,
    labels: labels,
  },
  options: {
    responsive: true,
    // aspectRatio: 3,
    title: {
      display: true,
      text: "Dev.to Article Stats",
    },
    legend: {
      // Our table is the legend
      display: false,
    },
    tooltips: {
      mode: "index",
      intersect: false,
    },
    hover: {
      mode: "nearest",
      intersect: true,
    },
    scales: {
      xAxes: [
        {
          display: true,
          scaleLabel: {
            display: true,
            labelString: "Date",
          },
        },
      ],
      yAxes: [
        {
          display: true,
          scaleLabel: {
            display: true,
            labelString: "Views",
          },
        },
      ],
    },
  },
};

// Create chart
let ctx = document.getElementById("chart").getContext("2d");
this.chart = new Chart(ctx, chartConfig);

// ...

Now we have the fronend and the server that serves it. Before we can deploy and run our app we only need a migration script to create the actual tables in the database.

Database migration

The app does not work without a articles and stats tables. We thus need a database migration that adds these.

// migrate.js

var r = require("rethinkdb");

r.connect(
  {
    host: process.env.RETHINKDB_HOST || "localhost",
    port: process.env.RETHINKDB_PORT || 28015,
    username: process.env.RETHINKDB_USERNAME || "admin",
    password: process.env.RETHINKDB_PASSWORD || "",
    db: process.env.RETHINKDB_NAME || "test",
  },
  async function (err, conn) {
    if (err) throw err;
    console.log("Get table list");
    let cursor = await r.tableList().run(conn);
    let tables = await cursor.toArray();

    // Check if articles table exists
    if (!tables.includes("articles")) {
      // Table missing --> create
      console.log("Creating articles table");
      await r.tableCreate("articles").run(conn);
      console.log("Creating articles table -- done");
    }

    // Check if stats table exists
    if (!tables.includes("stats")) {
      // Table missing --> create
      console.log("Creating stats table");
      await r.tableCreate("stats").run(conn);
      console.log("Creating stats table -- done");
      // Create index
      await r
        .table("stats")
        .indexCreate("article_date", [r.row("article_id"), r.row("date")])
        .run(conn);
      console.log("Creating article-date secondary index -- done");
    }

    await conn.close();
  },
);

This migration checks if the tables exists and creates them if they are missing. For the stats table we will also create a secondary index to make sure there is only ever one stats document for the same article_id and date.

Deploy the application to Heroku

To deploy our working application to Heroku we need to create a Procfile. This file basically tells Heroku what processes to run.

// Procfile

release: node migrate.js
web: node index.js
collect: node collect.js

The release and web processes are recognized by Heroku as the command to run upon release and the main web app respectively. The collect process is just a worker process that could have any name.

Deploy the app to Heroku with

$ echo "node_modules/" > .gitignore
$ git add .
$ git commit -m 'Working dev.to stats app'
$ git push heroku master

You will need to manually enable the collect process in your Heroku app. You can do so on the Resources tab.

Wrapping up

With this app running on Heroku I can finally go back to coding and writing articles without missing any move my article stats might be making.

Let me know if this app is useful to you, if there are any bugs, or if there are any features you would want me to add.

Bring RethinkDB's realtime magic to the frontend with GraphQL

Sascha — Mon, 07 Sep 2020 08:44:47 +0000

In a recent post we explored how RethinkDB's built-in reactivity is a perfect fit to write a chat app with Socket.io. In this article you will learn how to use GraphQL subscriptions instead, to access RethinkDB's reactive nature in the frontend.

RethinkDB is a realtime document database. It is easy to use and schema-less, just like MongoDB. In addition, you can subscribe to queries and get notified when data changes, making it the perfect choice for realtime applications.

You can also try the running app, or check out the code repository.

Application setup

We will use a simple Node.js server and a Vue.js frontend. Since the frontend needs to be build, we will create a Vue app with the Vue CLI:

$ vue create -d rethink-chat
$ cd rethink-chat

This will create a Node project, create a Vue.js skeleton, and initialize a git repository.

Prepare a Heroku app

In order to deploy the application to Heroku we need to create a Heroku app:

$ heroku create

We will also need a RethinkDB instance to store and subscribe to the chat messages sent between users. You can do this via the RethinkDB Cloud add-on as follows:

$ heroku addons:create rethinkdb

The RethinkDB Cloud add-on is currently in alpha. Request an invite for your Heroku account email.

Building the server

We will create our server in the server directory. So to start, lets create the directory and install the required dependencies:

$ mkdir server
$ npm install rethinkdb apollo-server-express graphql morgan lorem-ipsum

Now, let us set up the Node.js server. Create an index.js file and add the following server skeleton. We use an Express.js server to serve the frontend and the Apollo GraphQL server to access and subscribe to chat messages.

// server/index.js

// Setup Express server
const express = require("express");
const app = express();
const http = require("http").createServer(app);

// Logging middleware
var morgan = require("morgan");
app.use(morgan("combined"));

// Serve frontend
app.use(express.static("dist"));

// Lazy RethinkDB connection
// ...

// Setup Apollo (GraphQL) server
// ...

// HTTP server (start listening)
const listenPort = process.env.PORT || "3000";
http.listen(listenPort, () => {
  console.log("listening on *:" + listenPort);
});

This skeleton serves a static frontend from the dist folder. This is where the compiled Vue.js app is located which we will create later. In addition our server needs to do three things:

Handle connections to the RethinkDB database
Setup the Apollo server
Create a GraphQL schema including type definitions and resolvers

RethinkDB connection

// server/index.js
// ...

// Lazy RethinkDB connection
var r = require("rethinkdb");
let rdbConn = null;
const rdbConnect = async function () {
  try {
    const conn = await r.connect({
      host: process.env.RETHINKDB_HOST || "localhost",
      port: process.env.RETHINKDB_PORT || 28015,
      username: process.env.RETHINKDB_USERNAME || "admin",
      password: process.env.RETHINKDB_PASSWORD || "",
      db: process.env.RETHINKDB_NAME || "test",
    });

    // Handle close
    conn.on("close", function (e) {
      console.log("RDB connection closed: ", e);
      rdbConn = null;
    });

    console.log("Connected to RethinkDB");
    rdbConn = conn;
    return conn;
  } catch (err) {
    throw err;
  }
};
const getRethinkDB = async function () {
  if (rdbConn != null) {
    return rdbConn;
  }
  return await rdbConnect();
};

On Heroku, the RethinkDB Cloud add-on will set the environment variables. For a locally running instance of RethinkDB, the defaults should work.

Apollo GraphQL server setup

As mentioned earlier, the frontend is static. We do however need to access the data in a chat room. This will be handled by Apollo, the most used GraphQL server.

// server/index.js
// ...

// Setup Apollo (GraphQL) server
const { ApolloServer } = require("apollo-server-express");
const { typeDefs, resolvers } = require("./schema.js");
const graphqlServer = new ApolloServer({
  typeDefs,
  resolvers,
  context: async (arg) => {
    const conn = await getRethinkDB();
    return {
      conn: conn,
    };
  },
});
graphqlServer.applyMiddleware({ app });
graphqlServer.installSubscriptionHandlers(http);

This will create an Apollo server using the type definitions and resolves defined in our schema file (next section). We also connect to RethinkDB and pass the connection to our GraphQL context so it can be used in any incoming request.

Create a GraphQL schema

The main logic of the server resides in defining the GraphQL types and implementing their resolvers. We need to be able to preform three different actions, namely

Query chat messages in a room
Send a chat message to a room
Subscribe to new chat messages in a room

First, we create the GraphQL types. This consists of a Chat message type and the three mentioned actions, namely the chats query, the sendChat mutation, and the chatAdded subscription.

// server/schema.js

// GraphQL type definitions
const { gql } = require("apollo-server-express");
exports.typeDefs = gql`
  type Chat {
    user: String
    msg: String
    roomId: String
    ts: Float
  }

  type Query {
    chats(room: String!): [Chat]
  }

  type Mutation {
    sendChat(user: String!, message: String!, room: String!): Chat
  }

  type Subscription {
    chatAdded(room: String!): Chat
  }
`;

// GraphQL resolvers
// ...

Second, we need to resolve these actions, i.e., implement the code that they invoke. The query and the mutation are fairly straight-forward and are implemented as a simple RethinkDB query. The subscription however, requires an async iterator. This is basically a spell to turn the RethinkDB magic into GraphQL subscription magic. In more earthly terms, the async iterator wraps the RethinkDB change feed so we can subscribe to it via GraphQL.

// server/schema.js

// GraphQL type definitions
// ...

// GraphQL resolvers
const r = require("rethinkdb");
exports.resolvers = {
  Subscription: {
    chatAdded: {
      async subscribe(parent, args, context, info) {
        return new RethinkIterator(
          r.table("chats").filter({ roomId: args.room }),
          context.conn,
        );
      },
    },
  },
  Mutation: {
    async sendChat(root, args, context) {
      const chatMsg = {
        user: args.user,
        roomId: args.room,
        msg: args.message,
        ts: Date.now(),
      };
      await r.table("chats").insert(chatMsg).run(context.conn);
      return chatMsg;
    },
  },
  Query: {
    async chats(parent, args, context, info) {
      const cursor = await r
        .table("chats")
        .filter({ roomId: args.room })
        .orderBy(r.desc("ts"))
        .run(context.conn);
      return await cursor.toArray();
    },
  },
};

// Async iterator to access the RethinkDB change feed
const { $$asyncIterator } = require("iterall");
class RethinkIterator {
  constructor(query, conn) {
    this.cursor = query.changes().run(conn);
  }

  async next() {
    const val = await (await this.cursor).next();
    return { value: { chatAdded: val.new_val }, done: false };
  }

  async return() {
    await (await this.cursor).close();
    return { value: undefined, done: true };
  }

  async throw(error) {
    return Promise.reject(error);
  }

  [$$asyncIterator]() {
    return this;
  }
}

With the server set up, let's move to the frontend.

Creating the frontend

We already created the Vue.js app skeleton we will use for the frontend. However, since our server implements a standard GraphQL backend, you might as well use React or any other frontend framework that supports GraphQL.

Our frontend will use two views, one for the home page and one for the chat room as well as a router to navigate between the two. For this lets add a router to the Vue skeleton and install all required dependencies. Adding a router to the Vue app will warn you about uncommited changes (proceed anyway) and ask if you want history mode (no).

$ vue add router
$ npm install apollo-client apollo-link-http apollo-link-ws apollo-cache-inmemory vue-apollo
$ npm install sass sass-loader --save-dev

Our Vue app is located in the src folder and will be structured as follows: the entry point is in main.js and gets the GraphQL client configuration from graphql.js. Our main file also mounts App.vue which displays views selected by the router in router/index.js. Our app contains two views, views/Home.vue and views/ChatRoom.vue.

src
├── main.js
├── graphql.js
├── App.vue
├── router
│   └── index.js
└── views
    ├── Home.vue
    └── ChatRoom.vue

Main app and router

In a first step, let us modify the main app, home view, and router files that where initialized in the skeleton Vue app. In main.js we import the Apollo GraphQL client we will define further down and add it to our Vue app. In addition we will also create a random chat username for the user.

// src/main.js

import Vue from "vue";
import App from "./App.vue";
import router from "./router";
import apolloProvider from "./graphql";

Vue.config.productionTip = false;

// Initialize random username
window.username = Math.random().toString(36).substring(2, 8);

// Create and mount Vue app
new Vue({
  router,
  apolloProvider,
  render: (h) => h(App),
}).$mount("#app");

Our App.vue is even simpler than the skeleton, it just shows the router view and has some styling.

<!-- src/App.vue -->

<template>
  <div id="app">
    <router-view />
  </div>
</template>

<script>
export default {
  name: "App",
};
</script>

<style lang="scss">
// See styles at https://github.com/mostlytyped/rethink-chat-graphql/blob/master/src/App.vue
</style>

In our router/index.js we basically replace the "About" route with our "Room" route.

// src/router/index.js

import Vue from "vue";
import VueRouter from "vue-router";
import Home from "@/views/Home";
import ChatRoom from "@/views/ChatRoom";

Vue.use(VueRouter);

const routes = [
  { path: "/", name: "Home", component: Home },
  { path: "/:roomId", name: "Room", component: ChatRoom },
];

const router = new VueRouter({
  routes,
});

export default router;

In the home view we remove the HelloWorld component and add a form that allows us to join a room.

<!-- src/views/Home.vue -->

<template>
  <div class="main">
    <form v-on:submit.prevent="gotoRoom">
      <label>
        Username:
        <input v-model="user" type="text" />
      </label>
      <label>
        Room:
        <input v-model="room" type="text" />
      </label>
      <button>Join</button>
    </form>
  </div>
</template>

<script>
export default {
  name: "Home",
  data() {
    return {
      user: window.username,
      room: "lobby",
    };
  },
  methods: {
    gotoRoom() {
      window.username = this.user;
      this.$router.push({
        name: "Room",
        params: { roomId: this.room },
      });
    },
  },
};
</script>

<style scoped lang="scss">
// See styles at https://github.com/mostlytyped/rethink-chat-graphql/blob/master/src/views/Home.vue
</style>

Now that we stuffed the skeleton with the bits an pieces we need, let us tackle the real meat of the frontend, the GraphQL client and the chat room view.

GraphQL client

When our frontend loads we need to initiate the GraphQL client. In our example we use Apollo, the most used GraphQL client, which has good Vue.js integration with the vue-apollo package.

// src/graphql.js

import Vue from "vue";
import VueApollo from "vue-apollo";
import ApolloClient from "apollo-client";
import { createHttpLink } from "apollo-link-http";
import { InMemoryCache } from "apollo-cache-inmemory";
import { split } from "apollo-link";
import { WebSocketLink } from "apollo-link-ws";
import { getMainDefinition } from "apollo-utilities";

Vue.use(VueApollo);

// HTTP connection to the API
const httpLink = createHttpLink({
  // For production you should use an absolute URL here
  uri: `${window.location.origin}/graphql`,
});

// Create the subscription websocket link
const wsLink = new WebSocketLink({
  uri: `wss://${window.location.host}/graphql`,
  options: {
    reconnect: true,
  },
});

// Split link based on operation type
const link = split(
  ({ query }) => {
    const definition = getMainDefinition(query);
    return (
      definition.kind === "OperationDefinition" &&
      definition.operation === "subscription"
    );
  },
  wsLink, // Send subscription traffic to websocket link
  httpLink, // All other traffic to http link
);

// Create apollo client/provider with our link
const apolloClient = new ApolloClient({
  cache: new InMemoryCache(),
  link: link,
});

const apolloProvider = new VueApollo({
  defaultClient: apolloClient,
});

export default apolloProvider;

Since we will use GraphQL subscriptions, our Apollo setup is a bit more complicated than usual. This is because normal GraphQL should be performed over HTTP but subscription updates will be pushed over a WebSocket.

The chat room view

The final piece of the frontend will be the ChatRoom view. Here we actually get to use the GraphQL client we just initialized. This view basically shows a list with all the items in the chats variable and provides a form to send a chat message to the backend.

<!-- src/views/ChatRoom.vue -->
<template>
  <div class="chatroom">
    <ul id="chatlog">
      <li v-for="chat in chats" v-bind:key="chat.ts">
        <span class="timestamp">
          {{
            new Date(chat.ts).toLocaleString(undefined, {
              dateStyle: "short",
              timeStyle: "short",
            })
          }}
        </span>
        <span class="user">{{ chat.user }}:</span>
        <span class="msg">{{ chat.msg }}</span>
      </li>
    </ul>
    <label id="username"> Username: {{ user }} </label>
    <form v-on:submit.prevent="sendMessage">
      <input v-model="message" autocomplete="off" />
      <button>Send</button>
    </form>
  </div>
</template>
<script>
import gql from "graphql-tag";

export default {
  name: "ChatRoom",
  data() {
    return {
      chats: [],
      message: "",
      user: window.username,
      handle: null,
    };
  },
  methods: {
    sendMessage() {
      const msg = this.message;
      this.$apollo.mutate({
        mutation: gql`
          mutation($user: String!, $msg: String!, $room: String!) {
            sendChat(user: $user, room: $room, message: $msg) {
              ts
            }
          }
        `,
        variables: {
          user: this.user,
          msg: msg,
          room: this.$route.params.roomId,
        },
      });
      this.message = "";
    },
  },
  apollo: {
    chats: {
      query: gql`
        query FetchChats($room: String!) {
          chats(room: $room) {
            msg
            user
            ts
          }
        }
      `,
      variables() {
        return {
          room: this.$route.params.roomId,
        };
      },
      subscribeToMore: {
        document: gql`
          subscription name($room: String!) {
            chatAdded(room: $room) {
              msg
              user
              ts
            }
          }
        `,
        variables() {
          return {
            room: this.$route.params.roomId,
          };
        },
        // Mutate the previous result
        updateQuery: (previousResult, { subscriptionData }) => {
          previousResult.chats.unshift(subscriptionData.data.chatAdded);
        },
      },
    },
  },
};
</script>

<style scoped lang="scss">
// See styles at https://github.com/mostlytyped/rethink-chat-graphql/blob/master/src/views/ChatRoom.vue
</style>

The sendMessage method is bound to the sendChat GraphQL mutation. As for the chats variable, the binding is a bit more involved. We bind it to the GraphQL chats query and in addition we use the chatAdded subscription to keep the variable up to date.

Now we have a working server and frontend. The last thing we need is to make sure the chats table actually exists in the RethinkDB database when we run the app.

Database migration

The app does not work without a chats table. We thus need a database migration that adds the table.

// server/migrate.js

var r = require("rethinkdb");

r.connect(
  {
    host: process.env.RETHINKDB_HOST || "localhost",
    port: process.env.RETHINKDB_PORT || 28015,
    username: process.env.RETHINKDB_USERNAME || "admin",
    password: process.env.RETHINKDB_PASSWORD || "",
    db: process.env.RETHINKDB_NAME || "test",
  },
  function (err, conn) {
    if (err) throw err;

    r.tableList().run(conn, (err, cursor) => {
      if (err) throw err;
      cursor.toArray((err, tables) => {
        if (err) throw err;

        // Check if table exists
        if (!tables.includes("chats")) {
          // Table missing --> create
          console.log("Creating chats table");
          r.tableCreate("chats").run(conn, (err, _) => {
            if (err) throw err;
            console.log("Creating chats table -- done");
            conn.close();
          });
        } else {
          // Table exists --> exit
          conn.close();
        }
      });
    });
  },
);

This migration checks if the chats table exists, and if it is missing, it creates it.

A simple chat bot

As we saw, one of RethinkDBs great features is the baked in reactivity that allows us to subscribe to queries. This feature also comes in handy when creating a simple chat bot. The bot simply needs to subscribe to changes in the chats table and react to them whenever appropriate.

Our Lorem bot will reply with a random section of Lorem Ipsum whenever prompted with @lorem. The bot subscribes to the chats table and scans the beginning of the message. If it starts with @lorem, it will reply with a message in the same room.

// server/lorem-bot.js

const LoremIpsum = require("lorem-ipsum").LoremIpsum;
const lorem = new LoremIpsum({
  sentencesPerParagraph: {
    max: 8,
    min: 4,
  },
  wordsPerSentence: {
    max: 16,
    min: 4,
  },
});

// Run Lorem bot
const runBot = function (conn) {
  console.log("Lorem bot started");
  r.table("chats")
    .changes()
    .run(conn, (err, cursor) => {
      if (err) throw err;
      cursor.each((err, row) => {
        const msg = row.new_val.msg.trim().split(/\s+/);
        // Is the message directed at me?
        if (msg[0] === "@lorem") {
          let num = 10;
          if (msg.length >= 1) {
            num = parseInt(msg[1]) || num;
          }
          r.table("chats")
            .insert({
              user: "lorem",
              msg: lorem.generateWords(num),
              roomId: row.new_val.roomId,
              ts: Date.now(),
            })
            .run(conn, function (err, res) {
              if (err) throw err;
            });
        }
      });
    });
};

// Connect to RethinkDB
const r = require("rethinkdb");
const rdbConnect = async function () {
  try {
    const conn = await r.connect({
      host: process.env.RETHINKDB_HOST || "localhost",
      port: process.env.RETHINKDB_PORT || 28015,
      username: process.env.RETHINKDB_USERNAME || "admin",
      password: process.env.RETHINKDB_PASSWORD || "",
      db: process.env.RETHINKDB_NAME || "test",
    });

    // Handle close
    conn.on("close", function (e) {
      console.log("RDB connection closed: ", e);
      setTimeout(rdbConnect, 10 * 1000); // reconnect in 10s
    });

    // Start the lorem bot
    runBot(conn);
  } catch (err) {
    throw err;
  }
};
rdbConnect();

Deploy the application to Heroku

To deploy our working application and bot to Heroku we need to create a Procfile. This file basically tells Heroku what processes to run.

// Procfile

release: node server/migrate.js
web: node server/index.js
lorem-bot: node server/lorem-bot.js

The release and web processes are recognized by Heroku as the command to run upon release and the main web app respectively. The lorem-bot process is just a worker process that could have any name.

Deploy the app to Heroku with

$ git add .
$ git commit -m 'Working rethink-chat app'
$ git push heroku master

You will need to manually enable the lorem-bot process in your Heroku app. You can do so on the Resources tab.

Conclusion

In less than 15 minutes we managed to create and deploy a chat application with a simple bot. This shows the power and ease of use of RethinkDB. The ability to subscribe to queries makes it easy to build a reactive app and can easily be integrated with GraphQL. Further, Heroku makes deployment a breeze, and with the RethinkDB Cloud add-on you will never have to do the tedious work of managing a database server yourself.

Build a Chat app with Socket.IO and RethinkDB

Sascha — Wed, 26 Aug 2020 16:32:40 +0000

A lot of tutorials can be found that teach you how to build a chat app with Socket.io. However, have you ever wondered how to best persist those chat messages?

Enter RethinkDB, a realtime schema-less database. You can store and handle documents easily, just like in MongoDB, but it has reactivity built into it. That means you can subscribe to queries and get notified when data changes, making it the perfect choice when it comes to storing chat messages.

In this article you will learn how to create a simple chat app with Socket.io and persist the messages in RethinkDB. To show the usefulness of a reactive database, we will also add a simple bot that reacts when you address it.

You can also try the running app, or check out the code repository.

Application setup

To create the application, run the following in a terminal.

$ mkdir rethink-chat && cd rethink-chat
$ npm init -y
$ npm install rethinkdb express morgan http socket.io lorem-ipsum

This will initialize a Node.js app and install all required dependencies.

Prepare a Heroku app

In order to deploy the application to Heroku we need to create a Heroku app:

$ git init
$ heroku create

We will also need a RethinkDB instance to store and subscribe to the chat messages sent between users. You can do this via the RethinkDB Cloud add-on as follows:

$ heroku addons:create rethinkdb

The RethinkDB Cloud add-on is currently in alpha. Request an invite for your Heroku account email.

Building the server

To begin, let us set up the Node.js server. Create an index.js file and add the following server skeleton. We use an Express.js server to handle http traffic and Socket.io to handle WebSocket connections with clients.

// index.js

// Setup Express and Socket.io servers
var express = require("express");
const app = express();
var http = require("http").createServer(app);
var io = require("socket.io")(http);

// Logging middleware
var morgan = require("morgan");
app.use(morgan("combined"));

// Serve frontend
app.use(express.static("public"));

// Keep track of room subscriptions in RethinkDB
const watchedRooms = {};

// Lazy RethinkDB connection
// ...

// Route to access a room
// ...

// Socket.io (listen for new messages in any room)
// ...

// HTTP server (start listening)
const listenPort = process.env.PORT || "3000";
http.listen(listenPort, () => {
  console.log("listening on *:" + listenPort);
});

This skeleton serves a static frontend from the public folder. We will create the frontend code later. In addition our server needs to do three things:

Handle connections to the RethinkDB database
Create an Express.js route that will give a user access to the chat room
Configure the Socket.io server to listen to incoming chat messages

RethinkDB connection

// index.js
// ...

// Lazy RethinkDB connection
var r = require("rethinkdb");
let rdbConn = null;
const rdbConnect = async function () {
  try {
    const conn = await r.connect({
      host: process.env.RETHINKDB_HOST || "localhost",
      port: process.env.RETHINKDB_PORT || 28015,
      username: process.env.RETHINKDB_USERNAME || "admin",
      password: process.env.RETHINKDB_PASSWORD || "",
      db: process.env.RETHINKDB_NAME || "test",
    });

    // Handle close
    conn.on("close", function (e) {
      console.log("RDB connection closed: ", e);
      rdbConn = null;
    });

    console.log("Connected to RethinkDB");
    rdbConn = conn;
    return conn;
  } catch (err) {
    throw err;
  }
};
const getRethinkDB = async function () {
  if (rdbConn != null) {
    return rdbConn;
  }
  return await rdbConnect();
};

On Heroku, the RethinkDB Cloud add-on will set the environment variables. For a locally running instance of RethinkDB, the defaults should work.

Route to access room

As mentioned earlier, the frontend is static. We do however need a route to access a chat room. The route will return the message history of a given room, as well as a WebSocket handle to access it.

// index.js
// ...

// Route to access a room
app.get("/chats/:room", async (req, res) => {
  const conn = await getRethinkDB();

  const room = req.params.room;
  let query = r.table("chats").filter({ roomId: room });

  // Subscribe to new messages
  if (!watchedRooms[room]) {
    query.changes().run(conn, (err, cursor) => {
      if (err) throw err;
      cursor.each((err, row) => {
        if (row.new_val) {
          // Got a new message, send it via Socket.io
          io.emit(room, row.new_val);
        }
      });
    });
    watchedRooms[room] = true;
  }

  // Return message history & Socket.io handle to get new messages
  let orderedQuery = query.orderBy(r.desc("ts"));
  orderedQuery.run(conn, (err, cursor) => {
    if (err) throw err;
    cursor.toArray((err, result) => {
      if (err) throw err;
      res.json({
        data: result,
        handle: room,
      });
    });
  });
});

This is where the RethinkDB magic happens. The first time this route is called for a particular room (when the first person joins), we subscribe to a RethinkDB query to get notified whenever a new chat message is available. We send new chat messages via Socket.io to any clients listening for the room's handle.

Listen for new messages

The last puzzle piece of the server is to listen and save all incoming chat messages. Whenever a message comes in via the chats handle of the Socket.io connection, we save it to the chats table in RethinkDB.

// index.js
// ...

// Socket.io (listen for new messages in any room)
io.on("connection", (socket) => {
  socket.on("chats", async (msg) => {
    const conn = await getRethinkDB();
    r.table("chats")
      .insert(Object.assign(msg, { ts: Date.now() }))
      .run(conn, function (err, res) {
        if (err) throw err;
      });
  });
});

Saving a value in the chats table will trigger the subscription we added above, causing the message to be sent to all clients listening to this room, including the sender of the message.

Frontend

The layout file only serves as a mount point for the Vue app in addition to importing the dependencies.

  <!-- public/index.html -->

  <!DOCTYPE html>
  <html lang="en">
    <head>
      <meta charset="UTF-8" />
      <meta name="viewport" content="width=device-width, initial-scale=1.0" />
      <title>RethinkDB Chat with SocketIO</title>
      <link href="/css/main.css" rel="stylesheet" />
    </head>

    <body>
      <div id="app">
        <router-view></router-view>
      </div>
      <script src="/socket.io/socket.io.js"></script>
      <script src="https://cdn.jsdelivr.net/npm/vue/dist/vue.js"></script>
      <script src="https://unpkg.com/vue-router/dist/vue-router.js"></script>
      <script src="/js/app.js" language="javascript"></script>
    </body>
  </html>

The CSS asset contains the styling of the frontend. It is long, not very interesting, and can be found here.
The JavaScript asset app.js contains the actual Vue app.

  // public/js/app.js

  // Create random username
  let username = Math.random().toString(36).substring(2, 8);

  // Setup Socket.io
  var socket = io();

  // Main view
  // ...

  // Room view, holds a chat room component
  // ...

  // Chat room component
  // ...

  // Setup routes
  const router = new VueRouter({
    routes: [
      { path: "/", component: MainView },
      { path: "/:roomId", name: "room", component: RoomView },
    ],
  });

  // Mount Vue app
  var app = new Vue({
    router,
  }).$mount("#app");

The Vue app contains two routes. The / path points to the main view and the /:roomId path points to the room view.

Main view

The main view lets you choose a username (default is a random string) and allows you to join a room with a given name.

// public/js/app.js
// ...

// Main view
const MainView = Vue.component("main-view", {
  data() {
    return {
      room: "lobby",
      user: username,
    };
  },
  methods: {
    gotoRoom() {
      username = this.user;
      this.$router.push({ name: "room", params: { roomId: this.room } });
    },
  },
  template: `
<div class="main">
    <form class="main" v-on:submit.prevent="gotoRoom">
    <label>Username: <input v-model="user" type="text" /></label>
    <label>Room: <input v-model="room" type="text" /></label>
    <button>Join</button>
    </form>
</div>
    `,
});

Whenever you join a room, the Vue router will load the chat room view.

Chat room

The chat room, a room view containing a chat room component. makes a request to the Express route to join the given room when it is created. It also registers a Socket.io handler that listens for incoming chat messages and adds them to the list of messages.

The chat room allows the user to type and send a message which will then be sent to the server via the WebSocket handled by Socket.io.

// public/js/app.js
// ...

// Room view, holds a chat room component
const RoomView = Vue.component("room-view", {
  template: `<chat-room :roomId="$route.params.roomId"/>`,
});

// Chat room component
const ChatRoom = Vue.component("chat-room", {
  props: ["roomId"],
  data() {
    return {
      chats: [],
      message: "",
      username: username,
      handle: null,
    };
  },
  async created() {
    const url = new URL(document.location.protocol + "//" + document.location.host + "/chats/" + this.roomId);
    const chatsResp = await fetch(url);
    const { data, handle } = await chatsResp.json();
    this.chats = data;
    this.handle = handle;
    socket.on(this.handle, (msg) => {
      this.chats.unshift(msg);
    });
  },
  beforeDestroy() {
    socket.off(this.handle);
  },
  methods: {
    sendMessage() {
      socket.emit("chats", {
        msg: this.message,
        user: this.username,
        roomId: this.roomId,
      });
      this.message = "";
    },
  },
  template: `
<div class="chatroom">
    <ul id="chatlog">
        <li v-for="chat in chats">
            <span class="timestamp">
                {{ new Date(chat.ts).toLocaleString(undefined, {dateStyle: 'short', timeStyle: 'short'}) }}
            </span>
            <span class="user">{{ chat.user }}:</span>
            <span class="msg">{{ chat.msg }}</span>
        </li>
    </ul>
    <label id="username">Username:
        {{ username }}
    </label>
    <form v-on:submit.prevent="sendMessage">
        <input v-model="message" autocomplete="off" />
        <button>Send</button>
    </form>
</div>
    `,
});

Now we have a working server and frontend. The last thing we need is to make sure the chats table actually exists in the RethinkDB database when we run the app.

Database migration

The app does not work without a chats table. We thus need a database migration that adds the table.

// migrate.js

var r = require("rethinkdb");

r.connect(
  {
    host: process.env.RETHINKDB_HOST || "localhost",
    port: process.env.RETHINKDB_PORT || 28015,
    username: process.env.RETHINKDB_USERNAME || "admin",
    password: process.env.RETHINKDB_PASSWORD || "",
    db: process.env.RETHINKDB_NAME || "test",
  },
  function (err, conn) {
    if (err) throw err;

    r.tableList().run(conn, (err, cursor) => {
      if (err) throw err;
      cursor.toArray((err, tables) => {
        if (err) throw err;

        // Check if table exists
        if (!tables.includes("chats")) {
          // Table missing --> create
          console.log("Creating chats table");
          r.tableCreate("chats").run(conn, (err, _) => {
            if (err) throw err;
            console.log("Creating chats table -- done");
            conn.close();
          });
        } else {
          // Table exists --> exit
          conn.close();
        }
      });
    });
  },
);

This migration checks if the chats table exists, and if it is missing, it creates it.

A simple chat bot

// lorem-bot.js

const LoremIpsum = require("lorem-ipsum").LoremIpsum;
const lorem = new LoremIpsum({
  sentencesPerParagraph: {
    max: 8,
    min: 4,
  },
  wordsPerSentence: {
    max: 16,
    min: 4,
  },
});

// Run Lorem bot
const runBot = function (conn) {
  console.log("Lorem bot started");
  r.table("chats")
    .changes()
    .run(conn, (err, cursor) => {
      if (err) throw err;
      cursor.each((err, row) => {
        const msg = row.new_val.msg.trim().split(/\s+/);
        // Is the message directed at me?
        if (msg[0] === "@lorem") {
          let num = 10;
          if (msg.length >= 1) {
            num = parseInt(msg[1]) || num;
          }
          r.table("chats")
            .insert({
              user: "lorem",
              msg: lorem.generateWords(num),
              roomId: row.new_val.roomId,
              ts: Date.now(),
            })
            .run(conn, function (err, res) {
              if (err) throw err;
            });
        }
      });
    });
};

// Connect to RethinkDB
const r = require("rethinkdb");
const rdbConnect = async function () {
  try {
    const conn = await r.connect({
      host: process.env.RETHINKDB_HOST || "localhost",
      port: process.env.RETHINKDB_PORT || 28015,
      username: process.env.RETHINKDB_USERNAME || "admin",
      password: process.env.RETHINKDB_PASSWORD || "",
      db: process.env.RETHINKDB_NAME || "test",
    });

    // Handle close
    conn.on("close", function (e) {
      console.log("RDB connection closed: ", e);
      setTimeout(rdbConnect, 10 * 1000); // reconnect in 10s
    });

    // Start the lorem bot
    runBot(conn);
  } catch (err) {
    throw err;
  }
};
rdbConnect();

Deploy the application to Heroku

To deploy our working application and bot to Heroku we need to create a Procfile. This file basically tells Heroku what processes to run.

// Procfile

release: node migrate.js
web: node index.js
lorem-bot: node lorem-bot.js

Deploy the app to Heroku with

$ echo "node_modules/" > .gitignore
$ git add .
$ git commit -m 'Working rethink-chat app'
$ git push heroku master

You will need to manually enable the lorem-bot process in your Heroku app. You can do so on the Resources tab.

Conclusion

In less than 15 minutes we managed to create and deploy a chat application with a simple bot. This shows the power and ease of use of RethinkDB. The ability to subscribe to queries makes it easy to build a reactive app and a natural fit to interact with Socket.io. Further, Heroku makes deployment a breeze, and with the RethinkDB Cloud add-on you will never have to do the tedious work of managing a database server yourself.