Forem: Oladeji Femi

Creating a slack game using Serverless architecture & AWS - part 2

Oladeji Femi — Tue, 03 Mar 2020 18:29:07 +0000

In the previous article, we did a brief introduction to what serverless architecture is. We created a development environment with three lambda functions for starting a game, submitting words and ending the game. In this part, we’ll finalize what we started by making our app available online on AWS and create a Slack app so that people can interact with the game directly via slack.

We'll have to deploy our app to AWS and a new script command has to be added to package.json for that. The --stage option is used to specify the stage the app will be deployed to. You can use this to create a staging environment to test features and perform QA to prevent bugs from being deployed to prod.

"scripts": {
   "dev": "serverless offline start",
   "production": "serverless deploy --stage prod"
 }

The db.js file was configured to work with local DynamoDB but we need to modify it to support AWS DynamoDB also. The option object when running in the production environment can be an empty object but in development, it needs to be updated as in the snippet below. The snippet will replace the option declaration that was initially in the file.

let option = {};

if (process.env.NODE_ENV === 'development') {
 option = {
   region: 'localhost',
   endpoint: 'http://localhost:8000'
 };
}

Let’s run the yarn production command to get it deployed online. If the deployment was successful you should get the URL for two endpoints (start_game and submit lambda functions).

With the endpoints, we can get started with the creation of the app on Slack. Go to https://api.slack.com/apps and click on the Create New App.

To get our apps fully working, we need some slack permissions.

The first permission we need is to use the slash command. It will be used to start a new game. Click on the slash command link on the basic information tab. Enter any slash command of your choice and use the /start endpoint link as the redirect URL and save the entered details.

Go back to the basic information tab and click on event subscription. We need this to know when users submit words. In order to enable this, let’s modify the submit lambda function to handle slack's test POST request using the snippet below

module.exports.submit = async (event, _context, callback) => {
  const { body } = event;
  const { event: message, challenge } = JSON.parse(body);
  if (challenge) {
    // this is for slack verification
    return respond(callback, 200, challenge);
  }
  if (!message.thread_ts || message.text.trim().split(' ').length > 1) {
    return callback(null, { statusCode: 200 });
  }
  try {
    const id = `${message.team}${message.channel}`;
    await db.addWords(id, {
      user: message.user,
      word: message.text,
    });
    return callback(null, { statusCode: 200 });
  } catch (error) {
    console.log(error);
    if (error.code === 'ConditionalCheckFailedException') {
      return callback(null, { statusCode: 200, body: 'Game has ended' });
    }
    return callback(null, { statusCode: 200, body: 'An error occurred while ending the game' });
  }
};

Let's deploy the newly modified function with this command.

npx serverless deploy --stage prod --function submit_words

The /submit endpoint can now be set as the redirect url. Click on the workspace event and choose the message.channels event. With this being set up, every message posted on a public channel makes a POST request to the endpoint that was set up.

We need two more permissions to get things started. Click on the OAuth & Permissions link and choose chat.write:bot (to post the result on the slack channel) and users:read (to retrieve the full name of users who participated in a game).

We're almost there, we just need a few more modifications to get things going. In order to automatically invoke the end lambda function 60 seconds after a game is started, the AWS SQS will be used. We need to also give permissions to our lambda functions to use SQS and write/read data from our DynamoDB. Under the resources section in serverless.yml file. Add this snippet. It should be aligned with the gamesTable resource yml specification

gameQueue:
  Type: AWS::SQS::Queue
  Properties: 
    QueueName: games
    DelaySeconds: 60
    MessageRetentionPeriod: 120

Redeploy the app again by using the yarn production command. After the deployment, go to the SQS management console on AWS and select the queue. Copy the queue URL and ARN and create a new env variable QUEUE_URL with the URL copied as well as the QUEUE_ARN. The end game function has to be triggered whenever a message is delivered in the queue. Modify the end_game function declaration in the serverless yml to this snippet.

 end_game:
   handler: game.end
   name: end_game
   timeout: 20
   events:
     - sqs: ${env:QUEUE_ARN}

In the game.js file, the aws-sdk and dotenv package must be required.

const aws = require('aws-sdk');
require('dotenv').config();

Once a game is started, a message needs to be added to the queue that will be delivered in exactly 60 seconds due to the queue configuration. Add the snippet below after the await db.insert(gameItem) statement.

    await new aws.SQS().sendMessage({
      QueueUrl: process.env.QUEUE_URL,
      MessageBody: JSON.stringify(gameItem),
    }).promise();

The end lambda function will receive the event on the queue after 60 seconds. SQS messages are in a different format so the first statement in the function const game = event has to be changed to

  const game = JSON.parse(event.Records[0].body);

When users submit words, Slack only sends the user ID and we need a way to retrieve the user’s full name and profile picture also. The users:read permission allows us to do this. All we have to do is make a GET request to https://slack.com/api/users.info and pass the user ID and workspace token as query parameters. To get the workspace token, go to the auth & permissions link on the Slack dashboard and click on Install App to Workspace. Copy the access token and it as a new env variable TOKEN in the .env file. Let’s add a function to our app.js file specifically for this. It receives an object of userid:score pair and retrieves the user details. Make sure you install the axios npm package and require it in the file. Also, configure the dotenv package as we did in the game.js file

getUsers(users) {
    return new Promise(async (resolve) => {
      const slackUrl = `https://slack.com/api/users.info?token=${process.env.TOKEN}&user=`;
      const detailsRequest = Object.keys(users).map(each => axios.get(`${slackUrl}${each}`));
      let finalScore = await Promise.all(detailsRequest);
      finalScore = finalScore.map(({ data: { user }, status }) => {
        if (status === 200) {
          return {
            type: 'section',
            fields: [{
              type: 'plain_text',
              text: 'Name:',
            },
            {
              type: 'plain_text',
              text: user.real_name,
              emoji: true,
            },
            {
              type: 'plain_text',
              text: 'Username:',
            },
            {
              type: 'plain_text',
              text: user.name,
              emoji: true,
            },
            {
              type: 'plain_text',
              text: 'Score:',
            },
            {
              type: 'plain_text',
              text: `${users[user.id].totalScore}`,
            },
            {
              type: 'plain_text',
              text: 'words:',
            },
            {
              type: 'mrkdwn',
              text: users[user.id].words,
            }],
            accessory: {
              type: 'image',
              image_url: user.profile.image_72,
              alt_text: user.real_name,
            },
          };
        }
        return {};
      });
      resolve(finalScore);
    });
  }

All we have to do now is call this function from the computeResult function by replacing this statement const results = this.groupByUser(score); with

const results = await this.getUsers(this.groupByUser(score));

Another thing we need to do is post the result of each game to the Slack channel and to achieve that, the end game lambda function has to be modified. Replace the if(words.length) block with this snippet. Ensure that axios module has been required in this file (game.js)

if (words.length) {
      const results = await app.computeResults(words, letters);
      axios.post(game.response_url, JSON.stringify({
        response_type: 'in_channel',
        blocks: results,
      }), {
        headers: {
          Authorization: `Bearer ${process.env.TOKEN}`,
        },
      });
    }
    callback(null, {
      statusCode: 200,
    });

The last thing we need to do is give permissions to the lambda functions created to use some AWS resources (DynamoDB and SQS). The DynamoDB games table ARN has to be gotten from AWS. Go to the DynamoDB section on AWS, click on tables, select the games table and copy the Amazon Resource Name (ARN). Add it as DYNAMO_ARN env variable to the .env file. Now add this snippet to serverless.yml file just under the runtime property

iamRoleStatements: # permissions for all of your functions can be set here
    - Effect: Allow
      Action: # Gives permission to DynamoDB tables in a specific region
        - dynamodb:GetItem
        - dynamodb:PutItem
        - dynamodb:UpdateItem
        - dynamodb:DeleteItem
      Resource: ${env:DYNAMO_ARN}
    - Effect: Allow
      Action:
        - sqs:SendMessage
        - sqs:ReceiveMessage
      Resource: ${env:QUEUE_ARN}

Let's redeploy the app and see if we got everything right. Open your slack channel and use the slash command you created earlier. Submit as many words as you can by replying to the thread and wait for the result to be posted after 60 seconds of starting the game. If anything doesn't work quite right you can check Cloudwatch log groups on AWS to see if any error was logged.

Let’s do a quick recap of what we’ve done over the two posts. We’ve explored what serverless architecture is all about, we were able to set up the app locally and test some functions. We then created a Slack app and got the app set up on AWS so that the game can be played with other teammates on Slack.

P.S
Slack has released a way to give more granular permissions. Also, the conversations API has removed the need to subscribe to events on the Slack workspace. I've released a new version of the app to the Slack app directory with the changes.

If you have any questions or something isn’t working quite right, please leave a comment.
Complete codebase can be found here
To install the game on your slack workspace, go to http://slackwords.com/

Creating a slack game using Serverless architecture & AWS - part 1

Oladeji Femi — Tue, 03 Mar 2020 18:22:11 +0000

What is Serverless Architecture

Serverless architecture is a cloud computing model where the provider handles the infrastructure for your application. It allows you to develop applications without thinking so much about spinning up servers and it's related complexities. Serverless architecture aims to revolutionize the way applications are developed and maintained by giving developers the flexibility not to worry so much about infrastructure (cost and maintenance) but on the application itself.

There are two major types of serverless architecture; Backend as a service and Function as a service. Examples of BAAS are Firebase and Parse server. We'll be making use of FAAS in this post. In FAAS, your code runs in stateless containers and are triggered by pre-configured events such as HTTP requests, database reads/writes, scheduled events, etc.

Which game are we building

It's an interactive game called Wordsgame. Random letters are posted on a slack channel and members of the channel can respond with valid English words within 60 seconds. Each word is scored based on its length and the user with the highest score wins. Sounds interesting right?

Brief overview

A Slack slash command will be used to start a game. The command will be connected to an endpoint on the AWS API gateway. The endpoint will be set as a trigger for a lambda function that is responsible for starting a game. Once a game is started, the details are inserted in a DynamoDB and a message gets placed on a queue with a delay of 60 seconds (duration of the game). A slack event is also set up that listens to messages posted on the channel and makes a post request to another endpoint connected to a lambda function that will be responsible for saving every word users respond with. At the end of the 60 seconds, the message on the queue will trigger another lambda function that executes the functionality for closing out the game, calculating scores and announcing winners.

How do we do this?

We'll be using the serverless framework, NodeJs, AWS, and of-course Slack APIs 🙂 to achieve this.
Let's start by getting a cup of coffee ☕ cos we're in for a ride. Now create a folder in your favorite directory and let's initialize our node environment by running npm init or yarn init. We need to install the serverless node package yarn add serverless -D. The next thing is to create a new project/service and to do that run npx serverless create --template aws-nodejs. You could run serverless create --template aws-nodejs if you installed the package globally and follow the prompt. Once the command has finished executing, you should see the handler.js and serverless.yml files. The yaml file contains the configuration for your application that will eventually be transformed into AWS CloudFormation templates. The functions property has hello and the handler is set to handler.hello. It implies that when the hello lambda function is invoked, the hello function in the handler.js file is executed. How about we test that out? With serverless framework, you can invoke a function locally. To do that run the command npx serverless invoke local --function hello. Take a look at the handler file to see what it does. Before we can continue with the development, we have to set up an account on AWS and configure the serverless framework to use the access keys. Follow this instruction to set it up. The serverless framework will use the new user for all AWS activities like creating DynamoDB, lambda functions, and setting up the queue service.

Now that we've confirmed our environment is set up. Let's create a new function that will be used to start a new game. Edit the serverless.yml to contain the snippet below

service: wordsgame
plugins:
  - serverless-dynamodb-local
  - serverless-dotenv-plugin
  - serverless-offline

provider:
  name: aws
  runtime: nodejs10.x
  region: us-east-2

functions:
  start_game:
    handler: game.start
    name: start_game
    timeout: 3
    events:
      - http:
          path: start
          method: post

resources:
  Resources:
    gamesTable:
      Type: AWS::DynamoDB::Table
      Properties:
        TableName: ${env:DYNAMO_TABLE_NAME}
        AttributeDefinitions:
          - AttributeName: id
            AttributeType: S
        KeySchema:
          - AttributeName: id
            KeyType: HASH
        ProvisionedThroughput:
          ReadCapacityUnits: 1
          WriteCapacityUnits: 1

custom:
  dynamodb:
    stages:
      - dev
    start:
      migrate: true

package:
  exclude:
    -  dynamodb/**

Looks like a lot right? Let's analyze it a little. The plugins property is used to add some extra features to the serverless framework. We need to install those plugins using yarn add -D serverless-offline serverless-dynamodb-local serverless-dotenv-plugin. The serverless-offline emulates the AWS API gateway and Lambda function locally. Serverless-dynamodb-local allows us to use dynamo database locally while serverless-dotenv-plugin works like the dotenv package by allowing us to use variables in a .env file in the serverless yaml configuration.
In the functions property, we've created a new lambda function with a timeout of 3 seconds that can be invoked with a post request
In the resources section we've set up a dynamodb with a required id attribute and the table name has been saved in a .env file like this

DYNAMO_TABLE_NAME=games

The handler for the start_game function must be created in a game.js file like in the snippet below. Ensure the qs node package is installed (yarn add qs)

const qs = require('qs');
const db = require('./utils/db');
const app = require('./utils/app');

const respond = (callback, statusCode, body) => callback(null, {
  statusCode,
  body,
});

module.exports.start = async (event, _context, callback) => {
  const { body } = event;
  const gameItem = qs.parse(body);
  try {
    gameItem.id = `${gameItem.team_id}${gameItem.channel_id}`;
    gameItem.start = Date.now();
    gameItem.letters = app.generateLetters();
    gameItem.active = true;
    gameItem.words = [];
    gameItem.thread = ' ';
    delete gameItem.text;
    delete gameItem.token;
    delete gameItem.command;
    await db.insert(gameItem);
    return respond(callback, 200, JSON.stringify({
      text: `Game started, type as many English words in the thread within 60 seconds using \`${gameItem.letters}\``,
      response_type: 'in_channel',
    }));
  } catch (error) {
    console.log(error);
    return respond(callback, 200, JSON.stringify({
      text: 'Game was not started',
      response_type: 'ephemeral',
    }));
  }
};

Don't get overwhelmed, let's walk through the code. This is the lambda function that will be invoked when users use the slash command to start a game. A POST request will be made to an endpoint with the content-type header set as application/x-www-form-urlencoded. This is what a sample payload looks like according to the documentation.

token=gIkuvaNzQIHg97ATvDxqgjtO
&team_id=T0001
&team_domain=example
&enterprise_id=E0001
&enterprise_name=Globular%20Construct%20Inc
&channel_id=C2147483705
&channel_name=test
&user_id=U2147483697
&user_name=Steve
&command=/weather
&text=94070
&response_url=https://hooks.slack.com/commands/1234/5678
&trigger_id=13345224609.738474920.8088930838d88f008e0

Create a folder named utils and add two new files db.js and app.js with the code snippet below.

// db.js
const AWS = require('aws-sdk');
require('dotenv').config();

const option = {
  region: 'localhost',
  endpoint: 'http://localhost:8000'
};

module.exports = {
    insert(data) {
      return new Promise((resolve, reject) => {
        new AWS.DynamoDB.DocumentClient(option).put({
          TableName: process.env.DYNAMO_TABLE_NAME,
          Item: data,
        }, (error) => {
          if (error) {
            return reject(error);
          }
          return resolve(data);
        });
      });
    }
}

// app.js
const vowels = ['a', 'e', 'i', 'o', 'u'];
const consonants = ['b', 'c', 'd', 'f', 'g', 'h', 'j', 'k', 'l', 'm', 'n', 'p', 'q', 'r', 's', 't', 'w', 'x', 'y', 'z'];
const min = 15;
const max = 20;

const randomNumber = maxNum => Math.floor(Math.random() * maxNum);

module.exports = {
  generateLetters() {
    const length = Math.floor(Math.random() * (max - min + 1) + min);
    let shuffled = '';
    for (let i = 0; i < length; i += 1) {
      if (i % 4) {
        shuffled += `${consonants[randomNumber(consonants.length)]} `;
      } else {
        shuffled += `${vowels[randomNumber(vowels.length)]} `;
      }
    }
    return shuffled.trim().toUpperCase();
  },
}

The lambda function will create an id by combining the team_id and channel_id. The function will also call the app.generateLetters() function that is responsible for generating random characters. db.insert() inserts the record into the dynamo database.
It's time to run this and see if we’re still on the right track. We need to start by setting up our local db. Run npx serverless dynamodb install. Then add the snippet below to your script property in the package.json file so that the yarn dev or npm run dev command can be used. Ensure you have java development kit installed on your system (dynamodb requires it to run locally)

"scripts": {
  "dev": "serverless offline start"
},

Running the command above will set up a local endpoint and also make our db ready to receive values. We can test everything by making a post request using any REST client (e.g insomnia or postman) to http://localhost:3000/start using the slack sample payload above. Make sure the Content-Type header is set to application/x-www-form-urlencoded. If everything works perfectly you should get something like this as the response.

{
  "text": "Game started, type as many English words in the thread within 60 seconds using `U S Z N A G H Y U K K F I W N X I K G X`",
  "response_type": "in_channel"
}

This is the message that will get posted to our Slack channel. To confirm that the record was inserted in the database, open http://localhost:8000/shell/ on your browser to access the dynamodb shell interface. Paste the code below on the interface and hit the play button.

var params = {
    TableName: 'games'
};
dynamodb.scan(params, function(err, data) {
    if (err) ppJson(err); // an error occurred
    else ppJson(data); // successful response
});

You should see the record for the game we just started.

Starting a game is definitely not the EndGame but we’re also not in an Infinity War😉. We need a new lambda function to save users’ responses. Add the snippet below to the functions property of your yaml file (be mindful of the indentation)

  submit_words:
    handler: game.submit
    name: submit_words
    timeout: 3
    events:
      - http:
          path: submit
          method: post

In the game.js file, add the submit function with the snippet below.

module.exports.submit = async (event, _context, callback) => {
  const { body } = event;
  const { event: message } = JSON.parse(body);
  if (!message.thread_ts || message.text.trim().split(' ').length > 1) {
    return callback(null, { statusCode: 200 });
  }
  try {
    const id = `${message.team}${message.channel}`;
    await db.addWords(id, {
      user: message.user,
      word: message.text,
    });
    return callback(null, { statusCode: 200 });
  } catch (error) {
    if (error.code === 'ConditionalCheckFailedException') {
      return callback(null, { statusCode: 200, body: 'Game has ended' });
    }
    return callback(null, { statusCode: 200, body: 'An error occurred' });
  }
};

The event body sent by slack has to be parsed into JSON. Each word submitted by the user in the message thread will be added to the words array in our database. Also, we need to add a new function to our db.js file. The function will check if there's an existing game and push the new word into the array.

  addWords(id, word) {
      return new Promise((resolve, reject) => {
        new AWS.DynamoDB.DocumentClient(option).update({
          TableName: process.env.DYNAMO_TABLE_NAME,
          Key: { id },
          ConditionExpression: 'active = :status',
          UpdateExpression: 'set words = list_append(words, :word)',
          ExpressionAttributeValues: {
            ':word': [word],
            ':status': true
          },
        }, (err, data) => {
          if (err) {
            return reject(err);
          }
          return resolve(data);
        });
      });
    }

Alright let's test the new function. Run the dev script command (you'll need to start a new game because your db gets migrated; dynamodb tables are recreated every time you restart the app) and make a POST request to http://localhost:3000/submit with content-type header as application/json using the payload below according to slack documentation

{
    "event": {
        "type": "message_channel",
        "event_ts": "1234567890.123456",
        "user": "U1234567",
        "text": "test",
        "thread_ts": "123456789",
        "team": "T0001",
        "channel": "C2147483705"
    }
}

Run the dynamodb shell command we used earlier to see the new word you added.

You're probably wondering when will we start interacting with the Slack app. We're almost there, let's try to do as many things as possible locally before we move to AWS and Slack APIs.

The last thing we need is the function to end the game and score users. This is a crucial function and it’s a bit more complicated than what we’ve done earlier. A new function has to be added to the serverless.yml file

  end_game:
    handler: game.end
    name: end_game
    timeout: 20

We don't need to set up a HTTP event because we'll be invoking the function 60 seconds after the start game function has been invoked. In the local environment we can't achieve this effectively so we'll use the invoke-local command from the serverless framework. Now it's time to see what the end game function looks like.

module.exports.end = async (event, context, callback) => {
  const game = event;
  try {
    const { Attributes: { letters, words } } = await db.endGame(game.id);
    if (words.length) {
      const results = await app.computeResults(words, letters);
      callback(null, {
        statusCode: 200,
        body: JSON.stringify(results)
      });
    }
  } catch (error) {
    console.log(error);
    callback(error, {
      statusCode: 500,
    });
  }
}

Add this function to the db.js util file

endGame(id) {
  return new Promise((resolve, reject) => {
    new AWS.DynamoDB.DocumentClient(option).update({
      TableName: process.env.DYNAMO_TABLE_NAME,
      Key: { id },
      UpdateExpression: 'set active = :status',
      ReturnValues: 'ALL_NEW',
      ExpressionAttributeValues: {
        ':status': false,
      },
    }, (err, data) => {
      if (err) {
        return reject(err);
      }
      return resolve(data);
    });
  });
}

Add this to the app.js file to compute the result.

computeResults(entries, alphabets, token) {
    return new Promise(async (resolve, reject) => {
      const foundWords = [];
      let dictionaryCheck = entries.map(({ word }) => {
        if (foundWords.includes(word)) {
          // someone has already entered the word
          return Promise.resolve({
            status: 400,
          });
        }
        foundWords.push(word);
        return Promise.resolve({
          status: 200,
        });
      });
      try {
        dictionaryCheck = await Promise.all(dictionaryCheck);
        const score = entries.map((each, index) => {
          const { status } = dictionaryCheck[index];
          let wordValue = 0;
          if (status === 200) {
            wordValue = each.word.length;
          }
          return {
            user: each.user,
            score: wordValue,
            word: status === 200 ? each.word : `~${each.word}~`,
          };
        });
        const results = this.groupByUser(score);
        resolve(results);
      } catch (error) {
        reject(error);
      }
    });
  },
  groupByUser(scores) {
    const users = {};
    scores.forEach(({ user, score, word }) => {
      if (!users[user]) {
        users[user] = {
          totalScore: 0,
          words: '',
        };
      }
      users[user].totalScore += score;
      users[user].words += `${users[user].words === '' ? '' : ', '}${word}: ${score}`;
    });
    return users;
  },

Let's walk through the code to understand what's happening. When the end_game function is invoked, we pass the id of the game. The db.endGame function is called and it checks to see if there is an active game and then updates the status to false to indicate the game has ended. The app.computeResults function takes all users' submissions and ensures that the same word doesn't get scored twice. For the sake of brevity, I've removed some other validations (checking if every letter in the submitted word is actually among the random letter sent and confirming that it's a valid English word). You can see this in the full code repository. To test this, we have to run the yarn dev command, start a new game and submit some words. Now let's invoke this function in another terminal and pass the game id to see if it works.

npx serverless invoke local -f end_game -d '{"id":"T0001C2147483705"}'

You should get a response similar to this in your terminal

{
    "statusCode": 200,
    "body": "{\"U1234567\":{\"totalScore\":26,\"words\":\"you: 3, love: 4, especially: 10, different: 9\"}}"
}

Let’s recap what we’ve done so far. We’ve been able to set up our app locally, start a new game, submit words and end the game. The next thing we need to do is create a Slack app and deploy the app online so that it can be used online. This will be covered in the second part