Forem: Victory Akaniru

How to control an IoT device using AWS in a Node.js application.

Victory Akaniru — Mon, 04 May 2020 10:40:34 +0000

I recently have been participating in a hackathon in which we're required to build a smart meter solution for the growing population of prepaid meter users in Nigeria. This project is intended to solve some of the everyday problems of users of these prepaid meter devices, for example, a user should be able to turn the meter on and off from the software we're building. This, in fact, was exactly my second task as a backend engineer on the project.

I initially picked this story without having a detailed look at the specs so I felt it was going to be an easy endpoint which I could implement by using a device status enum on the DB, set it to ON by default and then create a PATCH endpoint that takes a meter ID and updates the status to OFF or ON depending... Boy was I wrong.

Read your spec guys...

Before I continue rambling tho,

What is an IoT device

It is an acronym which means the Internet of things...

The internet of things, or IoT, is a system of interrelated computing devices, mechanical and digital machines, objects, animals, or people that are provided with unique identifiers (UIDs) and the ability to transfer data over a network without requiring human-to-human or human-to-computer interaction.

Don't pay much attention to the boring long definition tho, were more focused on the Things part.

A thing, in the context of the Internet of things (IoT), is a representation of a specific device or logical entity. It can be a physical device or sensor (for example, a light bulb or a switch on a wall). It can also be a logical entity like an instance of an application or physical entity that does not connect to AWS IoT but is related to other devices that do (for example, a car that has engine sensors or a control panel).

The solution

The real solution was to connect my device(the meter) to a cloud-based service like AWS or GCP I went with AWS.

What we really want to do here is connect a physical IoT device to a cloud service like AWS and with some code magic build some form of ON and OFF switch into the application. Since most of us, won't have access to such hardware to use for full experimenting, The alternative is to create a thing on AWS. This thing mirrors a real device and if you ever want to go all the way you can get the hardware anytime and sync it to the thing we would be creating on AWS soon.

Setting up a device(thing) on AWS

Visit aws.amazon.com on the top right, click the my account drop-down and select AWS management console
Follow the next few steps to sign in to your account
After successful login on the top left click on services and search for our service of interest IoT core once found select and you'll be navigated to this page 👇🏽

From the side nav to your left, click secure => policies => create

On the Action input field, start typing * and select the suggested option. Make sure to fill the form as shown on the screenshot. Scroll to the bottom and click create.

Still from the side nav to your left click manage => create a single thing

Enter a name for your thing scroll down and click create type

After creating a type, you should see a page like 👇🏽, scroll to the bottom and click next

This last action will navigate you to a new page and you'll see a button to create certificate click on it and you'll be navigated to this page 👇🏽

Make sure to download all 4 keys, using the download buttons. The last download button should be opened in a new tab. we will need them later on. Click Activate, scroll down and click Attach a policy, this action would redirect you to this page 👇🏽

Select the policy we created earlier myIoTPolicy and click Register Thing

If you can see the screen below 👇🏽 then congratulations you just created a thing ready to be controlled by code!

Code

Next, we need to implement a switch that can control the device we just created. To do this we need a few things

An existing Node.js project(setup one or clone this to make your life easier as I would be using it throughout this article.
AWS-IoT-SDK for node see docs

On your console install the SDK by running



npm i aws-iot-device-sdk

Navigate to server => config, create a file called awsConfig.js and paste the following code




import awsIot from 'aws-iot-device-sdk';

const thingShadows = awsIot.thingShadow({
  keyPath: '../../Downloads/63efc683ec-private.pem.key',
  certPath: '../../Downloads/63efc683ec-certificate.pem.crt',
  caPath: '../../Downloads/AmazonRootCA1.pem',
  host: 'a1xfh88u91agm5-ats.iot.us-east-2.amazonaws.com',
  clientId: 'Meter-001',
  region: 'us-east-2',
});

thingShadows.on('status', (thingName, stat, clientToken, stateObject) => {
  console.log(JSON.stringify(stateObject.state));
});

export default { thingShadows };

From the code, we simply import aws-iot-device-sdk, create a thingShadow, and initialize it with the config keys(be sure to switch up the path to point to your own keys), and then we export that instance.

In case you're wondering how to get your CA_PATH remember the tab we opened on another window? well if you don't you can visit this link download the contents and save to a file called AmazonRootCA1.pem. The remaining information can be found on your AWS dashboard.

Next, we want to create a controller function. Navigate to the Controllers folder and create a file called meter.js, paste the following code



import awsService from '../config/awsSwitch';

/**
 * @class MeterController
 */
export default class MeterController {
  /**
   * @method on
   * @description Connects to a device
   * @param {*} req
   * @param {*} res
   * @returns {object} meter
   */
  static async on(req, res) {
    let clientTokenUpdate;

    awsService.thingShadows.register('USER_METER', {}, async () => {
      const userMeterState = {
        state: {
          desired: {
            status: 'ON',
          },
        },
      };
      clientTokenUpdate = awsService.thingShadows.update(
        'USER_METER',
        userMeterState
      );

      if (clientTokenUpdate === null) {
        return res.status(400).send({
          status: false,
          error: 'update shadow failed, operation still in progress',
        });
      }

      return res.status(200).json({
        status: true,
        message: 'Meter successfully connected',
      });
    });
  }

  /**
   * @method off
   * @description Disconnects a running instance of a device
   * @param {*} req
   * @param {*} res
   * @returns {object} meter
   */
  static async off(req, res) {
    awsService.thingShadows.end();

    return res.status(200).json({
      status: true,
      message: 'Meter successfully disconnected',
    });
  }
}

We have two controller functions here ON and OFF one registers a thingShadow and passes in the state ON and for OFF we forcefully close the connection.

A few things to note

A thing shadow is an instance of a physical device like we created on AWS.

We forcefully close the connection when we call the off endpoint because AWS IoT(to my knowledge) has no form of clean disconnect. what this means, in essence, is that after calling the off endpoint if we need to establish a connection again, we have to restart our server. I have an open issue questioning this design, click here

Also for simplicity, I won't be including a DB interaction after connecting to a thing, but in a real-world implementation(like what I'm building), you would want to save each things details to a DB and link it to a particular user. Your schema for doing that could look something like this

Finally, we have to create our routes for On and Off and test our implementation

Navigate to server => routes and add a new file meter.routes.js, paste the following code



import express from 'express';
import controllers from '../controllers';

const meterRoute = express.Router();

const {
  meterController: { on, off },
} = controllers;

meterRoute.patch('/meter/on', on);

meterRoute.patch('/meter/off', off);

export default meterRoute;

In the index.js file in the same directory replace the existing code with



import express from 'express';

// auth Routes

import authRoute from './auth.routes';

// meter Routes

import meterRoute from './meter.routes';

// express router

const router = express.Router();

router.use('/auth', authRoute);

router.use(meterRoute);

export default router;

Testing our implementation

Start the server by running npm run start:dev

If you hit an error along these lines Error: premature close... You may have missed something in your policy setup(Happened to me the first time) Follow this link to fix it. Or drop a comment 🙂

Finally, using your preferred API testing tool(Insomnia for me). hit the following endpoints

PATCH http://localhost:3333/v1/meter/on
PATCH http://localhost:3333/v1/meter/off

To verify the connection and disconnection of the device

From your AWS console side nav click on the activity button, you should see some changes

This updates real-time as well. so you can keep this tab open hit the endpoints and watch the changes happen.

Its a wrap 🎉

That's it guys, let me know what you think, how can we improve on this? Your feedback is important!. I will hopefully see this project through and write more articles on cool things we can do with AWS IoT SDK.

Stay safe and wash your hands!

Implementing an email queue with bullJs and hosting on Heroku!

Victory Akaniru — Fri, 20 Mar 2020 05:27:26 +0000

The problem...

If you've ever picked up a task that reads like this Setup an email service with SendGrid and nodemailer you probably did what most of us would do... Setup SendGrid for production and nodemailer for development and test environments. If you did that successfully and stopped no one would fault you but oftentimes email sending in some applications require just a bit more to keep em running smoothly. Few questions we often forget to ask is

How does the increase in volume affect my email setup?
Would it hold out or work when we have 10,000 users, sending at least one email every minute?
What happens when my email fails to send on the first attempt for various reasons...

Well, in this article we'll be looking at setting up an email queuing system using bullJs in a nodeJs application and also getting this setup to a production-like environment using Heroku.

Prerequisite

Basic knowledge of Javascript and nodeJs
Heroku account(you can sign up here)
Clone this repo 👉🏽 https://github.com/vic3king/bulljs-email-setup.git and follow the readme setup instructions.
Redis server running on localhost:6379. You can set one up quickly by following the instructions here

You can also set up your own nodeJs application the way you like it and integrate the code from this example and it should work regardless. But for this article, I will be working with the cloned repo.

The cloned repo contains some boilerplate code set up with node, express, and Sequelize. I added a registration endpoint that requires the user to input their username and email address(server/controllers/auth) to register. If you followed the setup instructions on the readMe you should be able to hit the registration endpoint at http://127.0.0.1:3333/v1/auth/register and register a user.

I have SendGrid and nodemailer preconfigured on the application as well in form of a function we can call whenever we need to send an email (server/config/emailSetup.js), I also added a base email template function that takes in a username, email, and a link as arguments. What we want to do now is to make sure that when a user registers they get an email that will be queued and processed by the bullJs package.

Bull is a Node library that implements a fast and robust queue system based on redis.

Although it is possible to implement queues directly using Redis commands, this library provides an API that takes care of all the low-level details and enriches Redis' basic functionality so that more complex use-cases can be handled easily. Read more

Bull Implementation

Navigate to the server/config folder and add a file called bullConfig.js.Paste in the following code.

import Queue from 'bull';
import dotenv from 'dotenv';

dotenv.config();

const { REDIS_URL } = process.env;

// Initiating the Queue with a redis instance
const sendMailQueue = new Queue('sendMail', REDIS_URL);

export default Sendmail queue;

Make sure your .env file contains REDIS_URL='redis://127.0.0.1:6379'. In this file, we're importing Queue from our package bull. with that, we create a new instance of the bull queue class and called it sendMailQueue. We could create multiple queues for a different use case like

const imageQueue = new Queue('image transcoding');

Make sure you have a Redis server running in the background

Next, we need to create a notifications.js file in the server/helper folder. In this file, we'll introduce two key concepts from bullJs producer and consumer. we'll also create a function that takes in a receiver's email address, name, and a link to verify their email address. we could extend this implementation by adding functions like a forgotPassword function that takes the same parameters as the registrationEmail function. for each new notification we send out, we need to trigger the producer by calling sendMailQueue.add(data, options);.

import emailService from '../config/emailSetup';
import sendMailQueue from '../config/bullConfig';
import template from './template';

/**
 *
 * @param {*} emailTo
 * @param {*} link
 * @param {*} name
 * @returns {*} sends an email to a new user
 */
const registrationEmail = (emailTo, link, name) => {
  const subject = 'Welcome to Karneek';
  const body = `<p>Dear ${name},</p>
  <p>We are thrilled to have you.</p>
  <p>Some random message with link</p>
      <a href="${link}" class="button">Confirm email</a>`;
  const message = template(subject, body, emailTo);

  const options = {
    attempts: 2,
  };
  const data = { emailTo, subject, message };

  // Producer: adds jobs to que, in this case emails to be sent out upon signup
  sendMailQueue.add(data, options);
};

// Consumer: this gets called each time the producer receives a new email.
sendMailQueue.process(async job => {
  emailService.mailSender(job.data);
});

const Notifications = { registrationEmail };

export default Notifications;

Finally, we need to call the registrationEmail function in our registration route. Navigate to server/controllers/auth.js. Import our notifications file

import notifications from '../helpers/notifications';

In the registerUser function, just after the verificatioToken variable add

      const { REGISTRATION_URL } = process.env;
      const verificationLink = `${REGISTRATION_URL}?token=${verificationToken}`;
      await notifications.registrationEmail(email, verificationLink, username);

That's it. To test things our run the server npm run start:dev and create a new user! On your console, you should see

Hosting this setup on Heroku

The challenge with this setup comes when we need to host this project on a remote server like Heroku. To get this working, we need to have an instance of Redis running by the side on Heroku. First, we need to add an addon by Heroku called Heroku Redis, we also need to add Heroku Postgres because this project uses a Postgres DB.

Navigate to your dashboard on Heroku, and create a new app. After that navigate to the overview tab of the newly created Heroku app and click on Configure Add-ons

This would navigate to a new page with a search box. Search for and add Heroku Redis, and Heroku Postgres

Copy the name of your Heroku app and on your terminal run the following commands.

heroku git:remote -a <name of heroku app>
heroku addons | grep heroku-redis
heroku addons:create heroku-redis:hobby-dev -a <name of heroku app>

After the last command, you should see something close to this

You need to copy your new Redis addon name at this point. for me, it was redis-deep-25660 (see screenshot)

run the next commands

heroku addons:info <your addon name> 
heroku config | grep REDIS
git push heroku master

If you're seeing anything like this on your logs, then you're done! 😳

To learn more about how Heroku Redis works ... https://devcenter.heroku.com/articles/heroku-redis#provisioning-the-add-on

Conclusion

In this article, we've been able to use bullJs for email queueing and also host our tiny app on Heroku by provisioning and running a Redis server on our Heroku app. I hope you found this useful, If you run into any problems you can check out the complete code here or leave a comment.

Building a SPA with React and Okta for authentication.

Victory Akaniru — Wed, 11 Mar 2020 02:45:43 +0000

The problem...

I recently had a close friend pitch me an idea to build a customized inventory management application for his organization.

After the idea pitch, I began doing some research on how this would come to life, part of this research(thinking and googling) centered around the authentication experience for new users in this case employees of the organization.

I came up with a couple of flows which all seemed like a good option but then would require a lot of resources and code to bring them to life and as with software the more code we write the more likely we are to face bugs etc.

Another issue I encountered was that this application was an organizational type application, meaning we could have multiple applications built and managed by the organization. would I have to go through these authentications and authorization dilemma each time we need to build a new app for the organization?

More research led me into considering an organizational-wide authentication microservice such that each application added to the organization can make use of this service to give or deny access to users for each of the organization's applications😰. I am but one man! Tho feasible, I knew my thoughts may not scale-out nicely and would require a decent amount of resources, time, funds, devs, etc, I'd basically have to form my own small tech company within my friends leather works startup 🤦🏽‍♂️.

I knew I had to come up with a better solution, thus I thought Okta!!!

Why Okta?

Okta is a user identity management platform that enables companies and developers to handle and manage authentication for their applications and their products.

Okta as a product has two major offerings as listed on the companies website

Workforce Identity: Protect and enable employees, contractors, and partners.

Workforce Identity allows you to easily sign-in to all the applications your organization uses through a single login. Once you sign in, your Okta home page displays all your applications in one location. Simply, click the application's corresponding icon and each application opens in a new browser window or tab and you are automatically logged-in. Read more

Customer Identity: Build secure, seamless experiences for your customers.

For this article I would like to prototype my solution, we would be focusing on a little bit on both Okta offering, basically how we manage users' authentication as an organization and how new users get to access our organization's application(s).

What we aim to achieve

create a simple react app bootstrapped by create-react-app
create a developers account on Okta and register our organization and our first organization-wide application
Build a custom Okta based authentication into our react app to enable assigned users to log in to our app.
Manage employees access and identity by assigning roles and application access to different users

Implementation

To get started we need to sign up for an Okta developers account here

You should see a form like this

Complete the form check your email for your activation email and follow the ACTIVATE MY ACCOUNT button. Change your password, fill the other information and click Create My Account.

You should have this dashboard on your screen at this point. 🎉

Now that we have our Okta account ready we need to create an application or select from the list of supported Okta applications like gsuite, slack, etc and then invite/assign users(employees) to use these applications.

With these next few steps, you can start seeing parts of our solution form. Organizations have countless applications they subscribe to and give access to their employees. with Okta we can decide who uses what.

we can add a user(employee) and also assign an organizational app to that user.

Add a user

To do this select Users > People from the second navigation tab on your Okta dashboard.
You would be redirected to this page

Click on Add Person. Fill up the form on the modal that pops up and hit Save.

You can add as many users as you like. If you want those users grouped for access control you can click the groups button on the modal and create different groups. eg Admin, Management, Visitors

Add an App

To do this select Users > People from the second navigation tab on your Okta dashboard.

You would be redirected to this page

Notice that on the right-hand side we only have one user, which is me. If you followed the above step you would have multiple users listed here.

Click on any user > Assign apps.

we should see a screen saying we don't have any application

Click Add Application. And you'll be redirected to this page

with this, we can either select an application our organization members can have access to like gsuite and or add our first custom organization application!

Click on Create New App on the top left, a modal should pop up like so

Since we'll be building a SPA select SPA from the drop-down and click create. By default, all SPA's on Okta uses an industry-standard OpenId connect. Click Create app which would redirect you to this page

Provide the name of your app and add a redirect URL as I have. you can use any preferred port... Finally hit Save

On the new dashboard for your app click the Assign button. You'll notice we can assign to individuals or groups. Okta automatically creates an, Everyone group for you, this happens to be one of Okta's solutions I like a lot because I could create more groups and add my users to any of these groups when I invite them and they would have access to all applications available to that group. I could also assign to people individually. for now, click on Groups and assign to everyone

Finally, navigate to the general tab, scroll to the bottom and copy your clientId because it's time to write some code 🎉

Setup sometimes may endure for a night but code comes in the morning... still UNKNOWN 😰

Building out the react app 🏌🏼

Now we need to create a react app and add the Okta authentication SDK to it such that only users we invite to our Okta org or assign to an app can have access to it!

Open up your terminal

- cd documents
- npx create-react-app spa-okta --template typescript
- cd spa-okta
- npm i semantic-ui-react semantic-ui-css @okta/okta-react @okta/okta-signin-widget dotenv
- cd src
- touch config.js Home.jsx Profile.jsx NavBar.jsx

We created a new project called spa-okta which is bootstrapped by the create-react-app template. This enables us to skip all the tooling and configurations for webpack and focus on what really matters.

We installed

semantic UI so we can change the appearance of the Okta form to suit our needs
@okta/okta-react this gives us access to some components from Okta which we would use on our app.
Okta Sign-In Widget is a JavaScript library that gives you a fully-featured and customizable login experience that can be used to authenticate users on any website.
dotenv to enable access to environmental variables

We also created some files which would hold our components
your project structure should look like this at this point

configurations

In src/index just above the ReactDOM.render function add

import 'semantic-ui-css/semantic.min.css';

This ensures global access to semantic UI's properties within our application.

Add the following code in your src/config.js file

const CLIENT_ID = process.env.CLIENT_ID;
const ISSUER = process.env.ISSUER

export default {
    clientId: CLIENT_ID,
    issuer: ISSUER,
    redirectUri: 'http://localhost:8082/implicit/callback',
    scopes: ['openid', 'profile', 'email'],
    pkce: true,
    disableHttpsCheck: false,
};

Here we are exporting an object with the basic configurations needed to get Okta running smoothly.

In your .env file add

CLIENT_ID=
ISSUER='issuerId/oauth2/default'

Remember your CLIENT_ID? paste it here. As for ISSUER,value you can get that from your Okta dashboard.

Building the components

With that done we need to create three components that show what we can do with Okta. we need to add a Home, Navbar, Login and finally a Profile component that would be protected and can only be accessed after successful sign-in.

we'll start with the Login component... Add the following code. since we'll be building our custom Login component we need to do a little bit more. see



import React, { useEffect } from 'react';
import * as OktaSignIn from '@okta/okta-signin-widget';
import '@okta/okta-signin-widget/dist/css/okta-sign-in.min.css';

import config from './config';

const Login = () => {
  useEffect(() => {
    const { pkce, issuer, clientId, redirectUri, scopes } = config;
    const widget = new OktaSignIn({
      /**
       * Note: when using the Sign-In Widget for an OIDC flow, it still
       * needs to be configured with the base URL for your Okta Org. Here
       * we derive it from the given issuer for convenience.
       */
      baseUrl: issuer ? issuer.split('/oauth2')[0] : '',
      clientId,
      redirectUri,
      logo: '/react.svg',
      i18n: {
        en: {
          'primaryauth.title': 'Sign in to React & Company',
        },
      },
      authParams: {
        pkce,
        issuer,
        display: 'page',
        scopes,
      },
    });

    widget.renderEl(
      { el: '#sign-in-widget' },
      () => {
        /**
         * In this flow, the success handler will not be called because we redirect
         * to the Okta org for the authentication workflow.
         */
      },
      (err) => {
        throw err;
      },
    );
  }, []);

  return (
    <div>
      <div id="sign-in-widget" />
    </div>
  );
};
export default Login;

Here we created a Login component and initialized an instance of OktaSignIn when the component renders, using a hook useEffect and passed in the destructured variables from our config object. Finally, we return a div to render the widget.

Next, we need to add our NavBar component which would display different items depending on if our user is authenticated or not.

In your NavBar.tsx file add the following code

import { useOktaAuth } from '@okta/okta-react';
import React from 'react';
import { Container, Image, Menu } from 'semantic-ui-react';
import logo from './logo.svg';

const Navbar = () => {
  const { authState, authService } = useOktaAuth();

  const login = async () => authService.login('/');
  const logout = async () => authService.logout('/');

  return (
    <div>
      <Menu fixed="top" inverted>
        <Container>
          <Menu.Item as="a" header href="/">
            <Image size="mini" src={logo} />
            &nbsp;
            Okta-React Sample Project
          </Menu.Item>
          {authState.isAuthenticated && <Menu.Item id="profile-button" as="a" href="/profile">Profile</Menu.Item>}
          {authState.isAuthenticated && <Menu.Item id="logout-button" as="a" onClick={logout}>Logout</Menu.Item>}
          {!authState.isPending && !authState.isAuthenticated && <Menu.Item as="a" onClick={login}>Login</Menu.Item>}
        </Container>
      </Menu>
    </div>
  );
};
export default Navbar;

Here we create a NavBar component using semantic UI and we also conditionally render items on the navbar depending on if the user is authenticated or not and we can tell if a user is authenticated by destructuring authState from the useOktaAuth function. we also created a login and logout redirect function based off of the authService destructured from the useOktaAuth.

Next up is our simple Home page or landing page component. In your src/Home.jsx add the snippet


import React from 'react';
import { useOktaAuth } from '@okta/okta-react';


const Home = () => {
  const { authState } = useOktaAuth();

  return (
    authState.isAuthenticated ? <p>Welcome! Click the profile button on the navBar to view your profile and some details returned by Okta!</p> : <p>This is the landing page of our tiny app.</p>
  )
}

export default Home

We're close!

In your src/Profile.tsx file add the following code


import React, { useState, useEffect } from 'react';
import { useOktaAuth } from '@okta/okta-react';
import { Table } from 'semantic-ui-react';

const Profile = () => {
  const { authState, authService } = useOktaAuth();
  const [userInfo, setUserInfo] = useState(null);

  useEffect(() => {
    if (!authState.isAuthenticated) {
      // When user isn't authenticated, forget any user info
      setUserInfo(null);
    } else {
      authService.getUser().then((info) => {
        setUserInfo(info);
      });
    }
  });

  if (!userInfo) {
    return (
      <div>
        <p>Fetching user profile...</p>
      </div>
    );
  }

  return (
    <div>
      <div>
        <p>
          Below is the information from your ID token which was obtained during the &nbsp;
          <a href="https://developer.okta.com/docs/guides/implement-auth-code-pkce">PKCE Flow</a> and is now stored in local storage.
        </p>
        <p>This route is protected with the <code>&lt;SecureRoute&gt;</code> component, which will ensure that this page cannot be accessed until you have authenticated.</p>
        <Table>
          <thead>
            <tr>
              <th>Claim</th><th>Value</th>
            </tr>
          </thead>
          <tbody>
            {Object.entries(userInfo).map((claimEntry) => {
              const claimName = claimEntry[0];
              const claimValue = claimEntry[1];
              const claimId = `claim-${claimName}`;
              return <tr key={claimName}><td>{claimName}</td><td id={claimId}>{claimValue}</td></tr>;
            })}
          </tbody>
        </Table>
      </div>
    </div>
  );
};

export default Profile;

When this component is rendered we first check if the user is authenticated. if the user is authenticated we fetch their profile details from authService.getUser() which we have access to via Okta. when we get that info back we use it to update the userInfo state. finally, we loop over that information and form a semantic UI table using those details.

Bringing it all together

In your src/App.jsx file update, its contents with the following code

import React from 'react';
import { BrowserRouter as Router, Route, useHistory } from 'react-router-dom';
import { Security, SecureRoute, LoginCallback } from '@okta/okta-react';
import { Container } from 'semantic-ui-react';
import config from './config';
import CustomLoginComponent from './Login';
import Navbar from './NavBar';
import Profile from './Profile';
import Home from './Home';

const HasAccessToRouter = () => {
  const history = useHistory(); // example from react-router

  const customAuthHandler = () => {
    // Redirect to the /login page that has a CustomLoginComponent
    history.push('/login');
  };

  return (
    <Security
      {...config}
      onAuthRequired={customAuthHandler}
    >
      <Navbar />
      <Container text style={{ marginTop: '7em' }}>
        <Route path="/" exact component={Home} />
        <Route path="/implicit/callback" component={LoginCallback} />
        <Route path="/login" exact component={CustomLoginComponent} />
        <SecureRoute path="/profile" component={Profile} />
      </Container>
    </Security>
  );
};

const App = () => (
  <div>
    <Router>
      <HasAccessToRouter />
    </Router>
  </div>
);

export default App;

In this file, we import all our components and the Okta config file. we create a component HasAccessToRouter which returns the Okta Security component. The security component takes two arguments, the first being our config object and the second being a callback function which redirects a user to a particular page when the user has not been authenticated and tries to access a SecureRoute in our case /profile. Finally using react-router we sandwich our HasAccessToRouter component.

That's it! To test our application

npm run start

We now have our prototype app ready! only users within an organization can access this app, also users within that organization have to be assigned to this application in order to use it. You can assign more users to this application from your Okta dashboard. Okta has great but very large product documentation which played a good role in helping me complete this article, feel free to have a look here.

Find full code here. ⭐️

if you encounter cors issues, go to your dashboard on Okta, from the navbar navigate to API > Trusted Origins click Add Origin and provide the details... Origin URL has to match the port on your react app

The one where we build a web scrapper and a slackbot - Part 2

Victory Akaniru — Sun, 23 Feb 2020 14:55:57 +0000

The problem

In part one of this two-part series we set out on a quest to automate these tasks

Visit Brainyquote
Find and copy a random quote about excellence from the site.
Post the quote to a slack channel.

we were able to build a web scraper that enabled us to scrape a website for quotes and save them to a JSON file. In this second part of the article series, we would be bringing it all together by building a slackbot that would periodically post one of the quotes we scrapped to a slack channel!

If you missed part one you can find it here

The Product

Slackbot: Here we would visit slack apps page, and create a bot associated with a slack channel.
Slackapi backend: Making use of slacks open API we would write some python functions that when called posts the message to a channel.
Apscheduler: Finally we would write another function that would help us call those API functions at intervals.

Prerequisites

Building up from part one, all you'll need at this point is a slack workspace, you could easily create one for free here or just build the bot into any workspace by creating a channel where you can play around with.

Part 2: The slackbot

To get started visit slack a modal should pop up like this

Type in your app name
Select a development workspace(slack workspace where the bot would be installed)
Click create App and you'll be navigated to this page
Now we need to set a scope for our bot

Scopes give your bot permission to do things like sending a message in your development workspace.
Navigate to OAuth & Permissions on the sidebar to add scopes to your app
Scroll down to the Bot Token Scopes section and click Add an OAuth Scope.
Select or search for chat:write scope to grant your bot the permission to post messages in channels it's a member of. you should see a success banner after this ✅
Next, we have to install the bot to our workspace, by scrolling to the top of the OAuth & Permissions pages and clicking the green Install App to Workspace button.
After the above step, you should be navigated to the below page. All you have to do is click the allow button.
Finally 🎉, copy your bot token, you'll need it to communicate with slack API.

Next steps... We need to write our backend code to interface with slack. we will be doing this in the same folder where we created our web scrapper you can find that code if you don't have it by clicking here

- cd clonedOrExistingProject
- mkdir bot
- cd scrapping-tool
- touch __init__.py builder.py post_message.py

Create and activate a virtual environment if you don't have one already

- virtualenv --python=python3 venv
- source venv/bin/activate

In the builder.py file add the following code

def construct_payload(HEADER_BLOCK, DIVIDER_BLOCK, MESSAGE_BLOCK):

    return {
        "channel": "your-channel-id-goes-here",
        "username": "name-of-your-app",
        "icon_emoji": ":robot_face:",
        "text": ":wave: Your daily dose of motivation has arrived",
        "blocks": [
            HEADER_BLOCK,
            DIVIDER_BLOCK,
            MESSAGE_BLOCK
        ],
    }

Slack has a tool called the block builder kit, it's basically used to generate markdowns that can be used to style the appearance of messages on slack. Try it out here.

For us, we created a function that accepts various blocks from the block builder kit and in the end, we want to have something looking like this

From the code, we passed in

Channel: The id of a Slack channel where we want to invite our bot(you can copy it from your slack channel, make sure to invite the bot to this channel)
Username: which would be the name of the app you created earlier
Text: would be what appears on the slack notification popup.
Blocks: contains the various parts of the message, the header, a divider line, and the actual message.

Let's start by creating our random quote generator function, this file will read our previously scrapped json quotes and return any random one when the function is called. from the root of your project folder, run the command

- mkdir utils
- touch __init__.py random_quote.py

In the random_quote file, add the following code

import random
import json


def random_quotes_generator():
    with open('quotes.json', 'r') as f:
        quotes_dict = json.load(f)


        quote = random.choice(quotes_dict)
        return quote

This file creates a function random_quotes_generator reads our quotes from the json file and with python's random module returns one random quote when called.

Next, we create our post message function. Install these packages

pip3 install slack dotenv

slack to interface with slack API and dotenv to load in our bot secret token.

Navigate to the post_message.py file and post the following code.

import os
from slack import WebClient
from dotenv import load_dotenv
from bot.builder import construct_payload
from utils.random_quote  import random_quotes_generator

load_dotenv()

SLACK_BOT_TOKEN = os.getenv("SLACK_BOT_TOKEN")

# Initialize a Web API client
slack_web_client = WebClient(token=SLACK_BOT_TOKEN)


def post_quote_to_channel():
    quote = random_quotes_generator()
    author = quote['author']
    link = f"https://www.brainyquote.com{quote['author-link']}"


    # Get the onboarding message payload
    message = construct_payload({
        "type": "section",
        "text": {
                "type": "mrkdwn",
                "text": "*Your daily dose of motivation* :blush:"
        }
    },
        {
        "type": "divider"
    },
        {
        "type": "section",
        "text": {
            "type": "mrkdwn",
            "text": f"The test of the artist does not lie in the will with which he goes to work, but in the excellence of the work he produces... *By <{link}|{author}>*"
        }
    })
    # Post the onboarding message in Slack
    response = slack_web_client.chat_postMessage(**message)

So what are we doing here...

We imported a couple of 3rd party packages, our message builder function and our random_quotes_generator.

We also load in our SLACK_BOT_TOKEN, create a .env file use the key SLACK_BOT_TOKEN and set it to your bot token.

SLACK_BOT_TOKEN=token

We initialized slack WebClient and created the post_quote_to_channel inside which we call our random generator function and save it to a variable quote.
We construct a link to each author's page.
Next, we construct our message by calling the construct_payload function exported from the message builder file and passing in all required parameters.
Finally, we call a function chat_postMessage from slack WebClient and pass in our constructed message.

You can test that this works now by calling the function on your terminal running this command

python3 post_message.py

The final step in this process would be to create our scheduler to periodically call the post_quote_to_channel. to do this we need to install some packages, create a flask server and our scheduler function.

On the root of your project folder create a new file app.py and run the following commands

pip3 install APScheduler Flask

- cd bot
- mkdir jobs
- touch __init__.py scheduler.py

In bot/jobs folder locate the scheduler.py file and paste in the following code

from apscheduler.schedulers.background import BackgroundScheduler

# jobs
from bot.post_message import post_quote_to_channel

scheduler = BackgroundScheduler()

scheduler.add_job(
    post_quote_to_channel,
    'interval',
    seconds=10
)

Here we import BackgroundScheduler from the scheduler package and initialize it.

We also import the post_quote_to_channel function from the file post_message in the bot folder.
With that, we call add_job function from the scheduler, pass the post_quote_to_channel function, the string interval and time interval in seconds. to read more about app scheduler click here

with that, we have our scheduler function ready.

Bringing it all together

In the app.py file we created earlier, post the following code

import os
import time
from flask import Flask
from bot.jobs.scheduler import scheduler

# Initialize a Flask app to host the events adapter
app = Flask(__name__)

if __name__ == "__main__":
    scheduler.start()
    print('Press Ctrl+{0} to exit'.format('Break' if os.name == 'nt' else 'C'))

    try:
        # This is here to simulate application activity (which keeps the main thread alive).
        while True:
            time.sleep(5)
    except (KeyboardInterrupt, SystemExit):
        # Not strictly necessary if daemonic mode is enabled but should be done if possible
        scheduler.shutdown()

Here we import our scheduler, create a flask server and call our scheduler function. we also use a nifty trick from StackOverflow to keep our main thread alive.

Run the program

python app.py

If we've done everything right we would start receiving quotes in our slack channel every 10 seconds.

You can tweet the time interval to suit your needs. e.g minutes=69 || hours=24

That's it guys, through these series we've been able to cover a few micro-skills

building a slackbot
scraping a website
scheduling tasks

These skills can be applied separately or brought together to build a cool product.

I'll like to hear your feedback and question, feel free to leave a comment.

To view the full code click here

The one where we build a web scrapper and a slackbot - Part 1

Victory Akaniru — Sun, 23 Feb 2020 09:37:55 +0000

The problem

As software engineers, part of what we do revolves around making seemingly easy things a little bit easier. who would imagine doing these three things would be a chore?

Visit Brainyquote
Find and copy a random quote about excellence from the site.
Post the quote to a slack channel.

It seems simple enough to do but if done every day for a year becomes boring and tedious.

Python is a scripting language built for things like this! With python, we could automate this whole process and not have to do the same thing every day.

The Product

Well, that's exactly what we would be doing 🎉. In this two-part series, we would be building a slackbot that periodically sends a random quote about excellence to a specified slack channel. Some of our MVP features would include

Scraping tool: This would be responsible for getting a whole lot of quotes and saving them to a JSON file for future use
A Slack bot: That would be responsible for periodically(maybe every morning?) sending one random quote to a slack channel. This part of the project would require us to write some simple code for posing the message to a Slack channel at intervals.

Prerequisites

A python environment and some basic knowledge of Python. That's it

Part 1: The scrapping tool

First off we need to get some groundwork done by creating a basic project setup, a virtual environment and installing some packages

- cd newly_created_folder
- mkdir scrapping-tool
- cd scrapping-tool
- touch __init__.py main.py scroll.py selenium_driver.py

At this point, we're good to go but I strongly recommend you create a virtual environment for this project. If you have virtualenv installed on your PC all you have to do is run the following commands

- virtualenv --python=python3 venv
- source venv/bin/activate

If you don't or have questions around what a virtualenv is... you may want to Read this

Next, install the following 3rd party packages

BeautifulSoup to help us scrape any website for data
selenium to automate browser interactions while doing so and lxml to interface with BeautifulSoup and parse data to LXML.

run the following command on your terminal

pip3 install BeautifulSoup selenium lxml

Finally, download chrome driver by following basic instructions here. This would enable us to run a headless version of chrome when using selenium for automation. If you're on a mac you can simply run

brew cask install chromedriver

Setup sometimes may endure for a night but code comes in the morning... UNKNOWN

Let's write some code!

In the scrapping-tool folder you created, locate the selenium_driver.py file and paste the following code in

from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--incognito')
options.add_argument('--headless')
driver = webdriver.Chrome("/usr/local/bin/chromedriver", chrome_options=options)

This piece of code imports webdriver from selenium and adds some configuration options for webdriver like incognito, headless mode, etc, finally we make use of the chromedriver we installed earlier by pointing to the path where it was downloaded to. we save this in a driver variable for future use.

By adding the __init__.py file in our folder we told python to consider every file in that folder a package. This means functions, variables, etc are exposed by default from any location in our app 😎.

Part of the hassles that come with automation in web browsers comes up when human interaction is needed. For example, the website we are trying to scrape has some functionalities you would notice once you open the site.

On the first visit to the website, you would have to click and accept the privacy policy
After that, we see the page with all those quotes we would like to get, but then this page implements an infinite scroll.

We won't be doing much automation if we were to help our browser click that button or help the browser scroll when it gets to the bottom of the page. These problems bring us to our next step scroll.py.

The key to scrapping a website properly lies in your ability to hit inspect and find that class or id with which you can access that element

In the file scroll.py Copy and paste the code below.

import time


def scroll(driver, timeout):
    scroll_pause_time = timeout

    # wait for terms modal to popup and then click
    driver.implicitly_wait(timeout)
    privacy_button = driver.find_elements_by_css_selector(".qc-cmp-buttons > button:nth-child(2)")
    privacy_button[0].click()
    time.sleep(2)

    # Get scroll height
    last_height = driver.execute_script("return document.body.scrollHeight")

    while True:
        # Scroll down to bottom
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

        # Wait to load page
        time.sleep(scroll_pause_time)

        # Calculate new scroll height and compare with last scroll height
        new_height = driver.execute_script("return document.body.scrollHeight")
        if new_height == last_height:
            # If heights are the same it will exit the function
            break
        last_height = new_height

A few things to note

We create a scroll function which takes two parameters, driver(our page source) and timeout(wait time).
We make use of some methods available on the driver object like find_elements_by_css_selector this would help us locate elements. like in our case locate the privacy button and where to start our infinite scrolling.
We also make use of execute_script method which takes our browsers window object as a parameter to enable us to scroll the website, determine page height, etc
Notice the while loop? This loop checks our browser and calculate's new scroll height by comparing our current height with the last scroll height. if both heights are the same we break the loop meaning we are at the end of the page.

Bringing it all together, we build the scrapper itself.

In main.py still within the scrapping-tool folder, add the following code

import re
import json
from bs4 import BeautifulSoup
from selenium_driver import driver # here we import the driver we configured earlier
from scroll import scroll # the scroll method


def get_quotes(url):
    try:
        # implicitly_wait tells the driver to wait before throwing an exception
        driver.implicitly_wait(30)
        # driver.get(url) opens the page
        driver.get(url)
        # This starts the scrolling by passing the driver and a timeout
        scroll(driver, 5)
        # Once scroll returns bs4 parsers the page_source
        soup = BeautifulSoup(driver.page_source, "lxml")
        # Them we close the driver as soup_a is storing the page source
        driver.close()

        # Empty array to store the links
        quotes = []

        regex_quotes = re.compile('^b-qt')
        regex_authors = re.compile('^bq-aut')

        quotes_list = soup.find_all('a', attrs={'class': regex_quotes})
        authors_list = soup.find_all('a', attrs={'class': regex_authors})

        quotes = []
        zipped_quotes = list(zip(quotes_list, authors_list))
        for i, x in enumerate(zipped_quotes):
            quote = x[0]
            author = x[1]
            quotes.append({
                "id":  f"id-{i}",
                "quote": quote.get_text(),
                "author": author.get_text(),
                "author-link": author.get('href')
            })

        with open("quotes.json", 'w') as json_file:
            json.dump(quotes, json_file)
    except Exception as e:
        print(e, '>>>>>>>>>>>>>>>Exception>>>>>>>>>>>>>>')


get_quotes('https://www.brainyquote.com/topics/excellence-quotes')

What do we have here?

We import the BeautifulSoup4 library, some inbuilt python packages like re(regular expression ) and json.
We also import the functions packages we created earlier like scroll and driver.
We create a get_quotes function that takes in a URL as a parameter.
With this, we tell our browser to wait a Lil before throwing an error(sometimes network issues may slow things down).
We called the scroll function to do its thing.
And once that is done we pass driver.page_source to BeautifulSoup4. printing driver.page_source at this point would show a bunch of HTML tags -We call close to stop browser interactions, we have all we need now

The goal is to scrape a quote, its author and a link to get all of that author's quotes. at this point, we have all of that data albeit in a format we cannot work with yet(HTML tags) also notice from the code that we are extracting data for the author separately and the same for quotes. How do we link each quote to its author? we also need to create a python dictionary containing all those pieces of information, give them unique id's and also form the author's links. Python zip function to the rescue, to put it simply this function takes two lists and generates a series of tuples containing elements from each list. We also made use of enumerate function this means we can unpack index and data from the tuples returned from the zip function. With that, we unpack and loop over the returned tuple, create a python dictionary containing the data we want and append that to the quotes array. We also called a BeautifulSoup4 method get_text() on the author and quote to enable us to return actual texts from our HTML tags. we also called get('href') which returns any property of a tag we specify, in our case href, this is how we get the link to the author's quotes. Finally, we save the contents of our quotes list to a json file by creating a quotes.json file and dumping our data into it by calling json.dump.

To run the scrapper

python scrapping-tool/main.py

To see all this in action, you can comment out this piece of code options.add_argument('--headless') in the file selenium_driver.py.

Yo! That’s it for now. Feel free to leave a comment, feedback or opinions in the comments. In part two of this article, we will go through creating a slackbot that would display these scrapped quotes on a slack channel. That would also mean we configure a flask project that would enable us to run a server and implement a scheduler!

To view the full code for this article click here

The one where I start writing again...

Victory Akaniru — Tue, 18 Feb 2020 08:38:09 +0000

Hi, my name is Victory and I love writing but not as much as I love procrastinating the act of writing and worse publishing my writings...

Writing has always been a means for me to express myself and my thoughts, two things I struggle to do to my satisfaction when I'm talking... I remember in my University days I had this series I called "30 mins rush" basically for 30 mins I'll pull out my phone and go to work on my notepad(I still do it when my thoughts go crazy), I would write about all I was thinking, feeling and experiencing at the time for 30 mins and stop. This provided me a much needed safe place to express myself the way I really wanted to. A few of those writeups others got to see but most of those write-ups and a lot more never saw the light of day, sometimes I never even went back to reading them myself!

Now I'm ready to write again, but this time I want you to see it. I want to know what you think when you see it, what you understood or didn't. I want your unadulterated feedback basically. I want us to have conversations bordering on those posts, I want to connect with you, I want us to be F.R.I.E.N.D.S...

“There is no failure. Only feedback.”

What will I be writing about? Everything. My Opinions on various subject matters, Technical writeups on what I've learned/know, personal(tongue in cheek)... you get the gist.

Cheers to a not so fun time of having to deal with my fear of public scrutiny... 🎉🥂

Want to get in touch with me? Click me