Forem: Hatem Hassan 👨‍💻☕️💻🌺😎

That time I built a tool that helps 1000 CS students every month

Hatem Hassan 👨‍💻☕️💻🌺😎 — Tue, 07 Jan 2020 22:52:39 +0000

Originally posted on my personal blog (better formatting)

TLDR: I built an online tool in 2013 at AUC that gets ~1000 visits a month from students all over the world and is mentioned in a couple of reputable universities slides including Columbia University.

Website

7 years ago, while I was pursuing my bachelor's of Computer Engineering at The American University in Cairo, I had a course called Digital Design, which in retrospect I think was supposed to be called Digital Electronics but who am I to name courses!!
The course had nothing to do with design in the common known sense; it was mainly about the design of electronic logic gates to build processors and basic computer systems.

Part of the course was studying the Quine McCluskey algorithm and learning how it works, then the semester project was building a calculator to solve for it in C++.

Luckily after finishing the assignment I checked the domain name quinemccluskey.com and to my surprise, it was available 😯😯

I immediately bought the domain name and started converting my C++ program to a website using my limited knowledge of PHP at the time. I also used one of the earliest versions of Bootstrap for styling. Please don't judge the UI, it was 2013 😅

Quine Mccluskey Calculator

Adoption

After publishing the website, I connected it to Google Analytics then I posted the link on several social media websites and added it to the Quine Mccluskey Wikipedia page.

Since this algorithm is a standard topic taught in most CS and Electronics undergraduate programs around the world, it had a relatively wide demand across students all over the world.

Looking at Google Analytics data we can see that over the past 6 years, almost 50k users visited the website.

Also there's a 28 day average of **943 "Active Users":**

Google defines an Active Users as "The number of unique users who had at least one session within a 28-day period."

Did you notice any patterns in the visitors timeline?

That's because students usually use the tool during the assignment period or the Midterms season, which is usually in November or March depending on the semester.

Where do those visitors come from?

Geographically we can see that the heavy load is coming from the US and India.

It's also worth noting that doing a simple search in the sources datatable we can find visitors coming from these universities internal systems:

Texas State University (USA)
MSA (Egypt)
Auburn University (USA)
National Ilan University (Taiwan)
Hunan Normal University (China)
Ming Chuan University (Taiwan)
Columbia University (USA)

What is Quine Mccluskey?

In the world of digital logic design, everything is a logical function that can be boiled down to one of the basic arithmetic operations: addition, subtraction, multiplication or division.

Even the most complex processes like image/video processing or machine learning algorithms are converted at the very low level to these simple arithmetic operations so it can run through the physical gates and flip-flops inside the processor/GPU card.

Earlier in computing history, engineers used to design and build such processors manually by finding out how to map a real problem to a series of boolean functions that outputs the correct result when electricity runs through the processing board.

Theoretically a boolean function is an equation that can be represented in multiple ways; however, there should be one optimal way to implement it. Here comes the process of Logic Optimization. One of the most straightforward ways to optimiza boolean functions is the Quine–McCluskey algorithm, which serves the same purpose as the K-maps method but for more variables.

Quine-McCluskey is a tabular and tedious method based on the concept of prime implicants. A prime implicant is a product or sum term, which can’t be further reduced by combining with any other terms of the given Boolean function.

Basically you have to do the following 6 steps to simplify a Boolean Function using Quine–McCluskey.

I omitted explaining the "don't care" notion for simplicity but you can read about it in depth here.

Step 1:
List down all of the minterms in ascending order in binary representation, then group them by the number of 1s in each.
This should create a maximum of n+1 groups for n minterms.

Step 2:
Compare the minterms in each group with the group after it. If there is a change in only one bit position, replace that one bit with ‘_’.

Step 3:
Repeat Step 2 with the First Comparison results to come up with the prime implicants.

Step 4:
Create the prime implicants coverage table. Prime implicants are be placed in rows and min terms are placed in columns.
Then cross out (X) the cells which correspond to a min term that is covered with a prime implicant.

Step 5:
Now we need to get the Essential Prime Implicants. An Essential Prime Implicant is a minterm that is covered by only one prime implicant. These essential prime implicants are the pre-final form the simplified Boolean function.

Step 6:
Reduce by removing the row of each essential prime implicant and the columns corresponding to the min terms that are covered in that essential prime implicant.

Repeat Step 5 for prime implicants till all min terms of given Boolean function are completed.

Fullstack NLP: Building & Deploying End-to-end Fake News Classifier

Hatem Hassan 👨‍💻☕️💻🌺😎 — Sat, 07 Dec 2019 10:44:09 +0000

Originally posted on my personal blog (better formatting)

TLDR: This is a tutorial on building the API+UI of NLP text classification web application and deploying it to production.

Github | Notebook | Demo

Note: The classification model used is by no mean accurate and it's definitely overfitted over the training dataset. The focus of this post is implementing the fullstack web application. Ref 1. Ref 2.

While pursuing my Master's degree at IE, I worked on this Text Classification model as part of the NLP course. The notebook is extra messy but I will try to briefly explain what's going on in the first section.

NLP Classifier

Our problem here is to define whether or not a certain news article is fake news. The dataset is comprised of 3997 news articles each includes a title, text, and the target label as a REAL/FAKE binary label. Part of the course was also testing the model on a test dataset but I never received target for this dataset. The accuracy score of cross validation testing within the training dataset was 94%.

I won't go through the coding details of the Notebook, but I will briefly describe the main steps taken before coming up with the final model.

Text Preprocessing

We can clearly see that the target is balanced. So we jump ahead into the preprocessing by doing the following:

Lowercase the text: This preprocessing step is done so words can later be cross checked with the stopwords and pos_tag dictionaries.
Remove words with just one letter
Remove words that contain numbers
Tokenize the text and remove punctuation
Remove stopwords: Proper analysis of text usually relies on the most recurring words. Stopwords including words as "the", "as" and "and" appear a lot in a text, but eech word doesn't really give relevant explanation, so one of NLP comon practices is to remove such words.
Remove empty tokens: After tokenization, we have to make sure all tokens taken into account contribute to the label prediction.
Lemmatize the text: In order to normalize the text, we apply lemmatization. In this way, words with the same root are processed equally e.g. when the words "took" or "taken" are found in the text, they are lemmatized to "take", infinitive of the verb.

You can find the preprocess() function in python_helper.py:79.

Baselining

In this part of the notebook, you can see how I split and vectorize the dataset and run several classifiers on the dataset and measure baseline accuracy.

There was a significant improvement in accuracy for Max Entropy text classifier.

Feature Engineering & Pipeline

After we have a baseline score, we do some extra work:

Pos-tag text:

Adding a prefix to each word with its type (Noun, Verb, Adjective,...). e.g: I went to school => PRP-I VBD-went TO-to NN-school

Also, after lemmatization it will be 'VB-go NN-school', which indicates the semantics and distinguishes the purpose of the sentence.

This will help the classifier differentiate between different types of sentences.

TF-IDF weighting:

TF-IDF or Term Frequency–Inverse Document Frequency tries to calculate the importance of each word based on its recurrence in text.

Use Trigram Vectorizer instead of regular vectorizer:

I opt to use a Trigram vectorizer here, which vectorizes triplets of words rather than each word separately. In this short example sentence, the trigrams are "In this short", "this short example" and "short example sentence".

Finally I do a Grid Search for different combinations of the past steps along with different models to optimize for the best hyper parameters.

If you look just above the predictions here, you will find that this is the best performing model (Vectorizer + TF-IDF + Classifier):

trigram_vectorizer = CountVectorizer(analyzer = "word", ngram_range=(1,3))
tf_idf = TfidfTransformer(norm="l2")
classifier = LogisticRegression(C=1000, class_weight=None, dual=False, fit_intercept=True,
          intercept_scaling=1, max_iter=100, multi_class='warn',
          n_jobs=None, penalty='l2', random_state=None, solver='warn',
          tol=0.0001, verbose=0, warm_start=False)

I wrap everything inside a Pipeline that can train (fit()) and classify (predict()) input easily.

from sklearn.pipeline import Pipeline
pipeline = Pipeline([
     ('trigram_vectorizer', trigram_vectorizer),
     ('tfidf', tf_idf),
     ('clf', classifier),
 ])

Pickling

Next step is fitting the pipeline on train data like so:

pipeline.fit(train.clean_and_pos_tagged_text, encoder.fit_transform(train.label.values))

At this stage, the pipeline object can predict() new input. You can find the test predicting step in the notebook here.

Now comes the magic. If you want to encapsulate your whole model along with the training data in one single file to be used anywhere, you use Pickle library.

import pickle
pickle.dump( pipeline, open( "pipeline.pkl", "wb" ) )

Any object in python can be pickled to be saved on disk. Then it can be transferred to be imported in another python environment. What pickle does is that it serializes the Python object before writing it to file. Pickling is just a way to convert any python object (list, dict, etc.) into a character stream. The whole idea behind it is that this serialized textual stream contains everything needed to reconstruct the object in another python script.

In this case we are pickling our pipeline object so we can use it later in the API to pipeline.predict().

This pipeline.pkl file is now ready to predict any news article and classify it to either REAL or FAKE.

Web Application Architecture

So we need a web interface in which the user can enter some news text and click on a button for the application to preprocess the input and feed it to the trained model and show the classification back on screen.

Keep in mind that for the pickled pipeline to predict correctly we need to feed it with text preprocessed exactly in the same way the train dataset was preprocessed.

Frontend/UI:

This is going to be a simple one page with one input and a button. No need to complicate stuff. As I mentioned in this post, React would be a better fit for such straightforward function.

Backend/Server:

Since we need to "unpickle" or pickle.load() our model (the pipeline) to use it, the best choice would be a python web server that can receive the input over HTTP and return back the prediction result. One of the easiest and most straightforward frameworks for this is Flask.

Another addition to the server would be implementing a "random picker" that can fetch one random news article at a time from the test dataset. This will be used to populate the UI input field.

The goal is to make it easier for the user to test the application, without actually writing a news article.

Flask API Server

Assuming you have a ready Python environment and are familiar with Python development in general, installing pythons is as easy as:

pip install flask

If you want to use a virtualenv you can check the steps here.

A simple Flask API that serves the static React app (next section) when you hit root path / would look like this:

from flask import Flask, render_template

app = Flask(__name__, static_folder="./public/static", template_folder="./public")

@app.route("/")
def home():
    return render_template('index.html')

# Only for local running
if __name__ == '__main__':
    app.run()

You can only make use of template_folder now in which you can add a simple hello world HTML file. In the next section when we build the React app we will use the static_folder store assets like CSS and JavaScript files.

If you run the command python app.py and hit http://localhost:5000 you should see the /public/index.html displayed in your browser.

As discussed in previous section, we mainly have 2 functions to implement here: predict and random. Let's start with the easier random picker endpoint.

random/:

Since we have the test dataset in the /data folder, we just need to load it and pick a random index and send it back to UI.

from flask import jsonify
import pandas as pd
from random import randrange

@app.route('/random', methods=['GET'])
def random():
    data = pd.read_csv("data/fake_or_real_news_test.csv")
    index = randrange(0, len(data)-1, 1)
    return jsonify({'title': data.loc[index].title, 'text': data.loc[index].text})

We use Pandas here to read the CSV file and then send back the data in JSON format using flask jsonify() helper function.

Now whenever you send a GET request to http://localhost:5000/random/ using Postman or even open the link in your browser you should see a random news entry from the test dataset file.

predict/:

For the prediction function we need to do some work than just exposing the endpoint. We need to take the input, preprocess it, pos-tag it, then feed it to the pipeline and return back the result.

So we create a PredictionModel class in a separate file:

from nltk.corpus import wordnet
import string
import nltk
nltk.data.path.append('./nltk_data') # local NLTK data
from nltk.corpus import stopwords
from nltk import pos_tag
from nltk.stem import WordNetLemmatizer

class PredictionModel:
    output = {}

    # constructor
    def __init__(self, text):
        self.output['original'] = text

    def predict(self):

        self.preprocess()
        self.pos_tag_words()

        # Merge text
        clean_and_pos_tagged_text = self.output['preprocessed'] + \
            ' ' + self.output['pos_tagged']

        self.output['prediction'] = 'FAKE' if pipeline.predict(
            [clean_and_pos_tagged_text])[0] == 0 else 'REAL'

        return self.output

    # Helper methods
    def preprocess(self):
        # check file on github

    def pos_tag_words(self):
        # check file on github

And then we use the class in app.py endpoint like so:

@app.route('/predict', methods=['POST'])
def predict():
    model = PredictionModel(request.json)
    return jsonify(model.predict())

So basically when you send a POST request to http://localhost:5000/predict, the script will initialize a new PredictionModel instance with the input text and return the prediction.

For example PredictionModel("Some news article text"), will return a JSON object like this:

{
  "original":"Some news article text",
  "preprocessed":"news article text",
  "pos_tagged":"NN-news NN-article IN-text",
  "prediction":"FAKE"
}

Note: I exported the NLTK data and you can see I'm loading it from a local path here. The reason for this is both for better version management as I had issues with installing the right NLTK corpse on Docker.

But we used the pipeline object without actually declaring it...

Loading the pickle:

The key idea we need to understand here is that we need to load the .pickle file before using it predict. Note that the pickle file is *~350 MB*.

In the top of the file we can add:

with open("pickle/pipeline.pkl", 'rb') as f:
        pipeline = pickle.load(f)

Keep in mind this loads when you first run the app.py script and it loads the 350MB pipeline object into your machine RAM.

After initial run and when the script is in standby mde waiting for requests, Whenever you call pipeline.predict() in your script it will run instantly because the object is already in your RAM.

Note: This will work fine when you run it locally, but when you deploy it into production you need to consider your server RAM and what happens if the script crashes. I will discuss this in the Deployment section.

React User Interface

To initialize the React app we will use create-react-app. You can install that globally by running npm i -g create-react-app.

Then to initialize the actual code template we run create-react-app fake-news-client. This will create a folder with the name "fake-news-client" and npm install the needed packages inside it.

If you cd into the folder and run a simple react-scripts start you should have a simple react app built and served in your browser on http://localhost:3000/.

Note that the backend files will be the root folder of the project and the frontend will live in a subdirectory inside it with the name webapp.

Now we need to install an extra package we will use in the project. cd into the folder and run npm i -s node-sass.

I'm using node-sass to write the app styles which means you need to rename all the .css files into .scss and change the reference in the index.js file.

Let's build the UI

In the /src/index.js file we can start adding the state object along with the HTML template that consists of:

Input: textarea for the news text
Actions: button to get /random data and another to /predict
State Labels: App can be in one of theses states: idle, loading, or error.
Result label: Prediction, REAL/FAKE.

import React from 'react';
import ReactDOM from 'react-dom';
import './index.scss';

class NLPInterface extends React.Component {

  state = {
    text: '',         // Textarea value
    loading: false,   // Loading state flag
    error: false,     // Error state flag
    prediction: null, // Prediction result
  }

  render() {
    return (
      <div className="container">
        <h1>NLP Fake News Classifier</h1>

        <textarea disabled={this.state.loading} value={this.state.text} onChange={this.updateInput}>
        </textarea>

        <button disabled={this.state.loading} className="random" onClick={this.random}>
          Load random News from test dataset ? <span>Click here</span>.
        </button>

        <button disabled={this.state.loading} className="cta" onClick={this.predict}> Predict </button>

        {this.state.loading ? <h1>Classifying ...</h1> : ''}

        {this.state.error ? <h1>ERROR</h1> : ''}

        <h1>
          {this.state.prediction}
        </h1>

      </div>
    )
  }
}

ReactDOM.render(
  <NLPInterface />,
  document.getElementById('root'));

You can find the styles in the SCSS file.

Next step would be writing down the UI logic needed to to wire the template with the backend.

Let's start by the textarea, we can see it reads it's value from this.state.text, and onChange we are binding the this.updateInput event handler which will update state.text

  updateInput = ({ target: { value } }) => this.setState({
      text: value,
      prediction: null // To reset previous prediction whenever the text in the textarea changes
    });

Next we have the Random news picker button, which fires the following function onClick:

  random = () => fetch("/random")
    .then(response => response.json())
    .then(article => article.title + '\n\n' + article.text)
    .then(text => this.setState({
      text,
      prediction: null
    }))
    .catch(error => this.setState({ error: true }));

Finally, we have the actual prediction function binded to another button:

  predict = () => {
    this.setState({ loading: true, prediction: null }); // Start loading

    fetch("/predict", {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify(this.state.text)
    })
      .then(response => response.json())
      .then(({ prediction }) => {
        this.setState({ loading: false, prediction })
      })
      .catch(error => this.setState({ loading: false, error: true }))
  }

This will get the prediction result, store it to state.prediction which is displayed in the H1 tag at the bottom of the render() function.

Voila! we have a working webpage that can communicate with our Flask backend.

Bundling

To bundle the frontend app and move it to the /public folder which the Flask app will serve, we will edit the package.json file:

  "scripts": {
    "prebuild": "rm -r ../public",
    "build": "react-scripts build",
    "postbuild": "mv build/ ../public",
  }

Now if you run npm run build inside webapp folder then go up one directory
and run python app.py then visit http://localhost:5000 you will find the app up and running 🎉🎉

Dockerizing & Deployment

To run this app successfully on a remote server we need to consider that this will be a service with a relatively slow boot time, as we load the 350 MB pickle once the script runs.

For this we will use gunicorn HTTP server to run the python app. gunicorn will basically:

Run the server script several times in parallel using multiple async workers
Make sure the server is always running and accepting requests

After running pip install gunicorn you can run the following to run the script using gunicorn:

gunicorn -t 120 -b :8080 app:app

This will run the server on port 8080 and I added the -t 120 flag to tell it to wait 120 seconds before timing out.

Now we move to deployment. I chose to use Google Cloud Run for this application also because our application has a heavy stateless workload as it needs to load the large pickle file.

To use Cloud Run we need to containerize our application by writing down the Dockerfile:

FROM python:3.7-stretch

RUN apt-get update -y
RUN apt-get install -y python-pip python-dev build-essential

COPY . /app
WORKDIR /app

RUN python -m pip install --upgrade pip

RUN pip install -r requirements.txt

CMD gunicorn -t 120 -b :$PORT app:app

You can see also that I exported all of the needed python packages into a requirments.txt file.

Finally, to get this to work on Google Cloud you need to:

Start a Google Cloud Project
Activate billing
Open Google online console
Pull the code into your account server
cd into the project folder and run gcloud builds submit --tag gcr.io/[your project ID]/fake-news-service This will deploy the docker container image into that URL.
Go to the Cloud Run dashboard and click on "Create Service"
Type the image URL you created in step 5
Set the memory allocated to 1GB
"Create"

💃🏼 💃🏼 The app will be up and running in less than 5 minutes. 💃🏼 💃🏼

Demo

Related Links:

Realtime Data Visualization with Peak Detection using Socket.IO, D3 and React 📈📈

Hatem Hassan 👨‍💻☕️💻🌺😎 — Wed, 30 Oct 2019 11:06:02 +0000

Originally posted on my personal blog (better formatting)

TLDR: This post is UX research + fullstack programming tutorial for building a web application that streams time series data to be visualized in realtime using multiple Javascript frameworks.

Github | Demo

The Why?

IoT is growing rapidly these days and one aspect that's key to the success of this kind of projects is data visualization. Design and UX is actually one of the most critical parts in any IoT project.

Any SME may have a very good infrastructure and complex integrations, but what would that do if their systems lack the tools to transform this gathered into actionable insights that can deliver real business value.

One thing is to collect some data from those new cool devices, another is to visualize it and make it accessible for your business or even your customer.

Use case: what is happening here?

In this project I'm trying to build a React app that can consume data using Socket.io from a live source of data (we'll call it a Sensor) and visualize the time series data (we'll call it Readings) in realtime using D3.js

To simulate the Sensor, I created a very simple server using Express.js that rotate over a JSON file and keeps on continuously sending the Readings one by one every 1.5 seconds.

TBH I didn't spend much time on the server development and didn't implement proper error handling/reporting because that's not the scope of the project.

It's worth noting that while this project is built to work with a stateful WebSockets API assuming a realtime streaming use case. We could also make it support batch streaming REST API with a periodic server pinging on a longer interval. This really depends on the nature of the data and what level of granularity you want to have. I'm assuming the customer wants to do deep analysis and monitoring down to the interval of 1.5 seconds.

Peak detection

In the world of Real-Time Signal Analysis, peak detection is a hot topic for many engineering fields including chemistry, biology, biomedical, optics, astrophysics and energy systems.

To add some spice to this sample project, I included in the sample data the Moving Z-score of each data-point beside the actual sensor value.

The Moving Z-score is a mathematical model for measuring the anomalousness of each point in sequential time series. One of its main parameters is the window size (w). Given that the moving Z-score is the number of standard deviations each data-point is away from the mean, (w) limits the window in which we calculate the such statistical parameters. In this specific use case, the mean and standard deviation are computed only over the previous (w) observations.

In this scenario the Z-score would be a binary (0/1) value that tells whether there's a "peak" at this point or not.

In this project I got hold of a dataset with manually pre-calculated z-score to detect if the sensor readings increase suddenly (aka peaks) over a certain period of time. I included the scores in the mock data on server side.

Later, I'm planning to revisit this project to do this calculation on client-side, but given we need a (w) of data-points to get the first value so there would be a little bit of lag in initializing the visualization.

UX Research

So now we know that we have two pieces of information at any certain point of time:

Actual sensor reading
A binary value for whether there's a peak or not

Let's start by asking ourselves a few questions:

How to show those 2 values effectively and clearly?
How to create readable and meaningful time series chart?
How to design unique and usable experience for the user interacting with the visualization?

Data Visualization should be "beautiful"

This is a must. Whichever use case you have or application you are building, if it's an IoT monitoring tool or a fancy marketing dashboard, an ugly looking visualization will discourage your end users from looking at it and may actually prevent them from understanding the meaning behind it and what insights the data entails.

But what is an "ugly" visualization? What makes it "beautiful" and visually appealing instead?

Well, this is subjective. There is no silver bullet. But with some conventional wisdom and lots of experience you realize that you should consider following:

Use the right type of diagram:

Each type of a diagram was designed to represent data in a different way and focus on one aspect of the data. Obviously, graphing population of distant cities on world map wouldn't be the best option and using a pie chart to display more than 2-3 variables is a big no, and so on.

Although there's some interesting visualizations for time series like Stream and Gantt charts and there's always room of creativity and artistic improvisation in data visualization, customers tend to like what they are familiar with. And we only have two variables in this project.

Well...this is a time series. It's gonna be a line graph.

Avoid overcrowded areas:

Too much elements in a tiny space can only be justified in very limited cases. Points in a scatter plot for example can be tolerated but also, it's called a scatter plot for a reason. Only when the crowded (or unscattered) data points have the same meaning you can allow crowding them together to show density, but if they have different meaning and you can't them apart because of the crowd jam ,you are doing something wrong.

This is why I started my visualization by drawing it in its simplest form then adding on top of it. I removed all of the chart junk for now and will add whatever I need along the way.

Avoid mismatching or vague colors:

Colors are very subjective too, and they are associated with different feelings. Some feelings are obvious like hot red and warm blue, but if your data doesn't represent temperature? Also, some feelings or ideas associated with certain colors are cultural and differ from one target group to another.

There's lots of science behind color theory and why we perceive colors the way we do.
So, for this challenge what I do is that I stick with some of the famous palettes that have proved to work over time. You can use this cool color wheel by Adobe to find some preset palettes or create your own based on color harmonies like Analogous, Triad, or Complementary colors, then they have an Amazing feature where you can Copy CSS or Less.

For this project, I went with this simple palette that has 2 shades of green and 2 shades of red.

Compare & contrast

Comparisons and contrasts lead to conclusions

A visualization has to reflect the meaning of data and be built as simple as possible to make comparisons easier so the user can draw conclusions.

First thing we need to contrast here is the Readings series to the Z-scores series, so instead of showing the two series in different graphs we can overlay the peaks over the original signal (Readings) and decreased **opacity* to 10%*.

We face a problem here with scales or the unit of each value. You can't place a binary value in a line chart along a numerical value like the sensor readings.

In this case we need to improvise. To show the Z-score as a pink region over the line series in D3, I converted it to an area series to span the whole height of the graph. I normalized the 0-1 values to be 0-X where X is the highest value of readings displayed currently in view.

We also need to provide the user with a way to compare the sensors data to each other. Why? So, the customer can see if the peak pattern is happening in one sensor or in all of them, and most importantly if the pattern is happening across all sensors at the exact time or if there's a shift.

Since I'm assuming there's only 3 sensors we are visualizing, we can't really use a small multiple. What we can do is stack the 3 graphs we have on top of each other, making sure that all graphs are horizontally aligned to each other.

Usability & Interaction

Usability is the ease of access of an interface. It’s a sub-discipline of UX. Although UX design and usability are sometimes used interchangeably, usability has grown to have to be more than the ease of access. Usability now is measurable. Measuring usability is out of the scope of this blog post, so will will take a holistic approach towards increasing usability in general.

Since we are here, we need to introduce a new terms: Dashboards and Widgets. A dashboard shows various semi-related visualizations that deliver a shared business value but not necessary from the same data source. Widgets are the building blocks of a dashboard.

The cards you've been seeing throughout the previous sections are all widgets. What do we need to consider now to make each single card/widget user friendly and most importantly relate them to each other ?

Labels & Controls

We need to show several labels to guide users where to look and help them understand what are they looking at, for this project we need to include the following:

Titles: dashboard title and sensor title.
Connectivity indicator: Here I'm assuming that the dashboard may get disconnect from a sensor for any reason. This happens a lot in IoT applications. We need to inform the user if one chart is outdated.
Time series legend: This will have 2 functions, it will tell the user which is the actual reading and which is the peak area, and it will act as a toggle so the user can show/hide one of the two variables.
Axes: Beside the fact that we need to show the units and values of each dimension, we need to make it clear in which direction time is moving.
(Extra element) Last reading Timestamp: Since for the x-axis, I'm truncating the time stamp to show only the seconds (:20, :30,...). I added the Full timestamp of the last reading in the bottom right corner of the widget.

States

Any frontend component goes through a cycle of several states. These states are driven by the business logic, in our case we have the following states for each Widget:

Connecting:
This is the initial state when the page is loading and we don't have enough information to show to the user.
Disconnected:
This is when a widget is disconnected for a server or client error. We also show the HTTP error message for debugging and helping users report their issues.

In this scenario we can't only rely on the Connectivity indicator, we need to explain to the user that the current data in view is not active. So we set the whole line graph opacity to 50%.

Connected: Everything is perfect 👌🏽
(Extra UI state) Mouseout:
This is mainly to make the visualization less cluttered and more visually appealing.

Although this is debatable and some designers don't favor it, I removed the x-axis and last Reading timestamp if the user is not hovering on a Widget.

My rationale behind this is that the customer is not really concerned with the exact time of each point but rather the main focus points of this visualization are the pink shaded Peak areas.
If users really want to know when that happened, they can hover on the graph.

**Final design:

Live Demo

The How

Engineering the solution

The frontend app contains two main classes:

Chart: src/components/Chart.js
- This is the main React component that connects to the relevant sensor to stream readings, store it then does some data manipulation logic and finally initialize and update the D3 chart.
- The React component has 1 required prop sensorID and optional x-ticks which has a default value of 20 and a max value of 50.
D3TsChart: src/d3-helpers/d3-ts-chart.js
- This is the custom class that handles the Time Series Chart graphics, and everything related to the chart SVG.
- Readings are passed to this class to be rendered in DOM using D3 but never stored in the class itself. Data lives in the Chart component state.

File organization:

:root             // React component
  > api           // Express JS App
  > src
    > components
    > d3-helpers
    > styles
  > public

Backend

The backend server is very simple, it's just a single Express.js file along with the the data.json file.

The data file contains mock data for 3 sensors. You can connect to the socket by pinging https://localhost:4001?socket={sensorId}. sensorId can only be 1 or 2 or 3 for now.

You can begin by creating the api folder and installing the 2 needed packages:

npm install -s socket.io express

First thing we need to import the server requirements and initialize the Express.js server (app) and wrap it by the socket.IO server (io). We will also import the JSON data and set a const INTERVAL of 1.5 seconds. This is how frequent we will emit data to each client connected to the server

const http = require('http');
const express = require('express');
const socketIO = require('socket.io');

const app = express();
const server = http.createServer(app);
const io = socketIO(server);

const port = process.env.PORT || 4001;

const INTERVAL = 1500;
const sensorData = require('./data.json');

To keep track of each client connected to the server we will create a custom object that will keep track of 1. which sensor data was requested by the client, 2. index of the next data point to serve, and 3. the setInterval reference that will emit data each 1.5 seconds (INTERVAL). Then we will store one object like this in a custom dictionary attached to the Socket.IO io object.

// Connection object interface
// {
//     sensorId,
//     index,
//     interval
// }

io.connections = {};

The idea behind storing the setInterval is that we need to emit the data periodically and we will also need to stop (clearInterval) this interval when a client disconnects from the server.

Now we need to listen to and handle the clients connect and disconnect in the Socket.IO server, and then emit data accordingly using the emitData function:

io.on('connection', (socket) => {
  const connectionId = socket.id;
  const sensorId = Number(socket.handshake.query['sensor']); //parse the sensorID
  console.log(`New client connected with id:${connectionId}`);

  // Add a client connection to the custom dictionary
  io.connections[connectionId] = {
    sensorId,
    index: 0,
    interval: setInterval(() => emitData(connectionId, socket), INTERVAL)
  };

  // Remove connection
  socket.on('disconnect', () => {
    clearInterval(io.connections[connectionId].interval)
    io.connections[connectionId] = undefined;

    console.log(`Client ${connectionId} disconnected`)
  });
});

After that we need to implement the emitData() function which basically:

Selects the relevant sensor data from the data file
calls getReading() which gets One Reading from the data file
Store the next Reading index in the connection object we created in io.connections dictionary last snippet.
Emit the reading with the event name 'reading'. We will listen to this in the client app next section.

const emitData = (connectionId, socket) => {
  let conn = io.connections[connectionId]
  const { newIndex, response } = getNextReading(sensorData[conn.sensorId - 1], conn.index);

  console.log(`Emitted to client: ${connectionId}, sensor id:${conn.sensorId}, index:  ${conn.index}`);

  socket.emit("reading", JSON.stringify(response));

  conn.index = newIndex;
}

// Get the next reading for the selected socket
const getNextReading = (data, index) => {
  response = {
    timestamp: Date.now(),
    value: data.readings[index],
    zscore: data.zScores[index]
  };

  return { newIndex: (index + 1) % data.readings.length, response };
}

Now if you add "start": "node index.js" to the scripts property in package.json file and then run npm start in the api folder, the server will be up and running and read to serve clients.

We can test the server using this awesome Electron app ...or go to the next section and start implementing the React app 👏🏽👏🏽👏🏽

Frontend

As mentioned before the client app will basically contain the main React component Chart that renders one chart and is responsible of controlling and passing data to the D3.js chart that lives inside the a separate custom d3-ts-chart class.

React App and `Chart` component

To initialize the React app we will use create-react-app. You can install that globally by running npm i -g create-react-app.

Then to initialize the actual code template we run create-react-app realtime-client. This will create a folder with the name "realtime-client" and npm install the needed packages inside it.

If you cd into the folder and run a simple react-scripts start you should have a simple react app built and served in your browser on http://localhost:3000/.

Note that this will be the root folder of the project and the backend server will live in a subdirectory inside it with the name api.

Now we need to install the extra packages we will use in the project. cd into the folder and run npm i -s socket.io node-sass d3.

I'm using node-sass to write the app styles which means you need to rename all the .css files into .scss and change the reference in the index.js file.

Let's build a component

The final Chart component is a big one. I will focus on the important parts here.

We will need to define some basics stuff:

The series list: which is a list of information about the series/lines that will be graphed. This is we will pass to the D3TsChart later to initialize the chart.
tsChart is the D3TsChart object that we will code later and it's the one responsible of all D3 related operations.
socket is the socketId object that we will use to connect to the server and listen to the data.
State: The React component state in which we will store the data and some info and flags about the chart.

So the initial Component should start as the following:

import React from 'react';
import ReactDOM from 'react-dom';

import socketIOClient from 'socket.io-client';
import D3TsChart from '../d3-helpers/d3-ts-chart';

export class Chart extends React.Component {

  seriesList = [
    {
      name: 'sensor-data',
      type: 'LINE',
      stroke: '#038C7E',
      strokeWidth: 5,
      label: 'Readings',
      labelClass: 'readings',
    },
    {
      name: 'z-score',
      type: 'AREA',
      fill: 'rgba(216, 13, 49, 0.2)',
      stroke: 'transparent',
      strokeWidth: 0,
      label: 'Peaks',
      labelClass: 'z-score',
    }
  ]

  tsChart = new D3TsChart();

  socket;

  state = {
    data: [],
    lastTimestamp: null,
    connected: false,
    error: ''
  }

  componentDidMount() { }

  render = () => (
    <div className="card">
      <div className='chart-container'></div>
    </div>
  )
}
export default Chart;

Now we need to connect to the socket.IO server and fetch data for one sensor by its id. We will pass sensorId to the component as a prop. This should be done in the componentDidMount() function. After passing the component HTML element reference to tsChart and initializing the 2 lines to be drawn by D3, it will call the connect() function and will disconnect() in componentWillUnmount().

Also notice that we listen to the "reading" event coming from the server and attach the storeReading handler to it.

componentDidMount():

componentDidMount() {
    if (this.props['sensorId'] === undefined) throw new Error('You have to pass "sensorId" prop to Chart component');

    // Component enclosing DIV HTML reference.
    const parentRef = ReactDOM.findDOMNode(this);

    this.tsChart.init({
      // Let D3 draw the chart SVG inside .chart-container div
      elRef: parentRef.getElementsByClassName('chart-container')[0],
      classList: {
        svg: 'z-chart'
      }
    });

    this.tsChart.addSeries(this.seriesList[0]); // readings
    this.tsChart.addSeries(this.seriesList[1]); //z-score

    this.connect();
  }

  connect = () => {
    this.socket = socketIOClient(`/?sensor=${this.props.sensorId}`);
    this.socket.on('reading', this.storeReading);

    // Various Errors handling
    SOCKETIO_ERRORS.forEach(errType => {
      this.socket.on(errType, (error) => this.setError(errType, error));
    });
  }

  componentWillUnmount() {
    this.socket.disconnect();
  }

The Socket.IO several errors strings and other constants are to be found in the top of the file:

const SOCKETIO_ERRORS = ['reconnect_error', 'connect_error', 'connect_timeout', 'connect_failed', 'error'];
const MAX_POINTS_TO_STORE = 20;

Now we need to implement the storeReading function that will store the readings into the component state and pass the new data to the tsChart object.

We first push the new reading to the current data then we update the state.data with the last MAX_POINTS_TO_STORE items. We also store some meta data like the connected indicator and the lastTimestamp to be displayed in UI. LAstly, we call the updateChart() method.

storeReading():

  storeReading = (response) => {
    const reading = JSON.parse(response);
    this.setState((prevState) => {
      const data = prevState.data;
      const pointsToStore = Math.max(data.length - MAX_POINTS_TO_STORE, 0);

      data.push(reading);

      return {
        data: data.slice(pointsToStore),
        connected: true,
        error: false,
        lastTimestamp: new Date(data[data.length - 1].timestamp).toLocaleTimeString()
      };
    });

    this.updateChart();
  }

updateChart() is implemented as a separate function because this is where we calculate highestValueInView from the Readings series. This is done so we can normalize the zscores 0/1 and replaces the 1s with the highest value. This will essentially make the Peaks Area series take the whole height of the current data in view.

updateChart():

  updateChart() {
    const data = this.state.data;
    const highestValueInView = Math.max(...data.map(p => p.value));
    const zLine = data.map(p => ({
        timestamp: p.timestamp,
        value: p.zscore ? highestValueInView : 0
        })
      );

    this.tsChart.adjustAxes(data);
    this.tsChart.setSeriesData('sensor-data', data, false);
    this.tsChart.setSeriesData('z-score', zLine, false);
  }

This all the basic logic needed to pass the data to the D2TsChart class object.

Now we need to update the render() function to display the meta data we store in state:

final render():

 render = () => (
    <div className="card">

      <h2>{!this.state.lastTimestamp ? 'Connecting...' : `Sensor ${this.props.sensorId}`}</h2>

      <span className={'status ' + (this.state.connected ? 'success' : 'danger')}>
        {this.state.error}
        <i className="pulse"></i>
        {this.state.connected ? 'Connected' : 'Disconnected'}
      </span>

      <div className={'chart-container ' + (this.state.error ? 'faded' : '')}></div>

      <span className={'timestamp ' + (this.state.connected ? 'success' : 'danger')}>
        {this.state.connected ? '' : 'Last reading was at '}
        {this.state.lastTimestamp}
      </span>

    </div>
  )

Finally we need to update the React index.js to include the charts for the 3 Sensors we can fetch from API.

index.js:

import React from 'react';
import ReactDOM from 'react-dom';
import './styles/main.scss';
import Chart from './components/Chart';

ReactDOM.render(
  <div>
    <h1>Peak Detection Dashboard</h1>

    <Chart sensorId="1" />
    <Chart sensorId="2" />
    <Chart sensorId="3" />
  </div>
  , document.getElementById('root'));

You can find all the needed scss styles in the styles directory.

D3 Time Series line graph

Here's where all the actual "graphing" happens. This is the class where we import D3.js library and use it to append the different SVG elements to the HTML element stored inelRef.

We need to set some constants like TRANSITION_DURATION and MAX_Y_TICKS, and for now we only support two SERIES_TYPES in graphing: LINE and AREA.

So this is how we start with the basic class:

import * as d3 from 'd3';

const SERIES_TYPES = ['LINE', 'AREA'];
const TRANSITION_DURATION = 100;
const MAX_Y_TICKS = 6;

export default class D3TsChart {
  margin = { top: 10, right: 30, bottom: 30, left: 30 };

  outerWidth; outerHeight;

   init({ elRef, width, height, classList }) {
    this.elRef = elRef;

    // If no width/height specified, SVG will inherit container element dimensions
    if (width === undefined) this.responsiveWidth = true;
    if (height === undefined) this.responsiveHeight = true;

    this.outerWidth = width || this.elRef.offsetWidth;
    this.outerHeight = height || this.elRef.offsetHeight;

    this.classList = classList || {};

    this.draw();
  }
}

You will notice that we pass some initial config to the chart in the init function including a width and height which are used to set up the graph layout according to the Margin Convention.

draw():

  draw() {
    // Main SVG
    this.svg = d3.select(this.elRef)
      .append('svg')
      .attr('width', this.outerWidth)
      .attr('height', this.outerHeight)
      .classed(this.classList.svg || null, true);

    //Inner box group  (deducting margins)
    this.group = this.svg.append('g')
      .attr('width', this.outerWidth - this.margin.left - this.margin.right)
      .attr('height', this.outerHeight - this.margin.top - this.margin.bottom)
      .attr('transform', `translate(${this.margin.left} , ${this.margin.top})`)
      .classed(this.classList.group || null, true);

    // X Axis init
    this.xScale
      .range([0, this.outerWidth - this.margin.left - this.margin.right]);
    this.xAxisRef = this.group.append('g')
      .attr('transform', `translate(0,${this.outerHeight - this.margin.bottom})`)
      .classed('x-axis', true);

    // Y Axis init
    this.yScale
      .range([this.outerHeight - this.margin.bottom, 0]);
    this.yAxisRef = this.group.append('g')
      .attr('transform', 'translate(0, 0)')
      .classed('y-axis', true);
  }

Here we set the main SVG (with margins) and the inner group then we set the scales for X-axis and Y-axis.

Now we need to add the functions that will draw the series (line and area) inside the SVG.

Add Series:

  addSeries({ name, type, fill, stroke, strokeWidth, id }) {
    if (this.seriesDict[name]) throw new Error('Series name must be unique!');
    if (!SERIES_TYPES.includes(type)) throw new Error('Series type not supported!');

    this.seriesDict[name] = {
      type,
      ref: this.group.append('path')
        .attr('fill', fill || 'none')
        .attr('stroke', stroke || 'black')
        .attr('stroke-width', strokeWidth || 2)
        .classed('series', true)
        .classed('hidden', false)
    };
  }

  setSeriesData(name, data, adjustAxes = true) {
    const series = this.seriesDict[name];

    switch (series.type) {
      case 'AREA':
        this.updateAreaSeries(series, data);
        break;
      case 'LINE':
      default:
        this.updateLineSeries(series, data);
        break;
    }
  }

Updating data of a single series:


  updateLineSeries(series, data) {
    series.ref
      .datum(data)
      .transition().duration(TRANSITION_DURATION).ease(d3.easeQuadIn)
      .attr('d', d3.line()
        .x((d) => { return this.xScale(d.timestamp); })
        .y((d) => { return this.yScale(d.value); })
      );
  }

  updateAreaSeries(series, data) {
    series.ref
      .datum(data)
      .transition().duration(TRANSITION_DURATION).ease(d3.easeQuadIn)
      .attr('d', d3.area()
        .x((d) => { return this.xScale(d.timestamp); })
        .y0(this.yScale(0))
        .y1((d) => {
          return this.yScale(d.value);
        })
      );
  }

Then finally we will have a function to adjust the axes to the current data in view.

adjustAxes():

 adjustAxes(data) {
    const maxValue = d3.max(data, (d) => d.value);

    this.xScale.domain(d3.extent(data, (d) => d.timestamp));
    this.xAxisRef
      .transition().duration(TRANSITION_DURATION).ease(d3.easeLinear)
      .call(d3.axisBottom(this.xScale));

    this.yScale.domain([0, maxValue]);
    this.yAxisRef
      .transition().duration(TRANSITION_DURATION).ease(d3.easeLinear)
      .call(
        d3.axisLeft(this.yScale)
          .ticks(maxValue < MAX_Y_TICKS ? maxValue : MAX_Y_TICKS)
          .tickFormat(d3.format('d'))
      );
  }

You can have a look a deeper a look at this class in the D3TsChart definition file src/d3-helpers/d3-ts-chart.js.

Deploying to Heroku

To deploy to Heroku this app to heroku we need to setup the app so it:

Builds the react app and move the static webapp to api/public
Run the Express.js server

We can do so by adding the proper commands in the root package.json file.

Package.json:

...
  "scripts": {
    "start": "node api/index.js",
    "prebuild": "rm -r api/public",
    "build": "react-scripts build",
    "postbuild": "mv build api/public"
  }
...

Heroku will automatically detect that this is a Node.js app and will run the npm commands correctly.

The final step here is to set up the ExpressJS app so it serves the static app in api/public directory.

ExpressJS index.js:

  app.use(express.static(__dirname + '/public'));

That's all folks.

Related Links:

Designing Data Intensive Applications - Book Review

Hatem Hassan 👨‍💻☕️💻🌺😎 — Sun, 13 Oct 2019 11:29:57 +0000

Originally posted on my personal blog

This book is definitely a classic. The book basically touches upon the surface of a wide array of topics related to handling data in a distributed environment; ranging from basic database theory, ACID, replication and partitioning to more complex (and "modern") topics like stream and batch processing on the cloud.

"Data outlives code.”

- Martin Kleppman

Martin Kleppman lays down all the needed information on what every engineer needs to know about designing systems that deal with any kind of data.

The first part is about basic database concepts:

Relational vs. NoSQL
Different query languages
How data is actually persisted on storage devices. (B-Trees / LSM Trees)
Encoding and serialization/deserialization

The second part goes deeper and discusses the following concepts (and their issues):

CAP Theorem
Replication
Partitioning
Transactions
Consistency

The final chapter discusses derived data and aims to tackle the issues discussed in the previous part, as well as introducing more "modern" concepts like:

Batch and Stream processing (MapReduce/Spark)
Eventual consistency and "Change Data Capture"

Although the book doesn't dive into any deep technical or implementation details, it has a very good bibliography and footnotes that leads you to all of the academic papers you need. Overall, the book is an essential read for any software/data engineer in 2019.

Instagram Art:

What I learned in 40 hours switching from Angular to React!

Hatem Hassan 👨‍💻☕️💻🌺😎 — Sat, 12 Oct 2019 23:01:22 +0000

Originally posted on my personal blog

I’ve been building webapps with Angular 2+ and Ember.js since 2016 but I never got the chance to start a React project before.

I once did some changes in an existing react app when I was working at CrazyEgg but they were mostly CSS. I really never had the full experience. I never dug deep into React.

So I decided to do so and I allocated 40 hours to build a simple React CRUD app and dig my way into the source code and to learn how it really works throughout the journey.

I use Toggl to log my time. Awesome tool.

React is a library

Angular is not.

Angular is an opinionated MVC/MVVM (or MVW) framework that comes with several smaller packages (or libraries) that handle a wide array of functionalities like routing, http requests, animations, forms,…etc.

On the other hand, react is just a V. It's really only concerned with the View layer. Woah woah hold on! Don't get me wrong, I'm not trying to put down React. Actually that kind of specialization gave React a couple of advantages and it definitely beat Angular in some areas that are discussed in the rest of this post.

One could say that React gives you much more freedom. Less boilerplate and faster initial setup. But however if you are going to build a complex webapp using react you need to rely on other libraries like react-router, react-promise, or Redux. Maybe Enzyme for testing.
It's common in React that you may rely on state management libraries like Redux or MobX.
Of course we still rely on external libraries in Angularverse like RxJS or NgRx for state management.

One Angular disadvantage that it has a much steep learning curve. Angular architecture is kinda influenced by backend Java frameworks. It is built using TypeScript (a must actually) and it has a clear-ish standard file structure. If you come from backend background, that would be great. Otherwise, you have to understand what is dependency injection and what are static types, add to that some basic OOP concepts like inheritance and polymorphism.

React's architecture ? It doesn't have any.

There are some best practices though.

Which can be great if you want to do a simple SPA or crazy animation magic without worrying about the project structure hassle. But it also means a whole lot more decision making if you wanna build a full-fledged app, an analytics dashboard for example. These decisions mostly include picking third-party libraries, choosing how to go about code organization and file structure, sticking with JS or using TypeScript (yes you can TS in react).

Markup

"Let's have everything in one place" - A wild Facebook Engineer, 2013

React introduced JSX and everyone went crazy. It was controversial for brief 2 years then the frontend community realized it's actually not that bad.

In JSX you have JS logic inside your HTML …or maybe you actually write HTML inside JS .. wut?! 🤔🤔🤔

// Something like this
const element = <h1>Hello, world!</h1>;
// or even this
const element = (
  <h1>
    Hello, {formatName(user)}!
  </h1>
);


// Look at this
const greetings = ['hola', 'Hello', 'Hey', 'sup'];

const listElement = (
            <ul>
              {greetings.map(
                   greeting => <li>greeting</li>  // 🤯🤯🤯
              )}
            </ul>)

On the other hand the conservative Angular has a clear separation of UI and logic. You write your template in the HTML file in TypeScript in the .ts file like how our parents raised us.

This distinction is highly debatable. While the Angular way is more readable and easy to understand if you come from any formal framework-y experience. JSX has better code completion and compile-time checks as everything is one file. Keep in mind, you also have to learn how to use custom directives like ng-for and ng-if to make use of Angular special "HTML on steroids" version.

Data binding

Whether you have a separated template file and TS file or you have them mixed up in a JSX file, at the end of the day you have a UI and some logic that runs it. Say you have an input field in the template and it pulls its value from a JS variable.

Angular MVVM data binding.

In Angular this value can flow in both directions, if you change it programmatically in your TS code or if you changed in your browser writing in the input field. This is called two-way data binding.

React works a little bit differently. You can only change this state (value of the JS variable) programmatically, and then the UI gets updated automatically. But if user changed the HTML input field value from the browser, that doesn't reflect on the JS variable. This is one-way binding. To achieve two-way binding in React we rely on callbacks and DOM events.

// Updating state in React
<input value={someData} onChange={({ target: { value }}) => this.setState({ someData: value })} />

DOM Manipulation

If you know what you are doing, React is fast. Really fast.

The reason behind this is that React uses a Virtual DOM which basically diffs the DOM tree looking for changes and then only updates the changes (like how git works).

Angular uses Regular DOM that traverses the whole tree till it finds the target element it wants to change and then edit it.

In the grand scheme of things this difference in performance is negligible unless you are doing crazy stuff with lots of elements in the page (Say more than 1500). This can actually block the main thread of the JS engine and freeze the UI if you are not careful.

Unit Testing

Using Angular CLI you can set up your project to include the boilerplate file provided by Jasmine testing framework which runs by Karma test runner. You can also use Karma to replace Jasmine with other frameworks like Mocha or QUnit. Karma plays very well too if you want to integrate with continuous integration services like Jenkins, TravisCI, or CircleCI.

React uses Jest out of the box with no need to do any configuration. You can also use AirBnb's Enzyme library next to Jest. Enzyme mimics JQuery API to facilitate DOM interactions in tests.

That's all folks!

You Don't Know JS - Book Review

Hatem Hassan 👨‍💻☕️💻🌺😎 — Sat, 12 Oct 2019 22:40:14 +0000

Originally posted on my personal blog

Just finished You don't Know JS series by Kyle Simpson. It took me almost 4 months to get through the 6 books, and I will definitely go back to reread certain parts of it.

I think this series is essential for anyone who want to dive deep into Javascript. From syntax gotchas and quirks to how coercion works to asynchronicity (or the illusion of asynchronicity).

Kyle Simpson has a unique way in explaining complex CS concepts and ideas in simple words.

Types & Grammar

This book is not as interesting or intriguing as other books in the series because it goes through the basic stuff then it's basically a list of Gotchas and quirks in the language logic. I gave it 5 stars because Kyle's explanations and examples are amazing even if the material itself is dull.

`this` & Object Prototypes

The book explains the deep quirks and gotchas of the dynamic nature of JavaScript embodied in the [[Prototype]] concept. It gives you a general understanding of how objects are created and how behavior is executed or rather "delegated" in JS.

Kyle in the last chapter proposes his own design pattern OLOO (Objects Linked to Other Objects) which aims to solve many of the issues of trying to code JS in an traditional OO pattern. I have my reservations on the pattern and how readable it's syntax is.

The fact that he wants each object to have it's unique method names and prohibits overriding at all is beyond me, it makes code more complex, less intuitive and also doesn't make it any easier for new engineers to work with an already existing codebase. However, he lays down some strong arguments against writing JS code in traditional OOP style, which ignore or "hide" the dynamic nature of the language and try to tame it into a static one.

Async & Performance

Amazing book if you want to understand how asynchronicity in JavaScript works. Kyle has a unique way of explaining deep technical CS concepts in a simple and straightforward manner. He lays down the differences between concurrency, serialism and parallelism and then use those differences to explain the JS Event loop and then goes into the technicalities of implementation using callbacks, promises, and generators. Finally, he gives some overview of assessing performance and benchmarking.

ES6 & Beyond

I consider this book to be the most important one in the You don't know JS series, especially for those who have some experience and have been working with Javascript for years. It goes through all of the tricks and gotchas of ES6 and as usual Kyle's way of explanation is very clear and straightforward.

Overall, the books are amazing and mind opening, and I highly recommend them.

Data driven Frontend development using RNN and Markov Chains

Hatem Hassan 👨‍💻☕️💻🌺😎 — Sat, 12 Oct 2019 22:13:31 +0000

Originally posted on my personal blog

The dark ages of the Web

Throughout my career as a frontend engineer, I've worked with many libraries, packages, and dependencies. I admit that when I used JQuery for the first time almost 10 years ago I never really thought about what was happening behind this innocent <script> tag. I was amazed with how easy it to $('.cool-logo').slideUp(), I didn't even consider how it works behind the scenes.

  <script src="https://code.jquery.com/jquery.min.js"></script>
  <script src="/assets/slideshow.js"></script>
  <script src="/assets/form-validations.js"></script>
  <script src="/assets/moment.js"></script> <!-- We need those `a year ago` strings, don't we? -->
  <script src="/assets/thatAnimationThingWeUseInOnePage.js"></script>

-- Part of a very cool website <head> tag.

Not only is this way hard to manage, because if one library depends on another we will probably mess it up, it also can make your website size grow rapidly without even noticing. You don't know how many of these libraries imports loadash or which version does it import. Are these libraries updated and secure? Are there duplicates? 🤷🏽‍♂️

We need a change

Now the web has evolved a lot and we don't only have libraries, we have frameworks, like Angular. We've seen lots of changes and innovation in the way we build web applications. JavaScript bundling is a major component of any Frontend framework in 2019. Basically what a Static File Bundler does is put your JavaScript files (and assets) and all its dependencies together in one (or more) file(s). Two of the most popular bundlers are browserify and webpack.

Webpack

Webpack is wildly adopted because it is the bundler used by Angular CLI to build production assets. In case of static websites, what it does is finding and eliminating all those random <script> tags in HTML all over your project and only includes one single JavaScript file (or a few).

Getting started with Webpack configuration can have a steep learning curve but it is nothing compared to manually managing dependencies. The basic concept a beginner engineer needs to understand about Webpack is that you give it an entry file; which Webpack will access to recursively look for all those imports and requires to figure out all of the dependencies in the project. Then it builds a dependency tree, which is very useful for various reasons, one of which is removing duplicate libraries. Finally it compiles (and possibly compress) everything into bundle(s).

It doesn't stop here!

Bundle optimization is another hot topic right now. Frameworks like Angular are trying optimize how these bundles are built and splitted. For example, if you use one big graphing library in some pages of a dashboard Webapp, it doesn't really make sense to load this library in all pages and right away. What would you do with it in the /login page? Always remember that our goal is to eventually decrease page load time.

That's one reason we divide our Angular app into modules. Angular and Webpack will create a specific bundle for all of the pages contained in that module and load it on demand. Assuming the modules structure is mapped to the path structure, this can be called Route-level code splitting.

There are lots in play when it comes to bundle optimization. Lazy loading, Eager loading and Preloading are all strategies used to optimize bundling and decrease page load time. You can read more about it in this sweet article

Machine Learning comes to the rescue

Googlers from the Angular team started a very cool project called Guess.js to tackle the bundling issues in Angular as well as static sites.

Google's Guess.js is optimizing code bundling and prefetching using TensorFlow.js RNN Machine Learning model (or Markov Chain) to learn navigation patterns. These patterns are used to predict users' next transition, as in next possibly visited page (or pages). Why? to prefetch these pages and provide instant transitions with your application. Cool, right?

WTH is RNN?

RNN is a Recurrent Neural Network that uses its internal memory to process a sequence of inputs. In this case, think about a web navigation sequence:

User 1: /login => /dasboard => /dasboard/report/1 => /dashboard/report/2 => /logout
User 2: /login => /account  => /dashboard/add/user => /dashboard/add/user/success/ => /logout
User 3: ...

What RNN does is that it learns the common patterns in such sequences then given a sequence of inputs it can predict the next item in the sequence.

But why RNN ?

The output of an RNN unit does not only depend on the current input but it also considers the previous hidden state which carries past information. This means the network learns from its past to come up with better target (prediction).

**Note:* I'm not sure how RNNs are implemented exactly inside Guess.js, this is just a description of how RNNs are relevant to navigation patterns prediction*.

There are limitations of RNN though:

These are the two variants of RNN that tackle these limitations:

The most obvious difference between those two is GRU has output and hidden state, while LSTM has output, hidden state and cell state. Check the list in the end of this post for more details about RNNs.

So in summary RNN contains internal state that gets updated every time we feed it with new input. During output sequence prediction, the knowledge of the past is used through hidden states.

What about Markov Chains?

Once again, this is general description of Markov Chains, not Guess.js implementation.

Markov chain is named after mathematician Andrey Markov and it is a probabilistic model that simulates the flow from one "state" to another. In other words, in a space of multiple events/states, the model can tell us how likely it is that we "hop" from event A to event B, or from B => C, what about B => C => A, and so on.

In our case, a Markov Model would give us for instance the probability of a certain user jumping from the /product page to the /checkout page. So if there is a high probability of the user "transitioning" from this product page to the checkout page, Guess.js can start loading that Stripe payment JS bundle in the background.

Data-driven predictions

So how do we get those sequences? Where is the data?

The brilliant thing about Guess.js is that it grabs its data from Google Analytics to better train the RNN model and perform data-driven route-level JavaScript parsing and code splitting optimizations. That's how it predicts which JavaScript bundle it should load next and when.

Here comes the cool part. Google Analytics has been widely used by many websites for years and it provides exactly the kind of data we need to feed Guess.js.

This combination of Google Analytics + Guess.js automagically figures out the best way to bundle your application and optimize its resources for better performance.

It even goes so much deeper that it can predict the next piece of content (article, product, video) a user is likely to want to view and adjust or filter the user experience to account for this. It also can predict the types of widgets an individual user is likely to interact with more and use this data to tailor a more custom experience.

I honestly think this is a breakthrough in Machine Learning empowering Customer Experience and Web Performance. It is basically instant page transitions.

Guess.js Github
You can read more about RNNs here.
Check this awesome visual playground of Markov Chains.
Much Math

Forem: Hatem Hassan 👨‍💻☕️💻🌺😎

That time I built a tool that helps 1000 CS students every month

Adoption

Did you notice any patterns in the visitors timeline?

Where do those visitors come from?

What is Quine Mccluskey?

Fullstack NLP: Building & Deploying End-to-end Fake News Classifier

NLP Classifier

Text Preprocessing

Baselining

Feature Engineering & Pipeline

Pickling

Web Application Architecture

Flask API Server

React User Interface

Bundling

Dockerizing & Deployment

Realtime Data Visualization with Peak Detection using Socket.IO, D3 and React 📈📈

The Why?

Use case: what is happening here?

Peak detection

UX Research

Data Visualization should be "beautiful"

Compare & contrast

Usability & Interaction

The How

Engineering the solution

Backend

Frontend

React App and Chart component

D3 Time Series line graph

Deploying to Heroku

Designing Data Intensive Applications - Book Review

"Data outlives code.”

What I learned in 40 hours switching from Angular to React!

React is a library

Markup

Data binding

DOM Manipulation

Unit Testing

You Don't Know JS - Book Review

Types & Grammar

this & Object Prototypes

Async & Performance

ES6 & Beyond

Data driven Frontend development using RNN and Markov Chains

The dark ages of the Web

We need a change

Webpack

Machine Learning comes to the rescue

WTH is RNN?

But why RNN ?

What about Markov Chains?

Data-driven predictions

React App and `Chart` component

`this` & Object Prototypes