Forem: Harshad Ranganathan

Spark SQL - DataFrames & Datasets

Harshad Ranganathan — Tue, 05 Nov 2019 00:00:00 +0000

Spark SQL

Spark SQL lets you query structured data inside Spark programs, using either SQL or a familiar DataFrame API.

DataFrames and SQL provide a common way to access a variety of data sources, including Hive, Avro, Parquet, ORC, JSON, and JDBC. You can even join data across these sources.

Spark SQL supports the HiveQL syntax as well as Hive SerDes and UDFs, allowing you to access existing Hive warehouses.

Spark SQL includes a cost-based optimizer, columnar storage and code generation to make queries fast.

DataFrames

A DataFrame is a Dataset organized into named columns. It is conceptually equivalent to a table in a relational database or a data frame in R/Python but with richer optimizations under the hood.

DataFrame is represented by a Dataset of Rows. In the Scala API, DataFrame is simply a type alias of Dataset[Row].

DataFrames are untyped and the elements within DataFrames are Rows.

Below is a sample dataframe which is composed of rows & columns -

scala> df.show
+---------------+--------+-----------+--------------------+--------+
| capital|currency|independent| name| region|
+---------------+--------+-----------+--------------------+--------+
| Canberra| AUD| true| Australia| Oceania|
| London| GBP| true| United Kingdom| Europe|
|Washington D.C.| USD| true|United States of ...|Americas|
| Paris| EUR| true| France| Europe|
| Tokyo| JPY| true| Japan| Asia|
| Dublin| EUR| true| Ireland| Europe|
+---------------+--------+-----------+--------------------+--------+

Datasets

A Dataset is a distributed collection of data. Dataset is a new interface added in Spark 1.6 that provides the benefits of RDDs (strong typing, ability to use powerful lambda functions) with the benefits of Spark SQL’s optimized execution engine.

Datasets are typed distributed collections of data and it unifies DataFrame and RDD APIs.

Datasets require structured/semi-structured data. Schemas and Encoders are part of Datasets.

Comparisons

RDD vs DataFrames vs Datasets

RDD	DataFrames	Datasets
Functional transformations	Relational	Functional & Relational transformations
No type safety	No type safety	Type-safe
No built-in optimization	Catalyst query optimization	Catalyst query optimization**
In-memory JVM objects resulting in GC & java serialization overheads	Tungsten execution engine - Off-Heap Memory Management using binary in-memory data representation	Tungsten execution engine - Off-Heap Memory Management using binary in-memory data representation
	JIT code generation	JIT code generation
	Sorting/suffling without deserialization	Sorting/suffling without deserialization
	No encoders	Encoders - generate byte code and provide on-demand access to attributes without the need for deserialization

DataFrames vs Datasets

DataFrames	Datasets
DataFrames are untyped - No type check by the compiler	Datasets are typed distributed collections of data - compile-time type safety
Transformations on dataFrames are untyped	Datasets include both untyped and typed transformations

Catalyst Optimizer

At the core of Spark SQL is the Catalyst optimizer, which leverages advanced programming language features (e.g. Scala’s pattern matching and quasi quotes) in a novel way to build an extensible query optimizer.

Catalyst supports both rule-based and cost-based optimization.

Image from https://databricks.com/wp-content/uploads/2018/05/Catalyst-Optimizer-diagram.png

On our sample dataset of countries and currencies, let’s find the most used currency. We will perform a group by operation on the currency column and then order it in descending order based on the counts.

scala> df.groupBy("currency").count().orderBy(desc("count")).show()
+--------+-----+
|currency|count|
+--------+-----+
| EUR| 2|
| JPY| 1|
| AUD| 1|
| GBP| 1|
| USD| 1|
+--------+-----+

Let’s see the catalyst optimization for the above query.

scala> df.groupBy("currency").count().orderBy(desc("count")).explain(extended=true)
== Parsed Logical Plan ==
'Sort ['count DESC NULLS LAST], true
+- Aggregate [currency#149], [currency#149, count(1) AS count#202L]
   +- Relation[capital#148,currency#149,independent#150,name#151,region#152] json

== Analyzed Logical Plan ==
currency: string, count: bigint
Sort [count#202L DESC NULLS LAST], true
+- Aggregate [currency#149], [currency#149, count(1) AS count#202L]
   +- Relation[capital#148,currency#149,independent#150,name#151,region#152] json

== Optimized Logical Plan ==
Sort [count#202L DESC NULLS LAST], true
+- Aggregate [currency#149], [currency#149, count(1) AS count#202L]
   +- Project [currency#149]
      +- Relation[capital#148,currency#149,independent#150,name#151,region#152] json

== Physical Plan ==
*(3) Sort [count#202L DESC NULLS LAST], true, 0
+- Exchange rangepartitioning(count#202L DESC NULLS LAST, 200)
   +- *(2) HashAggregate(keys=[currency#149], functions=[count(1)], output=[currency#149, count#202L])
      +- Exchange hashpartitioning(currency#149, 200)
         +- *(1) HashAggregate(keys=[currency#149], functions=[partial_count(1)], output=[currency#149, count#207L])
            +- *(1) FileScan json [currency#149] Batched: false, Format: JSON, Location: InMemoryFileIndex[file:/C:/Users/ranganathanh/Downloads/countries.json], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<currency:string>

Here,

Analyzed Logical Plan - Resolved relations, references, aliases and updated attribute nullability.

Optimized Logical Plan - Introduced a projection of column “currency” as we are interested in only that field for the rest of our operations.

Physical Plan - In the final physical plan, notice the ReadSchema operation as part of FileScan. It only reads in the “currency” field and ignores the rest.

Code references:

https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala

https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala

Tungsten Execution Engine

Tungsten improves the efficiency of memory and CPU usage.

Spark explicitly manages memory and converts most operations to operate directly against binary data.

Columnar layout for memory data avoids unnecessary I/O and accelerates analytical processing performance on modern CPUs and GPUs.

Vectorization allows the CPU to operate on vectors, which are arrays of column values from multiple records. This takes advantage of modern CPU designs, by keeping all pipelines full to achieve efficiency.

Image from https://arrow.apache.org/img/simd.png

Tungsten also performs -

[1] Whole-Stage Code-Generation - Instead of having each operator as an individual function, combine and compile all of those stages into single function.

[2] Cache Locality - Tungsten uses algorithms and cache-aware data structures that exploit the physical machine caches at different levels - L1, L2, L3.

[3] Loop unrolling and SIMD - Optimize Apache Spark’s execution engine to take advantage of modern compilers and CPUs’ ability to efficiently compile and execute simple for loops.

Structured Streaming

Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine.

The key idea in Structured Streaming is to treat a live data stream as a table that is being continuously appended. This leads to a new stream processing model that is very similar to a batch processing model.

For a word count example, when the query is started, Spark will continuously check for new data from the socket connection. If there is new data, Spark will run an “incremental” query that combines the previous running counts with the new data to compute updated counts.

Image from https://spark.apache.org/docs/latest/img/structured-streaming-example-model.png

You can use the common entry point SparkSession to create streaming DataFrames/Datasets from streaming sources, and apply the same operations on them as static DataFrames/Datasets.

There are a few built-in sources, e.g. for files, supported file formats are text, csv, json, orc, parquet.

Encoders

Encoder is the fundamental concept in the serialization and deserialization (SerDe) framework.

Encoders are used to convert a JVM object of type T to and from the internal Spark SQL representation.

Encoders know the schema of the records. This is how they offer significantly faster serialization and deserialization (comparing to the default Java or Kryo serializers).

So every data set has encoders to go along with it. And these encoder things are extremely specialized, optimized code generators that generate custom bytecode for serialization and deserialization of your data.

And this special representation is Sparks internal Tungsten’s binary format which allows these operations to happen on already serialized data greatly improving the memory utilization.

What to use ?


RDD	You have unstructured data If your unstructured data cannot to be reformulated to adhere to a schema You need to manage low-level details of RDD computations You have complex data types which can’t be serialized with Encoders
DataFrames	You have structured/semi-structured data You need best performance
Datasets	You have structured/semi-structured data You need type-safety You need functional APIs You need good performance

Example

RDD, DataFrames & Datasets have their pros & cons. Let’s see with an example where we will utilize all three of them (Interoperability) to achieve the best performance.

Our datasource is a json file where each line contains json data.

{"name":"Australia","independent":true,"currency":"AUD","capital":"Canberra","region":"Oceania"}
{"name":"United Kingdom","independent":true,"currency":"GBP","capital":"London","region":"Europe"}
{"name":"United States of America","independent":true,"currency":"USD","capital":"Washington D.C.","region":"Americas"}
{"name":"France","independent":true,"currency":"EUR","capital":"Paris","region":"Europe"}
{"name":"Japan","independent":true,"currency":"JPY","capital":"Tokyo","region":"Asia"}
{"name":"Ireland","independent":true,"currency":"EUR","capital":"Dublin","region":"Europe"}

We will make use of Scala language for demonstrating Spark SQL here.

Let’s read the json file using DataFrameReader with the session available in our spark shell.

Spark session available as 'spark'.
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_202)

scala> val df = spark.read.json("C:\\Users\\ranganathanh\\Downloads\\countries.json")
df: org.apache.spark.sql.DataFrame = [capital: string, currency: string ... 3 more fields]

scala> df.printSchema()
root
 |-- capital: string (nullable = true)
 |-- currency: string (nullable = true)
 |-- independent: boolean (nullable = true)
 |-- name: string (nullable = true)
 |-- region: string (nullable = true)

Notice that the schema is automatically inferred here. For better performance, specify the schema instead of using schema inference.

DataFrameReader accepts a StructType object for schema. So, we construct it as below.

scala> import org.apache.spark.sql.types.{BooleanType, StringType, StructField, StructType}
import org.apache.spark.sql.types.{BooleanType, StringType, StructField, StructType}

scala> val schema = StructType(
     | Array(
     | StructField("name", StringType),
     | StructField("independent", BooleanType),
     | StructField("currency", StringType),
     | StructField("capital", StringType),
     | StructField("region", StringType)
     | )
     | )
schema: org.apache.spark.sql.types.StructType = StructType(StructField(name,StringType,true), StructField(independent,BooleanType,true), StructField(currency,StringType,true), StructField(capital,StringType,true), StructField(region,StringType,true))

Now specify the schema while reading the json file.

scala> val df = spark.read.schema(schema).json("C:\\Users\\ranganathanh\\Downloads\\countries.json")
df: org.apache.spark.sql.DataFrame = [name: string, independent: boolean ... 3 more fields]

scala> df.printSchema()
root
 |-- name: string (nullable = true)
 |-- independent: boolean (nullable = true)
 |-- currency: string (nullable = true)
 |-- capital: string (nullable = true)
 |-- region: string (nullable = true)

Now, we want to get the list of countries which have currency as “EUR”. We’ll explore how this can be achieved using both DataFrame & Dataset API’s.

We already have our dataframe from our previous file reading operation, so, we’ll apply filter and select transformations on it to achieve our result.

scala> df.filter(df("currency").contains("EUR")).select(df("name"), df("currency"), df("capital")).show()
+-------+--------+-------+
| name|currency|capital|
+-------+--------+-------+
| France| EUR| Paris|
|Ireland| EUR| Dublin|
+-------+--------+-------+

You can select and work with columns in three-ways:

[1] Using $ notation e.g. df.filter($"currency".contains("EUR"))

[2] Referring to the dataframe e.g. df.filter(df("currency").contains("EUR"))

[3] Using SQL query string e.g. df.filter("currency == 'EUR'")

Let’s checkout the physical plan of our previous query.

scala> df.filter(df("currency").contains("EUR")).select(df("name"), df("currency"), df("capital")).explain()
== Physical Plan ==
*(1) Project [name#245, currency#247, capital#248]
+- *(1) Filter (isnotnull(currency#247) && Contains(currency#247, EUR))
   +- *(1) FileScan json [name#245,currency#247,capital#248] Batched: false, Format: JSON, Location: InMemoryFileIndex[file:/C:/Users/ranganathanh/Downloads/countries.json], PartitionFilters: [], PushedFilters: [IsNotNull(currency), StringContains(currency,EUR)], ReadSchema: struct<name:string,currency:string,capital:string>

We infer two things from our DataFrame approach:

[1] Catalyst query optimization got applied - ReadSchema was reading only the required fields from the file and not all the columns

[2] There is no type checking on the column names we used in the filter and select operations. This will possibly cause runtime errors if the mentioned fields are not present.

Let’s try the same query using Datasets.

To convert the structured data from the file to a Dataset, you need to define case classes in scala.

scala> case class Country(
     | name: String,
     | independent: Boolean,
     | currency: String,
     | capital: String,
     | region: String
     | )
defined class Country

We redefine our file read operation to return a Dataset of type [Country] using the as method.

scala> val ds = spark.read.schema(schema).json("C:\\Users\\ranganathanh\\Downloads\\countries.json").as[Country]
ds: org.apache.spark.sql.Dataset[Country] = [name: string, independent: boolean ... 3 more fields]

Now, we try the same filter and select operations using the Dataset.

scala> ds.filter(country => country.currency.equals("EUR")).select($"name", $"currency", $"capital").show()
+-------+--------+-------+
| name|currency|capital|
+-------+--------+-------+
| France| EUR| Paris|
|Ireland| EUR| Dublin|
+-------+--------+-------+

And the corresponding query plan -

cala> ds.filter(country => country.currency.equals("EUR")).select($"name", $"currency", $"capital").explain()
== Physical Plan ==
*(1) Project [name#297, currency#299, capital#300]
+- *(1) Filter <function1>.apply
   +- *(1) FileScan json [name#297,independent#298,currency#299,capital#300,region#301] Batched: false, Format: JSON, Location: InMemoryFileIndex[file:/C:/Users/ranganathanh/Downloads/countries.json], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<name:string,independent:boolean,currency:string,capital:string,region:string>

We can infer below when using a Dataset -

[1] Catalyst query optimization isn’t applied - ReadSchema is reading all the fields from the file. Reason for this is that catalyst cannot infer the operation being performed when you use lamba functions in your filter operations.

[2] You get compile time type safety checks e.g. ds.filter(country => country.currency.equals("EUR")) utilizes the case class.

[3] In your select query you are ending up having to specify column expressions. This is because, Dataset supports both typed and untyped transformations. While filter is a typed transformation, select is an untyped transformation so you lose the type information.

Now we have seen both approaches, I would prefer to stick with the DataFrame API’s for reading files and performing selective transformations to achieve read performance.

I had earlier mentioned about interoperability. Let’s say you have your list of countries which have EUR as their currency.

You want to perform map operations to transform the data so that it can be inserted in a DB. In such a case, it makes sense to convert your DataFrame to a Dataset so that you can reap the benefits of having a case class (Encoders!!!).

Encoders help to perform operations directly on the serialized data and they use internal Spark SQL representation.

All you have to do is to convert your dataframe once you have projected your fields to a dataset using the as method and import the spark implicits object available in your spark session.

import spark.implicits._ automatically creates encoders for your case classes.

df
  .filter(df("currency").contains("EUR"))
  .select(df("name"), df("currency"), df("capital")).as[Country]
  .map(country => {
     // your map operation
  })

Now, what will you return in your map function ?

Primitive types (Int, S tring, etc) and Product types (case classes) are supported by importing spark.implicits. Support for serializing other types will be added in future releases.


Returning case class with ciruclar dependency	Will fail as there is no supported encoder for it
Returning case class with Option fields	Will fail as there is no supported encoder for it
Return `null` value	You need to wrap it in a `Tuple1` object and return it

You can create custom encoders as well if there is no supported encoder for your return type. You can also create encoders from case class, java beans etc. and specify it in your transformations.

Now, after you have done your transformations you might want to write to a DB and need low level access (or) you don’t have a suitable encoder.

In that case, you can convert your Dataset to a RDD.

df
  .filter(df("currency").contains("EUR"))
  .select(df("name"), df("currency"), df("capital")).as[Country]
  .map(country => {
     // your map operation
  })
  .rdd

You can also convert your RDD to Dataset / DataFrame using toDS() and toDF() methods.

References

https://spark.apache.org/sql/

https://spark.apache.org/docs/2.2.0/sql-programming-guide.html

https://www.coursera.org/learn/scala-spark-big-data/home/week/4

https://mapr.com/ebook/getting-started-with-apache-spark-v2/assets/Spark2018eBook.pdf

https://mapr.com/blog/tips-and-best-practices-to-take-advantage-of-spark-2-x/

https://databricks.com/blog/2015/04/13/deep-dive-into-spark-sqls-catalyst-optimizer.html

https://spoddutur.github.io/spark-notes/second_generation_tungsten_engine

Connecting to Github with SSH (Windows)

Harshad Ranganathan — Sun, 22 Sep 2019 00:00:00 +0000

Pre-requisites

Git Bash

When to use SSH keys instead of HTTPS

Your corporate firewall blocks port 22, then even if you have set the remote origin to use HTTPS url, the authentication will fail as it is done via SSH.

Remedy is to set up SSH keys and use SSH over HTTPS port 443.

Below are some of the errors different applications throw when the port is blocked and HTTPS remote url is used.

Git Bash

When you try to push your changes, both Github Login and OpenSSH prompt asking for your username and password will fail with error remote no anonymous write access. fatal authentication failed for github.

Github Desktop Application

You will be able to login to your github desktop application as the authentication is done via HTTPS. However, when you try to push your changes it will result in authentication failure.

Below is the logs captured by the desktop application showing that the git push is trying to authenticate via SSH.

2019-09-22T13:16:46.927Z - info: [ui] Executing push: git -c credential.helper= -c protocol.version=2 push origin master:master --progress (took 21.740s)
2019-09-22T13:16:46.927Z - error: [ui] `git -c credential.helper= -c protocol.version=2 push origin master:master --progress` exited with an unexpected code: 128.
stderr:
ssh: connect to host github.com port 22: Connection timed out
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

(The error was parsed as 2: Authentication failed. Some common reasons include:

- You are not logged in to your account: see File > Options.
- You may need to log out and log back in to refresh your token.
- You do not have permission to access this repository.
- The repository is archived on GitHub. Check the repository settings to confirm you are still permitted to push commits.
- If you use SSH authentication, check that your key is added to the ssh-agent and associated with your account.)

Generate new SSH key

Run Git Bash.
Generate SSH key with your email id as comment.

$ ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

When you’re prompted to “Enter a file in which to save the key,” press Enter to save the key in the default location (/c/Users/username/.ssh/id_rsa). Your public key will also get saved here.
Copy the public key to clipboard.

$ clip < ~/.ssh/id_rsa.pub

Add public SSH key to your GitHub account

Go to Settings in your Github account to add the SSH public key.

Under SSH keys tab, select New SSH key.

Give a title and paste the key in the text area.

Add private SSH key to the ssh-agent

Git bash tool comes with a ssh-agent.

Create a new ~/.profile (or) ~/.bashrc file by running below command in git bash.

$ vi ~/.profile

Paste below script into your ~/.profile (or) ~/.bashrc file to auto launch the ssh-agent whenever you run your git bash shell.

env=~/.ssh/agent.env

agent_load_env () { test -f "$env" && . "$env" >| /dev/null ; }

agent_start () {
    (umask 077; ssh-agent >| "$env")
    . "$env" >| /dev/null ; }

agent_load_env

# agent_run_state: 0=agent running w/ key; 1=agent w/o key; 2= agent not running
agent_run_state=$(ssh-add -l >| /dev/null 2>&1; echo $?)

if [! "$SSH_AUTH_SOCK"] || [$agent_run_state = 2]; then
    agent_start
    ssh-add
elif ["$SSH_AUTH_SOCK"] && [$agent_run_state = 1]; then
    ssh-add
fi

unset env

This script will load the identities in the ssh agent from your default location ~/.ssh/id_rsa.

Identity added: /c/Users/username/.ssh/id_rsa (your_email@example.com)

Use SSH over HTTPS

This step is required only if your corporate firewall is blocking port 22.

Create a new config file in your .ssh directory i.e. /c/Users/username/.ssh/config

Paste below contents in the file to use port 443 for SSH connections to host ssh.github.com.

Host github.com
  Hostname ssh.github.com
  Port 443

Run below command in git bash to verify that the configuration is working. Ignore any authentication failures.

$ ssh -vT git@github.com

OpenSSH_8.0p1, OpenSSL 1.1.1c 28 May 2019
debug1: Reading configuration data /c/Users/username/.ssh/config
debug1: /c/Users/username/.ssh/config line 1: Applying options for github.com
debug1: Connecting to ssh.github.com [192.30.253.122] port 443.
debug1: Connection established.

You can see that the SSH connection is now established via port 443.

Verification

To check if everything works as expected perform below steps:

Run new git bash shell.
Check if the identity has been added to the ssh agent.

$ ssh-add -l

Check that the key is being used by trying to connect to git@github.com.

$ ssh -vT git@github.com

OpenSSH_8.0p1, OpenSSL 1.1.1c 28 May 2019
debug1: Reading configuration data
debug1: Offering public key
debug1: Server accepts key
debug1: Authentication succeeded (publickey).
Authenticated to ssh.github.com

Clone the repo using the SSH url.

Going forward, every push/pull will use the SSH keys to authenticate with Github.

References

https://help.github.com/en/articles/connecting-to-github-with-ssh

https://help.github.com/en/articles/using-ssh-over-the-https-port

Website Things To Do

Harshad Ranganathan — Tue, 09 Jul 2019 00:00:00 +0000

Secure

Make your site secure by enabling HTTPS on your website.

HTTPS helps prevent intruders from tampering with the communications between your websites and your users’ browsers. Intruders include intentionally malicious attackers, and legitimate but intrusive companies, such as ISPs or hotels that inject ads into pages.

Intruders exploit every unprotected resource that travels between your websites and your users. Images, cookies, scripts, HTML … they’re all exploitable. Intrusions can occur at any point in the network, including a user’s machine, a Wi-Fi hotspot, or a compromised ISP, just to name a few.

Github Pages

If you have your website on Github, you can enable HTTPS in the Settings section of your repository.

Let’s Encrypt

Let’s Encrypt is a free, automated, and open certificate authority which you can make use of to enable SSL for your website.

You can make use of Certbot client to automate certificate issuance and installation with no downtime.

Site Audit

It’s important to measure how your website performs in terms of performance, accessibility, best practices and SEO.

https://web.dev/measure - analyses your website and provides useful guidance

https://developers.google.com/web/tools/lighthouse/ - performs audits for performance, accessibility, progressive web apps, and more

https://developers.google.com/speed/pagespeed/insights/ - analyzes the content of a web page, then generates suggestions to make that page faster

All of the above are tools offered by Google and use data from Lighthouse project.

Discoverable

Google Search Console

Search Console provides tools and reports help you measure your site’s Search traffic and performance, fix issues, and make your site shine in Google Search results.

You can submit your sitemap.xml to google so that it can be crawled and indexed.

Analytics

Google Analytics

Google Analytics is a web analytics service offered by Google that tracks and reports website traffic.

Inside the console, you need to create new analytics accounts and apps with the option of custom views.

Once you have it set up, you can track your audience and real time views. You can also set include/exclude filters for your website.

To enable analytics for your site, add below script to your webpage with the ID of your google analytics property.

<script async src="https://www.googletagmanager.com/gtag/js?id=GA_MEASUREMENT_ID"></script>
<script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());
  gtag('config', 'GA_MEASUREMENT_ID');
</script>

Monetization

Google AdSense Ads

Google AdSense is a program run by Google through which website publishers in the Google Network of content sites serve text, images, video, or interactive media advertisements that are targeted to the site content and audience.

You need to submit your site for review with Google. It will take few days to weeks to have your account activated.

Once your account is activated, you can either choose enable auto ads or create custom ad units.

I would suggest to create a custom ad unit so that you can place your ads at appropriate places without affecting the user experience.

Auto ads issue: Adsense auto ads showing on mobile but not on the desktop

Also make sure to add ads.txt file to your site at root domain level.

GDPR

The General Data Protection Regulation (GDPR) and the ePrivacy Directive (ePR) affect how you as a website owner may use cookies and online tracking of visitors from the EU.

One of the most tangible requirements of the GDPR is in the definition of what constitutes a proper cookie consent, meaning, that the consent has to be:

Informed: Why, how and where is the personal data used? It must be clear for the user, what the consent is given to, and it must be possible to opt-in and opt-out of the various types of cookies.

Given by means of an affirmative, positive action that cannot be misinterpreted.

Given prior to the initial processing of the personal data.

Withdrawable. It must be easy for the user to change his or her mind and withdraw the consent.

There are several sites which offer GDPR compliant consent function for websites.

Open Graph Tags

Open graph meta tags allow you to control what content shows up when a page is shared on Facebook and Twitter.

<meta property="og:title" content="The Rock" />
<meta property="og:type" content="video.movie" />
<meta property="og:url" content="http://www.imdb.com/title/tt0117500/" />
<meta property="og:image" content="http://ia.media-imdb.com/images/rock.jpg" />

Proxy Filters

Lot of companies use proxy filters to restrict access to websites. Check if your website has been properly categorized to avoid being blocked.

https://sitereview.bluecoat.com/#/

DynamoDB Local in Docker

Harshad Ranganathan — Tue, 09 Jul 2019 00:00:00 +0000

Introduction

DynamoDB local Docker image enables you to get started with DynamoDB local quickly by using a docker image with all the DynamoDB local dependencies and necessary configuration built in. The new Docker image also enables you to include DynamoDB local in your containerized builds and as part of your continuous integration testing.

Image is available at: https://hub.docker.com/r/amazon/dynamodb-local

Docker Compose

We’ll create a docker compose file to launch dynamo db local as a container which can then be accessed by other containers.

version: '2'
services:
  dynamodb:
    image: amazon/dynamodb-local:latest
    ports:
      - "8000:8000"
    command: ["-jar", "DynamoDBLocal.jar", "-sharedDb", "-inMemory"]

Here,

image -

We pull the latest version of amazon/dynamodb-local image

ports -

Expose the container port 8000 to local port 8000 as the local dynamo db runs on this port

command -

By default the local dynamo db starts with inMemory setting. We override the command here with an additional argument -sharedDb. This is to ensure that the dynamo db uses a single database file instead of separate files for each credential and region. If you didn’t enable this setting then if in case your app container and local CLI use different AWS creds then they won’t be accessing the same dynamo db state. Table created via your CLI won’t be visible to your app container.

Run docker compose and access your local dynamo db by using the endpoint url option in your CLI commands.

docker-compose up

aws dynamodb list-tables --endpoint-url http://localhost:8000

If you’re not using the default network mode and instead using bridge network mode, this is how your docker compose will look like:

version: '2'
services:
  dynamodb:
    image: amazon/dynamodb-local:latest
    ports:
      - "8000:8000"
    command: ["-jar", "DynamoDBLocal.jar", "-sharedDb", "-inMemory"]
    networks:
      test-network:
        ipv4_address: 172.16.231.1
networks:
  test-network:
    driver: bridge
    driver_opts:
      com.docker.network.enable_ipv6: "false"
    ipam:
      driver: default
      config:
        - subnet: 172.16.231.0/24
          gateway: 172.16.231.1

Accessing DynamoDB Local Container From Another Container

Let’s say you have a webapp written in Java running in a tomcat container and you want to access the local dynamo db container for your integration tests.

You can make use of the hostname identical to the container name to access the local dynamodb from your webapp container, in this case, the endpoint url will be http://dynamodb:8000.

In your webapp, when you want to access the local dynamo db container, construct the client by setting the endpoint url to http://dynamodb:8000:

public AmazonDynamoDB dynamoDbClient(final String region) {
  return AmazonDynamoDBClientBuilder.standard()
      .withClientConfiguration(defaultClientConfiguration)
      .withCredentials(new DefaultAWSCredentialsProviderChain)
      .withRegion(region)
      .build();
}

/**
 * Returns Dynamo DB client to access local copy
 * where serviceEndpoint is http://dynamodb:8000 and region can be acceptable region names e.g. us-east-1
*/
public AmazonDynamoDB dynamoDbClient(final String region, final String serviceEndpoint) {
  return AmazonDynamoDBClientBuilder.standard()
      .withClientConfiguration(defaultClientConfiguration)
      .withCredentials(new DefaultAWSCredentialsProviderChain)
      .withRegion(region)
      .withEndpointConfiguration(new EndpointConfiguration(serviceEndpoint, region))
      .build();
}

Then in your docker compose file, you need to set the AWS creds and region for your webapp container.

version: '2'
services:
  dynamodb:
    image: amazon/dynamodb-local:latest
    ports:
      - "8000:8000"
    command: ["-jar", "DynamoDBLocal.jar", "-sharedDb", "-inMemory"]
  tomcat:
    image: webapp:latest
    environment:
      - AWS_ACCESS_KEY_ID=dummy
      - AWS_SECRET_ACCESS_KEY=dummy
      - AWS_REGION=us-east-1
    ports:
      - "80:8080"

Here,

image -

Docker image of your web app running in tomcat

environment -

Set AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY to any dummy value otherwise the credential chain will fail mentioning that it couldn’t detect any credentials.

Creating Tables

Before your application can read/write any data to the local dynamodb, you would have to create the required tables.

Either you can create the tables as part of your webapp code (or) you can bundle a script to your web app image which will create the tables using AWS CLI and start up your application.

Your webapp docker build file can be something like below:

FROM openjdk:8-jre-alpine

ENV TOMCAT_VERSION 8.0.36
RUN \
    wget https://archive.apache.org/dist/tomcat/tomcat-8/v${TOMCAT_VERSION}/bin/apache-tomcat-${TOMCAT_VERSION}.tar.gz && \
    tar xvzf apache-tomcat-${TOMCAT_VERSION}.tar.gz && \
    mv apache-tomcat-${TOMCAT_VERSION} tomcat && \
    rm -f apache-tomcat-${TOMCAT_VERSION}.tar.gz

RUN pip install awscli --upgrade --user
ENV PATH="$PATH:/root/.local/bin"

COPY start.sh .

ENV JPDA_ADDRESS 8000
ENV JPDA_TRANSPORT dt_socket
ENV JDPA_OPTS -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000

EXPOSE 8080 8000 8009

ENTRYPOINT ["start.sh"]

And in your start up script, create the tables and run catalina.

#!/usr/bin/env bash
aws dynamodb create-table --table-name <table_name> --attribute-definitions <attribute_definitions> \
--key-schema <key_schema> --billing-mode <billing_mode> --endpoint-url http://dynamodb:8000 --region <region>

./tomcat/bin/catalina.sh jpda run

Blockchain

Harshad Ranganathan — Sun, 09 Jun 2019 00:00:00 +0000

Introduction to Blockchain

Blockchain is a decentralized, distributed and public digital ledger that is used to maintain a continuously growing list of records, called blocks, which are linked using cryptography. Any involved record cannot be altered retroactively, without the alteration of all subsequent blocks.

Benefits of Blockchain

Decentralized - Not owned by a single entity. Every participant in the network can access the history of transactions or confirm new transactions.
Transparency - Public verifiability of transactions.
Security - With no central point to be exploited the system is protected against hacking attacks and fraud.
Immutability - Blocks added to the blockchain can’t be tampered.
No Intermediary - Peer-to-Peer and business-to-business transactions are completed without the need for a third party.

How Blockchain Works

Blockchain starts with a single block called genesis block.

Each block stores the following information:

Field	Description
Index	Genesis block will have an index of 0.
Timestamp	A record of when the block was created. Timestamp helps to keep the blockchain in order.
Hash	Digital fingerprint of data. Hash is of fixed length, easy to compute and infeasible to convert back to data.
Previous Hash	Hash of the previous block.
Data	Each block can store data.
Nonce	Number used to find a valid hash.

We’ll explain these with the mining of a new block.

Index

| Each new block is given an incremental index value, in this case, it’s 1. |

Timestamp

Block creation timestamp in epoch time, 1560082985.203.

Hash

SHA256 hash of (index, previous hash, timestamp, data, nonce)

CryptoJS.SHA256(1 + '0000018035a828da0878ae92ab6fbb16be1ca87a02a3feaa9e3c2b6871931046' 
+ 1560082985.203 + 'HarshadRanganathan' + 29877).toString()

000056f493569b609d6484da94b9031238e80076a8b6a373ae76d3db2746c211

Previous Hash

Hash of the previous block which is ‘0000018035a828da0878ae92ab6fbb16be1ca87a02a3feaa9e3c2b6871931046’.

Data

Block data.

Nonce

We start with a nonce of ‘1’ and keep incrementing it until we find a valid hash.

A Hash is considered to be valid if the number of leading zeros matches the difficulty.

For example, here we have set the difficulty as ‘4’. We will continue to re-generate the hash until it has 4 leading zeros.

As the difficulty increases, the number of possible valid hashes decreases, so it will take more processing power to find a valid hash.

A new block is added to the blockchain only if it meets the following requirements:

New block has a valid index i.e. block index should be greater than latest block index.
New block’s previous hash is valid i.e. previous hash equals latest block hash.
New block has valid hash i.e. it has been correctly calculated.
New block’s hash meets difficulty requirement.

As the blocks are added and linked using cryptography, they continuously grow the blockchain.

Immutability

Blockchain provides immutability of records. If an adversary tampers with any of the data in the entire chain, it will invalidate all the subsequent blocks.

Let’s assume we have 3 blocks and we want to tamper the data in block #1.

If you change the data in block #1 then the SHA256 hash value of the block changes as it is calculated based on the data.
Block #2’s hash changes as it is based on block #1’s hash.
Block #3’s hash changes as it is based on block #2’s hash.
Now, all the 3 blocks are invalid as they don’t meet the difficulty requirement of having 4 leading zeros.
You will then have to mine all the invalidated blocks again by finding new nonce values to make them valid.

This will be a compute intensive and unfeasible operation as the chain continuously grows with new blocks.

References

https://www.npmjs.com/package/blockchain-cli

https://blockchaindemo.io

https://anders.com/blockchain/blockchain.html

Microsoft Teams Webhook Integration

Harshad Ranganathan — Fri, 17 May 2019 00:00:00 +0000

Originally published at rharshad.com

Microsoft Team's Incoming webhook connector allows you to publish messages to teams channel.

You can even add actions to the content so that users can complete tasks within the channel.

Adding Webhook Connector to a Channel

In Microsoft Teams, choose the More options (⋯) button next to the channel name in the list of channels and then choose Connectors.

Add Incoming webhook connector to the channel.

Now choose configure button next to the Incoming Webhook connector.

Provide a name and upload an image for the connector. Now select Create button.

Copy the webhook URL for later reference.

We have now set up the webhook for the channel.

Sending Messages to Webhook

To send a message to the webhook connector we have to send it as a JSON payload. Refer message card template on how the payload needs to be structured.

Note: Adaptive cards are the recommended card type for new Teams development. They are the new cross product specification for cards in Microsoft products including Bots, Cortana, Outlook, and Windows. However, they are not supported yet in teams connectors.

You can experiment your card design at Card Playground.

We will send below sample payload to the connector and check if it works.

Send this JSON payload as a POST request to the Webhook URL.

You can either use Send via WebHook option available in the Card Playground (or) Postman to send the message to the webhook.

There is a python library which you can make use of for building the messages dynamically.

https://pypi.org/project/ms-teams/

Sending Messages to Webhook from Jenkins Build

You can make use of the office 365 connector plugin to send build messages to the webhook.

However, if you want to send a custom message as part of your jenkins job then we can't make use of this plugin.

Below is a sample which sends a custom card message to the webhook using httpRequest step in jenkins pipeline code.

This card shows the build parameter values with an action button to view the build job.

References:

https://docs.microsoft.com/en-us/microsoftteams/platform/concepts/connectors

https://docs.microsoft.com/en-us/outlook/actionable-messages/actionable-messages-via-connectors

Getting started with PySpark on Windows and PyCharm

Harshad Ranganathan — Mon, 13 May 2019 00:00:00 +0000

Pre-Requisites

Both Java and Python are installed in your system.

Getting started with Spark on Windows

Download Apache Spark by choosing a Spark release (e.g. 2.2.0) and package type (e.g. Pre-built for Apache Hadoop 2.7 and later).

Extract the Spark tar file to a directory e.g. C:\Spark\spark-2.2.0-bin-hadoop2.7

GIT clone winutils to your system e.g. cloned to directory C:\winutils

Add below system environment variables where HADOOP_HOME is set to the winutils hadoop binary location (depending on the version of pre-built chosen earlier) and SPARK_HOME is set to the Spark location which we had extracted in step 2.

HADOOP_HOME=C:\winutils\hadoop-2.7.1
SPARK_HOME=C:\Spark\spark-2.2.0-bin-hadoop2.7

Create a new folder tmp/hive in your C: drive.

Provide permissions for the folder tmp/hive using winutils.exe by running below command in your command prompt

C:\winutils\hadoop-2.7.1\bin\winutils.exe chmod 777 C:\tmp\hive

Now validate the setup by running spark-shell from your SPARK_HOME directory in your command prompt

C:\Spark\spark-2.2.0-bin-hadoop2.7>bin\spark-shell

PyCharm Configuration

Configure the python interpreter to support pyspark by following the below steps

Create a new virtual environment (File -> Settings -> Project Interpreter -> select Create Virtual Environment in the settings option)
In the Project Interpreter dialog, select More in the settings option and then select the new virtual environment. Now select Show paths for the selected interpreter option.
Add the paths for Spark Python and Spark Py4j to this virtual environment as shown in the screenshot below.

Create a new run configuration for Python in the dialog Run\Debug Configurations.

In the Python interpreter option select the interpreter which we had created in the first step. Also in the Environment variables option make sure Include parent environment variables is checked.

You can now add your pyspark script to the project and use this run configuration to execute it in a Spark context.

Jenkins Pipeline as Code

Harshad Ranganathan — Sun, 12 May 2019 00:00:00 +0000

Originally published at rharshad.com

Jenkins Pipeline is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins.

Need for Jenkins Pipeline as Code

Over time, Jenkins, like most other self-hosted CI/CD tools resulted in:

Accumulation/tendency to create vast number of jobs
Hard and costly maintenance
Heavy reliance on UI
Lack of powerful ways to specify conditional logic

Features of Pipeline as Code

Pipelines are implemented as code and typically checked into source control, serving as a single source of truth enabling the team members to edit, review, and iterate upon their pipeline.
Pipelines can optionally stop and wait for human input or approval before continuing the Pipeline run.
Pipelines support complex real-world requirements
- Conditional executions
- Chaining jobs
- Linear/Complex flows
The Pipeline plugin supports custom extensions to its DSL and multiple options for integration with other plugins.

https://imgur.com/WshehVd

Pipeline Autocompletion in IntelliJ

To enable code completion follow below steps:

Download Pipeline GDSL (http://(yourjenkinsurl)/job/(yourpipelinejob)/pipeline-syntax/gdsl) from your jenkins.
Add Groovy SDK support to Intellij.
Place the GDSL file inside src directory of your project and mark the src folder as Sources Root.
Open the GDSL file. Now, IntelliJ will show a message DSL descriptor file has been changed and isn't currently executed.. Click Activate.
Now create your .groovy jenkins file and write the code, auto completion will work.

Snippet Generator

You can also generate snippets of code for the pipeline steps using the Snippet Generator feature available in Jenkins (http://(yourjenkinsurl)/job/(yourpipelinejob)/pipeline-syntax/) which you can then make use of in your pipeline.

Replay

Replay feature allows you to test your pipeline code without having to commit the changes to your Repo. Once you have tested the code you can then push your changes.

Jenkins Declarative Pipeline Example

References

https://www.cloudbees.com/blog/need-jenkins-pipeline

Web Push Notifications for your React app using Firebase

Harshad Ranganathan — Tue, 07 May 2019 00:00:00 +0000

Push Notifications

Push notifications allow users to get updates and engage with your content. You can send push notifications via browsers (Desktop/Mobile) which support Push & Notification API.

Below browsers support Push & Notification API:

Chrome Desktop and Mobile (version 50+)
Firefox Desktop and Mobile (version 44+)
Opera on Mobile (version 37+)

Overview

Let’s see what’s involved in setting up a push notification service for a site.

Service Worker

Push is based on service workers because service workers operate in the background. A service worker is a “special” JavaScript file.

The browser can execute this JavaScript without your page being open. This means the only time code is run for a push notification (in other words, the only time the battery is used) is when the user interacts with a notification by clicking it or closing it.

It’s inside the service worker’s ‘push’ event that you can perform any background tasks. You can make analytics calls,cache pages offline and show notifications.

You must check if the browser supports service workers.

Push Manager

The PushManager interface of the Push API provides a way to receive notifications from third-party servers as well as request URLs for push notifications.

You must check if the browser supports push manager.

Service Worker Registration

Once we know that the browser supports service worker and push manager we register our service worker javascript file.

The browser will then run the file in a service worker environment.

User Permission

Once we register the service worker we need to get permission from the user to send push notifications.

If the user blocks the permission request then they have to manually unblock the site in the browser settings panel.

User Subscription

After registering the service worker and getting user permission, we need to subscribe the user.

We need to generate VAPID keys and submit to the push service. These keys are used by the push service to identify the application subscribing the user and ensure that the same application is the one messaging the user.

Once you subscribe you will receive an endpoint, associated with the app’s public key and an identifier (push subscription).

Later, when you want to send a push message, you’ll need to create an Authorization header which will contain information signed with your application server’s private key and submit to that endpoint.

Subscription Storage

We need to store the push subscription details by sending it to our server so that we can use it to send messages to a user.

Push Notification

To send a push message we need to do a web push by sending a payload with an Authorization header signed with the private key.

The push service will use the public key to decrypt the authorization header and verify that it is the same application that subscribed the user which is trying to send a message.

It will then send the push message to the user’s device when the browser becomes active.

Firebase Cloud Messaging

Firebase Cloud Messaging (FCM) is a cross-platform messaging solution that lets you reliably deliver messages at no cost.

We’ll see how we can use FCM to send notification messages to the client.

Below is how our application will work. We won’t go into the details of how to build and deploy a react app as it is outside the scope of this guide.

Source code of the application can be found here -

HarshadRanganathan / gnib-visa-app

Find appointment slots for GNIB (Irish Residence Permit) and Re-Entry Visa without hassle

gnib-visa-app

Find appointment slots for GNIB (Irish Residence Permit) without hassle.

App is hosted at https://gnib-visa-app.rharshad.com/

This App utilizes:

React
- axios for promise based http requests
Redux
- redux-thunk for middleware
Express minimalist web framework
Webpack for bundling
Babel for transpiling

Running the App locally

You'll need Node & NPM installed on your local development machine.

Install the project dependencies by running below commands.

# install react project dependencies
npm install
# install notification project dependencies
cd notifications/
npm install

Development mode

`npm run dev`

Runs the app in development mode with webpack recompiling and tests re-run whenever any files change.

Open http://localhost:8080 (defaults to port 8080) to view the app in the browser.

The page will automatically reload if you make changes to the code.

Production mode

`npm run build`

Builds the app for production to the public folder.

It bundles React in production mode, minifies the files and the…

View on GitHub

Sample notification page:

Client Side

Push API relies on a few different pieces of technology, including Web App Manifests and Service Workers.

Let’s see the steps involved in enabling Push API for your react app. We’ll use Firebase SDK to facilitate instant messaging.

Add Firebase to your app

To add Firebase to your app, you’ll need a Firebase project.

Create a firebase project in the Firebase console.
Get your app config which we will use to initialize firebase in your react app.

Install firebase npm module.

npm install --save firebase

Create a firebase.js file where we will initialize our app with the configuration details we got from our firebase project and also export the messaging component.

We will use the messaging reference later to register our service worker and handle incoming push notifications.

import firebase from 'firebase/app';
import '@firebase/messaging';

const config = {
    messagingSenderId: '<your-app-messaging-sender-id>'
};

firebase.initializeApp(config);

let messaging;

// we need to check if messaging is supported by the browser
if(firebase.messaging.isSupported()) {
    messaging = firebase.messaging();
}

export {
    messaging
};

Subscription Page

Next, we will create a notifications.js react component page which will check if the browser supports Service Worker and Push API. Based on it we will either display a message mentioning that push notifications are not supported or our subscription options.

Here, we have used material-ui for styling the page. Also, we import our firebase.js file which we had created previously.

We are following the passive approach here, which is to have a button or toggle switch that enables / disables push messages in a location on the page that is consistent throughout a site. Good UX design is to offer a way out for the user from the push messages and also not to ask for push permissions as soon as the user visits the page.

import React, { Fragment, Component } from 'react';
import PropTypes from 'prop-types';
import { withStyles, Card, CardContent, Typography, Switch, FormControlLabel } from '@material-ui/core';
import { messaging } from '../component/firebase';

const styles = theme => ({
    card: {
        marginTop: theme.spacing.unit * 2,
        marginBottom: theme.spacing.unit * 2,
        marginLeft: theme.spacing.unit * 2,
        marginRight: theme.spacing.unit * 2
    },
    noteTextPos: {
        marginTop: theme.spacing.unit * 2
    }
});

class Notifications extends Component {
    constructor(props) {
        super(props);
    }

    renderSubscriptionOptions(classes) {
        if(!('serviceWorker' in navigator) && !('PushManager' in window)) {
            return(
                <Typography className={classes.noteTextPos}>
                    Notification feature is supported only in:<br/>
                    Chrome Desktop and Mobile (version 50+)<br/>
                    Firefox Desktop and Mobile (version 44+)<br/>
                    Opera on Mobile (version 37+)
                </Typography>
            );
        } else {
            return (
                <Fragment>
                    <FormControlLabel 
                        control={<Switch />}
                        label="Enable/Disable GNIB(IRP) Appointment Notifications"
                        onChange={this.gnibApptSubscriptionToggle}
                        checked={this.state.gnibApptSubscriptionToggleSwitch}
                    />
                </Fragment>
            );
        }
    }

    render() {
        const { classes } = this.props;
        return (
            <Fragment>
                <Card className={classes.card}>
                    <CardContent>
                        {this.renderSubscriptionOptions(classes)}
                    </CardContent>
                </Card>
            </Fragment>
        );
    }
}

Notifications.propTypes = {
    classes: PropTypes.object.isRequired
};

export default withStyles(styles)(Notifications);

Service Worker

Next, we need to create a service worker file and register it.

Create a new file firebase-messaging-sw.js inside the pwa directory with below contents.

// import firebase scripts inside service worker js script
importScripts('https://www.gstatic.com/firebasejs/5.7.2/firebase-app.js');
importScripts('https://www.gstatic.com/firebasejs/5.7.2/firebase-messaging.js');

firebase.initializeApp({
    'messagingSenderId': '<your-app-messaging-sender-id>'
});

const messaging = firebase.messaging();

We then register the service worker file by updating firebase.js file.

// register service worker
if ('serviceWorker' in navigator) {
    window.addEventListener('load', async () => {
        const registration = await navigator.serviceWorker.register('/firebase-messaging-sw.js', {
            updateViaCache: 'none'
        });
        messaging.useServiceWorker(registration);
    });
}

Few points:

We mentioned the path of the worker file as /firebase-messaging-sw.js but we had placed the file inside pwa directory. This is because, later, we will be updating webpack config to copy the files to the public folder from where the static assets will be served. Based on your app design and bundler used you might have to update the path accordingly.
We have set updateViaCache to none, so that the HTTP cache will not be consulted when making requests for either the top-level /service-worker.js or for any imported scripted. Prior to Chrome 68, the update request for /service-worker.js would be made via the HTTP cache. Starting in 68, the HTTP cache will be ignored when requesting updates to the service worker script, so existing web applications may see an increase in the frequency of requests for their service worker script. Requests for importScripts will still go via the HTTP cache.
Every time our page loads, the browser will download the latest copy of service worker and will do a byte-to-byte comparison to see if anything has changed. If so, it will activate the new copy otherwise it won’t perform the update.
Also, if it notices any difference with the latest copy it won’t immediately activate the new service worker. The browser will wait until the current service worker controlls zero clients. You can however, force an update to be done.

Notification Message

Notification messages are handled differently depending on whether the page is in the foreground (has focus), or in the background, hidden behind other tabs, or completely closed.

Background Message Handling

We had earlier defined firebase-messaging-sw.js file which imported the firebase scripts and initialised the messaging component. That will take care of background message handling.

Whenever, a push event is sent, the service worker will get activated. The JS script is then run and the messaging component will make use of the Notification API to display the message in a standard format.

We will look at how the message is constructed with the content and action links in our server side section.

Foreground Message Handling

When your page in focus, then you need to explicity handle how your message gets displayed. For example, we need to make use of the onMessage function to handle the incoming message and show the notification using the service worker.

We update firebase.js file with the onMessage handler.

// register service worker & handle push events
if ('serviceWorker' in navigator) {
    window.addEventListener('load', async () => {
        const registration = await navigator.serviceWorker.register('/firebase-messaging-sw.js', {
            updateViaCache: 'none'
        });
        messaging.useServiceWorker(registration);
        messaging.onMessage((payload) => {
            const title = payload.notification.title;
            const options = {
                body: payload.notification.body,
                icon: payload.notification.icon,
                actions: [
                    {
                        action: payload.fcmOptions.link,
                        title: 'Book Appointment'
                    }
                ]
            };
            registration.showNotification(title, options);           
        });
    });
}

Here, we get the message, icon, link and use the service worker to display the notification. We also create a notification action to handle click events.

Refer:

[1] FCM message structure

Update firebase-messaging-sw.js with the notificationclick listener:

importScripts('https://www.gstatic.com/firebasejs/5.7.2/firebase-app.js');
importScripts('https://www.gstatic.com/firebasejs/5.7.2/firebase-messaging.js');

firebase.initializeApp({
    'messagingSenderId': '<your-app-messaging-sender-id>'
});

const messaging = firebase.messaging();

self.addEventListener('notificationclick', (event) => {
    if (event.action) {
        clients.openWindow(event.action);
    }
    event.notification.close();
});

Whenever, the notification is clicked, it will open the link in a new window and will close the notification.

Refer:

[1] Notification Click Listener

User Permissions

So far, we have seen how to register service workers, display notifications etc. Inorder to display push notifications the user has to accept it. If incase, the user rejects or blocks the site we won’t be able to send any notifications until the user manually revokes it.

Let’s update notifications.js file with logic to handle user permissions.

/**
 * Check if user has already given permission for sending notifications
 * If not, request permission from user, generate instance token and store it in firestore
 */
async notificationPermission() {
    let permissionGranted = false;
    try {
        /* request permission if not granted */
        if (Notification.permission !== 'granted') {
            await messaging.requestPermission();
        }
        /* get instance token if not available */
        if (localStorage.getItem(INSTANCE_TOKEN) !== null) {
            permissionGranted = true;
        } else {
            const token = await messaging.getToken(); // returns the same token on every invocation until refreshed by browser
            await this.sendTokenToDb(token);
            localStorage.setItem(INSTANCE_TOKEN, token);
            permissionGranted = true;
        }
    } catch (err) {
        console.log(err);
        if (err.hasOwnProperty('code') && err.code === 'messaging/permission-default') console.log('You need to allow the site to send notifications');
        else if (err.hasOwnProperty('code') && err.code === 'messaging/permission-blocked') console.log('Currently, the site is blocked from sending notifications. Please unblock the same in your browser settings');
        else console.log('Unable to subscribe you to notifications');
    } finally {
        return permissionGranted;
    }
}

We do couple of things here:

Check if the user has already granted the permission to send push notifications. If not, we request for it.
Also we check if the token is available in the local storage (we will store it later).
Finally, if above conditions aren’t satisfied we request for a token. If you request for it multiple times you will only get the same instance token back. We then send this token to our server to have it stored in firestore (or any file/database) as we need it to send push notifications. Also, we store the token in localStorage to quickly identify if the user has subscribed for notifications or not and display appropriate toggle switches.
If incase, the user has refused the permission then we can display the required messages to the user perhaps as a toast/snackbar.
Registration token may change when:
- The app deletes Instance ID
- The app is restored on a new device
- The user uninstalls/reinstall the app
- The user clears app/site data

You need to call ‘onTokenRefresh’ to send the refreshed token to your server.

Subscribe/Unsubscribe

We need to provide subscription options to our user based on whether he has already subscribed / not.

For example, in our notifications.js we could do something like below:

import axios from 'axios';

const ROOT_URL = ''; // define your server subscription url (sample express server setup for handling subscriptions described at the end)

class Notifications extends Component {

    constructor(props) {
        super(props);
        this.state = { gnibApptSubscriptionToggleSwitch: false }; // we set the toggle switch to false on component load
        this.gnibApptSubscriptionToggle = this.gnibApptSubscriptionToggle.bind(this);
        this.subscribeGnibApptNotifications = this.subscribeGnibApptNotifications.bind(this);
        this.unsubscribeGnibApptNotifications = this.unsubscribeGnibApptNotifications.bind(this);
        this.notificationPermission = this.notificationPermission.bind(this);
    }

    /**
     * If registration token is available in localStorage we enable the subscription option to indicate that the user has 
     * already subscribed
     */
    componentDidMount() {
        localStorage.getItem(GNIB_APPT_NOTIFICATION_SUBSCRIBED) === "TRUE" ? this.setState({ gnibApptSubscriptionToggleSwitch: true }) : this.setState({ gnibApptSubscriptionToggleSwitch: false });
    }

    /**
    * Send the subscription details (token and topic) to the server endpoint 
    */
    async subscriptionActions(mode, token, topic) {
        try {
            return await axios.post(`${ROOT_URL}/${mode}`, { token, topic });
        } catch(error) {
            if (error.response) {
                console.log(error.response.status);
                console.log(error.response.data);
            } else if (error.request) {
                console.log(error.request);
            } else {
                console.log('Error: ', error.message);
            }
            return null;
        }
    }

    /**
     * Subscribe app instance to notification topic if user permissions given
     */
    async subscribeGnibApptNotifications() {
        const notificationPermission = await this.notificationPermission();
        if(notificationPermission) {
            const isSubscribed = await this.subscriptionActions(SUBSCRIBE, localStorage.getItem(INSTANCE_TOKEN), GNIB_APPT_NOTIFICATIONS_TOPIC);
            if(isSubscribed) {
                localStorage.setItem(GNIB_APPT_NOTIFICATION_SUBSCRIBED, "TRUE");
                this.setState({ gnibApptSubscriptionToggleSwitch: true });
                this.displayMessage(<span>GNIB(IRP) appointment notifications have been enabled for your device</span>);
            } else {
                this.displayMessage(<span>Unable to subscribe you to notifications</span>);
            }
        }
    }

    /**
     * Unsubscribe app instance from notification topic
     */
    async unsubscribeGnibApptNotifications() {
        const isUnSubscribed = await this.subscriptionActions(UNSUBSCRIBE, localStorage.getItem(INSTANCE_TOKEN), GNIB_APPT_NOTIFICATIONS_TOPIC);
        if(isUnSubscribed) {
            localStorage.removeItem(GNIB_APPT_NOTIFICATION_SUBSCRIBED);
            await this.deleteTokenFromDb();
            this.setState({ gnibApptSubscriptionToggleSwitch: false });
            this.displayMessage(<span>You have been unsubscribed from notifications</span>);
        } else {
            this.displayMessage(<span>Unsubscribe failed</span>);   
        }
    }

    /**
     * Subscribe/UnSubscribe appointment notifications
     */
    gnibApptSubscriptionToggle(event ,checked) {
        if(checked) this.subscribeGnibApptNotifications();
        else this.unsubscribeGnibApptNotifications();
    }
}

Initially, we set the subscription toggle switch state to false. In our previous function, notificationPermission we had set the token in localStorage.

When the component mounts, we check if that token is available, if so we enable the toggle switch to indicate to the user that he has already subscribed.

If incase, this is a new subscription, we first get the permissions from the user and generate the registration token. In our notificationPermission function, we store the registration token in our database and also in localStorage.

We are also subscribing the user to a topic in function call subscriptionActions.

You can either send notifications to a set of users using their registration tokens (or) subscribe them to a topic and just publish a message to the topic. If incase, of publish-subscribe model, the message will get sent to all subscribers of topic.

We will look into them more in detail later.

Manifest file

Create a pwa directory in src folder which will contain the manifest.json file.

{
    "gcm_sender_id": "103953800507"
}

This indicates that FCM is authorized to send messages to this app.

Link the manifest in your index file.

<link rel="manifest" href="manifest.json">

If you are using webpack to generate your build files then you can use copy-webpack-plugin to copy the manifest file to the build directory.

npm install --save copy-webpack-plugin

const path = require('path');
const CopyPlugin = require('copy-webpack-plugin');
const srcPath = path.join(__dirname, 'src');
const buildPath = path.join(__dirname, 'public');
module.exports = () => {
    plugins: [
        new CopyPlugin([
            { from: path.join(srcPath, 'pwa'), to: buildPath }
        ])
    ]
};

If you are using Nginx to serve your static assets then you can specify to gzip the manifest file and specify content expiry time.

// virtual.conf
map $sent_http_content_type $expires {
    default off;
    application/json 2628000;
    application/manifest+json 2628000;
}

// gzip.conf
gzip on;
gzip_static on;
gzip_comp_level 6;
gzip_types
    text/plain
    text/css
    text/js
    text/javascript
    application/javascript
    application/manifest+json

Testing

We now have everything wired up on our client side. So, how can we test everything works fine on our client side before proceeding to the server side of things.

Run your react app in your local. In your subscription page, provide the user permissions and get the registration token from localStorage or your database.

Once you have the registration token we can then publish a message to test if the notification gets displayed when the app is in focus or in background.

Check Registration Token Status

Send a curl request to Google IID API with the registration token in the url and apiKey (from your firebase project configuration) in the Authorization header.

curl -X GET \
  'https://iid.googleapis.com/iid/info/<instance_token>?details=true' \
  -H 'Authorization: key=<apiKey>'

You will get back details like below:

{
    "connectDate": "2019-03-04",
    "application": "com.android.chrome",
    "scope": "*",
    "rel": {
        "topics": {
            "notifications": {
                "addDate": "2019-02-23"
            }
        }
    },
    "connectionType": "WIFI",
    "platform": "BROWSER"
}

Sample Notification Message

We then send a message to test if the notification gets shown to the user. We had previously used the project apiKey.

But, for sending the message we will use the FCM v1 HTTP protocol instead of the legacy protocol which will require JWT access token to be passed along the request.

To generate the access token you can make use of fcm-http-oauth. Follow the steps in the README to generate a service account in your firebase project.

Let’s now send the message to a particular user by providing firebase project id, jwt and the registration token:

curl -X POST \
  https://fcm.googleapis.com/v1/projects/<firebase_projectId>/messages:send \
  -H 'Authorization: Bearer <jwt_token>' \
  -H 'Content-Type: application/json' \
  -d '{
    "message":{
        "token": "<registration_token>",
        "notification": {
                "title": "New Appointments",
                "body": "14 February 2019 - 15:00\n18 February 2019 - 11:00\n18 February 2019 - 12:00"
                },
                "webpush": {
                    "headers": {
                        "TTL": "0"
                    },
                    "notification": {
                        "icon": "https://img.icons8.com/color/96/e74c3c/ireland.png"
                    },
                    "fcm_options": {
                        "link": "https://gnib-visa-app.rharshad.com"
                    }
                }
        }
    }'

Refer:

[1] FCM Message Structure

Server Side

We had looked so far on how to subscribe a user to receive push notifications. We will now look at storing the registration tokens in firestore and sending notification messages using firebase SDK.

Firestore

Cloud Firestore is a flexible, scalable database for mobile, web, and server development from Firebase and Google Cloud Platform.

It keeps your data in sync across client apps through realtime listeners and offers offline support for mobile and web so you can build responsive apps that work regardless of network latency or Internet connectivity.

Creating New Database

Create a new firestore database in test mode inside your firebase project.

In the Rules tab add below rule to allow reads/writes only from your account:

service cloud.firestore {
  match /databases/{database}/documents {
    match /{document=**} {
      allow read, write: if request.auth.uid != null;
    }
  }
}

Storing Registration Token

You can create a node js project which will be sending notifications to your users/use cloud functions/any other framework. In this guide, we will see how we can store tokens in firestore using node.js.

Generate a service account by following these steps:

In your firebase project, under project settings, choose Service accounts tab.

Select Generate new private key to download your service account file.

We will use the service account to access firestore.

Install firebase-admin SDK to your node.js project.

npm install --save firebase-admin

We will define firebase.js file which will perform the required operations.

const serviceAccount = require('./service-account.json');
const admin = require('firebase-admin');

/* initialise app */
admin.initializeApp({
    credential: admin.credential.cert(serviceAccount)
});

/* initialise firestore */
const firestore = admin.firestore();
firestore.settings({
    timestampsInSnapshots: true
});
const FIRESTORE_TOKEN_COLLECTION = 'instance_tokens';

async function storeAppInstanceToken(token) {
    try {
        return await firestore.collection(FIRESTORE_TOKEN_COLLECTION)
        .add({ token, createdAt: admin.firestore.FieldValue.serverTimestamp() });
    } catch(err) {
        console.log(`Error storing token [${token}] in firestore`, err);
        return null;
    }
}

async function deleteAppInstanceToken(token) {
    try {
        const deleteQuery = firestore.collection(FIRESTORE_TOKEN_COLLECTION).where('token', '==', token);
        const querySnapshot = await deleteQuery.get();
        querySnapshot.docs.forEach(async (doc) => {
            await doc.ref.delete();
        });
        return true;
    } catch(err) {
        console.log(`Error deleting token [${token}] in firestore`, err);
        return null;
    }
}

module.exports = {
    storeAppInstanceToken,
    deleteAppInstanceToken
}

This script exports two functions -

[1] storeAppInstanceToken - You will pass in the token which needs to be stored in a firestore collection. Also, adds a server timestamp to the document.

[2] deleteAppInstanceToken - Gets the docs which match the token and deletes them.

Sending User Notifications

We update the script firebase.js to export below functions to be able to send push notifications -

const messaging = admin.messaging();

function buildCommonMessage(title, body) {
    return {
        'notification': {
            'title': title,
            'body': body
        }
    };
}

/**
* Builds message with platform specific options
* Link: https://firebase.google.com/docs/reference/fcm/rest/v1/projects.messages
*/
function buildPlatformMessage(token, title, body) {
    const fcmMessage = buildCommonMessage(title, body);

    const webpush = {
        'headers': {
            'TTL': '0'
        },
        'notification': {
            'icon': 'https://img.icons8.com/color/96/e74c3c/ireland.png'
        },
        'fcm_options': {
            'link': 'https://gnib-visa-app.rharshad.com'
        }
    };

    fcmMessage['token'] = token;
    fcmMessage['webpush'] = webpush;
    return fcmMessage;
}

async function sendFcmMessage(fcmMessage) {
    try {    
        await messaging.send(fcmMessage);
    } catch(err) {
        console.log(err);
    }
}

module.exports = {
    buildPlatformMessage,
    storeAppInstanceToken,
    deleteAppInstanceToken,
    subscribeAppInstanceToTopic,
    unsubscribeAppInstanceFromTopic,
    sendFcmMessage
}

We can use buildPlatformMessage to generate a message and then pass it on to sendFcmMessage to notify the user.

Topic Subscription

You can also subscribe/unsubscribe users to topics by calling subscribeToTopic & unsubscribeFromTopic methods.

async function subscribeAppInstanceToTopic(token, topic) {
    try {
        return await messaging.subscribeToTopic(token, topic);
    } catch(err) {
        console.log(`Error subscribing token [${token}] to topic: `, err);
        return null;
    }
}

async function unsubscribeAppInstanceFromTopic(token, topic) {
    try {
        return await messaging.unsubscribeFromTopic(token, topic);
    } catch(err) {
        console.log(`Error unsubscribing token [${token}] from topic: `, err);
        return null;
    }
}

We had used firebase SDK for sending FCM messages. You can also make use of webpush or send the messages to the FCM HTTP App server endpoint.

Express Server

So far, we had defined subscription, firebase and fcm actions.

We will use Express to expose them as API’s in a web server so that our client app can access them.

Install express as a dependency.

npm install --save express

Create a new file named index.js and define below API’s.

const { storeAppInstanceToken, deleteAppInstanceToken, subscribeAppInstanceToTopic, unsubscribeAppInstanceFromTopic } = require('./firebase');

const 
    express = require('express'),
    bodyParser = require('body-parser'),
    app = express().use(bodyParser.json());

app.post('/storetoken', async (req, res) => {
    if (!req.body) res.sendStatus(400);
    if(req.body.token) {
        result = await storeAppInstanceToken(req.body.token);
        result?res.sendStatus(200):res.sendStatus(500);
    } else {
        res.sendStatus(400);
    }
});

app.delete('/deletetoken', async(req, res) => {
    if (!req.body) res.sendStatus(400);
    if(req.body.token) {
        result = await deleteAppInstanceToken(req.body.token);
        result?res.sendStatus(204):res.sendStatus(500);
    } else {
        res.sendStatus(400);
    }
});

app.post('/subscribe', async(req, res) => {
    if (!req.body) res.sendStatus(400);
    if(req.body.token) {
        result = await subscribeAppInstanceToTopic(req.body.token, req.body.topic);
        result?res.sendStatus(200):res.sendStatus(500);
    } else {
        res.sendStatus(400);
    }
});

app.post('/unsubscribe', async(req, res) => {
    if (!req.body) res.sendStatus(400);
    if(req.body.token) {
        result = await unsubscribeAppInstanceFromTopic(req.body.token, req.body.topic);
        result?res.sendStatus(200):res.sendStatus(500);
    } else {
        res.sendStatus(400);
    }
});

app.listen(process.env.PORT || 1338, () => {
    console.log('Server is running');
});

You can run the server with below command and access the endpoints via localhost e.g. send a POST request to http://localhost:1338/subscribe with appropriate JSON body content.

node index.js

There are various cloud platforms available where you can deploy the node express server.

References

https://developers.google.com/web/fundamentals/push-notifications/

https://developers.google.com/web/fundamentals/primers/service-workers/#update-a-service-worker

https://developers.google.com/web/fundamentals/primers/service-workers/lifecycle

https://developer.mozilla.org/en-US/docs/Web/API/ServiceWorkerRegistration/update

https://serviceworke.rs/

https://developers.google.com/instance-id/reference/server#get_information_about_app_instances

https://developer.mozilla.org/en-US/docs/Web/API/Notification

https://firebase.google.com/docs/cloud-messaging/concept-options#collapsible_and_non-collapsible_messages

https://firebase.google.com/docs/reference/fcm/rest/v1/projects.messages

https://firebase.google.com/docs/cloud-messaging/js/client

Windows Subsytem for Linux (WSL) on Windows 10

Harshad Ranganathan — Tue, 26 Mar 2019 00:00:00 +0000

WSL was introduced with Windows 10 Fall Creators Update and later (Windows build 16215 or later).

Pre-requisites

Ensure that “Windows Subsystem for Linux” feature is enabled by running below command in PowerShell as an administrator.

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

You can also enable it via Turn Windows features on or off control panel option by checking Windows Subsystem for Linux.

Install Linux Distribution

Open the Microsoft Store and choose your favorite Linux distribution.
From the distro’s page, select “Get” to install the distribution.

You might get below error if incase your corporate policies prevent install from Microsoft store.

In that case, download the distros and manually install them.

Running the .appx file will still fail due to the corporate policy. To overcome the issue follow below steps:

Rename the extension of the .appx file to .zip.
Extract all the contents.
Run the .exe file inside the extracted folder.

Once the installation is complete, you will be asked to create a new user account with password.

When you launch your distro, you won’t be asked for a password unless you elevate your access using sudo.

Upgrade Distro

Windows does not automatically update or upgrade your Linux distro(s). On Debian/Ubuntu, you can use apt to upgrade the packages:

sudo apt update && sudo apt upgrade

Launch Distro

You can launch the distribution in multiple ways:

Launch from windows store
Running wsl command from command prompt wsl [command]
Running wsl.exe which allows you to manage your distributions, set default distribution and uninstalling distributions.

Re-install Distro

You will get an error 0x80070003 if you manually delete the ubuntu folder and start WSL. It occurs when the path where you had installed your distro is not present/deleted/removed. E.g. when you move your distro folder to your C: drive.

In that case, you will have to unregister the distro and re-install it.

List the distros available to WSL:

C:\Users\harshad>wslconfig.exe /l
Windows Subsystem for Linux Distributions:
Ubuntu-18.04 (Default)

Unregister the distro with the distribution name:

C:\Users\harshad>wslconfig.exe /u Ubuntu-18.04

Re-install the distro again.

Change Mount Path

WSL mounts your machine’s fixed drives under the /mnt/<drive> folder in your Linux distros.

To change the mount path to / so that you don’t have to change directory to your drive everytime you run wsl, create a wsl.conf file.

sudo nano /etc/wsl.conf

Paste below contents and save it. On your next restart, the changes should get reflected.

[automount]
root = /
options = "metadata"

Install Docker

To install docker and docker compose in WSL run below script:

set -e

# Update package lists
echo "# Updating package lists"
sudo apt-add-repository -y ppa:git-core/ppa
sudo apt-get update

# Ensure that CA certificates are installed
sudo apt-get -y install apt-transport-https ca-certificates

# Add Docker repository key to APT keychain
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

# Update where APT will search for Docker Packages
echo "deb [arch=amd64] https://download.docker.com/linux/ubuntu ${CODENAME} stable" | \
    sudo tee /etc/apt/sources.list.d/docker.list

# Update package lists
sudo apt-get update

# Verifies APT is pulling from the correct Repository
sudo apt-cache policy docker-ce

# Install Docker
echo "# Installing Docker"
sudo apt-get -y install docker-ce

# Add user account to the docker group
sudo usermod -aG docker $(whoami)

# Install docker compose
echo "# Installing Docker-Compose"
sudo curl -L "https://github.com/docker/compose/releases/download/1.13.0/docker-compose-$(uname -s)-$(uname -m)" \
    -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

# Print installation details for user
echo ''
echo 'Installation completed, versions installed are:'
echo ''
echo -n 'Docker: '
docker --version
echo -n 'Docker Compose: '
docker-compose --version

# Print reminder of need to logout in order for these changes to take effect!
echo ''
echo "Please logout then login before continuing."

Enable Expose daemon on tcp://localhost:2375 without TLS in your Docker for Windows settings.

We then configure WSL to connect to the remote docker daemon running in Docker for Windows.

echo "export DOCKER_HOST=tcp://localhost:2375" >> ~/.bashrc && source ~/.bashrc

We then run docker commands in WSL.

Install Git, Node & Python

# Exit on any failure
set -e

# Update package lists
echo "# Updating package lists"
sudo apt-add-repository -y ppa:git-core/ppa
sudo apt-get update

# Install Git
echo "# Installing Git"
sudo apt-get install -y git

# Install nvm dependencies
echo "# Installing nvm dependencies"
sudo apt-get -y install build-essential libssl-dev

# Execute nvm installation script
echo "# Executing nvm installation script"
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.2/install.sh | bash

# Set up nvm environment without restarting the shell
export NVM_DIR="${HOME}/.nvm"
[-s "${NVM_DIR}/nvm.sh"] && . "${NVM_DIR}/nvm.sh"
[-s "${NVM_DIR}/bash_completion"] && . "${NVM_DIR}/bash_completion"

# Install node
echo "# Installing nodeJS"
nvm install 8
nvm use 8

# Ensure that CA certificates are installed
sudo apt-get -y install apt-transport-https ca-certificates

# Install python v2 if required
set +e
COUNT="$(python -V 2>&1 | grep -c 2.)"
if [${COUNT} -ne 1]
then
   sudo apt-get install -y python-minimal
fi

# Print installation details for user
echo ''
echo 'Installation completed, versions installed are:'
echo ''
echo -n 'Node: '
node --version
echo -n 'npm: '
npm --version
echo -n 'Python: '
python -V

# Print reminder of need to logout in order for these changes to take effect!
echo ''
echo "Please logout then login before continuing."

References

https://docs.microsoft.com/en-us/windows/wsl/about

Building and Publishing your Messenger Bot

Harshad Ranganathan — Tue, 01 Jan 2019 00:00:00 +0000

Originally published at rharshad.com

Building your Bot

In this example, we will use Node.js to build our bot which will respond to user's messages.

Create a new Node.js project

mkdir dbei-bot
cd dbei-bot

// create package.json file
npm init
touch index.js

// install project dependencies
npm install express body-parser --save // http server
npm install axios --save // promise based http client
npm install dotenv --save // loads environment variables from .env file

Configure an Environment File

We will create a .env file which will have below entries:

SEND_API=https://graph.facebook.com/v3.0/me/messages
VERIFY_TOKEN=<VERIFY_TOKEN>
PAGE_ACCESS_TOKEN=<PAGE_ACCESS_TOKEN>

VERIFY_TOKEN

It can be a random string. We will configure this token in the messenger platform at a later stage.

Purpose - Messenger platform will send a challenge request to our webhook and we must verify whether it matches the configured token. Only then, our webhook will be subscribed.

PAGE_ACCESS_TOKEN

We will generate this token at a later stage in the messenger platform and replace it here.

Purpose - We need this token to send API requests to messenger.

Create an HTTP server

We will create an HTTP server that listens for requests. In the index.js file which we had created earlier add below code:

// configure dotenv so that process.env has the keys and values you defined in your .env file
require('dotenv').config();

const 
    express = require('express'),
    bodyParser = require('body-parser'),
    app = express().use(bodyParser.json());

app.listen(process.env.PORT || 1337, () => console.log('webhook is listening'));

Add webhook verification endpoint

In index.js file we add below code to expose /webhook GET endpoint.

/**
 * Webhook challenge endpoint
 */
app.get('/webhook', (req, res) => {
    let VERIFY_TOKEN = process.env.VERIFY_TOKEN;
    let mode = req.query['hub.mode'];
    let token = req.query['hub.verify_token'];
    let challenge = req.query['hub.challenge'];
    if(mode && token) {
    // Checks the mode and token sent is correct
        if(mode == 'subscribe' && token == VERIFY_TOKEN) {
            res.status(200).send(challenge);
        } else {
            res.sendStatus(403);
        }
    }
});

We will configure this endpoint in the messenger platform later which will be used for challenge verification using the VERIFY_TOKEN we provide.

If the token sent by the messenger platform matches the one we had configured then we need to send 200 response otherwise 403.

On successful challenge verification, messenger platform will subscribe our webhook.

Test your webhook by running below commands:

node index.js // starts the express HTTP server to serve at port 1337

curl -X GET "localhost:1337/webhook?hub.verify_token=<YOUR_VERIFY_TOKEN>&hub.challenge=CHALLENGE_ACCEPTED&hub.mode=subscribe"

Above should return HTTP status code 200 with response as CHALLENGE_ACCEPTED if everything was set up properly.

Add webhook endpoint

In index.js file we add below code to expose /webhook POST endpoint which will be used by the Messenger platform to send webhook events.

/**
 * Messenger webhook endpoint
 */
app.post('/webhook', (req, res) => {
    let body = req.body;
    // Checks this is an event from a page subscription
    if(body.object === 'page') {
        // Iterates over each entry - there may be multiple if batched
        body.entry.forEach(function(entry) {
            // entry.messaging is an array, but will only ever contain one message
            let webhook_event = entry.messaging[0];
            // A person is assigned a unique page-scoped ID (PSID) 
            // for each Facebook Page they start a conversation with. 
            // The PSID is used by your Messenger bot to identify a person when sending messages.
            let sender_psid = webhook_event.sender.id;
            messenger.handleMessage(sender_psid, webhook_event.message); 
        });
        res.status(200).send('EVENT_RECEIVED');
    } else {
        res.sendStatus(404);
    }
});

Handler functions

Previously we had mentioned messenger.handleMessage(sender_psid, webhook_event.message); function call. We will define this function now to handle and respond to text messages from uers.

/**
 * Incoming Message Handler
 */
function handleMessage(sender_psid, received_message) {
    // check if the input is text message
    // users might send smiley which can be filtered out
    if(typeof received_message.text !== "undefined") {
        let message = received_message.text.toLowerCase();
        callSendAPI(sender_psid, { text: message });
    }    
}

We will define callSendAPI to send response messages via the SEND API.


// access environment variables
const PAGE_ACCESS_TOKEN = process.env.PAGE_ACCESS_TOKEN;
const SEND_API = process.env.SEND_API;

/**
 * Calls the Messenger API to send the message
 */
function callSendAPI(psid, message) {
    let data = { 
        "recipient": { "id": psid }, 
        "message": message 
    };
    return axios({
        method: 'POST',
        url: SEND_API,
        params: { access_token: PAGE_ACCESS_TOKEN },
        data: data
    })
    .catch((error) => {
        if (error.response) {
            console.log('PSID: ', psid);
            console.log('Status code: ', error.response.status);
            console.log('Response: ', error.response.data);
        } else if (error.request) {
            console.log('Request: ', error.request); 
        } else {
            console.log('Error: ', error.message);
        }
    });
}

Here,

recipient - Sets the intended message recipient. In this case, we identify the person by their PSID.

message - Sets the details of the message to be sent.

We are sending our message to the Send API whose URL we had defined previously in a .env file. We are also appending PAGE_ACCESS_TOKEN in the access_token parameter of the URL query string.

If in case you are sending a subscription message, you need to include NON_PROMOTIONAL_SUBSCRIPTION tag in your data request.

let data = { 
    "recipient": { "id": psid }, 
    "message": message, 
    "messaging_type": "MESSAGE_TAG", 
    "tag": "NON_PROMOTIONAL_SUBSCRIPTION" 
};

Sending structured message

Messenger Platform provides a set of useful message templates, each designed to support a different, common message structure, including lists, receipts, buttons, and more.

Below is sample templates.js helper script which can be used to send structured messages to the Send API.

/**
 * link https://developers.facebook.com/docs/messenger-platform/reference/template/generic
 */
function genericTemplate(elements) {
    return {
        "attachment": {
            "type": "template",
            "payload": {
                "template_type": "generic",
                "elements": elements
            }
        }
    };
}

/**
 * link https://developers.facebook.com/docs/messenger-platform/reference/template/list
 */
function listTemplate(elements) {
    return {
        "attachment": {
            "type": "template",
            "payload": {
                "template_type": "list",
                "top_element_style": "compact",
                "elements": elements,
                "buttons": [
                    {
                        "title": "Donate", 
                        "type": "web_url", 
                        "url": 'https://www.paypal.me/harshadranganathan'
                    }
                ]
            }
        }
    };
}

/**
 * link https://developers.facebook.com/docs/messenger-platform/send-messages/quick-replies/
 */
function quickRepliesTemplate(text, quickReplies) {
    return {
        "text": text,
        "quick_replies": quickReplies
    }
}

module.exports = {
    genericTemplate: genericTemplate,
    listTemplate: listTemplate,
    quickRepliesTemplate: quickRepliesTemplate
};

Messenger Quick Replies Sample:

For example, to make use of quickRepliesTemplate, import the templates script and pass in the reply options referring the template documentation:

const _ = require('lodash');
const templates = require('./templates');

function quickReplyOptions(text) {
    let quickReplies = [];
    _.forEach(['Subscribe', 'Unsubscribe'], (option) => {
        quickReplies.push({ content_type: 'text', title: option, payload: option });
    });
    return templates.quickRepliesTemplate(text, quickReplies);
}   

callSendAPI(sender_psid, quickReplyOptions("Hi! How can I help you today?"));

Deploy your webhook

Now that we have built our sample bot, we have to deploy it on a server with a valid SSL certificate so that it can accept requests over HTTPS.

HTTPS connection is a must have for integrating our bot with Messenger platform.

We have couple of options:

If you need to integrate and test it from your local you can make use of ngrok.
You can remix messenger quick start project on glitch which will provide a public URL served over HTTPS for your webhook.
You can host the node.js application in your linux instance with Nginx as a proxy and configure SSL certificates with Let's Encrypt.

Once you have deployed your app and configured HTTPS connection, your webhook url will be https://<server-ip>/webhook (or) https://<domain>/webhook if you had mapped a domain to your server IP.

Proceed to next steps for integrating it with Messenger platform.

Set Up Your Facebook App

Follow below steps to set up your facebook app for use with the Messenger Platform and subscribe your webhook to receive events.

Facebook Page

A Facebook Page will be used as the identity of your bot. When people chat with your app, they will see the Page name and the Page profile picture.

Create a facebook page for your bot first.

For our bot, we are going to create a business page with category as Internet Company.

You can then choose a profile picture and cover page.

Additional page configurations:

Create Page Username Create a custom url which people can use to search and visit your page.
Visitor Posts You can disable posts by other people on the page.
Page Info You can provide a description and email address for contact.
Notifications Turn off notifications for messages so that you don't receive an alert for every message sent to the bot via messenger.
Add a Button to Your Page You can add a contact button so that people can send a message from your page.

Whitelist Domain & Subscription Messaging

We will be integrating your webhook URL with messenger platform soon. In order for your webhook to be invoked by messenger you need to whitelist your domain.

Add your domain in the Whitelisted Domains section under Messenger Platform settings.

If your bot needs the ability to send regular content to a person on Messenger you need to raise Subscription Messaging permission at a page level.

You will able to send messages to the person outside Standard Messaging only after this permission is reviewed and approved.

https://developers.facebook.com/docs/messenger-platform/policy/policy-overview/

Facebook Developer Account

You need a developer account to create new apps which serve as an integration between your page and bot.

Create a developer account through Facebook for Developers platform.

Facebook App

Facebook app contains the settings for your Messenger bot, including access tokens.

Create a new app using the app dashboard.

Your new app will be created in Development Mode.

Under Development Mode only users who have been granted Administrator, Developer, or Tester role in the app will be able to interact with the bot once we complete the integration with messenger.

Add Messenger Platform to your App

In the sidebar of your app settings under 'PRODUCTS', click '+ Add Product'.
Hover over 'Messenger' to display options.
Click the 'Set Up' button.

Setup Webhook

In the 'Webhooks' section of the Messenger settings console, click the 'Setup Webhooks' button.

Configure the Callback URL and Verify Token.

Here, the Callback URL will be https://<server-ip-or-domain>/webhook and Verify Token will the value which you had given in the .env file previously.

The Messenger Platform will send a GET request to your webhook with the verify token you provided. If your webhook is valid and properly set up to respond to the verification request, your webhook settings will be saved.

Also, in the subscription field choose messages and save the webhook config.

Subscribe your Facebook Page

In the 'Webhook' section of the Messenger settings console, click the 'Select a Page' dropdown and select the facebook page which you had created earlier.

This will subscribe your app to receive events when people on Messenger chat with your page and forward them to your webhook.

Generate Page Access Token

In the 'Token Generation' section of the Messenger settings console, click the 'Select a Page' dropdown and select the Facebook Page that you had created before.
Copy the token that appears in the 'Page Access Token' field.
Add this token to property PAGE_ACCESS_TOKEN which we had defined in the .env file.

The generated token will NOT be saved in this UI. Each time you select a Page from the dropdown, a new token will be generated. If a new token is generated, previously created tokens will continue to function.

Test App Subscription

Send a message to your Page from facebook.com or in Messenger. If your webhook receives a webhook event then everything has been wired up properly.

Publishing your App

While in Development mode, apps are automatically approved for all login permissions, features, and product-specific features.

Once you switch your app to Live Mode, however, your app can only use permissions and features that it has been approved for.

Messenger platform requires below review and approval steps:

Permissions and Features Review
Business Information and Verification Documents
Accept Supplemental Terms and Sign Tech Provider Agreement

Permissions and Features Review

Once you are ready to publish your bot, ensure your app is in Development Mode.

Provide basic information on the app console under Settings > Basic:

App Icon
Privacy Policy URL
Category
Business Use

You can generate a privacy policy using this bot.

Select app permissions

Request the pages_messaging permissions under App review section in messenger settings, which allows your bot to send messages in live mode.

Submit your bot for review once you have chosen all the required permissions.

Your bot will then be reviewed by messenger test users who will post the command which you had mentioned while submitting your app for review. Your bot is expected to respond to their messages within 20 seconds.

If no response is returned within the stipulated time then your app submission will be sent back.

Gotchas:

I had sent a message to my page and received a response back from my bot. However, when the test users had sent a message they aren't getting any response.

This could be a platform bug as I had faced the same issue. When the test user had sent their first message the webhook wasn't called however when the same user sent the message the subsequent time everything worked fine. I had raised a bug report and it was fixed by messenger team.

Business Information and Verification Documents

If you use the messenger API then your business needs to be verified.

You need to create a Business manager account and link it to your app.

Once your bot has been reviewed for responsiveness, functionality, and policy compliance, you will receive a message asking you to complete business verification in the 'Alerts' tab of the app console.

You then need to provide the proofs for verification. If you are not an actual business, you can submit your home utility bill (must match your name in business manager account) as proof.

Business verification is required once per Business Manager account.

Accept Supplemental Terms and Sign Tech Provider Agreement

After your business information and verification documents are submitted, you will receive an email from the Facebook legal team asking you to digitally sign the supplemental terms and apply for review.

Once all the review is completed, your bot will be approved. You can then make it live and anyone will be able to message and get the response back from your page.

References:

https://developers.facebook.com/docs/messenger-platform/getting-started/webhook-setup

https://developers.facebook.com/docs/messenger-platform/getting-started/quick-start

https://developers.facebook.com/docs/messenger-platform/getting-started/app-setup

https://developers.facebook.com/docs/messenger-platform/policy/policy-overview/

https://developers.facebook.com/docs/apps/review/

https://developers.facebook.com/docs/messenger-platform/app-review/

Setting up Kubernetes on Windows with Minikube

Harshad Ranganathan — Sun, 25 Nov 2018 00:00:00 +0000

Originally published at rharshad.com

Minikube is a tool that makes it easy to run Kubernetes locally. Minikube runs a single-node Kubernetes cluster inside a VM on your laptop for users looking to try out Kubernetes or develop with it day-to-day.

Minikube can be installed in multiple operating systems (Linux, MacOS & Windows) and supports multiple drivers (VirtualBox, Hyper-V in case of Windows).

We are going to look at Windows setup of Minikube using both VirtualBox & Hyper-V drivers.

If you have Docker in your local system then you would have enabled Hyper-V. In that case, it makes sense to make use of Hyper-V driver instead of VirtualBox as the latter doesn't work with Hyper-V enabled.

So, in such a case if you choose VirtualBox as your driver you might end up toggling Hyper-V and restarting your system based on whether you want to use Docker or VirtualBox for Kubernetes.

Install Chocolatey

Chocolatey is a package manager for Windows similar to how homebrew is for MacOS. It simplifies our installation process so we will use it here.

Open PowerShell as an administrator and run below commands.

# Check the execution policy and ensure that it is not restricted
Get-ExecutionPolicy

# If execution policy is restricted then set the policy to All-Signed
Set-ExecutionPolicy AllSigned

# Install chocolatey
iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

Install Minikube

Once chocolately is installed, we run below command to install Minikube

choco install minikube --version 0.27 -y

Note: Versions above 0.27 have an issue with shutdown

Install VirtualBox

If you're planning to use VirtualBox as the driver for Minikube then run below command. Otherwise skip this step.

choco install virtualbox -y

We'll look at running Kubernetes cluster in both VirtualBox and Hyper-V next.

Running Kubernetes Cluster in VirtualBox

Start Minikube

Run this command in PowerShell as an administrator.

minikube start --alsologtostderr

Minikube will perform below steps

Downloads MinikubeISO and places it in .minikube folder in your user directory
Connects to VirtualBox and runs a minikube virtual machine
Downloads the necessary files and moves them to the cluster
Runs a single-node Kubernetes cluster inside the VM

Jump to Interacting With Your Cluster section next.

Troubleshooting

If you face any issues during the setup process perform below steps:

Delete the minikube VM

minikube delete

Delete .minikube and .kube folders in your home directory

Run minikube start command with appropriate options.

Running Kubernetes Cluster in Hyper-V

Create vSwitch

First step is to create a vSwitch in Hyper-V.

Open Hyper-V Manager in your windows system. On the right pane, select Virtual Switch Manager option.

Select External and choose Create Virtual Switch option in the Virtual Switch Manager window.

Give a name for the virtual switch e.g. Minikube with connection type as External and below option enabled

Allow management operating system to share this network adapter

Now select Apply to create a new vSwitch.

If you see an error message Failed while adding virtual Ethernet switch connections it means that you have enabled sharing on your wifi connection which needs to be disabled.

Go to Network and Sharing Center and select your Wi-Fi connection. In the Wi-Fi status window select Properties.

In the Wi-Fi properties window under Sharing tab uncheck this option

Allow other network users to connect through this computer's Internet connection

Start Minikube

Run this command in PowerShell as an administrator.

minikube start --vm-driver "hyperv" --hyperv-virtual-switch "Minikube" --disk-size 10g --memory 4096 --alsologtostderr

Minikube will perform below steps

Downloads MinikubeISO and places it in .minikube folder in your user directory
Connects to Hyper-V and runs a minikube virtual machine
Downloads the necessary files and moves them to the cluster
Runs a single-node Kubernetes cluster inside the VM

You will see below logs if everything went fine.

Connecting to cluster...
Setting up kubeconfig...
I1125 23:05:12.192115    4588 config.go:125] Using kubeconfig:  
Starting cluster components...
I1125 23:05:12.196100    4588 ssh_runner.go:80] Run with output:
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml 
ts --ignore-preflight-errors=DirAvailable--data-minikube 
FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml 
fests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable
 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml
flight-errors=CRI  &&
sudo /usr/bin/kubeadm alpha phase addon kube-dns

Kubectl is now configured to use the cluster.
Loading cached images from config file.

Troubleshooting

If you face any issues during the setup process perform below steps:

Delete the minikube VM

minikube delete

Delete .minikube and .kube folders in your home directory

Run minikube start command with appropriate options.

Interacting With Your Cluster

To check if minikube is running use below command

minikube status

which outputs

PS C:\WINDOWS\system32> minikube status
minikube: Running
cluster: Running
kubectl: Correctly Configured: pointing to minikube-vm at

To ensure all the cluster components are running we use below command

PS C:\WINDOWS\system32> kubectl get pods -n kube-system
NAME                                    READY   STATUS    RESTARTS   AGE
coredns-c4cffd6dc-565mc                 1/1     Running   0          12m
etcd-minikube                           1/1     Running   0          11m
kube-addon-manager-minikube             1/1     Running   0          11m
kube-apiserver-minikube                 1/1     Running   0          11m
kube-controller-manager-minikube        1/1     Running   0          12m
kube-dns-86f4d74b45-vxrr9               3/3     Running   0          12m
kube-proxy-fn5c9                        1/1     Running   0          12m
kube-scheduler-minikube                 1/1     Running   0          11m
kubernetes-dashboard-6f4cfc5d87-hgdnq   1/1     Running   5          12m
storage-provisioner                     1/1     Running   0          12m

If any of the above pods are in failed status you can get it's logs as follows for debugging

kubectl logs kubernetes-dashboard-6f4cfc5d87-hgdnq -n kube-system

If you notice any of the pods in CrashLoopBackOff status then perform the steps as mentioned in the troubleshooting guide earlier.

To access the Kubernetes dashboard run below command

minikube dashboard