Forem: Aditya Sridhar

How to get started with Kafka Streams

Aditya Sridhar — Tue, 15 Jun 2021 19:04:12 +0000

This post was originally published in adityasridhar.com

In this article we will see how to use kafka streams to do some real time data processing on data present in a kafka topic.

What is kafka streams

Kafka streams helps build real time applications on top of data which is present in kafka.

kafka streams reads a continuous stream of data from kafka, does some processing on this stream of data and outputs the result back to kafka.

Pre-requisites

Please ensure that you have Maven and Java installed in your System

What kafka streams application are we planning to build?

We will build a system which will work as follows:

There will be an input kafka topic in which you will get video titles with number of views per video
There will be a kafka streams application which reads the video views data and counts the number of views each video is getting.
Kafka streams will write the total video view data into an output topic
The output topic at any point will have the total number of views each of our videos have got.

For example if the input is as follows ( video and count are separated by a :)

Video1:1
Video2:20
Video1:23
Video3:8
Video2:5

Then the output will be as follows:

Video1:24
Video2:25
Video3:8

Setup

Download Kafka

Download kafka 2.8.0 from https://kafka.apache.org/downloads

wget https://apachemirror.wuchna.com/kafka/2.8.0/kafka_2.12-2.8.0.tgz

Extract kafka

tar xzf  kafka_2.12-2.8.0.tgz

Open the kafka folder. All kafka commands should be run in the kafka folder

cd kafka_2.12-2.8.0

Setup zookeeper and Kafka

Start zookeeper using the following command:

bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

Zookeeper starts on port 2181

Start kafka using the following command:

bin/kafka-server-start.sh -daemon config/server.properties

Kafka runs on port 9092. For this example we will use only one kafka instance.

Create the input and output topic for the application

Let us create an input topic from which the kafka streams application will read data.

This is done using the following command:

bin/kafka-topics.sh --create --topic input-topic --partitions 1 --replication-factor 1 --bootstrap-server localhost:9092

The topic name is input-topic and it has 1 replica and 1 partition

Let us also create an output topic, into which the kafka streams application will write the result to.

This is done using the following command:

bin/kafka-topics.sh --create --topic output-topic --partitions 1 --replication-factor 1 --bootstrap-server localhost:9092

The topic name is output-topic and it has 1 replica and 1 partition

We can list out all the topics created using the following command:

bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

The above command will print the following:

input-topic
output-topic

Writing the kafka streams application

Create a new Java Maven project in your IDE. If you want to know more about maven you can read this beginners guide to maven

pom.xml

In the pom.xml file, add the following dependencies. This includes kafka streams library and the log4j library

<dependencies>
    <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-streams -->
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-streams</artifactId>
        <version>2.8.0</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-log4j12 -->
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-log4j12</artifactId>
        <version>1.7.30</version>
    </dependency>
</dependencies>

Below is the complete pom.xml file.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>kafka-streams-demo</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>
        <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-streams -->
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-streams</artifactId>
            <version>2.8.0</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-log4j12 -->
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
            <version>1.7.30</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.6.1</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                        <configuration>
                            <archive>
                                <manifest>
                                    <mainClass>KafkaStreamsDemo</mainClass>
                                </manifest>
                            </archive>
                            <descriptorRefs>
                                <descriptorRef>jar-with-dependencies</descriptorRef>
                            </descriptorRefs>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>

In the above file we have added the maven compiler plugin to support java 1.8

We have also added the maven assembly plugin so that we can build a standalone executable jar for the kafka streams application.

Kafka Streams Java Code

Here is the complete Java code for the kafka streams application

import org.apache.kafka.common.serialization.Serde;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.kstream.*;
import org.apache.log4j.BasicConfigurator;
import org.apache.log4j.Level;
import org.apache.log4j.Logger;
import java.util.Properties;
import java.util.concurrent.CountDownLatch;

public class KafkaStreamsDemo {
    public static void main(String[] args) {
        BasicConfigurator.configure();
        Logger.getRootLogger().setLevel(Level.INFO);

        final Serde<String> stringSerde = Serdes.String();
        final Serde<Long> longSerde = Serdes.Long();

        final StreamsBuilder builder = new StreamsBuilder();

        KStream<String, String> views = builder.stream(
                "input-topic",
                Consumed.with(stringSerde, stringSerde)
        );

        KTable<String, Long> totalViews = views
                .mapValues(v -> Long.parseLong(v))
                .groupByKey(Grouped.with(stringSerde, longSerde))
                .reduce(Long::sum);

        totalViews.toStream().to("output-topic", Produced.with(stringSerde, longSerde));

        final Properties props = new Properties();
        props.putIfAbsent(StreamsConfig.APPLICATION_ID_CONFIG, "streams-totalviews");
        props.putIfAbsent(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");

        final KafkaStreams streams = new KafkaStreams(builder.build(), props);

        final CountDownLatch latch = new CountDownLatch(1);

        try {
            streams.start();
            latch.await();
        } catch (final Throwable e) {
            System.exit(1);
        }

        Runtime.getRuntime().addShutdownHook(new Thread("streams-totalviews") {
            @Override
            public void run() {
                streams.close();
                latch.countDown();
            }
        });

        System.exit(0);
    }
}

Let me go over the code step by step.

First we initialise a default logging mechanism using the following lines of code:

BasicConfigurator.configure();
Logger.getRootLogger().setLevel(Level.INFO);

Next we setup String, Long Serdes to help with serialising and deserialsing the kafka data

final Serde<String> stringSerde = Serdes.String();
final Serde<Long> longSerde = Serdes.Long();

Next we build the kafka streams topology. This is where we specify the logic:

final StreamsBuilder builder = new StreamsBuilder();

KStream<String, String> views = builder.stream(
        "input-topic",
        Consumed.with(stringSerde, stringSerde)
);

KTable<String, Long> totalViews = views
        .mapValues(v -> Long.parseLong(v))
        .groupByKey(Grouped.with(stringSerde, longSerde))
        .reduce(Long::sum);

totalViews.toStream().to("output-topic", Produced.with(stringSerde, longSerde));

First we create a StreamsBuilder object

Then we are creating a KStream object called views to read data from the kafka topic input-topic. In this topic both the key and value are strings. Hence we use string serde for both the key and the value

Next we are created a KTable object with key as string and value as long. This is the object where we will store our computation result.

We are using java lambda expressions to achieve the following

Convert the value to Long, since the value here is the video view count
Group all the same video titles together.
For the Grouped video title, calculate the sum.

Finally the total video view count is written to a kafka topic called output-topic. In this topic, the key is a string and the value is a Long.

Next we create a KafkaStreams object using the following code:

final Properties props = new Properties();
props.putIfAbsent(StreamsConfig.APPLICATION_ID_CONFIG, "streams-totalviews");
props.putIfAbsent(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");

final KafkaStreams streams = new KafkaStreams(builder.build(), props);

In the above code we are giving the application an id of streams-totalviews. This will help us in identifying the application.

We are also giving the kafka bootstrap server details as localhost:9092 since this is where our kafka is running.

Then we create a KafkaStreams object called streams

Until now we have just created the kafka streams flow, but we have not yet started it.

The Next few lines of code, starts the kafka streams application

final CountDownLatch latch = new CountDownLatch(1);

try {
    streams.start();
    latch.await();
} catch (final Throwable e) {
    System.exit(1);
}

Runtime.getRuntime().addShutdownHook(new Thread("streams-totalviews") {
    @Override
    public void run() {
        streams.close();
        latch.countDown();
    }
});

System.exit(0);

We start the flow using streams.start()

We also created a Shutdown Hook in the above lines of code. This ensures that if the application is abnormally shutdown, then the kafka streams application will close properly using the streams.close() function.

Running the Kafka streams application

Build the kafka streams application using the following command:

mvn clean package

This will create a file called kafka-streams-demo-1.0-SNAPSHOT-jar-with-dependencies.jar in the target folder

Run the application in a terminal using the following command

java -jar target/kafka-streams-demo-1.0-SNAPSHOT-jar-with-dependencies.jar

Ensure you give the full path to your jar file

In a second terminal, start a kafka producer using the following command

bin/kafka-console-producer.sh --bootstrap-server localhost:9092 \
    --topic input-topic \
    --property key.separator=":" \
    --property parse.key=true

We will be producing into the kafka topic input-topic. Also we will separate the video title from the view count by a :

In a third terminal, start a kafka consumer using the following command

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
    --topic output-topic \
    --property print.key=true \
    --property key.separator="-" \
    --property key.deserializer=org.apache.kafka.common.serialization.StringDeserializer \
    --property value.deserializer=org.apache.kafka.common.serialization.LongDeserializer

We will be consuming from the topic output-topic. In the output, the video title will be separated from the total video count with a -. Also in the output, the key is a string and the value is a Long.

In the Producer terminal send the following

Video1:5
Video2:10
Video1:4
Video1:0
Video2:7
Video3:12
Video1:7

In the Consumer terminal you will get the following output

Video2-17
Video3-12
Video1-16

If we further produce the following in the producer:

Video1:51
Video2:12
Video1:41
Video3:15

We will get the following in the consumer terminal

Video2-29
Video1-108
Video3-27

We can see that the kafka streams application is aggregating all the view counts based on the video title.

Code

The code for the kafka streams application created in this blog can be found in this Github Repo

Congrats 😊

You have now learnt how to use kafka streams

If you are interested in realtime analytics using kafka and Apache Pinot, feel free to checkout my course in Udemy by clicking on the Image below

Feel free to connect with me in LinkedIn or follow me in twitter.

How to easily install kafka without zookeeper

Aditya Sridhar — Mon, 07 Jun 2021 14:37:55 +0000

This post was originally published in adityasridhar.com

In this article we will see how kafka can be setup without using zookeeper. We will setup a 3 node kafka cluster and create a test topic. We will use a kafka producer to produce data into the test topic and also use a kafka consumer to consume data from the kafka topic.

Why is zookeeper used in kafka

Zookeeper is used to store kafka cluster metadata information. Zookeeper stores information like topic configuration, topic partition locations and so on.

How will kafka work without zookeeper

Zookeeper acts as a external metadata management system for kafka. This creates multiple problems such as data duplication, increased system complexity and also leads to additional java processes being used.

In order to run kafka without zookeeper, it can be run using Kafka Raft metadata mode ( KRaft ). In KRaft the kafka metadata information will be stored as a partition within kafka itself. There will be a KRaft Quorum of controller nodes which will be used to store the metadata. The metadata will be stored in an internal kafka topic @metadata .

This is available in an experimental mode in kafka 2.8.0

Right now using KRaft is experimental and should not be used in production

Setting up a kafka cluster without zookeeper

Download kafka

Download kafka 2.8.0 from https://kafka.apache.org/downloads

wget https://apachemirror.wuchna.com/kafka/2.8.0/kafka_2.12-2.8.0.tgz

Extract kafka

tar xzf  kafka_2.12-2.8.0.tgz

Open the kafka folder. All kafka commands should be run in the kafka folder

cd kafka_2.12-2.8.0

Kafka cluster configuration

If you go to config/kraft folder inside the kafka home directory, you will see a file called server.properties. This is a sample file which is provided by kafka, to show how kafka can be started without zookeeper

Create 3 new files from server.properties. This is because we will be creating a 3 node cluster

cd config/kraft
cp server.properties server1.properties
cp server.properties server2.properties
cp server.properties server3.properties

In server1.properties, modify the following properties. Please keep the other properties as is.

node.id=1

process.roles=broker,controller

inter.broker.listener.name=PLAINTEXT

controller.listener.names=CONTROLLER

listeners=PLAINTEXT://:9092,CONTROLLER://:19092

log.dirs=/tmp/server1/kraft-combined-logs

listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

controller.quorum.voters=1@localhost:19092,2@localhost:19093,3@localhost:19094

Let me explain what these properties do:

node.id: This will act as the node Id in the cluster. This will help us identify which broker this is. It will also help us identify which kraft controller node this is.

process.roles: A node can act as a broker or controller or both. Here we are indicating that this node can be both a kafka broker and a kraft controller node.

inter.broker.listener.name: Here the broker listener name is set to PLAINTEXT

controller.listener.names: Here the controller listener name is set to CONTROLLER

listeners: Here we indicate that the broker will use port 9092 and the kraft controller will use port 19092

log.dirs: This is the log directory where kafka will store the data

listener.security.protocol.map: Here the connection security details are added

controller.quorum.voters: This is used to indicate all the kraft controllers which are available. Here we are indicating that we will have 3 kraft controller nodes running on ports 19092, 19093 and 19094

For server2.properties modify the following properties. Please keep the other properties as is.

node.id=2

process.roles=broker,controller

controller.quorum.voters=1@localhost:19092,2@localhost:19093,3@localhost:19094

listeners=PLAINTEXT://:9093,CONTROLLER://:19093

inter.broker.listener.name=PLAINTEXT

controller.listener.names=CONTROLLER

log.dirs=/tmp/server2/kraft-combined-logs

listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

In Server 2, the broker port is 9093 and controller port is 19093. Also the log.dirs is different

For server3.properties modify the following properties. Please keep the other properties as is.

node.id=3

process.roles=broker,controller

controller.quorum.voters=1@localhost:19092,2@localhost:19093,3@localhost:19094

listeners=PLAINTEXT://:9094,CONTROLLER://:19094

inter.broker.listener.name=PLAINTEXT

controller.listener.names=CONTROLLER

log.dirs=/tmp/server3/kraft-combined-logs

listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

In Server 3 the broker port is 9094 and the controller port is 19094. Also the log.dirs is different

The complete server property files are available in this Git repo

Kafka cluster id creation and log directory setup

First we need to create kafka cluster id before starting the servers. This can be done using the following command:

./bin/kafka-storage.sh random-uuid

Note down the uuid that comes after running the above statement. In my case I got the following uuid

 ./bin/kafka-storage.sh random-uuid
 9dJzdGvfTPaCY4e8klXaDQ

Next we need to format all the storage directories. This is basically the directory that we put in log.dirs property.

This can be done with the following command

./bin/kafka-storage.sh format -t <uuid> -c <server_config_location>

Replace <uuid> with the uuid that you got in the previous step. Replace <server_config_location> with the server property files

In my case, I will be running the following commands

For Server 1:

./bin/kafka-storage.sh format -t 9dJzdGvfTPaCY4e8klXaDQ -c ./config/kraft/server1.properties

For Server 2:

./bin/kafka-storage.sh format -t 9dJzdGvfTPaCY4e8klXaDQ -c ./config/kraft/server2.properties

For Server 3:

./bin/kafka-storage.sh format -t 9dJzdGvfTPaCY4e8klXaDQ -c ./config/kraft/server3.properties

This is the result which you get when you run the 3 commands

$ ./bin/kafka-storage.sh format -t 9dJzdGvfTPaCY4e8klXaDQ -c ./config/kraft/server1.properties
Formatting /tmp/server1/kraft-combined-logs
$ ./bin/kafka-storage.sh format -t 9dJzdGvfTPaCY4e8klXaDQ -c ./config/kraft/server2.properties
Formatting /tmp/server2/kraft-combined-logs
$ ./bin/kafka-storage.sh format -t 9dJzdGvfTPaCY4e8klXaDQ -c ./config/kraft/server3.properties
Formatting /tmp/server3/kraft-combined-logs

Starting the kafka servers

The kafka servers can be started in daemon mode using the following commands:

First setup the heap properties

export KAFKA_HEAP_OPTS="-Xmx200M –Xms100M"

Here we are giving a very small max heap of 200M since we are running all the servers in a single local machine. If you have bigger servers you can give heap size of 1GB or Above.

Start Server 1:

./bin/kafka-server-start.sh -daemon ./config/kraft/server1.properties

Start Server 2:

./bin/kafka-server-start.sh -daemon ./config/kraft/server2.properties

Start Server 3:

./bin/kafka-server-start.sh -daemon ./config/kraft/server3.properties

Create a kafka topic

Let us create a topic kraft-test in this cluster

The topic can be created using the following command:

./bin/kafka-topics.sh --create --topic kraft-test --partitions 3 --replication-factor 3 --bootstrap-server localhost:9092

We are creating a topic with 3 partitions and 3 replicas since we have 3 nodes.

We can list the topics present in the cluster using the following command:

bin/kafka-topics.sh --bootstrap-server localhost:9093 --list

Running the above command gives the below result

$ bin/kafka-topics.sh --bootstrap-server localhost:9093 --list
kraft-test

We can describe the topics present in the cluster using the following command:

bin/kafka-topics.sh --bootstrap-server localhost:9093 --describe --topic kraft-test

Running the above command gives the below result

$ bin/kafka-topics.sh --bootstrap-server localhost:9093 --describe --topic kraft-test
Topic: kraft-test       TopicId: vZKswHHlQk2mEOw0yzAGAA PartitionCount: 3       ReplicationFactor: 3    Configs: segment.bytes=1073741824
        Topic: kraft-test       Partition: 0    Leader: 3       Replicas: 3,2,1 Isr: 3,2,1
        Topic: kraft-test       Partition: 1    Leader: 2       Replicas: 2,3,1 Isr: 2,3,1
        Topic: kraft-test       Partition: 2    Leader: 2       Replicas: 2,3,1 Isr: 2,3,1

Exploring the kafka metadata using metadata shell

Similar to zookeeper cli, there is a metadata shell provided by kafka so that we can read the data in the @metadata internal topic.

Open the shell using the following command

./bin/kafka-metadata-shell.sh  --snapshot /tmp/server1/kraft-combined-logs/\@metadata-0/00000000000000000000.log

The structure here is very similar to what we see in zookeeper

In order the list the brokers you can type ls brokers/

>> ls brokers/
1  2  3

In order to list the topics you can type ls topics/

>> ls topics/
kraft-test

In order to see the topic metadata you can type cat topics/kraft-test/0/data

>> cat topics/kraft-test/0/data
{
  "partitionId" : 0,
  "topicId" : "vZKswHHlQk2mEOw0yzAGAA",
  "replicas" : [ 3, 2, 1 ],
  "isr" : [ 3, 2, 1 ],
  "removingReplicas" : null,
  "addingReplicas" : null,
  "leader" : 3,
  "leaderEpoch" : 0,
  "partitionEpoch" : 0
}

In order to find the controller leader node you can type cat metadataQuorum/leader

>> cat metadataQuorum/leader
MetaLogLeader(nodeId=2, epoch=4)

Type exit to get out of the metadata shell

Producing and consuming data from kafka

Use the following command to start a kafka producer in the terminal

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kraft-test

In a different terminal, use the following command to start a kafka consumer

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic kraft-test

In the producer terminal send a bunch of messages as shown below

$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kraft-test
>message 1
>message 2
>message 3
>hello
>bye

In the consumer terminal you would see the messages coming as shown below

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic kraft-test                                                                       
message 1
message 2
message 3
hello
bye

Congrats 😊

You have now learnt how to setup kafka without zookeeper. This is still an experimental feature and should not be used in production.

If you are interested in realtime analytics using kafka and Apache Pinot, feel free to checkout my course in Udemy by clicking on the Image below

References

https://kafka.apache.org/

https://cwiki.apache.org/confluence/display/KAFKA/KIP-500

Feel free to connect with me in LinkedIn or follow me in twitter.

What is the use of Synchronized keyword in Java

Aditya Sridhar — Mon, 16 Nov 2020 15:53:04 +0000

This post was originally published in adityasridhar.com

This article will focus on what the synchronized keyword does in Java and when to use it.

In order to understand this article better please go through my article on Introduction to Multithreading in case you are new to multithreading in Java.

What is the use of Synchronized?

Let's take the following code snippet:

public void doSomething(){
    //Execute Logic 1
    //Execute Logic 2
    ......
}

Let's say that 2 threads t1 and t2 will be calling doSomething() method.

Here the main thing we care about is that if one thread calls doSomething(), then it needs to finish everything in the method before another thread gets to call doSomething().

Basically something like this:

t1 executes Logic 1
t1 executes Logic 2
t2 executes Logic 1
t2 executes Logic 2

t2 executes Logic 1
t2 executes Logic 2
t1 executes Logic 1
t1 executes Logic 2

But reality is they can run in any order.. The below scenario is highly possible:

t1 executes Logic 1
t2 executes Logic 1
t1 executes Logic 2
t2 executes Logic 2

But what if we want to ensure that t1 fully finishes doSomething() before t2 starts doSomething()?

This is where synchronized comes in. Synchronized ensures that any method or piece of code which is synchronized should be fully completed by one thread before another thread can access that block.

If we make doSomething() synchronized as shown below, then only one thread can access doSomething() at a time

public synchronized void doSomething(){
    //Execute Logic 1
    //Execute Logic 2
    ......
}

Different ways of using Synchronized

Normal Synchronized Method

In this section we will create 3 threads which will print numbers from 0 to 5.

First let's create a simple utility which will print numbers from 0 to 5:

class PrintUtil {
    public synchronized void printNumbers() {
        for (int i = 0; i <= 5; i++) {
            System.out.println(Thread.currentThread().getName() + ": " + i);
            try {
                Thread.sleep(100);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }
    }
}

This util has a method called printNumbers(). This method prints numbers from 0 to 5 and waits for 100ms after every print.
The method is marked as synchronized so that at any time only one thread can access printNumbers() method.

Next let's create a worker which will implement Runnable Interface:

class RunnableWorker implements Runnable {
    PrintUtil pu;
    public RunnableWorker(PrintUtil pu) {
        this.pu = pu;
    }

    @Override
    public void run() {
        pu.printNumbers();
    }
}

This is very simple worker which calls the printNumbers() method in PrintUtil class.

Finally let's create the Main class which will tie all this together:

public class SynchronizedMethodDemo {
    public static void main(String[] args) {
        PrintUtil pu = new PrintUtil();
        Runnable r = new RunnableWorker(pu);
        Thread t1 = new Thread(r);
        Thread t2 = new Thread(r);
        Thread t3 = new Thread(r);

        t1.start();
        t2.start();
        t3.start();
    }
}

The main class creates an instance of PrintUtil.

It then creates 3 threads t1, t2 and t3 by passing the RunnableWorker as an argument and then starts the 3 threads.

The output of the above program is given below:

Thread-0: 0
Thread-0: 1
Thread-0: 2
Thread-0: 3
Thread-0: 4
Thread-0: 5
Thread-2: 0
Thread-2: 1
Thread-2: 2
Thread-2: 3
Thread-2: 4
Thread-2: 5
Thread-1: 0
Thread-1: 1
Thread-1: 2
Thread-1: 3
Thread-1: 4
Thread-1: 5

As seen above, Synchronized has ensured that only when one thread completes printing from 0 to 5, the other thread starts it's execution.

How does this work ?

In this example the PrintUtil object instance pu has a lock. When one thread executes the printNumbers() method, it gets the lock from pu. This thread releases the lock only when printNumbers() execution is fully done. The other threads can execute printNumbers() only when they get the lock.

Since each thread holds the lock until printNumbers() is fully complete, we get the output shown above.

In General the lock is held by the object which calls the synchronized method. In this case that object was pu

Synchronized with Static methods

In this example we use a static method to print numbers from 0 to 5 instead of an instance method.

The Printer Utility code is shown below:

class PrintUtilExample2 {
    public static synchronized void printNumbers() {
        for (int i = 0; i <= 5; i++) {
            System.out.println(Thread.currentThread().getName() + ": " + i);
            try {
                Thread.sleep(100);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }
    }
}

Here the method is marked as static and synchronized.

Next let us create the worker. The code for the worker is shown below:

class RunnableWorkerExample2 implements Runnable {
    @Override
    public void run() {
        PrintUtilExample2.printNumbers();
    }
}

In this case the worker calls printNumbers() directly using the class name since it is static method.

Finally we have the main class which is shown below:

public class SynchronizedStaticMethodDemo {
    public static void main(String[] args) {
        Runnable r = new RunnableWorkerExample2();
        Thread t1 = new Thread(r);
        Thread t2 = new Thread(r);
        Thread t3 = new Thread(r);

        t1.start();
        t2.start();
        t3.start();
    }
}

Here also the main method creates three threads t1, t2 and t3 with argument as RunnableWorkerExample2 and starts the threads.

The output of this code is given below:

Thread-2: 0
Thread-2: 1
Thread-2: 2
Thread-2: 3
Thread-2: 4
Thread-2: 5
Thread-0: 0
Thread-0: 1
Thread-0: 2
Thread-0: 3
Thread-0: 4
Thread-0: 5
Thread-1: 0
Thread-1: 1
Thread-1: 2
Thread-1: 3
Thread-1: 4
Thread-1: 5

Again we can see that each thread completes printing from 0 to 5 before the next thread starts printing.

How does this work

In case of static methods, the lock is maintained by the PrintUtilExample2 class itself. Rest of the locking mechanism is exactly same as mentioned in the previous section.

Creating synchronized blocks with custom objects

In this section we will see how to make a block of code synchronized using a custom object.

The Print utility class is shown below:

class PrintUtilExample3 {
    final Object lockObject = new Object();
    public void printNumbers() {
        synchronized(lockObject) {
            for (int i = 0; i <= 5; i++) {
                System.out.println(Thread.currentThread().getName() + ": " + i);
                try {
                    Thread.sleep(100);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
            }
        }
    }
}

Here the printNumbers() method is not marked as synchronized.

We create an object called lockObject. We then create a synchronized block of code using synchronized(lockObject){}

The advantage in using this approach, is that we can mark any custom block of code as synchronized. ( It can be an entire method, or it can be even a single line in the entire method. )

The worker code is shown below:

class RunnableWorkerExample3 implements Runnable {
    PrintUtilExample3 pu3;
    public  RunnableWorkerExample3(PrintUtilExample3 pu3){
        this.pu3 = pu3;
    }
    @Override
    public void run() {
        pu3.printNumbers();
    }
}

The main class is shown below:

public class SynchronizedCustomObjectDemo {
    public static void main(String[] args) {
        PrintUtilExample3 pu3 = new PrintUtilExample3();
        Runnable r = new RunnableWorkerExample3(pu3);
        Thread t1 = new Thread(r);
        Thread t2 = new Thread(r);
        Thread t3 = new Thread(r);

        t1.start();
        t2.start();
        t3.start();
    }
}

The output of the above program is shown below:

Thread-0: 0
Thread-0: 1
Thread-0: 2
Thread-0: 3
Thread-0: 4
Thread-0: 5
Thread-2: 0
Thread-2: 1
Thread-2: 2
Thread-2: 3
Thread-2: 4
Thread-2: 5
Thread-1: 0
Thread-1: 1
Thread-1: 2
Thread-1: 3
Thread-1: 4
Thread-1: 5

In this approach as well we see that each thread completes printing from 0 to 5 before the next thread starts printing the numbers.

How does this work

In this case the custom object created lockObject holds the lock. Rest of the locking mechanism is same as mentioned in the previous sections.

Also in the previous sections the entire method was marked as synchronized. But in this approach any block of code can be marked as synchronized which gives a greater flexibility.

Code

All the code discussed in this article can be found in this GitHub repo.

Issues with Synchronized

Synchronized does come with some issues. There maybe scenarios where we want a block of code to be synchronized only for write operations. But for read operations any number of concurrent threads should be allowed. In order to handle these scenarios there are other better approaches in Java which I will cover in a future articles.

Congrats 😊

You now know what is synchronized and how to use it in Java.

Feel free to connect with me in LinkedIn or follow me in twitter.

If you liked this post, you can checkout my website https://adityasridhar.com for other similar posts

What are Callbacks in Javascript and how to use them

Aditya Sridhar — Tue, 14 Apr 2020 15:53:42 +0000

Callbacks are a very important topic to understand in javascript. In this article, We will be seeing what Callbacks are and how to use them.

What are Callbacks

First, we will see a regular javascript function, and from there we will see how callbacks are used.

Regular Javascript function

So first let us look at a normal Function in javascript.

function multiply(a, b) {
  var result = a * b;
  console.log("multiply Function Result:",result);
}
multiply(2, 4);

Here we have a simple function which multiplies 2 numbers. We are then calling the function with inputs 2 and 4.

Callbacks Example 1

Now imagine if you had to run another operation immediately after multiply computes the result. This is where we use a callback. The Below code shows this.

function multiply(a, b, callback) {
  var result = a * b;
  console.log("multiply Function Result:",result);
  callback(result);
}

function checkOddEven(result){
    var isEven = result % 2 == 0
    if(isEven){
        console.log('checkOddEven Function:',result,' is Even');
    }
    else
    {
        console.log('checkOddEven Function:',result,' is Odd');
    }
}

multiply(7, 9, checkOddEven);

Here in the multiply function, we accept a callback as well as the input.

when we call the multiply function we pass callback as checkOddEven. So basically a callback is nothing but a function. checkOddEven is a function which checks whether a number is odd or even.

In the multiply function, at the end, we have callback(result). This is where we ask the callback function to execute.

So in the above code, the sequence is as follows

First we call the multiply function and pass checkOddEven as the callback
multiply function executes and computes the multiplication result
once the result is calculated, multiply function asks the callback to execute.
In this case, the callback is checkOddEven function. So checkOddEven function will execute.

The result of the above code is shown below

multiply Function Result: 63
checkOddEven Function: 63  is Odd

We can pass any function to the callback.

Callbacks Example 2

Let's take the following script

function multiply(a, b, callback) {
  var result = a * b;
  console.log("multiply Function Result:",result);
  callback(result);
}

function checkPosNeg(result){
    var isPositive = result >= 0; 
    if(isPositive){
        console.log('checkPosNeg Function:',result,' is Positive');
    }
    else
    {
        console.log('checkPosNeg Function:',result,' is Negative');
    }
}
multiply(-7, 9, checkPosNeg);

Here we have a function called checkPosNeg which checks whether the number is positive or negative.

We are passing the callback as checkPosNeg in this Example.

The output of the above program is given below

multiply Function Result: -63
checkPosNeg Function: -63  is Negative

From this example, we see that any function can be passed to the callback.

Anonymous Callback Function

Another way of passing a callback is by using anonymous functions. The code for this is shown below.

function multiply(a, b, callback) {
  var result = a * b;
  console.log("multiply Function Result:", result);
  callback(result);
}

multiply(-7, 9, function(result) {

  if (result > 0) {
    console.log('checkPosNeg Function:', result, ' is Positive');
  } else {
    console.log('checkPosNeg Function:', result, ' is Negative');
  }
});

In this case, we see that the callback function is created at the same time we are calling the multiply function. This function basically checks whether the number is positive or negative but the function does not have any name.

Error Handling in Callbacks

The below code snippet shows how to do error handling in callbacks.

function divide(a, b, callback) {
  if (b != 0) {
    var result = a / b;
    console.log('divide Function Result', result);
    callback(null, result);
  } else
    callback(new Error('Divide by 0 Error:' + a + '/' + b))
}

function checkPosNeg(error, result) {
  if (error) {
    console.log('checkPosNeg Function cannot run');
    console.log(error);
  } else {
    var isPositive = result >= 0;
    if (isPositive) {
      console.log('checkPosNeg Function:', result, ' is Positive');
    } else {
      console.log('checkPosNeg Function:', result, ' is Negative');
    }
  }
}

divide(4, 0, checkPosNeg);

In this case, we have a function called divide which has a callback checkPosNeg.

Now when b is 0, then the division is not possible. If the division is not possible, then we cannot send any result to the callback.

So in this case we define the callback function as checkPosNeg(error,result).

Whenever division is possible we call callback(null,result) indicating there is no Error and everything is good.

If the division is not possible then we call callback(new Error('Error message')) which tells that there is an error.

Now in checkPosNeg function we need to check for error as well. In case error is not null then we need to take necessary action in the code. For example, here we are just printing the error message.

Why do we need Callbacks

The obvious question which you might have is why do we even need callbacks.

Let’s take the following code snippet

console.log('Task1');
makeServerCall(url,function(error,result){
    console.log('Task2');
});
console.log('Task3');

In the Above code first Task1 is printed.

Next makeServerCall function makes a network call.

Now Will Task3 be printed before or after Task2?

Generally, whenever we make a network call, the Code continues to the next statement and does not wait for the result in sync.

So the moment the network call is made, the code continues to the next statement and prints Task3.

Once the network call completes and the response comes back then Task2 is printed.

So here makeServerCall takes a callback as its input. So once the server call completes, it executes the callback.

In this case, the callback enables us to run some operation once the network call is complete without blocking the code ( i.e the future statements are not blocked until the network call is complete).

Callback of Callbacks

Callbacks can be Chained Together.

Take the following code snippet.

function1(input1, function(error, result1) {
  function2(result1, function(error, result2) {
    function3(result2, function(error, result3) {
      console.log('Callback Chain')
    })
  })
})

Here first we wait for function1 to complete the network call and execute the first callback.
The first callback, in turn, calls function2. Once function2 completes its network call, it executes the second callback.
The Second callback calls function3. Once function3 completes its network call, it executes the third callback.
The Third callback just prints a message.

More callbacks can be chained together as well.

Something doesn't seem right here

Well, as you might have already noticed in the above script, it becomes a little unreadable as the number of callbacks increase.

The above example shows just one-liner functions. If the functions were slightly bigger and the number of chained callbacks was more, then the code will be highly unreadable. Also, this means it is very very hard to debug the code.

A sample snippet is here to illustrate this

function1(input1, function(error, result1) {
  if (error) {
    console.log('Error')
  } else {
    function2(result1, function(error, result2) {
      if (error) {
        console.log('Error')
      } else {
        function3(result2, function(error, result3) {
          if (error) {
            console.log('Error')
          } else {
            function4(result3, function(error, result4) {
              if (error) {
                console.log('Error')
              }
            })
          }
        })

      }
    })
  }
})

This problem is known as the Pyramid of doom.

One way to get around this is to use Promises which I will be covering in a future article

Congrats 😄

You now know what are callbacks and how to use them.

Happy Coding 😄

This post was originally published in adityasridhar.com

Feel free to connect with me in LinkedIn or follow me in twitter.

If you liked this post, you can checkout my website https://adityasridhar.com for other similar posts

How to Effectively use GRPC Streams in NodeJS

Aditya Sridhar — Wed, 01 Apr 2020 16:22:13 +0000

This post was originally published in adityasridhar.com

This article will explain how to Use Streams in GRPC in a NodeJS Application.

To know the Basics of GRPC and Protocol Buffers you can read my Introduction to gRPC Article

What are Streams in GRPC

Streams in GRPC help us to send a Stream of messages in a single RPC Call.

In this article, we will be focussing on the following streams

Server Streaming GRPC: In this case, the client makes a single request to the server and the server sends a stream of messages back to the client.
Client Streaming GRPC: In this case, the client sends a stream of messages to the server. The server then processes the stream and sends a single response back to the client.

Server Streaming GRPC

Now let us create the server and client codes for a Server Streaming GRPC

Creating the .proto file

create a folder called proto. In the folder create a file called employee.proto. Copy the following into employee.proto

syntax = "proto3";

package employee;

service Employee {

  rpc paySalary (EmployeeRequest) returns (stream EmployeeResponse) {}
}


message EmployeeRequest {
  repeated int32 employeeIdList = 1;
}

message EmployeeResponse{
  string message = 1;
}

Refer to my grpc basics article to know more about .proto files and Protocol Buffers.

Here we are creating an rpc called paySalary which accepts EmployeeRequest as the request and sends stream of EmployeeResponse as the response. We use the keyword stream to indicate that the server will send a stream of messages

EmployeeRequest and EmployeeResponse are defined above as well. repeated keyword indicates that a list of data will be sent.

In this example, the request to paySalary will be a list of employee Ids. The server will respond with a stream of messages telling if Salary has been paid to an employee or not.

Creating Dummy data for the Server

Create a file called data.js and copy the following code into it.

//Hardcode some data for employees
let employees = [{
    id: 1,
    email: "abcd@abcd.com",
    firstName: "First1",
    lastName: "Last1"   
},
{
    id: 2,
    email: "xyz@xyz.com",
    firstName: "First2",
    lastName: "Last2"   
},
{
    id: 3,
    email: "temp@temp.com",
    firstName: "First3",
    lastName: "Last3"   
},
];

exports.employees = employees;

We will use it as the data source for the server.

Creating the Server

Create a file called as server.js. Copy the following code into server.js

const PROTO_PATH = __dirname + '/proto/employee.proto';

const grpc = require('grpc');
const protoLoader = require('@grpc/proto-loader');


let packageDefinition = protoLoader.loadSync(
  PROTO_PATH,
  {
    keepCase: true,
    longs: String,
    enums: String,
    defaults: true,
    oneofs: true
  });
let employee_proto = grpc.loadPackageDefinition(packageDefinition)

Refer to my grpc basics article to understand what the above script does.

Next, add the following piece of code into server.js

let { paySalary } = require('./pay_salary.js');

function main() {
  let server = new grpc.Server();
  server.addService(employee_proto.Employee.service, 
    { paySalary: paySalary }
  );
  server.bind('0.0.0.0:4500', grpc.ServerCredentials.createInsecure());
  server.start();
}

main();

In the above script, we are starting the GRPC Server and adding Employee Service into it along with paySalary implementation.

But paySalary function is defined in pay_salary.js file.

Let us create a pay_salary.js file.

Add the following script into pay_salary.js file

let { employees } = require('./data.js');
const _ = require('lodash');

function paySalary(call) {
    let employeeIdList = call.request.employeeIdList;

    _.each(employeeIdList, function (employeeId) {
      let employee = _.find(employees, { id: employeeId });
      if (employee != null) {
        let responseMessage = "Salary paid for ".concat(
          employee.firstName,
          ", ",
          employee.lastName);
        call.write({ message: responseMessage });
      }
      else{
        call.write({message: "Employee with Id " + employeeId + " not found in record"});
      }

    });
    call.end();

}

exports.paySalary = paySalary;

paySalary function takes call as the input. call.request will have the request sent by the client.

call.request.employeeIdList will have the list of employee Ids sent by the client.

We then loop over the EmployeeId's and for each employee Id we do some processing.

For each employee Id, we call the call.write function at the last. call.write will write a single message in a stream back to the client.

In this case for each employee, call.write will send back whether salary has been paid or not.

Once all the Employee Id's have been processed we call the call.end function. call.end indicates that the stream is complete.

Here is the final server.js file

const PROTO_PATH = __dirname + '/proto/employee.proto';

const grpc = require('grpc');
const protoLoader = require('@grpc/proto-loader');


let packageDefinition = protoLoader.loadSync(
  PROTO_PATH,
  {
    keepCase: true,
    longs: String,
    enums: String,
    defaults: true,
    oneofs: true
  });
let employee_proto = grpc.loadPackageDefinition(packageDefinition)

let { paySalary } = require('./pay_salary.js');

function main() {
  let server = new grpc.Server();
  server.addService(employee_proto.Employee.service, 
    { paySalary: paySalary }
  );
  server.bind('0.0.0.0:4500', grpc.ServerCredentials.createInsecure());
  server.start();
}

main();

Creating the Client

Create a file called client_grpc_server_stream.js. Copy the following code into the file.

const PROTO_PATH = __dirname + '/proto/employee.proto';

const grpc = require('grpc');
const protoLoader = require('@grpc/proto-loader');

let packageDefinition = protoLoader.loadSync(
    PROTO_PATH,
    {keepCase: true,
     longs: String,
     enums: String,
     defaults: true,
     oneofs: true
    });
let employee_proto = grpc.loadPackageDefinition(packageDefinition).employee;

The above script has already been explained in my grpc basics article.

Next, add the following piece of the script to the client.

function main() {
  let client = new employee_proto.Employee('localhost:4500',
                                       grpc.credentials.createInsecure());

  let employeeIdList = [1,10,2];
  let call = client.paySalary({employeeIdList: employeeIdList});

  call.on('data',function(response){
    console.log(response.message);
  });

  call.on('end',function(){
    console.log('All Salaries have been paid');
  });

}

main();

client variable will have the stub which will help us call the function in the server.

employeeIdList is the input given to the server.

let call = client.paySalary({employeeIdList: employeeIdList}); script calls the paySalary function in the server and passes employeeIdList as the request. Since the server is going to send a stream of messages, call object will help us listen to the stream events

We listen to the 'data' event in call object for any message coming from the server in the stream. This is shown in the below script

  call.on('data',function(response){
    console.log(response.message);
  });

Here we just print the response message whenever we receive any
message from the server

We listen to the 'end' event in the call object to know when the server stream ends. This is shown in the below script

  call.on('end',function(){
    console.log('All Salaries have been paid');
  });

Here when the stream ends we are printing 'All Salaries have been paid'.

Here is the complete code for client_gprc_server_stream.js

const PROTO_PATH = __dirname + '/proto/employee.proto';

const grpc = require('grpc');
const protoLoader = require('@grpc/proto-loader');

let packageDefinition = protoLoader.loadSync(
    PROTO_PATH,
    {keepCase: true,
     longs: String,
     enums: String,
     defaults: true,
     oneofs: true
    });
let employee_proto = grpc.loadPackageDefinition(packageDefinition).employee;

function main() {
  let client = new employee_proto.Employee('localhost:4500',
                                       grpc.credentials.createInsecure());

  let employeeIdList = [1,10,2];
  let call = client.paySalary({employeeIdList: employeeIdList});

  call.on('data',function(response){
    console.log(response.message);
  });

  call.on('end',function(){
    console.log('All Salaries have been paid');
  });

}

main();

Running the Code

Open a Command prompt and start the server using the following script.

node server.js

open a new command prompt and run the client using the following script.

node client_grpc_server_stream.js

On running the client we will get the following output.

Salary paid for First1, Last1
Employee with Id 10 not found in record
Salary paid for First2, Last2
All Salaries have been paid

In this case, the client has sent 3 Id's 1,10,2 to the server. The Server processes the Id's one by one and sends a stream of messages to the client. Once all the messages in the stream are completed, the message 'All Salaries have been paid' is printed.

Client Streaming GRPC

Now let us create the server and client codes for a Client Streaming GRPC.

Creating the .proto file

In the previously created employee.proto file add the following

service Employee {

  rpc paySalary (EmployeeRequest) returns (stream EmployeeResponse) {}

  rpc generateReport (stream ReportEmployeeRequest) returns (ReportEmployeeResponse) {}
}

message ReportEmployeeRequest {
  int32 id = 1;
}

message ReportEmployeeResponse{
  string successfulReports = 1;
  string failedReports = 2;
}

Here we have added a new rpc called generateReport which accepts a stream of ReportEmployeeRequest as request and returns ReportEmployeeResponse as the response.

so the input to the rpc is a stream of employee Id's and the response from the server will be a single response which says how many reports were generated and how many reports failed.

Here is the complete employee.proto file after our changes

syntax = "proto3";

package employee;

service Employee {

  rpc paySalary (EmployeeRequest) returns (stream EmployeeResponse) {}

  rpc generateReport (stream ReportEmployeeRequest) returns (ReportEmployeeResponse) {}
}


message EmployeeRequest {
  repeated int32 employeeIdList = 1;
}

message EmployeeResponse{
  string message = 1;
}

message ReportEmployeeRequest {
  int32 id = 1;
}

message ReportEmployeeResponse{
  string successfulReports = 1;
  string failedReports = 2;
}

Creating the Server

Here is the complete server.js code with the new rpc added

const PROTO_PATH = __dirname + '/proto/employee.proto';

const grpc = require('grpc');
const protoLoader = require('@grpc/proto-loader');


let packageDefinition = protoLoader.loadSync(
  PROTO_PATH,
  {
    keepCase: true,
    longs: String,
    enums: String,
    defaults: true,
    oneofs: true
  });
let employee_proto = grpc.loadPackageDefinition(packageDefinition).employee;


let { paySalary } = require('./pay_salary.js');
let { generateReport } = require('./generate_report.js');

function main() {
  let server = new grpc.Server();
  server.addService(employee_proto.Employee.service, 
    { paySalary: paySalary ,
      generateReport: generateReport }
  );
  server.bind('0.0.0.0:4500', grpc.ServerCredentials.createInsecure());
  server.start();
}

main();

In the above script, we can see that we have added generateReport function as well to the grpc server.
Also we can see that generateReport function comes from generate_report.js file.

Create a file called as generate_report.js.

Add the following script into the file

let { employees } = require('./data.js');
const _ = require('lodash');

function generateReport(call, callback){

    let successfulReports = [];
    let failedReports = [];
    call.on('data',function(employeeStream){
        let employeeId = employeeStream.id;
        let employee = _.find(employees, { id: employeeId });
        if (employee != null) {
          successfulReports.push(employee.firstName);
        }
      else{
          failedReports.push(employeeId);
      }

    });
    call.on('end',function(){
        callback(null,{
            successfulReports: successfulReports.join(),
            failedReports: failedReports.join()
        })
    })
}

exports.generateReport = generateReport;

The generateReport function takes two inputs, call and callback

In order to get the stream of messages from the client, we need to listen to the data event in the call object. This is done in the following script.

    call.on('data',function(employeeStream){
        let employeeId = employeeStream.id;
        let employee = _.find(employees, { id: employeeId });
        if (employee != null) {
          successfulReports.push(employee.firstName);
        }
      else{
          failedReports.push(employeeId);
      }

    });

The data event is called for every single message coming from the client. The message is present in the employeeStream variable. On receiving the message we try to generate a report and find out if it succeeded or failed.

The end event on the call object indicates that the client stream has ended. The following code shows how to listen to the end event.

 call.on('end',function(){
        callback(null,{
            successfulReports: successfulReports.join(),
            failedReports: failedReports.join()
        })
    })

In this case, when the end event happens, we combine all the success and failure reports into a single response object and send it back to the client using the callback object.

Creating the Client

Create a file called as client_grpc_client_stream.js. Add the following script into it.

const PROTO_PATH = __dirname + '/proto/employee.proto';

const grpc = require('grpc');
const protoLoader = require('@grpc/proto-loader');
const _ = require('lodash');

let packageDefinition = protoLoader.loadSync(
  PROTO_PATH,
  {
    keepCase: true,
    longs: String,
    enums: String,
    defaults: true,
    oneofs: true
  });
let employee_proto = grpc.loadPackageDefinition(packageDefinition).employee;

The above script does the same functionality as we saw in the server code.

Add the following script as well to the client_grpc_client_stream.js.

function main() {
  let client = new employee_proto.Employee('localhost:4500',
    grpc.credentials.createInsecure());

  let call = client.generateReport(function (error, response) {
    console.log("Reports successfully generated for: ", response.successfulReports);
    console.log("Reports failed since Following Employee Id's do not exist: ", response.failedReports);
  });

  let employeeIdList = [1, 10, 2];
  _.each(employeeIdList, function (employeeId) {
        call.write({ id: employeeId });
  })

  call.end();
}

main();

Let us see what the script above is doing.

let call = client.generateReport(function (error, response) {
    console.log("Reports successfully generated for: ", response.successfulReports);
    console.log("Reports failed since Following Employee Id's do not exist: ", response.failedReports);
  });

In this portion of the script, we are creating a call object and calling the generateReport function. Also inside the generateReport function, we are indicating what the client should do once it receives the response from the server. In this case, we are printing the successful and failed reports which the server sends back.

 let employeeIdList = [1, 10, 2];
  _.each(employeeIdList, function (employeeId) {
        call.write({ id: employeeId });
  })

In the above portion of the script, we are looping over the employee IDs and sending a stream of messages to the server. We use call.write to send the message in a stream to the server.

Finally, once we have sent all the messages in a stream, we indicate that the stream is complete using the call.end function as shown below

call.end();

The complete code for client_grpc_client_stream is given below.

const PROTO_PATH = __dirname + '/proto/employee.proto';

const grpc = require('grpc');
const protoLoader = require('@grpc/proto-loader');
const _ = require('lodash');

let packageDefinition = protoLoader.loadSync(
  PROTO_PATH,
  {
    keepCase: true,
    longs: String,
    enums: String,
    defaults: true,
    oneofs: true
  });
let employee_proto = grpc.loadPackageDefinition(packageDefinition).employee;

function main() {
  let client = new employee_proto.Employee('localhost:4500',
    grpc.credentials.createInsecure());

  let call = client.generateReport(function (error, response) {
    console.log("Reports successfully generated for: ", response.successfulReports);
    console.log("Reports failed since Following Employee Id's do not exist: ", response.failedReports);
  });

  let employeeIdList = [1, 10, 2];
  _.each(employeeIdList, function (employeeId) {
        call.write({ id: employeeId });
  })

  call.end();
}

main();

Running the Code

Open a Command prompt and start the server using the following script.

node server.js

open a new command prompt and run the client using the following script.

node client_grpc_client_stream.js

On running the client we will get the following output.

Reports successfully generated for:  First1,First2
Reports failed since Following Employee Id\'s do not exist:  10

In this case, the client has sent 3 Id's 1,10,2 to the server as a stream of messages. The server then processes the messages in the stream and sends a single response back to the client showing how many reports succeeded and how many reports failed.

Code

The Code discussed in this article can be found here

References

GRPC official Documentation : https://grpc.io/

Protocol Buffers Proto3 documentation : https://developers.google.com/protocol-buffers/docs/proto3

Congrats 😄

You now know how to use GRPC Streams in a NodeJS Application.

Happy Coding 😄

Feel free to connect with me in LinkedIn or follow me in twitter.

If you liked this post, you can checkout my website https://adityasridhar.com for other similar posts

How to Easily use GRPC and Protocol Buffers with NodeJS

Aditya Sridhar — Sat, 28 Mar 2020 14:35:51 +0000

This post was originally published in adityasridhar.com

This Article will give a basic introduction to GRPC and Protocol Buffers. Following that I will be showing how to use GRPC and Protocol Buffers in a NodeJS Application

What is GRPC

GRPC is a Open Source High Performance RPC Framework

So what exactly does a RPC do?

Take the following Example

function getEmployee() {
  return "ABCD";
}
function greetEmployee()
{
    let employee = getEmployee();
    console.log("Greet",employee) 
}

Here we have a function getEmployee which returns an Employee Name and another function greetEmployee which Calls getEmployee and gets the employee's Name and then prints a Greeting.

Here greetEmployee calling getEmployee is a regular function call.

Now if getEmployee and greetEmployee functions are located in different address spaces, or they are located in 2 different hosts which are separated by the network, then the function call is called a Remote Procedure Call. Here the System which has the getEmployee function acts like a Server and the System which has the greetEmployee function acts like a Client.

What is a Protocol Buffer

Protocol Buffer is the Interface Definition Language which is used by default in GRPC.

It helps to define the various services offered by a server.
It helps to define the Structure of the Payloads used in the System
It helps to serialize the message ( To a Special Binary Format ) and send it across the wire between the server and client.

We will see how to use protocol buffers when we are working on the NodeJS application later in this article.

What are the Different Types of RPC Supported

Unary RPC

This is the Simplest RPC available. Here the Client sends a Request Message to the Server. The Server processes the request and then sends a Response message back to the Client.

In this Article this is the grpc we will be focussing on.

Server Streaming RPC

In this RPC, the client sends a request message to the server, and the server sends a sequence of messages back to the Client in a stream fashion.

Client Streaming RPC

In this RPC, the client sends a sequence of messages to the server in a stream fashion. The server then processes all these requests and then sends a response message back to the client.

Bidirectional Streaming RPC

In this RPC, the client sends a sequence of messages to the server in a stream fashion. The server then processes the request and then sends a sequence of messages back to the client in stream fashion.

How to use GRPC and Protocol Buffers in NodeJS

Create a folder called as grpc-nodejs-demo and Initialize nodejs in it using the following commands

mkdir grpc-nodejs-demo
cd grpc-nodejs-demo
npm init

This will create a package.json file. To know more about NodeJS you can read my other article here

Modify the package.json file

Replace the package.json file with the following

{
  "name": "grpc-nodejs-demo",
  "version": "1.0.0",
  "description": "",
  "main": "server.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "dependencies": {
    "@grpc/proto-loader": "^0.1.0",
    "grpc": "^1.11.0",
    "lodash": "^4.6.1"
  },
  "author": "Name",
  "license": "MIT"
}

Here we are adding 3 dependencies

@grpc/proto_loader and grpc dependencies will help us use GRPC and Protocol Buffers in the Application.
lodash is a general utility dependency. It will help simplify some code logic

Once the package.json file has been updated. Run the following command to install the dependencies

npm install

Defining the Protocol Buffer

In this Example we will build a service which will take employee ID as the input, and give the employee details as the output.

The Service Interface and payloads needed will be specified in a protocol buffer file. Protocol buffer files have an extension of .proto

Now let's create the .proto file.

Create a folder called as proto within the project. Inside the proto folder, create a file with the name employee.proto and add the following code to it

syntax = "proto3";

package employee;

service Employee {

  rpc getDetails (EmployeeRequest) returns (EmployeeResponse) {}
}


message EmployeeRequest {
  int32 id = 1;
}

message EmployeeResponse{
  EmployeeDetails message = 1;
}
message EmployeeDetails {
  int32 id = 1;
  string email = 2;
  string firstName = 3; 
  string lastName = 4;
}

So what exactly have we done here?.

syntax = "proto3"; indicates that we want to use Protocol Buffer version 3.

package employee; indicates that we are creating a package called as employee within which we will define our services

service Employee {

  rpc getDetails (EmployeeRequest) returns (EmployeeResponse) {}
}

The Above script tells that We are creating a Service called as Employee. Within this service we are creating a function ( rpc ) called as getDetails which accepts input of type EmployeeRequest and provides response in the format EmployeeResponse

Next we need to define EmployeeRequest and EmployeeResponse. This is done in the following script

message EmployeeRequest {
  int32 id = 1;
}

message EmployeeResponse{
  EmployeeDetails message = 1;
}
message EmployeeDetails {
  int32 id = 1;
  string email = 2;
  string firstName = 3; 
  string lastName = 4;
}

Here we see that the message EmployeeRequest has a single field of type int32 and name id. The number 1 assigned here is a field number and it helps during encoding and decoding of the message. Every field defined should have a unique field number

We also see that EmployeeResponse has a custom field of type EmployeeDetails and name message having field number of 1. This means that even EmployeeDetails has to be defined which is shown above as well.

EmployeeDetails has 4 fields comprising types int32 and string. All of them have unique field numbers

Field numbers between 1 -15 use 1 byte of space during encoding. and field numbers from 2 - 2047 uses 2 bytes for encoding and hence will take up more space. So try to design in such a way that the field numbers are between 1 - 15 as much as possible

Creating the GRPC Server

Create a file called as server.js

First Let us include all the libraries we need and also define the location where the .proto file is present

const PROTO_PATH = __dirname + '/proto/employee.proto';

const grpc = require('grpc');
const protoLoader = require('@grpc/proto-loader');
const _ = require('lodash');

Next we need to load the .proto file. This is done using the protoLoader libary loadSync method.

let packageDefinition = protoLoader.loadSync(
    PROTO_PATH,
    {keepCase: true,
     longs: String,
     enums: String,
     defaults: true,
     oneofs: true
    });

Next from the loaded proto file package Definition we need to get the package we need. This is done using the following script

let employee_proto = grpc.loadPackageDefinition(packageDefinition).employee;

here we are getting the employee package into employee_proto variable.

employee_proto will now have all the proto definitions.

Next we need to create some dummy employees data for the server to work with. Create a file called as data.js and add the following script into it

let employees = [{
    id: 1,
    email: "abcd@abcd.com",
    firstName: "First1",
    lastName: "Last1"   
},
{
    id: 2,
    email: "xyz@xyz.com",
    firstName: "First2",
    lastName: "Last2"   
},
{
    id: 3,
    email: "temp@temp.com",
    firstName: "First3",
    lastName: "Last3"   
},
];

exports.employees = employees;

Next we need to import data.js into server.js. Add the following script in server.js for this

let {employees} = require('./data.js');

So employees will have the list of employees with their id, email, firstName and lastName

The next piece of script creates and starts the GRPC Server.

function main() {
  let server = new grpc.Server();
  server.addService(employee_proto.Employee.service, {getDetails: getDetails});
  server.bind('0.0.0.0:4500', grpc.ServerCredentials.createInsecure());
  server.start();
}

let server = new grpc.Server(); is the script which creates a new GRPC Server

In the .proto file we notice that we have a function called as getDetails inside Employee Service.

server.addService(employee_proto.Employee.service, {getDetails: getDetails}); is the script in which we add the Service implementation. This script says that we are adding the getDetails function in employee_proto.Employee Service. And Then We are adding this Service to the Server.

server.bind('0.0.0.0:4500', grpc.ServerCredentials.createInsecure()); is the script which tells that the server will start in port 4500 and have no Authentication

server.start(); is the script which actually starts the server.

The main thing pending now is to implement getDetails function. The below script shows the implementation

function getDetails(call, callback) {
  callback(null, 
    {
       message: _.find(employees, { id: call.request.id })
    });
}

here call has the request parameters and callback is where we need to define the implementation.

Inside callback we have message: _.find(employees, { id: call.request.id }) which says the following

Get the employee ID from Input - call.request.id
search the employees list to find the Employee who has that Id
Return that Employee Details

This completes the Server implementation. Here is the complete script for server.js

const PROTO_PATH = __dirname + '/proto/employee.proto';

const grpc = require('grpc');
const protoLoader = require('@grpc/proto-loader');
const _ = require('lodash');

let packageDefinition = protoLoader.loadSync(
    PROTO_PATH,
    {keepCase: true,
     longs: String,
     enums: String,
     defaults: true,
     oneofs: true
    });
let employee_proto = grpc.loadPackageDefinition(packageDefinition).employee;

let {employees} = require('./data.js');

function getDetails(call, callback) {
  callback(null, 
    {
       message: _.find(employees, { id: call.request.id })
    });
}

function main() {
  let server = new grpc.Server();
  server.addService(employee_proto.Employee.service, {getDetails: getDetails});
  server.bind('0.0.0.0:4500', grpc.ServerCredentials.createInsecure());
  server.start();
}

main();

Creating the GRPC Client

Create a file called as client.js

Copy the following script into client.js

const PROTO_PATH = __dirname + '/proto/employee.proto';

const grpc = require('grpc');
const protoLoader = require('@grpc/proto-loader');

let packageDefinition = protoLoader.loadSync(
    PROTO_PATH,
    {keepCase: true,
     longs: String,
     enums: String,
     defaults: true,
     oneofs: true
    });
let employee_proto = grpc.loadPackageDefinition(packageDefinition).employee;

The Above script loads the employee package into employee_proto variable in the same way we saw in server.js

Next we need a way in which we can call the RPC. In this case we need to be able to call the getDetails function which is implemented in the Server.

For this we need to create a stub in the client. This is done using below script.

let client = new employee_proto.Employee('localhost:4500',
                                       grpc.credentials.createInsecure());

This client Stub will help us call the getDetails function which is defined in Employee Service which runs on the Server. The Server in turn runs on port 4500. The Line of code also indicates that there is no Authentication used

finally we can call the getDetails function using the following script

let employeeId = 1;
 client.getDetails({id: employeeId}, function(err, response) {
    console.log('Employee Details for Employee Id:',employeeId,'\n' ,response.message);
  });

As mentioned earlier the client stub helps us call the getDetails function in the Server like a normal function call. To this we pass the employeeId as the input.

Finally the Response comes in the response variable. We are then printing the response message.

The complete client.js code is given below

const PROTO_PATH = __dirname + '/proto/employee.proto';

const grpc = require('grpc');
const protoLoader = require('@grpc/proto-loader');

let packageDefinition = protoLoader.loadSync(
    PROTO_PATH,
    {keepCase: true,
     longs: String,
     enums: String,
     defaults: true,
     oneofs: true
    });
let employee_proto = grpc.loadPackageDefinition(packageDefinition).employee;

function main() {
  let client = new employee_proto.Employee('localhost:4500',
                                       grpc.credentials.createInsecure());
  let employeeId;
  if (process.argv.length >= 3) {
    employeeId = process.argv[2];
  } else {
    employeeId = 1;
  }
  client.getDetails({id: employeeId}, function(err, response) {
    console.log('Employee Details for Employee Id:',employeeId,'\n' ,response.message);
  });
}

main();

Running The Server and Client

Running the Server

Open a command prompt and run the Server using the following command

node server.js

This will start the server

Open a new Command prompt and run the Client using the following command

node client.js

When we run the client. It will print the following output

Employee Details for Employee Id: 1 
 { id: 1,
  email: 'abcd@abcd.com',
  firstName: 'First1',
  lastName: 'Last1' }

So here the client has called the getDetails function in the server. The Client has passed the input for employeeId as 1. Then the server went over the data, found the employee with id as 1 and returned that Employee data back to the client.

In This example we have run the client and server in a single machine. But you can also test this out by having the server and client in different hosts.

Code

The Code discussed in this article can be found here

References

GRPC official Documentation : https://grpc.io/

Protocol Buffers Proto3 documentation : https://developers.google.com/protocol-buffers/docs/proto3

Congrats 😄

You now know what is GRPC And Protocol Buffers. You also know how to use them in a NodeJS Application

Happy Coding 😄

Feel free to connect with me in LinkedIn or follow me in twitter.

If you liked this post, you can checkout my website https://adityasridhar.com for other similar posts

How to get started with Gradle

Aditya Sridhar — Tue, 24 Mar 2020 13:30:40 +0000

This post was originally published in adityasridhar.com

Gradle is used very often in the industry and I felt it will be good to cover the basics of Gradle in this article so that it can be used efficiently😄

What is Gradle

Gradle is a Build Automation tool and is also Open source. This Article will cover the following topics in Gradle.

Creating a Java Application by using Gradle as the build tool
Running Unit tests for the Java Application
Gradle Build Script and Tasks

Installing Gradle

You can Install Gradle from https://docs.gradle.org/current/userguide/installation.html

Also ensure Gradle is set in the PATH so that gradle commands work.

you can verify if gradle is installed and can be accessed using the command

gradle -v

Also ensure JAVA_HOME is set.

Creating a Java Application by Using Gradle as Build Tool

Create a folder called as 'simple-gradle-java-app' and enter into the folder in command prompt. This will be our project folder.

mkdir simple-gradle-java-app
cd simple-gradle-java-app

Use the following Command to Initialize Gradle for this project

gradle init

This will ask you to select the type of project to generate. Choose 'application'. This is because we are creating an application

Next you will be asked to select Implementation Language. Choose 'Java'. This is because we are creating a Java Application.

Next you will be asked to Choose Build Script DSL. Choose 'Groovy'. In this Demo we will be using Groovy as the build script language.

Next you will be asked to Select a Test Framework. Choose 'Junit Jupiter'.

For now you can leave the project name and source package as default.

Folder Structure

Once the project is created. It would have the following Folder Structure

Some of the notable items created in this folder structure are

src/main/java: This is the folder which has all the Java Code in the Application
src/test/java: This is folder which has all the test cases for the Above Java code
gradlew and gradlew.bat: This is the gradle wrapper which is used the run various gradle tasks and Commands. gradlew works in Linux and MacOS and gradlew.bat is used for Windows.
settings.gradle: This file indicates which projects to include in the build. In our case we have only one project which is 'simple-gradle-java-app'. This is set in settings.gradle as rootProject.name = 'simple-gradle-java-app'
build.gradle: This is the file in which the Build Script is written. We will be discussing more on this later in this Blog.
gradle: This is the folder which has all the Gradle Wrapper files

Modifying the Java Class and Its Test Class

In src/main/java The Init Script has automatically Created a Java Class called as App.java.

This is the Java class which is Automatically generated for us.

package simple.gradle.java.app;

public class App {
    public String getGreeting() {
        return "Hello world.";
    }

    public static void main(String[] args) {
        System.out.println(new App().getGreeting());
    }
}

Replace the Above Java class with the following Class

package simple.gradle.java.app;

public class App {
    public static int getSum(int num1, int num2) {
        return (num1 + num2);

    }

    public static void main(String[] args) {
            System.out.println("Sum is " + getSum(4,5));
    }
}

We have a function here called as getSum which returns sum of 2 numbers. The main function calls this function.

In src/test/java the Init Script has automatically created a Java Test class for us Called as AppTest.java

The class this is generated is given below

package simple.gradle.java.app;

import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;

class AppTest {
    @Test void appHasAGreeting() {
        App classUnderTest = new App();
        assertNotNull(classUnderTest.getGreeting(), "app should have a greeting");
    }
}

Replace the Above java class with the following

package simple.gradle.java.app;

import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;

class AppTest {
    @Test void getSum() {
       int result = App.getSum(10, 12);
       int expected = 22;
       assertEquals(expected,result);
    }

    @Test void main(){
        App.main(new String[]{});
    }
}

Here we have written a unit test to check if getSum function is working as expected.

Running The Java Application

First let us build the project using Gradle. This can be done using the following command

Windows

gradlew.bat build

Linux and MacOS

./gradlew build

Once the build is done, a folder called as build is created which will have the built project and also it would contain information like the test reports.

Then we can run the Application using the following Command

Windows

gradlew.bat run

Linux and MacOS

./gradlew run

This will Run the application and Print the output as Shown Below

> Task: run
Sum is 9

If needed we can clean the build using the below Command

Windows

gradlew.bat clean

Linux and MacOS

./gradlew clean

This will delete the build folder.

Running The Unit Tests

When we Built the project using the build command, it would automatically run the Test classes as well and Generate the Test reports

The Test reports can be found in build/reports/tests/test/index.html. Open the index.html file using a browser to see the test results.

Now if you want to run only the tests, then you can use the following command.

Windows

gradlew.bat test

Linux and MacOS

./gradlew test

This will also generate the index.html file. The Test report will look as shown below

Now Let us make an Error on Purpose In the code

Replace App.java with the following

package simple.gradle.java.app;

public class App {
    public static int getSum(int num1, int num2) {
        return num1;

    }

    public static void main(String[] args) {
            System.out.println("Sum is " + getSum(4,5));
    }
}

As you can see here, we are always returning num1 instead of num1 + num2 in getSum function which is wrong.

Let's Run the tests now

Windows

gradlew.bat test

Linux and MacOS

./gradlew test

You would notice that now it says that the Task: Test failed. Let's check the report in index.html now to see what it is showing

The Error Clearly shows that it Expected the sum to be 22, But then our App logic is returning 10. Hence the Test has Failed.

Now modify getSum Function in App.java to return ( num1 + num2 ) and re-run the test and it will pass

Gradle Build Script and Tasks

build.gradle has the build script. In this demo we are using Groovy for the build script.

The build.gradle which was generated has the following snippets. All of these are already Automatically generated for us. So here I will be explaining all the snippets generated

Plugins

plugins {
    // Apply the java plugin to add support for Java
    id 'java'

    // Apply the application plugin to add support for building a CLI application.
    id 'application'
}

Plugins help to add additional functionality to Groovy. Here we have a java plugin to Support Java Classes. We also have an Application Plugin so that We can build Applications using Groovy

Dependencies

dependencies {
    // This dependency is used by the application.
    implementation 'com.google.guava:guava:28.1-jre'

    // Use JUnit Jupiter API for testing.
    testImplementation 'org.junit.jupiter:junit-jupiter-api:5.5.2'

    // Use JUnit Jupiter Engine for testing.
    testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine:5.5.2'
}

Dependencies is where we add additional libraries need in the project. For Example here we are mentioning that we need junit jupiter dependency for Unit Testing. If you wanted to make a Database connection, then you will need to use a Library like JDBC for it. In that case JDBC will be added as a dependency.

Repositories

repositories {
    // Use jcenter for resolving dependencies.
    // You can declare any Maven/Ivy/file repository here.
    jcenter()
}

Repositories is the location from where Groovy will fetch and download the dependencies. Here we have mentioned that the repository we are using is Jcenter. You can read more about Jcenter here. Another Commonly used Repository is Maven Central. You can read more about Maven Central here

Application

application {
    // Define the main class for the application.
    mainClassName = 'simple.gradle.java.app.App'
}

Since we have created a Java application, we need to mention the Java class containing the main class in the build script. In this case it is App.Java.

Tasks

A Task is a piece of work that the build needs to do. In build.gradle currently we do not see any tasks defined. But a lot of default tasks already exist. You can see them by typing the following command

Windows

gradlew.bat tasks

Linux and MacOS

./gradlew tasks

This will give the following output

Application tasks
-----------------
run - Runs this project as a JVM application

Build tasks
-----------
assemble - Assembles the outputs of this project.
build - Assembles and tests this project.
clean - Deletes the build directory.
jar - Assembles a jar archive containing the main classes.
testClasses - Assembles test classes.

Documentation tasks
-------------------
javadoc - Generates Javadoc API documentation for the main source code.

Help tasks
----------
tasks - Displays the tasks runnable from root project 'simple-gradle-java-app'.

Verification tasks
------------------
test - Runs the unit tests.

We have already used some of these tasks like build, clean, run and test.

The Flexibility of groovy is that we can add Custom Tasks as well. Let's add a Custom Task and then Run the Task. Add the following code snippet in build.gradle

task projectupper {
    group = 'Custom Tasks'
    description = 'Task to get Project Name and Convert it to upper case'
    doLast {
        String projectName = project.name
        println "Project Name: $projectName"
        println "Upper case: ${projectName.toUpperCase()}"
    }
}

We have written a simple task here which takes the project name , Coverts the name to upper case and then prints the Name in Upper case.

projectupper is name of the Task

group and description are very important since it will list this task in the tasks report whenever we run gradlew.bat tasks

doLast is where we define the task action. We first get the project Name using project.name, then we convert it to upper case and print it.

In order to run the task use the following command

Windows

gradlew.bat projectupper

Linux and MacOS

./gradlew projectupper

This will generate the following output

> Task :projectupper
Project Name: simple-gradle-java-app
Upper case: SIMPLE-GRADLE-JAVA-APP

This task is a very simple task. We can create much more complex tasks in groovy as well and I will cover those in future articles.

Code

The entire code discussed here is available in this repo https://github.com/aditya-sridhar/simple-gradle-java-app

References

Gradle Offical Guide: https://gradle.org/

Gradle Tasks : https://docs.gradle.org/current/userguide/tutorial_using_tasks.html

Building Java Projects with Gradle: https://docs.gradle.org/current/userguide/building_java_projects.html

Congrats 😄

You know how to use Gradle now. This Article covered some of the basic Concepts of Gradle. Please go over the documentation of Gradle to know more about Gradle.

Happy Coding 😄

Feel free to connect with me in LinkedIn or follow me in twitter.

If you liked this post, you can checkout my website https://adityasridhar.com for other similar posts

How to use Java Executor framework for Multithreading

Aditya Sridhar — Tue, 04 Jun 2019 16:47:38 +0000

This post was originally published in adityasridhar.com

In my Previous Blog I covered the basics of Multithreading in Java. Click here to read that blog.

The previous Blog covered how to create Threads by Extending the Thread class and implementing the Runnable Interface.

This article will be covering 2 topics.

Creating Threads by implementing the Callable Interface
Using the Executor Framework in Java

Implementing the Callable Interface

In order to create a Piece of code which can be run in a Thread, we create a class and then implement the Callable Interface. The task being done by this piece of code needs to be put in the call() function. In the below code you can see that CallableTask is a class which implements Callable Interface, and the task of summing up numbers from 0 to 4 is being done in the function.

import java.util.concurrent.Callable;
class CallableTask implements Callable<Integer> {

    @Override
    public Integer call() throws Exception {

        int sum = 0;
        for (int i = 0; i < 5; i++) {
            sum += i;
        }
        return sum;
    }

}

In the above code you would notice that Callable has a parameter of Integer. This shows that the return type of this Callable will be Integer. Also it can be seen that the call function returns Integer type.

Creating the Threads and running them

The code below shows how to create the Threads and then run them.

import java.util.concurrent.ExecutionException;
import java.util.concurrent.FutureTask;
public class CallableInterfaceDemo {

    public static void main(String[] args) {
        FutureTask<Integer>[] futureList = new FutureTask[5];

        for (int i = 0; i <= 4; i++) {
            Callable<Integer> callable = new CallableTask();
            futureList[i] = new FutureTask<Integer>(callable);
            Thread t = new Thread(futureList[i]);
            t.start();

        }

        for (int i = 0; i <= 4; i++) {
            FutureTask<Integer> result = futureList[i];
            try {
                System.out.println("Future Task" + i + ":" + result.get());
            } catch (InterruptedException e) {

                e.printStackTrace();
            } catch (ExecutionException e) {

                e.printStackTrace();
            }
        }

    }

}

In order to create a Thread, first we need to create an Instance of CallableTask which implements the Callable Interface as shown in

Callable<Integer> callable = new CallableTask();

Then we need to create an Instance of the FutureTask class and pass the instance of Callable task as an argument as shown in

futureList[i] = new FutureTask<Integer>(callable);

Then to create a Thread we create an instance of the Thread class and pass the Instance of the FutureTask class as an argument as shown in

Thread t = new Thread(futureList[i]);

Finally the Thread is started with the start() function.

Getting the result from the Thread

In case of Callables the Thread can actually return a value. In order to get this value we can call the get() function on the instance of the FutureTask. In our code, the return value of the thread is the sum of numbers from 0 to 4.

This is shown in the below code snippet

FutureTask<Integer> result = futureList[i];
try {
    System.out.println("Future Task" + i + ":" + result.get());
} catch (InterruptedException e) {

    e.printStackTrace();
} catch (ExecutionException e) {

    e.printStackTrace();
}

Also the thread may throw an Exception as well which can be handled with try catch blocks.

Complete Code

Here is the complete code discussed till now

import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.FutureTask;

class CallableTask implements Callable<Integer> {

    @Override
    public Integer call() throws Exception {

        int sum = 0;
        for (int i = 0; i < 5; i++) {
            sum += i;
        }
        return sum;
    }

}

public class CallableInterfaceDemo {

    public static void main(String[] args) {
        FutureTask<Integer>[] futureList = new FutureTask[5];

        for (int i = 0; i <= 4; i++) {
            Callable<Integer> callable = new CallableTask();
            futureList[i] = new FutureTask<Integer>(callable);
            Thread t = new Thread(futureList[i]);
            t.start();

        }

        for (int i = 0; i <= 4; i++) {
            FutureTask<Integer> result = futureList[i];
            try {
                System.out.println("Future Task" + i + ":" + result.get());
            } catch (InterruptedException e) {

                e.printStackTrace();
            } catch (ExecutionException e) {

                e.printStackTrace();
            }
        }

    }

}

The Executor Framework

Creating a Thread on the Fly everytime is Resource Intensive. One good alternative for this is to have some Threads already setup and then allocate our tasks to these threads. This is where the Executors Class and ExecutorService are very useful.

The above image shows a Thread pool with 4 threads. Whenever we want any task to be run, we can assign it to these threads. Once the task is complete, the Thread will be freed to take up other tasks.

How to use the Executor Framework

Here is a code which uses the Executor framework.

import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;

class Worker implements Callable<Integer> {

    @Override
    public Integer call() throws Exception {

        int sum = 0;
        for (int i = 0; i < 5; i++) {
            sum += i;
        }
        return sum;
    }

}

public class ExecutorDemo {

    public static void main(String[] args) {
        ExecutorService executors = Executors.newFixedThreadPool(4);
        Future<Integer>[] futures = new Future[5];
        Callable<Integer> w = new Worker();
        try {
            for (int i = 0; i < 5; i++) {
                Future<Integer> future = executors.submit(w);
                futures[i] = future;

            }

            for (int i = 0; i < futures.length; i++) {
                try {
                    System.out.println("Result from Future " + i + ":" + futures[i].get());
                } catch (InterruptedException e) {

                    e.printStackTrace();
                } catch (ExecutionException e) {

                    e.printStackTrace();
                }
            }
        } finally {
            executors.shutdown();
        }

    }

}

First we create a Worker Class which implements Callable and does the task which we need.

Next we need to create an ExecutorService.

The Executors class has multiple implementations of the ExecutorService.

Let us use the Executors class to create a fixed Thread pool of size 4. This is done as follows

ExecutorService executors = Executors.newFixedThreadPool(4);

Next we need to submit our task to the Executor Service. This is done using the following Line of code

Future<Integer> future = executors.submit(w);

On submitting the task we get an Instance of the Future Object. The Future Object is what will store the result of the Task.

Getting the result of the Thread

In order to get the result of each task, we can call the get() method of the Future Instance.
This is shown in the below code snippet.

try {
    System.out.println("Result from Future " + i + ":" + futures[i].get());
} catch (InterruptedException e) {

    e.printStackTrace();
} catch (ExecutionException e) {

    e.printStackTrace();
}

The thread can also Throw an Exception which can be handled using try catch.

Some possible Scenarios of the Fixed Thread Pool

In this example we created a fixed Thread pool of size 4
If we submit a total of 3 tasks to this ExecutorService, then all 3 tasks will be assigned to the thread pool and they will start executing.
If we submit 4 tasks to this ExecutorService, then again all 4 tasks will be assigned to the thread pool and they will start executing.
Now if we submit 5 tasks to this thread pool. Only 4 of the tasks will be assigned to the thread pool. This is because the size of the Thread pool is 4. The 5th Task will be assigned only when one of the threads in the pool gets freed

Shutting down the ExecutorService

The ExecutorService needs to be shutdown when the threads are not needed anymore. This will ensure that the JVM is not consuming Additional Resources.

The ExecutorService can be shutdown using the following command

executors.shutdown();

It can be seen that this shutdown is put within the finally block. This is to ensure that the shutdown is always executed at the end of the code even if any exception occured.

If the shutdown is not done in the right way, then in case any exception occurs then the ExecutorService will still be running and will be consuming Additional JVM resources.

Code

All the code discussed in this article can be found in this git repo

Congrats😃

Now you know the following concepts

Using the Callable Interface
Using the Java Executor Framework for Multithreading.

In my future Articles I will be covering more topics on Multithreading

Feel free to connect with me in LinkedIn or follow me in twitter.

If you liked this post, you can checkout my website https://adityasridhar.com for other similar posts

How to get started with Multithreading in Java

Aditya Sridhar — Thu, 18 Apr 2019 16:07:38 +0000

This post was originally published in adityasridhar.com

What is a Thread

A Thread is a lightweight Process. Any Process can have multiple Threads running in it.

For example in a web browser, we can have one thread Which will load the User Interface and we can have one more thread which will actually retrive all the data that needs to be displayed in the User Interface.

What is MultiThreading

Multithreading enables us to run Multiple Threads Concurrently.

For example in a web browser, we can have on thread which handles the User Interface and in Parallel we can have one more thread which is fetching the data to be displayed.

So multithreading improves the responsiveness of a System.

What is Concurrency

Concurrency in the Context of Threads enables us to run multiple Threads at the same time.

But do the Threads Really run at the same time?

Single Core Systems

The Thread Scheduler provided by the JVM decides which Thread runs at any given time. The scheduler gives a small time slice to each Thread.

So at any given time we have only thread which is actually running in the processor. But because of the time slicing we get the feeling that multiple threads are running at the same time.

Multi Core Systems

Even in Multiple Core systems the thread scheduler is involved. But since we have multiple cores, we can actually have multiple threads running at the exact same time.

For example if we have a dual core system, then we can have 2 threads running at the exact same time. The first thread will run in the first core, and the second thread will run in the second core.

Why is Multithreading needed?

Multithreading enables us to improve the responsiveness of a System.

For example in a web browser, if everything ran in a single Thread, then system will be completely unresponsive whenever data is being fetched to display. If it takes 10 seconds to fetch the data, then in that 10 seconds we wont be able to do anything else in the web browser like opening new tabs, or even closing the web browser.

So running different parts of a program in different threads concurrently helps improve the responsiveness of a system.

How to write Multithreaded Programs in Java

We can create Threads in java using the following

Extending the Thread Class
Implementing the Runnable Interface
Implementing the Callable Interface
By using the Executor Framework along with Runnable and Callable Tasks

We will look at Callables and the Executor Framework in a separate Blog. In this article I will be mainly focussing on Extending the Thread class and Implementing the Runnable Interface.

Extending the Thread Class

In order to create a Piece of code which can be run in a Thread, we create a class and then extend the Thread class. The task being done by this piece of code needs to be put in the run() function. In the below code you can see that Worker is a class which extends Thread Class, and the task of printing numbers 0 to 5 is being done inside the run() function.

class Worker extends Thread {

    @Override
    public void run() {
        for (int i = 0; i <= 5; i++) {
            System.out.println(Thread.currentThread().getName() + ": " + i);
        }
    }

}

In the Above code Thread.currentThread().getName() is used to get the name of the Current Thread which is running the code.

In order to create a Thread, we just need to create an instance of the Worker class. And then we can start the Thread using the start() function.

public class ThreadClassDemo {
    public static void main(String[] args) {
        Thread t1 = new Worker();
        Thread t2 = new Worker();
        Thread t3 = new Worker();
        t1.start();
        t2.start();
        t3.start();

    }
}

In the Above code, we are creating 3 threads t1,t2 and t3 from the Worker class.
Then we are starting the threads using the start() function.

Here is the Final Code for Creating a thread by Extending a Thread class

class Worker extends Thread {

    @Override
    public void run() {
        for (int i = 0; i <= 5; i++) {
            System.out.println(Thread.currentThread().getName() + ": " + i);
        }
    }

}

public class ThreadClassDemo {
    public static void main(String[] args) {
        Thread t1 = new Worker();
        Thread t2 = new Worker();
        Thread t3 = new Worker();
        t1.start();
        t2.start();
        t3.start();

    }
}

Here is the output we get by Running the above code

It can be seen that all the 3 threads have printed the numbers from 0 to 5.

It can be seen clearly from the output that the 3 threads do not run in any particular sequence

Implementing the Runnable Interface

In order to create a Piece of code which can be run in a Thread, we create a class and then implement the Runnable Interface. The task being done by this piece of code needs to be put in the run() function. In the below code you can see that RunnableWorker is a class which implements Runnable Interface, and the task of printing numbers 0 to 4 is being done inside the run() function.

class RunnableWorker implements Runnable{

    @Override
    public void run() {
        for (int i = 0; i <= 4; i++) {
            System.out.println(Thread.currentThread().getName() + ": " + i);
        }
    }

}

In order to create a Thread, first we need to create an Instance of RunnableWorker which implements the Runnable Interface.

Then we can create a new thread by creating an instance of Thread class and passing the Instance of RunnableWorker as the Argument. This is shown in the code below

public class RunnableInterfaceDemo {

    public static void main(String[] args) {
        Runnable r = new RunnableWorker();
        Thread t1 = new Thread(r);
        Thread t2 = new Thread(r);
        Thread t3 = new Thread(r);

        t1.start();
        t2.start();
        t3.start();

    }

}

The above code creates a Runnable Instance r. Then it create 3 threads t1, t2 and t3 and passes r as the argument to the 3 threads. Then the start() function is used to start all the 3 threads.

Here is the complete Code for creating a Thread by Implementing the Runnable Interface

class RunnableWorker implements Runnable{

    @Override
    public void run() {
        for (int i = 0; i <= 4; i++) {
            System.out.println(Thread.currentThread().getName() + ": " + i);
        }
    }

}

public class RunnableInterfaceDemo {

    public static void main(String[] args) {
        Runnable r = new RunnableWorker();
        Thread t1 = new Thread(r);
        Thread t2 = new Thread(r);
        Thread t3 = new Thread(r);

        t1.start();
        t2.start();
        t3.start();

    }

}

On Running the Above code, we will get the following output. The sequence of the output will change everytime the code is run.

Implementing the Runnable Interface is a better option than extending the Thread class since we can extend only one class, but we can implement multiple interfaces in java.

Runnable Interface in Java 8

In java 8, the Runnable Interface becomes a FunctionalInterface since it has only one function run().

The below code shows how we can create a Runnable Instance in java 8.

public class RunnableFunctionalInterfaceDemo {

    public static void main(String[] args) {

        Runnable r = () -> {
            for (int i = 0; i <= 4; i++) {
                System.out.println(Thread.currentThread().getName() + ": " + i);
            }
        };

        Thread t1 = new Thread(r);
        Thread t2 = new Thread(r);
        Thread t3 = new Thread(r);

        t1.start();
        t2.start();
        t3.start();
    }

}

Here instead of creating a class and then implementing Runnable Interface, we can directly use lambda expression to create a Runnable Instance as shown below

Runnable r = () -> {
        for (int i = 0; i <= 4; i++) {
            System.out.println(Thread.currentThread().getName() + ": " + i);
        }
    };

Code

The code in this article is available in the following github repo
https://github.com/aditya-sridhar/basic-threads-demo

Congrats 😃

You now know how to create threads by extending the Thread class and by implementing the Runnable interface.

I will discuss about the Thread life cycle and challenges while using threads in my next blog post.

Feel free to connect with me in LinkedIn or follow me in twitter.

If you liked this post, you can checkout my website https://adityasridhar.com for other similar posts

What is a Mutation in GraphQL and how to use it

Aditya Sridhar — Mon, 28 Jan 2019 14:11:53 +0000

This post was originally published in adityasridhar.com

This Blog post is a continuation to my previous blog post on GraphQL Basics. Click Here to checkout the GraphQL Basics post.

It is necessary to read the GraphQL Basics post to make the best use of this article.

What is a Mutation in GraphQL?

Whenever you want to write data back into the server, mutations are used.

How is Mutation and Query Different?

Query is used when you want to read some data from the server while mutation is used when you want to write data back to the server.

But wait. Can't I go to the resolver in the query and do a write operation?

Though it is possible to do a write operation in a query, it shouldn't be done. It is necessary to separate the read the write operations and hence mutations are needed.

Code

Click Here to get the code from my previous blog post. We will be adding the mutation logic to this code in this article.

Add Movie Mutation

Let us create a mutation which can be used to add a new movie.

create a new file called as mutation.js. Copy the following code into mutation.js

const { GraphQLObjectType
} = require('graphql');
const _ = require('lodash');

const {movieType} = require('./types.js');
const {inputMovieType} = require('./inputtypes.js');
let {movies} = require('./data.js');

const mutationType = new GraphQLObjectType({
    name: 'Mutation',
    fields: {
        addMovie: {
            type: movieType,
            args: {
                input: { type: inputMovieType }
            },
            resolve: function (source, args) {

                let movie = {
                    id: args.input.id, 
                    name: args.input.name, 
                    year: args.input.year, 
                    directorId: args.input.directorId};

                movies.push(movie);

                return _.find(movies, { id: args.input.id });
            }
        }
    }
});

exports.mutationType = mutationType;

You will notice that a mutation looks very similar to a query. The main difference is that the name of the GraphQLObjectType is Mutation.

Here we have added a mutation called as addMovie which has a return type of movieType ( movieType was covered in the previous blog )

In args we are mentioning that we need a parameter called input which is of type inputMovieType

So what is inputMovieType here?

Input Types

It is possible that multiple mutations need the same input arguments. So it is a good practise to create Input Types and reuse the Input Types for all these mutations.

Here we are creating a input type for the movie called as inputMovieType.

It is seen that inputMovieType in turn comes from inputtypes.js file. Let us create this now.

Create a new file called as inputtypes.js

Copy the following code into inputtypes.js

const {
    GraphQLInputObjectType,
    GraphQLID,
    GraphQLString,
    GraphQLInt
} = require('graphql');

inputMovieType = new GraphQLInputObjectType({
    name: 'MovieInput',
    fields: {
        id: { type: GraphQLID },
        name: { type: GraphQLString },
        year: { type: GraphQLInt },
        directorId: { type: GraphQLID }

    }
});

exports.inputMovieType = inputMovieType;

It is seen that an Input Type looks exactly like any other Type in GraphQL. GraphQLInputObjectType is used to create an Input Type, while GraphQLObjectType is used to create normal Types.

Resolve function of a mutation

The resolve function of a mutation is where the actual write operation happens.

In a real application this can be a Database write operation.

For this example, we are just adding the data to movies array and then returning the added movie back.

 resolve: function (source, args) {

                let movie = {
                    id: args.input.id, 
                    name: args.input.name, 
                    year: args.input.year, 
                    directorId: args.input.directorId};

                movies.push(movie);

                return _.find(movies, { id: args.input.id });
            }

The above code in resolve does the following actions

Gets the input movie parameters from input arg.
Adds the new movie to the movies array
Returns the new movie which was added, by fetching it from the movies array

Add Director Mutation

Let us create a mutation which can be used to add a new director

This would be exactly same as adding the movie Mutation.

inputtypes.js with director Mutation added

const {
    GraphQLInputObjectType,
    GraphQLID,
    GraphQLString,
    GraphQLInt
} = require('graphql');

inputMovieType = new GraphQLInputObjectType({
    name: 'MovieInput',
    fields: {
        id: { type: GraphQLID },
        name: { type: GraphQLString },
        year: { type: GraphQLInt },
        directorId: { type: GraphQLID }

    }
});

inputDirectorType = new GraphQLInputObjectType({
    name: 'DirectorInput',
    fields: {
        id: { type: GraphQLID },
        name: { type: GraphQLString },
        age: { type: GraphQLInt }

    }
});

exports.inputMovieType = inputMovieType;
exports.inputDirectorType = inputDirectorType;

mutation.js after adding addDirector mutation

const { GraphQLObjectType
} = require('graphql');
const _ = require('lodash');

const {movieType,directorType} = require('./types.js');
const {inputMovieType,inputDirectorType} = require('./inputtypes.js');
let {movies,directors} = require('./data.js');

const mutationType = new GraphQLObjectType({
    name: 'Mutation',
    fields: {
        addMovie: {
            type: movieType,
            args: {
                input: { type: inputMovieType }
            },
            resolve: function (source, args) {

                let movie = {
                    id: args.input.id, 
                    name: args.input.name, 
                    year: args.input.year, 
                    directorId: args.input.directorId};

                movies.push(movie);

                return _.find(movies, { id: args.input.id });
            }
        },
        addDirector: {
            type: directorType,
            args: {
                input: { type: inputDirectorType }
            },
            resolve: function (source, args) {
                let director = {
                    id: args.input.id, 
                    name: args.input.name, 
                    age: args.input.age};

                directors.push(director);

                return _.find(directors, { id: args.input.id });
            }
        }
    }
});

exports.mutationType = mutationType;

Enabling the mutations

Until now we have defined the mutation end points and their functionality. But we haven't enabled the mutations yet.

To enable them, the mutations have to added to the schema.

The mutation is added using the following code in server.js

 // Define the Schema
const schema = new GraphQLSchema(
    { 
        query: queryType,
        mutation: mutationType 
    }
);

Complete Code in server.js after adding the mutation

//get all the libraries needed
const express = require('express');
const graphqlHTTP = require('express-graphql');
const {GraphQLSchema} = require('graphql');

const {queryType} = require('./query.js');
const {mutationType} = require('./mutation.js');

//setting up the port number and express app
const port = 5000;
const app = express();

 // Define the Schema
const schema = new GraphQLSchema(
    { 
        query: queryType,
        mutation: mutationType 
    }
);

//Setup the nodejs GraphQL server 
app.use('/graphql', graphqlHTTP({
    schema: schema,
    graphiql: true,
}));

app.listen(port);
console.log(`GraphQL Server Running at localhost:${port}`);

Code

The complete code for this article can be found in this git repo

Testing The mutation End Points

Run the application using the following command

node server.js

Open your web browser and go to the following url localhost:5000/graphql

Test addMovie Mutation Endpoint

Input

mutation {
    addMovie(input: {id: 4,name: "Movie 4", year: 2020,directorId:4}){
    id,
    name,
    year,
    directorId
  }

}

Output

{
  "data": {
    "addMovie": {
      "id": "4",
      "name": "Movie 4",
      "year": 2020,
      "directorId": "4"
    }
  }
}

Input

mutation {
    addMovie(input: {id: 5,name: "Movie 5", year: 2021,directorId:4}){
    id,
    name,
    year,
    directorId
  }

}

Output

{
  "data": {
    "addMovie": {
      "id": "5",
      "name": "Movie 5",
      "year": 2021,
      "directorId": "4"
    }
  }
}

Test addDirector Mutation Endpoint

Input

mutation {
    addDirector(input: {id: 4,name: "Director 4", age: 30}){
    id,
    name,
    age,
    movies{
      id,
      name,
      year
    }
  }

}

Output

{
  "data": {
    "addDirector": {
      "id": "4",
      "name": "Director 4",
      "age": 30,
      "movies": [
        {
          "id": "4",
          "name": "Movie 4",
          "year": 2020
        },
        {
          "id": "5",
          "name": "Movie 5",
          "year": 2021
        }
      ]
    }
  }
}

Congrats 😃

You now know about Mutations in GraphQL

You can checkout the documentation to know more about GraphQL

Feel free to connect with me in LinkedIn or follow me in twitter.

If you liked this post, you can checkout my website https://adityasridhar.com for other similar posts

What is GraphQL and how to use it

Aditya Sridhar — Sun, 23 Dec 2018 12:27:39 +0000

This post was originally published in adityasridhar.com

What is GraphQL?

It is basically a query language for API's

GraphQL shows what are the different types of data provided by the server and then the client can pick exactly what it wants.

Also in GraphQL you can get multiple server resources in One call rather than making multiple REST API calls.

You can check https://graphql.org/ for the full list of benefits.

The thing is until you see GraphQL in action, it's hard to understand the benefits. So let's get started with using GraphQL.

We will be using GraphQL along with NodeJS in this Article.

Pre-requisites

Install NodeJS from https://nodejs.org/en/

How to use GraphQL with NodeJs

GraphQL can be used with multiple languages. Here we will focus on how we can use GraphQL with javascript using NodeJS.

Create a Folder called as graphql-with-nodejs. Go into the project folder and run npm init to create the NodeJS project. The command for this is given below.

cd graphql-with-nodejs
npm init

Install the Dependencies

Install Express using the following command

npm install express

Install GraphQL using the following command. We will be installing graphql and graphql for express.

npm install express-graphql graphql

NodeJS Code

Create a file called as server.js inside the project and copy the following code into it

const express = require('express');
const port = 5000;
const app = express();

app.get('/hello', (req,res) => {
    res.send("hello");
   }
);

app.listen(port);
console.log(`Server Running at localhost:${port}`);

The above code has a single http get end point called as /hello.

The end point is created using express.

Now let us modify this code to enable GraphQL.

Enabling GraphQL in the code

GraphQL will have a single url endpoint called as /graphql which will handle all requests.

Copy the following code into server.js

//get all the libraries needed
const express = require('express');
const graphqlHTTP = require('express-graphql');
const {GraphQLSchema} = require('graphql');

const {queryType} = require('./query.js');

//setting up the port number and express app
const port = 5000;
const app = express();

 // Define the Schema
const schema = new GraphQLSchema({ query: queryType });

//Setup the nodejs GraphQL server
app.use('/graphql', graphqlHTTP({
    schema: schema,
    graphiql: true,
}));

app.listen(port);
console.log(`GraphQL Server Running at localhost:${port}`);

Let us go through this code now

graphqlHTTP enables us to setup a GraphQL server at /graphql url. It basically knows how to handle the request that is coming in.

This setup is done in the following lines of code

app.use('/graphql', graphqlHTTP({
    schema: schema,
    graphiql: true,
}));

Now let us explore the parameters inside graphqlHTTP

graphiql

graphiql is a Web UI using which you can test the graphql endpoints. We will set this to true so that it is easier to test the various graphql endpoints we create.

schema

Though graphql has only one external endpoint /graphql, this in turn can have multiple other endpoints doing various things. These endpoints would be specified in the schema.

The schema would do things like:

Specify the endpoints
Indicate the input and output fields for the endpoint
Indicate what action should be done when an endpoint is hit and so on.

The Schema is defined as follows in the code

const schema = new GraphQLSchema({ query: queryType });

The schema can contain query as well as mutation types. This article will focus only on the query type.

query

It is seen the in the schema that query has been set to queryType.

We import queryType from query.js file using the following command

const {queryType} = require('./query.js');

query.js is a custom file which we will be creating soon.

query is where we specify the read only endpoints in a schema.

Create a file called as query.js in the project and copy the following code into it.

const { GraphQLObjectType,
    GraphQLString
} = require('graphql');


//Define the Query
const queryType = new GraphQLObjectType({
    name: 'Query',
    fields: {
        hello: {
            type: GraphQLString,

            resolve: function () {
                return "Hello World";
            }
        }
    }
});

exports.queryType = queryType;

query Explained

queryType is created as a GraphQLObjectType and given the name Query.

fields is where we specify the various endpoints.

So here we are adding one endpoint called as hello

hello has a type of GraphQLString Which means this endpoint has a return type of String. The type is GraphQLString instead of String since this a graphql schema. So directly using String will not work.

resolve function indicates the action to be done when the endpoint is called. Here the action is to return a String "Hello World".

Finally we export the querytype using exports.queryType = queryType. This is to ensure we can import it in server.js

Running the Application

Run the application using the following command

node server.js

The application runs on localhost:5000/graphql.

You can test the application by going to localhost:5000/graphql.

This url runs the Graphiql web UI as shown in the screen below.

The input is given in the left and the output is shown in the right.

Give the following input

{
  hello
}

This will give the following output

{
  "data": {
    "hello": "Hello World"
  }
}

Congrats 😃

You have created your first GraphQL endpoint.

Adding more endpoints

We will create 2 new endpoints:

movie: This endpoint will return a movie, given the movie ID
director: This endpoint will return a director given the director ID. It will also return all the movies directed by this director.

Adding Data

Usually an application will read data from a Database. But for this tutorial we will be hardcoding the data in the code itself for simplicity.

Create a file called data.js and add the following code.

//Hardcode some data for movies and directors
let movies = [{
    id: 1,
    name: "Movie 1",
    year: 2018,
    directorId: 1
},
{
    id: 2,
    name: "Movie 2",
    year: 2017,
    directorId: 1
},
{
    id: 3,
    name: "Movie 3",
    year: 2016,
    directorId: 3
}
];

let directors = [{
    id: 1,
    name: "Director 1",
    age: 20
},
{
    id: 2,
    name: "Director 2",
    age: 30
},
{
    id: 3,
    name: "Director 3",
    age: 40
}
];

exports.movies = movies;
exports.directors = directors;

This file has the movies and directors data. We will be using the data in this file for our endpoints.

Adding the movie endpoint to the query

The new endpoints will be added to queryType in query.js file

The code for the movie endpoint is shown below

movie: {
            type: movieType,
            args: {
                id: { type: GraphQLInt }
            },
            resolve: function (source, args) {
                return _.find(movies, { id: args.id });
            }
        }

The return type of this endpoint is movieType which we will be defining soon.

args parameter is used to indicate the input to the movie endpoint. The input to this endpoint is id which is of type
GraphQLInt

resolve function returns the movie corresponding to the id, from the movies list. find is a function from lodash library used to find an element in a list.

The complete code for query.js is shown below

const { GraphQLObjectType,
    GraphQLString,
    GraphQLInt
} = require('graphql');
const _ = require('lodash');

const {movieType} = require('./types.js');
let {movies} = require('./data.js');


//Define the Query
const queryType = new GraphQLObjectType({
    name: 'Query',
    fields: {
        hello: {
            type: GraphQLString,

            resolve: function () {
                return "Hello World";
            }
        },

        movie: {
            type: movieType,
            args: {
                id: { type: GraphQLInt }
            },
            resolve: function (source, args) {
                return _.find(movies, { id: args.id });
            }
        }
    }
});

exports.queryType = queryType;

From the above code it can be seen that movieType is actually defined in types.js

Adding the Custom Type movieType

Create a file called as types.js.

Add the following code into types.js

const {
    GraphQLObjectType,
    GraphQLID,
    GraphQLString,
    GraphQLInt
} = require('graphql');

// Define Movie Type
movieType = new GraphQLObjectType({
    name: 'Movie',
    fields: {
        id: { type: GraphQLID },
        name: { type: GraphQLString },
        year: { type: GraphQLInt },
        directorId: { type: GraphQLID }

    }
});

exports.movieType = movieType;

It can be seen that movieType is created as a GraphQLObjectType

It has 4 fields id, name, year and directorId. The types for each of these fields are specified as well while adding them.

These fields come directly from the data. In this case it will be from movies list.

Adding the query and type for director endpoint

Similar to movie, even the director endpoint can be added.

In query.js, the director endpoint can be added as follows

director: {
            type: directorType,
            args: {
                id: { type: GraphQLInt }
            },
            resolve: function (source, args) {
                return _.find(directors, { id: args.id });
            }
        }

directorType can be added as follows in types.js

//Define Director Type
directorType = new GraphQLObjectType({
    name: 'Director',
    fields: {
        id: { type: GraphQLID },
        name: { type: GraphQLString },
        age: { type: GraphQLInt },
        movies: {
            type: new GraphQLList(movieType),
            resolve(source, args) {
                return _.filter(movies, { directorId: source.id });
            }

        }

    }
});

Wait a minute. The directorType is slightly differnt from movieType. Why is this?

why is there a resolve function inside directorType. Previously we saw that resolve functions were present only in the query

The Speciality of directorType

When the director endpoint is called we have to return the director details, as well as all the movies the director has directed.

The first 3 fields id,name,age in directorType are straightforward and come directly from the data ( directors list)

The fourth field movies needs to contain the list of movies by this director.

For this we are mentioning that the type of movies field is a
GraphQLList of movieType ( List of movies )

But how exactly will be find all the movies directed by this director?

For this we have a resolve function inside the movies field.
The inputs to this resolve function is source and args.

source will have the parent object details.

Lets say the fields id =1, name = "Random" and age = 20 for a director. Then source.id =1, source.name = "Random" and source.age = 20

So in this example, resolve function finds out all the movies where directorId matches the Id of the required Director.

Code

The Entire code for this application is available in this github repo

Testing the Application

Now Let us test the application for different scenarios.

Run the application using node server.js

Go to localhost:5000/graphql and try the following inputs.

movie

Input:

{
  movie(id: 1) {
    name
  }
}

Output:

{
  "data": {
    "movie": {
      "name": "Movie 1"
    }
  }
}

From the above it can be seen that, the client can request exactly what it wants and GraphQL will ensure only those parameters are sent back. Here only name field is requested and only that is sent back by the server.

In movie(id: 1), id is the input parameter. We are asking the server to send back the movie which has an id of 1.

Input:

{
  movie(id: 3) {
    name
    id
    year
  }
}

Output:

{
  "data": {
    "movie": {
      "name": "Movie 3",
      "id": "3",
      "year": 2016
    }
  }
}

In the above example name, id and year fields are requested. So the server sends back all of those fields.

director

Input:

{
  director(id: 1) {
    name
    id,
    age
  }
}

Output:

{
  "data": {
    "director": {
      "name": "Director 1",
      "id": "1",
      "age": 20
    }
  }
}

Input:

{
  director(id: 1) {
    name
    id,
    age,
    movies{
      name,
      year
    }
  }
}

Output:

{
  "data": {
    "director": {
      "name": "Director 1",
      "id": "1",
      "age": 20,
      "movies": [
        {
          "name": "Movie 1",
          "year": 2018
        },
        {
          "name": "Movie 2",
          "year": 2017
        }
      ]
    }
  }
}

In the above example we see the power of GraphQL. We indicate we want a director with id 1. Also we indicate we want all the movies by this director. Both the director and movie fields are fully customizable and the client can request exactly what it wants.

Similarily this can be extended to other fields and types. For example we could run a query like Find a director with id 1. For this director find all the movies. For each of the movie find the actors. For each actor get the top 5 rated movies and so on. For this query we need to specify the relationship between the types. Once we do that, the client can query any relationship it wants.

Congrats 😃

You now know the basic concepts of GraphQL.

You can checkout the documentation to know more about GraphQL

Feel free to connect with me in LinkedIn or follow me in twitter.

If you liked this post, you can checkout my website https://adityasridhar.com for other similar posts

How to easily build Desktop Apps with HTML, CSS and Javascript

Aditya Sridhar — Thu, 29 Nov 2018 12:49:35 +0000

This post was originally published in adityasridhar.com

Can HTML, CSS and Javascript really be used to build Desktop Applications?

The Answer is Yes 😄

In this Article we will be focussing mainly on how Electron can be used to create desktop applications with Web Technologies like HTML, CSS and Javascript

Electron

Electron can be used to build Desktop Apps with HTML, CSS and Javascript. Also these apps work for multiple platforms like Windows, Mac, Linux and so on.

Electron Combines Chromium and NodeJS into a single Runtime. This enables Us to run the HTML, CSS and Javascript Code as a desktop application.

Electron Forge

If Electron is used directly, then some manual setup is needed before building your application. Also if you want to use angular, react, vue or any other framework or library, you will need to manually configure for that.

Electron Forge makes the above things much easier.

It provides template applications with angular, react, vue and other frameworks. This avoids the extra manual setups needed.

Also it provides an easy way to build and package the application. It also provides many other features which can be found in their documenation.

Pre-requisites

Ensure you have NodeJS installed. It can be installed from here

Install Electron Forge Globally using the following command

npm install -g electron-forge

Let's get started with the application

Use the following command to create your application

electron-forge init simple-desktop-app-electronjs

simple-desktop-app-electronjs is the name of the application

The above command will take some time to run.

Once it finishes running, Start the application using the following commands

cd simple-desktop-app-electronjs
npm start

This should open up a window like the one shown below

Understanding the Existing Folder Structure and Code

The application created has a folder structure. Here I will be mentioning some of the important things in this folder structure

package.json

It has information about the application you are creating, it has all the dependencies needed for the app and also it has some scripts. Some of the scripts are already pre configured and you can add new scripts as well.

The config.forge path has all the configurations which are specific to ElectronJS. For example make-targets is used specify the target make files for various platforms like Windows, Mac or Linux.

Also package.json has "main": "src/index.js" which indicates that src/index.js is the starting point of the application

src/index.js

According to package.json, index.js is the main script. The process which runs the main script is known as the Main Process. So Main Process runs the index.js script.

The main process is used to display GUI elements. It does this by creating Web Pages.

Each Web page created runs in a process called as renderer process

Main Process and Renderer process

The purpose of the Main Process is to create web pages using a BrowserWindow Instance.

The BrowserWindow Instance uses a renderer process to run each Web page.

Each App can have only one Main Process but can have many renderer processes

It is possible to communicate between the main and the renderer process as well. This will not be covered in this article.

Electron Architecture showing main and renderer process. The file names can vary

abcd.html is shown as a second webpage in the above architecture. But in our code we won't be having a second web page.

src/index.html

index.js loads the index.html file into a new BrowerWindow Instance.

What this basically means is that, index.js creates a new GUI Window, and loads it with index.html web page. The index.html web page runs in its own renderer process.

Code in index.js explained

Most of the code created in index.js has good comments explaining what it does. Here I will mention a few key points to note in index.js

mainWindow = new BrowserWindow({
    width: 800,
    height: 600,
  });

  // and load the index.html of the app.
  mainWindow.loadURL(`file://${__dirname}/index.html`);

The above code snippet basically creates a BrowserWindow Instance and loads index.html into the BrowserWindow.

You would see app used often in the code. For example take the below code snippet.

app.on('ready', createWindow);

app is used to control the applications's event life cycle.

The above code snippet says that when the application is ready, load the first window.

Similarily app can be used to perform other actions on various events. For example it can be used to perform some action right before the application closes and so on.

Let's create a Temperature Converter Desktop Application

Writing the Code

Let us use the same application we used before and modify it slightly to create a temperature converter application.

First Let us install Bootstrap. Install bootstrap using the following command

npm install bootstrap --save

Copy the following code into src/index.html

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <title>Temperature Converter</title>
    <link rel="stylesheet" type="text/css" href="../node_modules/bootstrap/dist/css/bootstrap.min.css">

  </head>
  <body>
    <h1>Temperature Converter</h1>
    <div class="form-group col-md-3">
      <label for="usr">Celcius:</label>
      <input type="text" class="form-control" id="celcius" onkeyup="celciusToFahrenheit()">
    </div>
    <div class="form-group col-md-3">
      <label for="pwd">Fahrenheit:</label>
      <input type="text" class="form-control" id="fahrenheit" onkeyup="fahrenheitToCelcius()">
    </div>
    <script src='./renderer.js'></script>
  </body>
  </body>
</html>

The above code does the following

Creates a text box with id celcius. whenever anything is typed in this textbox, celciusToFahrenheit() function is called.
Creates a text box with id fahrenheit. whenever anything is typed in this textbox, fahrenheitToCelcius() function is called.
Whenever a new value is typed in celcius text box, the value in the fahrenheit text box displays the same temperature in fahrenheit
Whenever a new value is typed in fahrenheit text box, the value in the celcius text box displays the same temperature in celcius

The 2 functions which do the temperature conversion are present in renderer.js

Create a file called renderer.js inside src. Copy the following code into it.

function celciusToFahrenheit(){
    let celcius = document.getElementById('celcius').value;
    let fahrenheit = (celcius* 9/5) + 32;
    document.getElementById('fahrenheit').value = fahrenheit;

}

function fahrenheitToCelcius(){
    let fahrenheit = document.getElementById('fahrenheit').value;
    let celcius = (fahrenheit - 32) * 5/9
    document.getElementById('celcius').value = celcius;
}

celciusToFahrenheit() function reads the value in celcius text box, converts it to fahrenheit and writes the new temperature into fahrenheit text box.

fahrenheitToCelcius() function does the exact opposite of this.

Running the application

Run the application using the following command

npm start

This should display the following window. Try it out with different values.

Packaging the application

The command to package the application is

npm run package

This command will take some time to run. Once it finishes check the out folder within the project folder.

I tested this in a windows machine. This creates a folder called simple-desktop-app-electronjs-win32-x64 inside out folder

So in the out/simple-desktop-app-electronjs-win32-x64 folder, the command creates an .exe file for this application. Clicking on the exe file automatically starts the desktop application.

The folder name simple-desktop-app-electronjs-win32-x64 can be broken down as appname-platform-architecture where

appname = simple-desktop-app-electronjs
platform = win32
architecture = x64

When you run this command without any parameters, by default it packages for the platform which you are using for development

Let's say you want to package for a different platform and architecture. Then you can use the following syntax

npm run package -- --platform=<platform> arch=<architecture>

For example, in order to package for linux you can use the following command

npm run package -- --platform=linux --arch=x64

This will create a folder called as simple-desktop-app-electronjs-linux-x64 inside out folder.

Creating a make File

In order to create a make file or an installer for the application, use the following command

npm run make

This command will take some time to run. Once it finishes check the out folder within the project folder.

The out/make folder will have a windows installer for the desktop application.

When you run this command without any parameters, by default it creates the installer for the platform which you are using for development

Code

The code for this desktop application is available in my github repo

https://github.com/aditya-sridhar/simple-desktop-app-electronjs

Congrats 😄

You now know how to create desktop applications using HTML, CSS and Javascript.

This article covered very basic concepts of electron and electron-forge.

To know more about them, you can checkout their documentation.