Forem: vikash-agrawal

Elastic Search

vikash-agrawal — Tue, 03 Aug 2021 12:02:15 +0000

What is Elastic Search

Elastic Search is one of the most powerful search engine, which provides following functionality:

Full Text Search
Analyze metric and performance data (APM: Application Performance Management)
Receiving events from any of the application

How does Elastic Search work

Elastic Search works with the document in the form of JSON, containing the list of fields and nested fields along with the corresponding values.
Elastic Search enables the functionality over the REST API, with request response format.
Elastic Search is written in java built on top of Apache Lucene.

Elastic Search Stack

Elastic Search
Kibana:
- An analytic and Visualization Platform to monitor the performance of application, Elastic Search Cluster,
- Also provides platform for machine learning.
Log Stash
- A data processing pipeline.
- The data received can be customer data, application logs from any kind of source like file, kafka, database etc.
- The data processed and shipped to any destination like kafka, elastic search.
- The Log stash goes through following:
- Input
- Filter
- Output
- Each of these steps is enabled through the plugins.
Beat: Light wight data shipper installed in the application server.
- File Beat: Send the logs.
- Metric Beat: Collects and sends the application/service level metrics
X-Pack: Adds additional functionalities on top of Elastic Search and Kibana, like:
- Authentication by integrating it with LDAP.
- Authorization
- Define alerts, trigger emails etc on performance monitoring.
- Machine learning e.g. Abnormality Detection, forecast future value etc.
- Elastic Search SQL, good to send SQL to ES over the http and JDBC.

Installation

Mac

Download the Elastic Search latest from the elasticsearch-download
Download the Kibana through one of the following steps:
- kibana-download
- Brew

brew tap elastic/tap
brew install elastic/tap/kibana-full

if you happen to see any error, follow the instructions as mentioned in the error message.
e.g. if the error is due to xcode then

xcode-select --install

Post Installation

GET /_cat/nodes?v
GET /_cat/nodes?v=true&h=id,ip,port,v,m&pretty

# return just process
GET /_nodes/process?pretty
# same as above
GET /_nodes/_all/process?pretty

Sharding and Scalability

Elastic search data is stored in the index.
Elastic Search Data in the given index is stored in the shards.
1 TB of data can be stored in a cluster of more than one nodes with 500 GB hard disk capacity.
Each shard is kind of independent index and actually an Apache Lucene Index.
The size of the shard would increase based on the index size.
It can store up to 2 Billion documents.
It also helps in performance improvement if each of the queries are being executed in different shard and hence more than 1 shard for the given indes can reside in the same node.
An index is created with 1 shard by default.
Elastic Search less than 7.0.0 used to create index with 5 shards by default.
During any of the queries, the shard is decided through: value of _routing % number of shard
- Once the index is created, it's advisable not to change the number of shards as the existing documents wouldn't be fetched if query using the _routing parameters.
- To change the number of shards, use Elastic Search split API to increase the shards and Elastic Search Shrink API to decrease the shards, it internally performs the reindex operations.
By default the routing is defined on the document id, which can be unpredictable on case the document id is randomly generated.
Based on the use cases and to have the predictable behaviour, it's better to define the routing on any of the existing field.
But the custom routing comes with more responsibility, e.g.
- if the value of the field on which the routing is defined, is changed then the reindexing is required to performed on the concerned document.
- GET search on document id, would need the routing also to passed otherwise the GET API might not return the results with the default routing.
Following factors should be considered to define number of shards:
- Number of Nodes.
- Size of the index.
- Size of Hard disk in the node.
- Number of index
- Number of queries.

Replication

Replication is created at the index level
Replica works by coping the shards called replication shards.
The original shard is called primary shard
A primary shards and its replica shard is called replication group.
An index is created with 1 replica by default.
The replica shard will always be in the different nodes.
In a cluster of one node, the status of any user index will be yellow.
As soon as the new node is added the status would turn to green.
There are following ways to add a new node to the cluster in local:
- Go inside the Elastic Search directory and execute:
```
 bin/elasticsearch -Nnode.name=node-2 -Epath.data=./data/node-2 -Epath.logs=./logs/node-2
```
- Extract the elastic zip file to another directory, change the value of node in conf/elasticsearch.yml file and start the process.
Number of replica for kibana index is changed based on the number of nodes.
Number of replica should depend upon many factors:
- It should be 2 for critical APIs else 1.
- Is the data stored in other DBs also?
Query pointing to the same shard can be served by the replicas as well. Hence replication increase the availability and throughput of the query.
Replicas is a recovery system in the given real time while snapshot is a recovery system in the given time

GET /_cluster/health
GET /_cat/indices?v
PUT /product-index
GET /_cluster/health

Node Roles

Master:
- node.master = true|false
- Responsible for creating and deleting indices, allotong shards/replicas to the node.
- This role is assigned to the node on rotation basis.
- In a big cluster, it make sense to declare one of the node as true.
Data Node
- node.data = true|false
- This is enabled for all nodes by default.
- If you have the dedicated master node, then don't make it as data node.
Ingest
- node.ingest = true|false
- Series of steps to perform some steps while ingesting the data, mainly used for log stash.
Machine Learning
- node.ml: true|false
- This lets the machine learning related jobs.
- xpack.ml.enabled = true|false
- This enables the Machine learning APIs
Coordinations
- Enabled by disabling all other node roles.
- This lets the coordinations related job like shards selection and aggregations.

Index

Create the Index

PUT /product-index

It supports many settings to define the number of shards, replicas etc.

Add the document to index

POST /product-index/_doc
{
  "product":"Dove Soap",
  "reviewsCount": 100,
  "price": 120
}
POST /product-index/_doc/100
{
  "product":"Dove Cream",
  "reviewsCount": 110,
  "price": 150
}
PUT /product-index/_doc/110
{
  "product":"Dove Cream",
  "reviewsCount": 120,
  "price": 150
}

Analyze the value of shards.total, it should be equal to 1 + no of replicas, 1 stands for primary shard as the data would go to only 1 primary shard and would get replicated in all replica shards in the __replication group_
Document can be added using POST and PUT both with only difference is PUT defines that the document to be added on the mentioned URI hence id is mandatory field for PUT.

Retrieve the documents by id

  GET /product-index/_doc/100

Update the document
- All the documents in the ES are immutable means the documents can not be changed.
- The update query retrieves the document and updates the fields and reindex the document with the same id. So it replaces the existing document not updates the document.

  POST /product-index/_update/100
{
  "doc": {"reviewsCount": 99, "inStock":120}

}

Scripted updates
- "ctx.op = 'noop'" will not cause any updates hence there is no change in the primary term and sequence number.
- Otherwise the response would say updates with the change in the primary term and sequence number regardless of the condition is met or not.

  POST /product-index/_update/100
{
  "script" : {
    "source": "ctx._source.price++"
  }
}

POST /product-index/_update/100
{
  "script" : {
    "source": "ctx._source.price = 500"
  }
}

POST /product-index/_update/100
{
  "script" : {
    "source": """
    if(ctx._source.price > 100){
      ctx.op = 'noop';
    }
    ctx._source.price = 150
    """
  }
}

POST /product-index/_update/100
{
  "script" : {
    "source": """
    if(ctx._source.price < 100){
      ctx._source.price = 150;
    }
    """
  }
}

POST /product-index/_update/100
{
  "script" : {
    "source": """
    if(ctx._source.price > 100){
      ctx.op = 'delete';
    }
    ctx._source.price = 150
    """
  }
}

Upserts
- If the document doesn't exist, it would index the new document.
- If the document exists, it would execute the script (make the updates)

#upserts
GET /product-index/_doc/101
POST /product-index/_update/101
{
  "script": {"source": "ctx._source.reviewsCount++"},
  "upsert": {
    "product":"Dove Soap",
    "reviewsCount": 125,
    "price": 123
  }
}
GET /product-index/_doc/101
POST /product-index/_update/101
{
  "script": {"source": "ctx._source.reviewsCount++"},
  "upsert": {
    "product":"Dove Soap",
    "reviewsCount": 125,
    "price": 123
  }
}
GET /product-index/_doc/101

Replace the Document

# Add the document to the index
PUT /product-index/_doc/101
{
  "product":"Dove Soap",
  "reviewsCount": 100,
  "price": 120
}

Delete the document

DELETE /product-index/_doc/101

Journey to Cloud Migration

vikash-agrawal — Fri, 25 Sep 2020 13:43:18 +0000

Background

Lucky to be part of cloud migration team in my organization, with the best thing is my involvement when it matters most for me.

I take this platform to share my learnings which I have learnt while talking to the people and exploring the solutions for the technical problem, which is eventually leading me to gain leadership and technical skills.

Some of the learnings and takeaway:

Split the tasks into 2 categories:

Tasks, which are required to be completed at any cost.
There is no end to improvement, so tasks, which qualify for continual improvement.

Phase wise migration

Migration would occur in following 2 categories:

Application
Data
Infrastructure Services (Database, Messaging Platform)
DevOps Pipelines
Most importantly, Alerts

Migration would generate to lot of unpredictable behaviour, so it might led to give a thought to the following factors:

Define phases in the context of traffic volume, it can a client or set of clients generating low traffic.
Give thought to the delta data migration
- Does the current infrastructure (database) supports delta data back up
- Proposed infrastructure support delta data restoration.`

Nothing but the set up from the scratch

So it's worth investing in having the local set up where you can debug the application from your IDE in your development environment.
It's a golden opportunity to upgrade all your existing services because of following reasons:
- Need to conduct all the possible testings irrespective of service upgrade.`
- Uncertainty on how the new target system would behave with the services.`

2P Dependencies

Make sure you highlight the dependencies on the dependent team first otherwise you might end up in a situation where you are ready to Go Live but you need to complete last moment thing like traffic diversion. We had the similar scenario where we had to communicate the team to add header element in their request URL so that the traffic can be diverted to new infrastructure.

Mistake, which you might make

Make sure to confirm in your source code repository and deployment pipeline if all services have been migrated. We missed some of the services because of following reasons:
- One service was missed as we searched with some patterns and this particular service was not full filled by this pattern.
- Root cause: One of the Git repo contains another un related micro service.
- Learnings: Have the common pattern and hence get this un related micro service to another new git repo.
- Root cause: Git repos were scanned based on patterns and one of the git repo was not following this pattern.
- Learnings:
  - Scan should happen to all possible places, but not limited to just following:
  - Git
  - Container Repository
  - Log Aggregator
- One deployment was missed as this deployment was part of already migrated module.
Don't ignore any single error message, which you see as part of testing, it can be functional testing or load testing. Because of the ignorance, we had to increase the meta space memory for the JVM from 128m to 160m post deployment.

What went well

After migration, there was traffic for delivery but not for ingestion. Upon highlight, the concerned team had to rectify their side.
Load Test, which gives the proof for the microservice scalability and API performance, was on top priority.

Challenges

While using Azure VM, we realized that adding data disk from the portal is not sufficient. It's required to be changed on the kernel level.
Upgrading the dependent infra (Database, messaging platform) client version directly is not sufficient as each of the springboot maps to specific infra version through dependencies module.

Learnings from the leaders

There is no end to the work so don't hesitate to give some relax time to the mates, if team member is tired, we can't achieve any turnaround result.
Irrespective of your busy schedule, explain all the interesting facts and root cause of any issues to the team, which you have gone through.

Database

Couchbase

couchbase database migration

Couchbase Database Migration

vikash-agrawal — Sun, 20 Sep 2020 11:42:53 +0000

Good Practices about the Couchbase

Create admin level users with all permission.
Create admin level users with just read permission, it might need if you want to collect some statistics about the couchbase.
Create separate user for buckets or each of the bucket with limited privileges.
Upgrading memory on the VM level is not sufficient, you need to change for each of the services in the setting present in Couchbase Console UI. Keep some memory in buffer in case if you want to run some agents like stats collector.
As the query services uses the temporary directory to scan the result, so change it to point to path in data disk not in OS disk.
Granting bucket level permission is not sufficient, you should grant all the permission for the bucket.
Keep each of the services like data service, index service, query service, search service, analytic service in separate nodes. It would provide the fault tolerant incase any of the node or service crashes.
Not sure on other database, but it doesn't support back up with query filter.
Couchbase 5.X onwards, it supports the replica index where it's not required to define the same index in each of the nodes to provide high availability rather it can be achieved with replica index.

We performed upgrade of Couchbase 5.X to 6.X

Strategy for upgrade:

Collect the following possible data from the previous installation:
- CPU Core Size for the entire cluster and for each of the services
- RAM for each of the nodes and services.
- Data Volume.
- Keep the specification for the VMs and Data Disks separate for each of the services as index node data disk capacity might be less than the data disk hard disk and query node doesn't use data disk significantly.
Licensing cost for Couchbase Enterprise edition is done on node level, so make sure to use the advanced node so that more than 1 service can be accommodated if required.

Challenges with Database migrations

No filter supports for cbbackup, we had to use the cbq script to take the query based on the timestamp and set of data.
As the above step is not the back up actually, so we had to write the spring boot application to perform UPSERT.
- USERT is suggested if no document in source database is updated post migration.
- Specific field update is suggested if there is updates on both the DB.
The above steps worked will but with some difference in meta data but can be ignored as no difference was observed when and DB operations were performed from using SDK or Couchbase Query Editor:
- Flag: 33554432(COUCHBASE_CFFMT_JSON) vs ‘0’(COUCHBASE_VAL_IS_STRING)
- rev, cas: value differs.

Advantages with Couchbase 6.0:

Better UI to manage the settings and view data.
Backup and restore with various options: There are 2 options from CB 6.0 CE to back up and restore:

Cbbackup and cbrestore

Backup:
- /opt/couchbase/bin/cbbackup -m full http://localhost:8091 /tmp/cb-bkp -u USER -p (The first full backup)
- /opt/couchbase/bin/cbbackup -m diff http://localhost:8091 /tmp/cb-bkp -u USER -p PASSWORD (The next incremental backup)
Restore
- /opt/couchbase/bin/cbrestore /tmp/cb-bkp/2020-07-21T125200Z/2020-07-21T125200Z-full http://localhost:8091 --u USER -p PASSWORD -x rehash=1
- /opt/couchbase/bin/cbrestore /tmp/cb-bkp/2020-07-21T125200Z/2020-07-21T125822Z-diff http://localhost:8091 -u USER -p PASSWORD -x rehash=1
Cons:
- While restoring, we need to restore full and then all incremental in the order.
- Difficult to manage all incremental dirs over the longer time, only solution is have the scheduled full back and remove all previous incremental backup dirs.

Cbbackupmgr (Preferred options)

Register archive (one time process): /opt/couchbase/bin/cbbackupmgr config -a /tmp/cb-bkp-mgr -r cluster
Back up /opt/couchbase/bin/cbbackupmgr backup -a /tmp/cb-bkp-mgr -r cluster -c couchbase://localhost -u USER -p PASSWORD First backup will be full and then next onwards would be incremental.
Merge all backup present under /tmp/cb-bkp-mgr/cluster/ on the day of back up /opt/couchbase/bin/cbbackupmgr merge -a /tmp/cb-bkp-mgr -r cluster --start 2020-07-21T13_43_19.471413454Z --end 2020-07-21T13_54_33.394966624Z
Restore /opt/couchbase/bin/cbbackupmgr restore -a /tmp/cb-bkp-mgr -r cluster -c couchbase://localhost -u USER -p PASSWORD
Pros:
- With the merge, we can merge all backups on the day of merge so that we will have only 1 dir.
- cbbackupmgr comes with lot of options, one of them is list to see all the backup directories.
- /opt/couchbase/bin/cbbackupmgr list -a /tmp/cb-bkp-mgr

Learnings

Give a thought to define bucket based on any category e.g. clients or set of clients.
Give thought to generate the document id based on the clients or set of clients.
Backup definition interval is business driven.

Multi Threading In Java

vikash-agrawal — Thu, 26 Sep 2019 15:04:30 +0000

Thread Lifecycle in java

Once a thread completes its execution, it reaches to Dead state that means the given thread cannot be started again, means you can’t invoke start method more than once.
The join() method waits for a thread to die. In other words, it causes the currently running threads to stop executing until the thread it joins with completes its task.

class TestJoinMethod1 extends Thread{  
 public void run(){   
  for(int i=1;i<=5;i++){  
   try{  
    Thread.sleep(500);  
   }catch(Exception e)
   {
    System.out.println(e);
   }  
  System.out.println(i);  
  }  
 }  

 public static void main(String args[]){  
  TestJoinMethod1 t1=new TestJoinMethod1();  
  TestJoinMethod1 t2=new TestJoinMethod1();  
  TestJoinMethod1 t3=new TestJoinMethod1();  
  t1.start();  
  try{   
   t1.join();  
  }catch(Exception e)
  {
   System.out.println(e);
  }   

 t2.start();  
 t3.start();  
}  
}

Output:1 2 3 4 5 1 1 2 2 3 3 4 4 5 5

Main thread will wait for all the user threads to complete then will exit.
If there are no user threads are running then daemon thread will also die.

ThreadPool

  public class TestThreadPool {  
     public static void main(String[] args) {  
        ExecutorService executor = Executors.newFixedThreadPool(5);//creating a pool of 5 threads  
        for (int i = 0; i < 10; i++) {  
            Runnable worker = new WorkerThread("" + i);  
            executor.execute(worker);//calling execute method of ExecutorService  
        }  
        executor.shutdown();  
        while (!executor.isTerminated()) {   }  

        System.out.println("Finished all threads");  
    }  
 }

Output

pool-1-thread-1 (Start) message = 0
pool-1-thread-2 (Start) message = 1
pool-1-thread-3 (Start) message = 2
pool-1-thread-5 (Start) message = 4
pool-1-thread-4 (Start) message = 3
pool-1-thread-2 (End)
pool-1-thread-2 (Start) message = 5
pool-1-thread-1 (End)
pool-1-thread-1 (Start) message = 6
pool-1-thread-3 (End)
pool-1-thread-3 (Start) message = 7
pool-1-thread-4 (End)
pool-1-thread-4 (Start) message = 8
pool-1-thread-5 (End)
pool-1-thread-5 (Start) message = 9
pool-1-thread-2 (End)
pool-1-thread-1 (End)
pool-1-thread-4 (End)
pool-1-thread-3 (End)
pool-1-thread-5 (End)
Finished all threads

Wait-notify

package com.learnings.synch;

public class Tutorial {
   private static Account account = new Account();

   public static void main(String[] args) {
       Thread[] depositThreads = new Thread[5];
       Thread[] withdrawThreads = new Thread[8];
       for (int i = 0; i < 5; i++) {
           depositThreads[i] = new Thread(new UserThread(account, "deposit"));
       }
       for (int i = 0; i < 8; i++) {
           withdrawThreads[i] = new Thread(new UserThread(account, "withdraw"));
       }
       for (Thread thread : withdrawThreads) {
           thread.start();
       }
       try {
           Thread.sleep(5000);
       } catch (InterruptedException e) {
           e.printStackTrace();
       }
       for (Thread thread : depositThreads) {
           thread.start();
       }

   }
}

/**
*
*/
package com.learnings.synch;

/**
* @author Vikash Agrawal 
*
*/
public class Account {
   private int balance = 50;
   private Object lock = new Object();

   private void withdraw(int amount) {
       balance = balance - amount;
   }

   private void deposit(int amount) {
       balance = balance + amount;
   }

   public int getBalance() {
       return balance;
   }

   public void deposit() {
       synchronized (lock) {
           System.out.println("Will be deposited by: " + Thread.currentThread().getName());
           deposit(10);
           lock.notify();
           System.out.println("After deposit: " + getBalance());
       }
   }

   public void withdraw() {
       synchronized (lock) {
           System.out.println("Will be withdrawn by: " + Thread.currentThread().getName());
           if(getBalance() < 10) {
               try {
                   lock.wait();
               } catch (InterruptedException e) {
                   e.printStackTrace();
               }
           }
           withdraw(10);
           System.out.println("After withdraw: " + getBalance());
       }
   }
}

/**
*
*/
package com.learnings.synch;

/**
* @author Vikash Agrawal 
*
*/
public class UserThread implements Runnable {
   private String operation;
   private Account account;

   public UserThread(Account account, String operation) {
       this.account = account;
       this.operation = operation;
   }

   public void run() {
       switch (operation) {
       case "deposit":
           account.deposit();
           break;
       case "withdraw":
           account.withdraw();
           break;
       }
   }
}

Java Updates

vikash-agrawal — Thu, 26 Sep 2019 14:50:01 +0000

Java 8 updates

Functional Interface

• An interface with just 1 abstract method is called Functional Interface.
• A Functional Interface only can be used for lambda as there is no way of combining more than 1 method to the single instance.

Lambda

• It’s a way to override a method in a simpler way.
• It can be single-statement without any curly braces.
• It can be multiline enclosed in the curly braces.
• Single statement outcome will be “return” value (No need of explicit return value)
• Content of lambda can be extracted to a private method.
• A lambda method can be with parameter or without parameters (in case of single parameter).

Default Method

• Java 8 onwards allows to have method definition with keyword default in the interface.
• 2 interfaces with same default method mandate the method to be overridden in the class implementing these 2 interfaces. (I’s the same reason that java doesn’t allow 2 classes to be extended.)

Static method

• Similar to static method in the given class, interface also allows to have the static method.
• The static method in interface is the property of the interface. Means this method can be accessed using the interface only not by another interface extending this interface or class implementing this interface.

Foreach method

• Iterable interface, a super interface of collection framework/interface, have the default method implementation of foreach method.
• To iterate through the element in the collection framework, the traditional way of doing it is getting the iterator and iterate through each of the elements and then perform the business logic inside the for loop.
• Foreach method will have the iteration logic within it, that means the client just needs to take care of business logic.
• Other difference between this approach is, foreach method will use the internal iterator as compared to external iterator in traditional logic. And this external iterator has to be taken care with the ConcurrentModificationException.

IPV4 vs IPV6

vikash-agrawal — Thu, 26 Sep 2019 14:44:56 +0000

IPV4:

• It’s a decimal based IP address.
• It contains 4 decimal number separated by dot.
• Each number ranges from 0 to 255, that means 00000000 to 11111111.
• Each of the decimal is 8 bits long, so it’s 32 bits IP address.

IPV6:

• It's a hexadecimal based IP address.
• It contains 8 hexadecimal number separated by colon.
• Each number ranges from 0 to that means 0000 0000 0000 0000 to FFFF FFFF FFFF FFFF
• Each of the hexa decimal is 16 bits long, so it’s 128 bits.

Distributed Memory Caching

vikash-agrawal — Thu, 26 Sep 2019 14:43:16 +0000

There are 2 kinds of in memory data storage system available:

Redis

Redis is a single threaded system.
It can be scaled horizontally easily.
Size of the key and value can go upto 512 MB each.
Following 5 types of data types available:
- String
- Hash
- List
- Set
- Sorted Set
Any kind of CRUD operations can be performed on a field level of the given data.
- which means the application doesn't need to load the entire data.
There are multiple eviction policy available:
- No Eviction: On reaching the memory limit, it throws the error.
- All Keys LRU: Remove keys by least recently used.
- Volatile LRU: Remove keys, that have an expiration time set, by the least recently used first.
- All Keys Random: Remove keys randomly.
- Volatile Random: Remove keys, that have an expiration time set, randomly
- TTL: Remove keys, that have expired wrt time.
There are 2 kind of back up supported:
- RDB Snapshot: This snapshot happens periodically.
- AOF: Append only file is perform for every write operation. It can slow down the write operation but ensures the data availability.

Memcached

Memcached is a multi threaded system.
It can be scaled vertically easily.
Size of the key can be upto 250B and value can be upto 1 MB.
Horizontal scaling is possible through client side complex algorithm.
It's just on String Data Type.
Not possible to perform any operations on a field level of the given data.
- which means the application needs to load the entire data.
It works on LRU eviction policy only. • It's a distributed memory cached object storage. • It has 4 components:

o Client software: fetch the list of available servers.
o Server software: A logic to store and retrieve the value for the passed key.
o Expiry: Mechanism to remove the older contents based on the time expiry.
o Client based hashing algorithm: to fetch the corresponding server.

• The cached data is stored in the RAM so when RAM runs out of memory, it does the clean up.
• There is no fault tolerance means it doesn't share the data with other servers.

Design Principles

vikash-agrawal — Thu, 26 Sep 2019 14:37:59 +0000

SOLID

• Single Responsibility Principle: A class should be defined to have only one responsibility to avoid compiling and testing the totally dependent changes in case the class is mixed of many responsibilities. Logger and java Persistence API (JPA) are the ideal example for this.
• Open/Closed Principle: Open for extension and closed for modification. It mandates to have the interface defined so that any new implementations can be provided independently and plugged without modifying the existing.

public class VehicleCalculations {
    public double calculateValue(Vehicle v) {
        if (v instanceof Car) {
            return v.getValue() * 0.8;
        if (v instanceof Bike) {
            return v.getValue() * 0.5;

    }
}

if there is a requirement to calculate the cost for another Vehicle type, say 'Cycle' then above method would required to be modified.
But if the above class is modified in the following way, then any new vehicle type can be added independetly

public class Vehicle {
    public double calculateValue() {...}
}
public class Car extends Vehicle {
    public double calculateValue() {
        return this.getValue() * 0.8;
}
public class Cycle extends Vehicle{
    public double calculateValue() {
        return this.getValue() * 0.2;
}

• Liskov Substitution Principle: With the interface-based implementations, you can override the method with parameter of the method being of sub class. Means the type of super class can be replaced with the object of child class.
• Interface Segregation Principle: it avoids implementing the unrequired method from the interface. e.g.

public interface Vehicle {
    public void drive();
    public void stop();
    public void refuel();
    public void openDoors();
}
public class Bike implements Vehicle {

    // Can be implemented
    public void drive() {...}
    public void stop() {...}
    public void refuel() {...}

    // Can not be implemented
    public void openDoors() {...}
}

Instead it mandates to have another interface inheriting the super interface and have this sub interface declared with specific method.

public interface Vehicle {
    public void drive();
    public void stop();
    public void refuel();
}
public interface FourWheeler extends Vehicle {
    public void openDoors();
}
public class Bike implements Vehicle {
    public void drive() {...}
    public void stop() {...}
    public void refuel() {...}
}

• Dependency Inversion: Dependency should be on abstractions (interfaces and abstract classes) instead of concrete implementations (classes).
Consider the example below. We have a Car class that depends on the concrete Engine class

public class Car {
    private Engine engine;
    public Car(Engine e) {
        engine = e;
    }
    public void start() {
        engine.start();
    }
}
public class Engine {
   public void start() {...}
}

The code will work, for now, but what if we wanted to add another engine type, let’s say a diesel engine? This will require refactoring the Car class.
However, we can solve this by introducing a layer of abstraction. Instead of Car depending directly on Engine, let’s add an interface:

public interface EngineInterface {
    public void start();
}
public class Car {
    private EngineInterface engine;
    public Car(EngineInterface e) {
        engine = e;
    }
    public void start() {
        engine.start();
    }
}
public class PetrolEngine implements EngineInterface {
   public void start() {...}
}
public class DieselEngine implements EngineInterface {
   public void start() {...}
}

DRY

Don't repeat yourself to avoid the duplication of the code so the maintainability becomes easy. It also helps to find the bugs.

REST

• Uniform Interface: A resource in the system should have only one logical URI and that should provide a way to fetch related or additional data.
• Client Server: This essentially means that client application and server application MUST be able to evolve separately without any dependency on each other.
• Stateless: If client application needs to be a stateful application for the end user, where user logs in once and do other authorized operations thereafter, then each request from the client should contain all the information necessary to service the request – including authentication and authorization details.
• Cacheable: Well-managed caching partially or completely eliminates some client-server interactions, further improving scalability and performance.
• Layer: REST allows you to use a layered system architecture where you deploy the APIs on server A, and store data on server B and authenticate requests in Server C
• Code on demand (optional): You are free to return executable code to support a part of your application e.g. clients may call your API to get a UI widget rendering code.

Microservice

Domain Driven Design: Domain driven design along with related technical logic makes sure to have single responsibility microservice.
Hide implementation detail: with so many microservices being part of the complete application hide the implementations of one microservice from other.
Decentralized: With so many microservices around, it has the flexibility to have a different DB or different schema of the same DB with each of the microservice. With this DB migration becomes easier.
Failure isolation: with one microservice being unavailable due to any reason being scalability or performance impact that particular functionality not the entire application.
Continuous Deliver: With so many microservices being part of single application mandates to have the CI/CD pipeline so it requires to have collaboration between the developers and dev ops.

Best Practices

• Encapsulate what changes: have all the methods and fields declared as private and then based on the need increase the accessibility, don't jump directly to public.
• Favor Composition Over Inheritance: Composition allows changing the behavior of a class at runtime by setting property.
• Programming for Interface Not Implementation: use interface type in variable, return type or argument of method.
• Delegation Principle: Don't do all the stuff yourself. Like if you want any objects to be compared using equals or hashcode, the client doesn't do it, it's the class which does it.

Kafka

vikash-agrawal — Wed, 25 Sep 2019 15:20:52 +0000

• There are 4 components of Kafka:

o Producer
o Consumer
o Broker: A broker is nothing but the server and a node, which manages the topic.
o Zookeeper: Zookeeper maintains the state of brokers, in case of any broker going down or coming up, it notifies the zookeeper and zookeeper informs to the producer and consumer so that producer and consumer are connected to live broker.

• Topic is a classification for the message type let's say data type.
• The given topic is splitter into many partitions.
• And these partitions are replicated across more than 1 broker, if the number of brokers is more than 1.
• For the given partitions, there will be only 1 lead node and rest will be the slave node.
• Messages from the producer can go into any partitions and hence gets replicated.
• If the messages are assigned with the key, then it ensures the message with the same key in the particular partitions.
• In this way, it makes sure that messages are sequenced in the partitions.
• Kafka client (producer/consumer) to Kafka data transfer happens through https which makes sure that data is encrypted using SSL certificate.
• Use case:

o Messaging: A messaging platform built in publisher-subscriber model.
o Website tracking: to get the real time report of user activity in the website.
o Log aggregation: To aggregate the logs from many services.
o Fault tolerance: Kafka platform enables the back up of the events along with regular synch up.
o Stream Processing: events coming from multiple services can be processed in real 
time and then can be transferred to other platform like storm for further processing.

• Eamples:

o Twitter uses it to take the post from the registered users to another registered 
user to repost and read and to unregistered user for read.
o LinkedIn uses it for online consumption along with offline analytics while being 
sorted in Hadoop.
o Netflix uses it for real time monitoring and event processing.

API Gateway

vikash-agrawal — Wed, 25 Sep 2019 15:11:48 +0000

Proxy

Proxy is a dedicated computer or software system running in the same computer or different computer acts as intermidiary between the requester of a website and the actual web server.
Type of proxy:
• Forward Proxy: it helps in connecting the user to the website.
• Reverse Proxy: it is used:

o To allow load balancing between the server.
o To stream internal content to its user. (SSO page)
o To block some particular website.
Nginx

Load balancers:

• With the many servers configured as upstreams, you can define proxy_pass to this upstreams.
• There are multiple types of load balancers:

o Round Robin: this is by default.
o least_conn: the one with the least client connection will be picked up.
o ip_hash: each of the client has some unique key, the first few characters of this key works as server chooser. It helps in speeding up the process.

• To provide high availability load balancers, you can plug in external load balancer also.
• Caching

o Various cache related information like path, key, validity can be configured to improve the performance.
o User data is suggested not to be cached to avoid sharing one user data with other user.

API Gateway

It is used to manage the API, also referred as translator, which includes
• combining more than 1 APIs into 1
• combining responses from more than 1 API into 1
• Providing the default data/message in case of error from any API invocation
• Security management

REST vs SOAP

vikash-agrawal — Wed, 25 Sep 2019 15:06:06 +0000

• REST (REpresentational State Transfer) is an architectural style while SOAP (Simple Object
Access Protocol) is a protocol.
• REST is all about simplicity, where it comes with the lighter data (JSON), although it
supports other format as well, e.g. simple TEXT., HTML, XML etc, while SOAP is built within
the envelop, where it contains data in XML.
• REST comes with SSL security and https while SOAP comes with WS-Security as well.
• SOAP has built in ACID compliance that’s why it makes it favorable choice when it comes
to sensitive data e.g. financial data.
• AS REST is lighter so it needs less bandwidth as compared to SOAP.
• Data can be cached only in REST not in SOAP.
• SOAP has strict communication contract.
• REST uses URL and URI to access the API while SOAP uses service interface (WSDL file) to
expose its functionality to client application.
• If the service is required to maintain the state from one request to another then SOAP is
preferred.
• The WSDL file is a contract between the client and server. The content in this file has all
the information about the services being exposed, which includes field name, field type. If
there is any change to any parameter, it would have the huge impact on all the client side
implementation.
• Using wsimport, you can generate all the classes from wsdl file. Wsdl file is a xml file which
contains the schema. It can be used to create data object including the Authentication
detail and sent it over the network. The signature for the request is similar between rest
based service and soap based service with only difference:

o SOAP Action
o Soapenv
o soapenv: Body

• Status Code:

o 200: Request got executed successfully.
o 201: resource has been created successfully.
o 202: Accept, means the request has been accepted for execution asynchronously.
o 203: Non Authoritative information, means the content has been delivered from 
third party
o 400: Bad request because of syntax error.
o 401: Authentication is needed.
o 403: Authenticated but don't have enough access for the given request.
o 404: Resource not found

Spring Boot

vikash-agrawal — Wed, 25 Sep 2019 15:00:54 +0000

What benefits Spring Framework provides:

• Dependency injection: it’s very critical to provide a loosely coupled application
• Duplication/Plumbing Code

o   Spring JDBC
o   Spring MVC
o   Spring AOP
o   Spring ORM
o   Spring JMS
o   Spring Test

Pillars of a spring boot project:

• Auto Configuration: When we use Spring MVC, we need to configure component scan, dispatcher servlet, a view resolver, web jars (for delivering static content) among other things.
Spring Boot looks at

o   Frameworks available on the CLASSPATH
o   Existing configuration for the application.

Based on these, Spring Boot provides basic configuration needed to configure the application with these frameworks.
All auto configuration logic is implemented in spring-boot-autoconfigure.jar.
As soon as we add Spring Boot Starter Web as a dependency in our project, Spring Boot Autoconfiguration sees that Spring MVC is on the classpath. It autoconfigures dispatcherServlet, a default error page and webjars.
• Spring Boot Starter project
All web application needs Spring MVC, Jackson Databind (for data binding), Hibernate-Validator (for server side validation using Java Validation API) and Log4j (for logging)along with compatible versions of all these frameworks.

o   spring-boot-starter-web-services - SOAP Web Services
o   spring-boot-starter-web - Web & RESTful applications
o   spring-boot-starter-test - Unit testing and Integration Testing
o   spring-boot-starter-jdbc - Traditional JDBC
o   spring-boot-starter-hateoas - Add HATEOAS features to your services
o   spring-boot-starter-security - 
 Authentication and Authorization using Spring Security.
 Basic Authentication is the default.
 Using default security password, Default userid is user. Default password is printed in the server start up log.
o   spring-boot-starter-data-jpa - Spring Data JPA with Hibernate
o   spring-boot-starter-cache - Enabling Spring Framework’s caching support
o   spring-boot-starter-data-rest - Expose Simple REST Services using Spring Data REST
o   spring-boot-starter-actuator - To use advanced features like monitoring & tracing to your application out of the box
o   spring-boot-starter-undertow, spring-boot-starter-jetty, spring-boot-starter-tomcat - To pick your specific choice of Embedded Servlet Container
o   spring-boot-starter-logging - For Logging using logback
o   spring-boot-starter-log4j2 - Logging using Log4j2

• Spring Boot Starter Parent

o   All Spring Boot projects typically use spring-boot-starter-parent as the parent in pom.xml.
o   Spring Boot Starter Parent defines spring-boot-dependencies as the parent pom. It inherits dependency management from spring-boot-dependencies. Details in the next section.
o   Parent Poms allow you to manage the following things for multiple child projects and modules:
 Configuration - Java Version and Other Properties
 Dependency Management - Version of dependencies
 Default Plugin Configuration
o   A few other settings related to encoding and source, target version are also set in the parent pom.
<java.version>1.6</java.version><resource.delimiter>@</resource.delimiter> <!-- delimiter that doesn't clash with Spring ${} placeholders --><project.build.sourceEncoding>UTF-8</project.build.sourceEncoding><project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding><maven.compiler.source>${java.version}</maven.compiler.source><maven.compiler.target>${java.version}</maven.compiler.target>
o   Spring Boot Starter Parent inherit from spring-boot-dependencies, it shares all these characteristics as well.
<properties><activemq.version>5.13.4</activemq.version>...<ehcache.version>2.10.2.2.21</ehcache.version><ehcache3.version>3.1.1</ehcache3.version>...<h2.version>1.4.192</h2.version><hamcrest.version>1.3</hamcrest.version><hazelcast.version>3.6.4</hazelcast.version><hibernate.version>5.0.9.Final</hibernate.version><hibernate-validator.version>5.2.4.Final</hibernate-validator.version><hikaricp.version>2.4.7</hikaricp.version><hikaricp-java6.version>2.3.13</hikaricp-java6.version><hornetq.version>2.4.7.Final</hornetq.version><hsqldb.version>2.3.3</hsqldb.version><htmlunit.version>2.21</htmlunit.version><httpasyncclient.version>4.1.2</httpasyncclient.version><httpclient.version>4.5.2</httpclient.version><httpcore.version>4.4.5</httpcore.version><infinispan.version>8.2.2.Final</infinispan.version><jackson.version>2.8.1</jackson.version>....<jersey.version>2.23.1</jersey.version><jest.version>2.0.3</jest.version><jetty.version>9.3.11.v20160721</jetty.version><jetty-jsp.version>2.2.0.v201112011158</jetty-jsp.version><spring-security.version>4.1.1.RELEASE</spring-security.version><tomcat.version>8.5.4</tomcat.version><undertow.version>1.3.23.Final</undertow.version><velocity.version>1.7</velocity.version><velocity-tools.version>2.0</velocity-tools.version><webjars-hal-browser.version>9f96c74</webjars-hal-browser.version><webjars-locator.version>0.32</webjars-locator.version><wsdl4j.version>1.6.3</wsdl4j.version><xml-apis.version>1.4.01</xml-apis.version></properties>

What is REST?

REST stands for REpresentational State Transfer. REST specifies a set of architectural constraints. Any service which satisfies these constraints is called RESTful Service.
The five important constraints for RESTful Web Service are

• Client - Server: There should be a service producer and a service consumer.
• The interface (URL) is uniform and exposing resources.
• The service is stateless.
• The service results should be Cacheable. HTTP cache, for example.
• Service should assume a Layered architecture. Client should not assume direct connection to server - it might be getting info from a middle layer - cache.

Fault Tolerance

When we build a microservices architecture, there are a large number of small microservices, and they all need to communicate with one another.
Let’s say Microservice5 is down at some point of time. The solution to this problem is to have a fallback in case of failure of a microservice. This aspect of a microservice is called fault tolerance.
A popular framework used to implement fault tolerance is Hystrix, a Netflix open source framework. Here is a code example of the same:

@GetMapping("/fault-tolerance-example")
@HystrixCommand(fallbackMethod="fallbackRetrieveConfguration")
public LimitConfiguration retrieveConfiguration() {
        throw new RuntimeException("Not Available");
}
public LimitConfiguration fallbackRetrieveConfiguration() {
        return new LimitConfiguration(999, 9);
}

Here, if retrieveConfiguration() fails, then fallbackRetrieveConfiguration is called, which returns a hardcoded LimitConfiguration instance:
Difference between catching exception and Hystrix
• Metrics and Dashboarding provided Out of the Box which can help you peek into your system and dependent connection
• Implements BulkHead by using different Thread Pools
• Lower maintenance cost
• Health check ability. It provides a health check class which plugins with Health monitoring APIs
• The main difference is that Hystrix opens the circuit (it is an analogy to electrical circuits) when it detects an error and does not invoke downstream services until sometime has elapsed. This behavior prevents an avalanche of errors in cascading. It is similar to a smart traffic light that turns red and doesn't let you pass because it knows that you are going to have an accident a little later. After a configurable time, the circuit is closed again. You can see 'Circuit opened / closed' at Hystrix dashboard:

load balancing algorithms

Ribbon offers wide variety of client side load balancing algorithms to choose from.

naming server

Eureka Naming server offers naming server to register all the microservices, which can be used to fetch the address of the registered microservices.

Auto scaling

You can containerize each microservice using Docker and create an image.
Kubernetes has the capability to manage containers. Kubernetes can be configured to auto scale based on the load. Kubernetes can identify the application instances, monitor their loads, and automatically scale up and down.

Centralized Logging

Using Log Streams is one way to implement centralized logging. The common way to implement it is to stream microservice logs to a common queue. Distributed logging server listens to the queue and acts as log store. It provides search capabilities to search the trace.
Popular Implementations
Some of the popular implementations include
• the ELK stack (Elastic Search, Logstash and Kibana) for Centralized Logging
• Zipkin, Open Tracing API And Zaeger for Distributed Tracing

API Gateway

In microservices, we route all requests - both internal and external - through API Gateways. We can implement all the common features like authentication, logging, auditing, and rate limiting in the API Gateway.
For example, you may not want Microservice3 to be called more than 10 times by a particular client. You could do that as part of rate limiting in the API gateway.
You can implement the common features across microservices in the API gateway. A popular API Gateway implementation is Zuul API Gateway.

Centralized Configuration

Configuration for all microservices (for all environments) is stored at one place - a centralized configuration store.
When a microservice needs its configuration, it provides an id at launch - a combination of microservice name and the environment.
Spring Cloud Config Server is one of the popular implementations of a cloud config server