Forem: vaibhav aggarwal

Automating Continuous Integration through Jenkins

vaibhav aggarwal — Thu, 11 Jun 2020 17:57:36 +0000

In the last post, Setting up Jenkins on AWS jenkins was installed and hosted on aws. It was accessible at a hosted public address.

Now, lets proceed to connect our jenkins to our vcs (read github in our case).

While implementing a project, there are many components that are being developed in parallel. These small components are (should be) loosely coupled for better scalability.

Let's understand this through an example;
Suppose, team is working on a ecommerce portal. After high-level-design and tech stack discussion, a basic portal will have services namely; UI, backend, database and a payment system.

Implementation will start and after a significant progress, services need to be integrated with each other. Code needs to be gathered, compiled and built to check the proper integration. Now imagine doing this without any tools everyday. Somebody has to sit down, gather the code at one place, compile it and test the working. And if any error occurs, person has to track down the implementor and coordinate the concerned errors. This will slow down the process and will add overhead to the functioning of the team.

A version control system with continuous integration pipeline will automate and fasten the process.

Jenkins with many of its plugin will pull the code, build and test it. If any erros occurs, concerned developer or team will be notified with the log outputs.

Connect with github

To automate the process of continuous integration, code should be compiled and built on every new change pushed to the vcs repository. Change has to be notified to the CI server (read jenkins here) via some mechanism. Here, webhooks come handy as they allow external services to be notified when certain change happens.

Most powerful or realiable aspect of jenkins is plugins available. Go to jenkins console and proceed to manage jenkins section. Select manage plugins and make sure jenkins have git plugin installed or otherwise install it.

Go to the concerned github project and proceed to settings. In the options, you will find webhooks section. Select add webhook and enter the http://<jenkins_url/github-webhook/>.

As soon as you save the webhook, github will make a dummy post call to the jenkins webhook. Make sure the call is successful to the jenkins. It can be verified from from the status icon on right of added webhook.

Pipeline steps

In the last section, github repository was connected to jenkins server. Everytime theres a new update (our case only push events), it will be notified to the jenkins github webhook.

Create a new freestyle project and name it build (your choice).
Its always better to give a meaningful description, so other team members can understand.
Under source code management, select git and enter the github repository. Provide the credentials if its a private repository (always better though).
In build triggers, select GitHub hook trigger for GITScm polling. It will trigger the build for this jenkins project anytime, jenkins webhook receive any PUSH event from github.
Under build section, select execute shell and enter the commands to build your project. For as of now just put something,

 echo "building"

This completes our build stage of the pipeline. Save and exit from the configuration.

Finally, everything is setup and ready to test. Lets test our first stage of pipeline by creating a commit in the corresponding github repo.

As soon as commit is published, git will post a PUSH event on to the jenkins webhook. That will trigger the build project that was created above.

Check console output and confirm our statement "building" stdout on the screen.

Notice that nothing was specified in the post build stage of jenkins build project. Above build stage was completed and on success pipeline should proceed to testing stage.

install docker

sudo amazon-linux-extras install docker
sudo service docker start
sudo usermod -a -G docker ec2-user

If your jenkins is getting error " dial unix /var/run/docker.sock: connect: permission denied". This is due to the permissions as jenkins does not have access to docker.sock. There are two workarounds for this

Change file access type (not recommended though) : change permission access of the file

   chmod 777 /var/run/docker.sock

Using groups:

After that configure your jenkins file build stage. In execete shell commands;

docker build desiredname -t .
docker push dockerhubname/desiredname

Final run

Everything is setup and its time to do a test run. Publish a commit to the github repo and pipeline will start running. At the end desired result can be cross checked at console output.

var vs let vs const

vaibhav aggarwal — Fri, 05 Jun 2020 18:20:45 +0000

Introduction

Let, var and const are ways to create a new varibale in javascript. Before ES2015 (or ES6) only var was available which provided limited scoping capabilities. let and const were instroduced in ES6.

There are two scopes in JS called global scope and function scope. Global scoped variable is accessible everywhere whereas function scoped variable is accessible only inside function declaration.

"In JavaScript, variables are initialized with the value of undefined when they are created.". The JavaScript interpreter will assign variable declarations a default value of undefined during what's called the "Creation" phase.

var

For var, it does not matter where it is declared first inside the function. Creation phase will happen before anything and var declaration will assigned a value as 'undefined' until initialized. (Think of every var in the function declaration coming up and being declared as unassigned on the first line.)

function app() {
  console.log(declare); //undefined
  console.log(i); //undefined

  var declare;
  declare  = "initialize";

  for(var i = 0; i < 5; i++){
    var sum = i;
  }

  console.log(declare); //initialize
  console.log(i); //5
  console.log(sum); //4
}

app();

Notice, declare was assigned a default value of unassigned and is accessible even before declaration. For variable i and sum, their values are accessible outside the loop as var is function scoped and not block scope. (Remember every var in the function coming up on the first line);

Also, I do not think its a good practice to access variable before declaring as it can lead to unknown issues.

To solve the problem, let and const was introduced in ES6.

let

let is block scoped rather than function scoped as in the case of var. Block scoped in most simple terms means inside {} and below nested code. Variables declared using let are not accessible before declaration. Imagine making a box starting from declaration of let till corresponding closing bracket.

function app() {
  console.log(declare); //undefined
  console.log(i); //ReferenceError: i is not defined

  var declare;
  declare  = "initialize";

  for(let i = 0; i < 5; i++){
    let sum = i;
  }

  console.log(declare); //initialize
  // console.log(i);

}

app();

As we tried accessing variable i before declaring it throws a reference error opposed to case with variables declared using var. This difference occurs because of the difference in scope of let and var.

function app() {
  console.log(declare); //undefined

  var declare;
  declare  = "initialize";

  for(let i = 0; i < 5; i++){
    let sum = i;
  }

  console.log(declare); //initialize
  console.log(i); //ReferenceError: i is not defined

}

app();

Notice, variable i is accessible only inside for loop. Outside its block, it throws a reference error of not being defined.

const

const is almost similar to let only difference being it can not be reassigned.

let firstName = "vaibhav";
const secondName = "aggarwal";

firstName = "changeMyName";
secondName = "youCantChangeMyName"; //TypeError: Assignment to constant variable.

Dont confuse reassigned with change. Its properties can be changed and only restriction is on reassigning.

const name = {
  firstName: "vaibhav",
  secondName: "aggarwal"
}

console.log(name);

name.firstName = "changeMyName";

console.log(name); 
// {
//   firstName: "changeMyName",
//   secondName: "aggarwal"
// }

name = {}; //TypeError: Assignment to constant variable.

There are many important concepts such as scope, hoisting etc... being involved here. I have tried explaining in simple terms for better understanding.

Reference

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide

Setting up Jenkins on AWS

vaibhav aggarwal — Thu, 04 Jun 2020 18:16:27 +0000

This tutorial walks you through the process of deploying a Jenkins application.

You will launch an EC2 instance (centOS), install and configure Jenkins.
Jenkins to automatically spin up Jenkins build slave instances if build abilities need to be augmented on the instance.

Launch and Setup an EC2 instance

First, launch an ec2 instance to install and run jenkins. Steps involved are:

launch an ec2 instance

Add secutiry group

ssh into the instance: select initiated instance, proceed to connect and follow the instructions. Make sure you are in the same folder as your downloaded .pem key.

    chmod 400 jenkins-ec2.pem //if you named the instance same
    ssh -i "jenkins-ec2.pem" ec2-user@ec2-13-127-31-75.ap-south-1.compute.amazonaws.com //please check your details

Install and run Jenkins

Here, we will connect to the instance and run the jenkins after all necessary installation.
Steps covered:

connect to your instance
download and install jenkins
configure jenkins

After connecting to your ec2 instance via ssh, jenkins need to be download and installed.

    sudo yum update -y //update system
    sudo yum install java-1.8.0-openjdk-devel //install jdk
    curl --silent --location http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo | sudo tee /etc/yum.repos.d/jenkins.repo //get GPG key
    sudo rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key //add repo to system 
    sudo yum install jenkins //install jenkins

    // start the service
    sudo systemctl start jenkins
    //check service status
    systemctl status jenkins.service

On checking jenkins service status, an output like this is expected.

Voila! Jenkins is installed and running at port 8080 on your instance.
But remember, there is no access to port 8080 to outside world. Security groups are used for the purpose of protecting our running instances from outside attack. Let's go to attached security group and edit inbound rules.

Open a browser window and access your jenkins instance running on your_ec2_public_ip:8080
access your initial administator password for jenkins using (ec2 terminal).

    sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Select install selected plugins and wait for it. Its almost done. Create a user if needed other than admin. Jenkins ready screen will appear. We are all set to automate our CI process.

Go to manage jenkins and toggle to install plugins.

Jenkins is up and running on AWS.
"Please note for security reasons, never expose your public IP. Always have it running behind a reverse proxy."

In next article, we will create a Jenkins project and integrate it with github. That will be the first step to create a Continuous Integration pipeline.

P.S. "Above written are my views. I am always learning and exploring new things. Please comment and criticize wherever possible. Stay Happy."

Helpful links

Sending emails in NodeJs with Nodemailer

vaibhav aggarwal — Thu, 04 Jun 2020 12:45:43 +0000

Introduction

While making a product communication with the customers is one of the most important factor. The formal and continuous communication for the upcoming events, newsletter and other cyclic events happen through email servers. There are events when email needs to be sent based on specific action performed by the customer.

Consider the mentioned specific actions by the user:

Register for the product
Buys or avails any service
Updates about transactions
Query related to the product

Hence, emails needs to be sent triggered through some APIs. A email server needs to contacted and carry out the communication. An SMTP (Simple Mail Transfer Protocol) server is an application that’s primary purpose is to send, receive, and/or relay outgoing mail between email senders and receivers. Read more about SMTP server.

After setting up a server (article for another day) a transported is needed to send emails through that.

Nodemailer is zero dependency module for Node.js applications that allows sending email in an easy manner. Its flexible and supports SMTP and other transport mechanism. It can be configured for aws-ses, sendgrid and other smtp providers. Read more about nodemailer at nodemailer.com

Some of the features of nodemailer:

Zero dependecny on other modules
Secure email delivery with TLS and DKIM email authentication
HTML content and embedded image attachments

Lets integrate nodemailer in our project and start sending some emails.

  npm install nodemailer 

  // If to include in package.json
  npm install --save nodemailer

Create a transporter

SMTP protocol is the most common transporter for sending mails. It makes integration very simple. Set host, port, authentication details and method.

SMTP can be setup on various services such as aws-ses, gmail etc. But here as our main focus is on to use nodemailer, lets use mailtrap. Mailtrap integrated as a SMTP server and enables you to debug emails in a pre-production environment.

Go to mailtrap.io and sign up in a second. On opening, a page in figure below will be seen.

Figure 1. Mailtrap account to setup and use as an pre production SMTP server

Now, we have SMTP credentials, lets setup our transporter for nodemailer.

//server.js
var transport = nodemailer.createTransport({
  host: "smtp.mailtrap.io",
  port: 2525,
  auth: {
    user: "2a591f5397e74b",
    pass: "c8115f6368ceb0"
  }
});

Nodemailer uses transporter to facilitate sending mails. Next steps will be setting up email configurations.

//server.js
/**
 * Sends mail through aws-ses client
 * @param options Contains emails recipient, subject and text
 */
const send = async options => {
  const message = {
    from: `${options.fromName} <${options.fromEmail}>`,
    to: ${options.userEmail},
    subject: ${options.subject},
    text: ${options.message}
  };

  const info = await transporter.sendMail(message);
  console.log(info.messageId);
  return info;
}

One thing to notice is that, here in message object text field will have normal text. But the emails, you generally receive are beautiful and formatted rather than just plain text. As mentioned above, nodemailer provides options to send HTML and image attachments.(I will write another article covering all features of nodemailer and how to send beautiful HTML based emails)

In the to section, take email from mailtrap.io. It provides temporary email addresses for testing.

Figure 2. Temporary Email address for testing by mailtrap

//server.js
const http = require('http');
const express = require('express');
const nodemailer = require('nodemailer');
const app = express.Router();
const port = 3000;

app.post('/email', async (req, res) => { 
  try{
   const response = await send(req);
   console.log(response);
   res.status(200).json({
    message: 'sucess',
    data: response
   })
  } catch (error) {
   console.log(error);
   res.status(400).json({
    message: error.message
   })
  }
})

var transporter = nodemailer.createTransport({
  host: "smtp.mailtrap.io",
  port: 2525,
  auth: {
    user: "2a591f5397e74b",
    pass: "c8115f6368ceb0"
  }
});

/*
Use in your req.body
 const options = {
   userEmail: <mailtrapEmail>,
   subject: 'Welcome to Auffr',
   message: 'We are excited to have you in the family'
 }
*/
/**
 * Sends mail through aws-ses client
 * @param options Contains emails recipient, subject and text
 */
const send = async options => {
  const message = {
    from: `${options.fromName} <${options.fromEmail}>`,
    to: ${options.userEmail},
    subject: ${options.subject},
    text: ${options.message}
  };

  const info = await transporter.sendMail(message);
  console.log(info.messageId);
  return info;
}

const server = http.createServer((req, res) => {
  res.statusCode = 200;
    res.setHeader('Content-Type', 'text/plain');
    res.end('This is the Main App!\n');
});

server.listen(port, () => {
    console.log(`Server running at http://localhost:${port}/`);
});

On running the file server.js, express server will be up and running at http://localhost:3000/ . As mentioned above, a post request made at the endpoint /email will send an email to the mailtrap account.

I used postman to make a simple post request at endpoint http://localhost:3000/email. Voila, email is received and visible in our mailtrap account inbox.

Figure 3. Email received through /email endpoint

In next article, I will share how nodemailer can achieve so much more than discusses here.

"I am not an expert, please comment and correct me if I am wrong anywhere. Always love to have a discussion."

How to Analyse Big Data Using the ELK Stack

vaibhav aggarwal — Tue, 03 Mar 2020 12:08:08 +0000

This article introduces users to the ELK stack. Apart from discussing how it helps in better data analytics, there are detailed explanations about the internal workings of Elasticsearch, as well as insights into how Logstash and Kibana complement Elasticsearch’s capabilities. The article was originally published in Open source for you, Asia’s largest open-source magazine: https://opensourceforu.com/2020/02/how-to-analyse-big-data-using-the-elk-stack/
Big Data refers to large data sets that combine structured, semi-structured and non-structured data from various sources. Big Data is not just big in size but can also provide various insights about running and improving businesses when it is analysed. It can give insights on user behavioural patterns, changing trends, a system’s health or any anomalies. The analysis helps improve customer integration, detect abnormalities and mitigate fraud.
Increased Internet penetration and mobile in every hand are generating loads of user data for companies. This data is of high velocity, volume and variety. A lot of data is being generated from mobiles, media devices, traffic surveillance systems, messages, etc. This data needs to be analysed in real-time to provide the user with consumable insights.

Imagine an e-commerce company that aims to provide high-quality customer service. When users search for a product on the website, they want the best-suited results to be shown. For better user experience, various other factors like user age, brand, budget and location are added to the user search.
Relational databases have predefined schema to store structured data, whereas data being generated today can be non-structured. The real power of data can be harnessed only when we are able to aggregate the data collected from various sources. Data can be structured (has a defined schema) or unstructured (has no defined schema). Examples like user number, location and email can be stored in an SQL database as structured. User information, such as payment mode, wish lists and previous orders, is stored in an unstructured format. To provide the best user experience, users should be shown recommendations and search results based on an analysis of the data collected about them — for example, combining users’ product search with their purchase history and budget range. No matter how complex data is, it needs to be combined and analysed properly for the best decision making.

Fig1. Sample logstash config file
ELK is an acronym for three open-source projects. The first is Elasticsearch, which is the distributed search and analytics engine. Then comes Logstash, which is the data pipeline that helps ingest data from multiple sources, transforms it and sends it to data stashes like Elasticsearch. And finally, Kibana lets users visualise data stored in Elasticsearch for better understanding and business interactions. ELK is easy to set up, scalable and efficient and provides near real-time search analytics. Its capabilities include data feature enhancing, visual analytics and alerting mechanisms with the help of Logstash and Kibana.
A hands-on guide on the ELK stack can be found at github.com/ELK-Tutorial.

Logstash Logstash is a server-side data processing pipeline used to ingest, parse and then transform data, before outputting it to Elasticsearch. It provides various filters that enhance the insights and human readability. Logstash ingests the structured and unstructured data from various sources such as logs, system metrics and Web applications, by using the input plugins available. It transforms and combines the data using many of the available filters to a common format for more powerful analysis and business value. Filter examples grok: Derives structure from unstructured data geo-ip: Extracts information from IP addresses date: Extracts date and time from standard Logstash structures and enhances the data collected from different sources to output it into Elasticsearch, which is a NoSQL database that can store data and provide real-time search over that multi-valued data. Logstash has a variety of outputs that let you route transformed data where you want it, flexibly.

fig 2. Data after enhancing through logstash

fig 3. Data after enhancing through logstash

Elasticsearch Elasticsearch is a distributed open source search engine released under the Apache licence. It is a REST API layer over Apache’s Lucene (https://lucene.apache.org/). It provides horizontal scalability, reliability and real-time search through the documents. Elasticsearch is able to search faster because it uses indexing to go over documents. Lucene (the storage engine of Elasticsearch) stores the data using a technique called inverted indexing. This storage architecture lets Elasticsearch provide functionalities like aggregation (similar to GROUPBY in SQL), which helps when doing complex queries on data. Indexing Indexing is jargon for inserting data in Elasticsearch, which stores data as documents contained under an index. The index can be thought of as a table in SQL. An Elasticsearch cluster can have multiple indices, which in turn contain multiple types. These types then have multiple documents. A document is similar to a row in SQL. Documents can have multiple fields that provide information about a data object. Here is an example: A POST statement is used for creating an index. Here, iiitb is an index. ‘Student’ is one of the types under iiitb. Others are professors, staff and security. Inside the brackets, we have a document under the type ‘student’. By looking at the example stated above, we observe that there wasn’t a need to perform any architectural work specifically, like creating the index or defining the types of a field as in an RDBMS. Elasticsearch stores this data as documents using inverted indexing. Inverted indexing Elasticsearch uses an inverted index to achieve a fast full-text search over documents. An inverted index consists of all the unique words in the document, and each unique word is mapped to the list of documents containing it.

fig 4. Data stored by Elastic Search before any indexing stage
Table 1 considers three documents and six words. Each column shows if the document contains the corresponding word. The way in which Elasticsearch stores this data is called inverted indexing.
The inverted index may hold a lot more information than just the list of the documents containing a unique term. It may have a count of documents, the number of times the term appears in the document, the length of the document, and many other valuable insights.
Indexing involves tokenisation and normalisation, which is essentially an analysis of the document. The data from each field is broken into terms, and then normalised into a standard form to improve ‘searchability’.
Token filters are used in the analysis to increase search efficiency are listed below.
Stop words: Removing frequently occurring words like the, is, they, etc.
Lower casing: All words are changed into a lower case for better results.
Stemming: This involves using only the root word. The extra tense-defining part is removed; e.g., swimming and swimmer can be stemmed to swim.
Synonyms: This filter merges the occurrences of words with the same meaning.
A separate word known as the token is mapped to the documents carrying that word. Therefore, when we are searching for a document with a particular word or phrase in our data, we don’t have to search the whole document. Instead, indexes will directly give the document associated with the desired word or phrase.
Indexed documents are then sent into a buffer, and wait for a certain period to get stored as segments. Segment data is divided and stored into small blocks called shards. This reduces the time taken for the document to become available for search after the process of indexing. In versions before 6.7.0, Elasticsearch is used to directly store to disks using the fsync() function, which is very costly as it involves directly writing to the disk. In Elasticsearch 6.7.0, the latest version, this has been changed to a file cache system that stores data in a buffer first, and then commits it to the disk, increasing the performance.

fig 5. Data indexing optimisation stage
As can be seen in Table 2, after analysis, stop words (like ‘the’) are removed, synonyms combined (like ‘sad’and ‘upset’) and stemming is done
(‘swimming’ and ‘swimmer’ to ‘swim’).
Shard is a low-level worker unit that holds just a slice of all the data. Index, as mentioned above, is just a logical namespace that points to one or more shards. Elasticsearch distributes the given data in clusters by storing them as shards, which are then allocated to nodes present in the cluster. Elasticsearch needs to find the shard where it has saved a particular document. Therefore, the process should be deterministic and can be described as follows:
shard number = Hash( _id ) % number_of_primary_shards
After fixing the shard, we have a way to find the location of any document. This means that any node in the cluster knows about every document and can forward the call to the correct node.

Fig 6. Data storage and fetching cycle
Under the hood
The Lucene engine stores data in shards, i.e., data blocks. A node can have many shards but this creates a big, single point of failure. Elasticsearch handles failure by doing replication across the cluster. We have to mention the number of primary index shards at the time of indexing. This fixes the number of primary nodes and helps to coordinate in finding the location of appropriate nodes.
Let us say that we have two primary indexes and a two-node architecture.
The master node finds the shard containing the required documents (here, Node 2) and forwards the request to the corresponding node, as shown in Figure 6.
The effect of replication on queries
Insert: We send the request to the master node (each cluster has one), which finds the shard containing the document with the help of _id. The document is saved to the primary shard. If successful, the request is forwarded to the replica nodes in parallel and while inserting a document, Elasticsearch will locate the primary shard for the index and insert the document there.
Search: The related shard will be retrieved based on the following formula. Using round robin scheduling on the primary or replica nodes, one is chosen to carry out the current query. Round robin scheduling between primary and replicated nodes optimises the load on the search engine. The explained architecture uses round robin scheduling to balance the load on nodes.
Delete: The delete request follows similar steps as inserting. The master node finds the associated primary shard and deletes the document from there. Afterwards, it will also be deleted from replicated shards.

Fig 7. Stored data as a index in ES

Kibana Kibana is a tool to visualise Elasticsearch data in real-time. It provides a variety of visualisations to get the best meaning out of your data. Dashboards can be created according to use cases, and get updated with new data automatically.

Fig 8. Visualisations based on data stored
Kibana is a visualisation tool built on top of Elasticsearch, which connects to the latter using Elasticsearch’s REST APIs. It helps us understand data better from a non-developer perspective by providing various forms such as tables, rows, charts, geo maps, heat maps, and so on.
Some of the main features of Kibana are:
Leverages the aggregation capabilities of Elasticsearch
Has many predefined visualisations and it is very easy to define new ones using Vega
Can plot complex data like geo-data, time series metrics, relationship graphs, etc
Unsupervised machine learning has been added to detect anomalies in Elasticsearch data
Dashboards can be made and transported in the form of PDF, PNG or level-controlled links
Alerts and notifications can be set up for any specific query
The ELK stack helps users to collect data from various sources, enhance it and store it in a self-replicating distributed manner. Stored data can be accessed in real-time and can be presented to the user in various forms such as pie-charts or geo-mapping, with the help of Kibana. It helps enterprises in collecting data to provide it to businesses and users in real-time.