Forem: Rushal Verma

Hello, I'm a Junior DevOps looking for a job. Is there anyone who is looking for Junior DevOps?

Rushal Verma — Mon, 23 Jul 2018 11:39:21 +0000

Here is the link to my resume: Resume

I recently graduated and currently residing in India, but willing to relocate or work remotely! I have a great interest in learning about infrastructure, containers, Data Intensive Technologies, and DevOps. I am currently reading Kubernetes: Up and Running and posting notes from it on my Github which you can find here: Kubernetes: Up and Running.

Also, for paying the bills I am currently doing an internship with The Linux Foundation under there outreach program.

My mail id: rusrushal13@gmail.com

So if anyone is interested in hiring, please let me know

Learning about the Druid Architecture

Rushal Verma — Thu, 11 Jan 2018 18:59:57 +0000

Learning about the Druid Architecture
This post distills the material presented in the paper titled “Druid-A Real-time Analytical Data Store” published in 2014 by F Yang and others.

The paper presents the architecture of Druid and what problem it solves in the world of analytical processing and details how it supports fast aggregations, flexible filters, and low latency data ingestion.

1. Introduction

The paper starts with talking about how much Hadoop has grown in time but the pain in dealing with Hadoop performance. Hadoop excels at storing and providing access to large amounts of data, however, it does not make any performance guarantees around how quickly that data can be accessed. Hadoop works well for storing data, it is not optimised for ingesting data and making that data immediately readable.

The system combines a column-oriented storage layout, a distributed, shared-nothing architecture, and an advanced indexing structure to allow for the arbitrary exploration of billion-row tables with sub-second latencies.

Every section of the paper describes bits and pieces of the problem and how to solve it with the help of Druid storage, it also describes what they learned leaving Druid on production.

2. Problem

Druid solves the problem around the ingesting and exploring large quantities of transactional events(log data). The need for Druid was facilitated by the fact that existing open source Relational Database Management Systems (RDBMS) and NoSQL key/value stores were unable to provide a low latency data ingestion and query platform for interactive applications. In addition to the query latency needs, it supports the system to be multi-tenant and highly available. It tries to solve the problem of data exploration, ingestion, and availability span multiple industries.

3. Architecture

A Druid cluster consists of different types of nodes and each node type is designed to perform a specific set of things. The different types of nodes are loosely coupled so that cluster could support distributed, shared-nothing architecture so that intra-cluster communication failures have minimal impact on availability.

Real-Time Nodes

Real-Time Nodes encapsulates the functionality to ingest and query event streams. Events indexed via these nodes are immediately available for querying. The nodes announced their online state and the data they serve in Zookeeper(uses for coordination). Druid behaves as a row store for queries and processes the events in a buffer to a column-oriented storage format. Each persisted index is immutable and nodes load the indexes into off-heap memory for querying.

We refer segments here as the immutable block which contains all the events that have been ingested by a real-time node for some span of time. Also, Deep Storage mostly refers to S3 or HDFS. The ingest, persist, merge, and handoff steps are fluid; there is no data loss during any of the processes.

Real-Time Nodes are a consumer of data and require a corresponding producer to provide the data stream, Kafka(or any other message bus) sits between the producer and the real-time node. Real-time nodes ingest data by reading events from the message bus. The time from event creation to event consumption is ordinarily on the order of hundreds of millisecond. The purpose of message bus acts as a buffer for incoming events and to act as a single endpoint from which multiple real-time nodes can read events.

Historical Nodes

Historical Nodes encapsulate the functionality to load and serve the immutable blocks of data(segments) created by real-time nodes. Most of the data in Druid is immutable and typically Historical Nodes are the main workers in the cluster. Nodes also follow shared-nothing architecture i.e. the nodes have no knowledge of one another; they simply know how to load, drop and serve immutable segments. They also use Zookeeper of coordination.

Historical Nodes can support read consistency because they only deal with immutable data. Immutable data blocks also enable a parallelization model; can concurrently scan and aggregate immutable blocks without blocking.

Broker Nodes

Broker Nodes are query routers to historical and real-time nodes. Broker nodes understand the metadata published in Zookeeper about what segments are queryable and where those segments are located. Broker nodes route the incoming queries such that the queries hit the right historical or real-time nodes. Broker nodes also merge partial results from historical and real-time nodes before returning a final consolidated result to the caller. Nodes also contain a cache with an LRU invalidation strategy. The cache can use local heap memory or an external distributed key/value store such as Memcached. Real-time data is never cached. In the time of outages, broker nodes use their last state and forward queries to the real-time nodes and historical nodes.

Coordinator Nodes

Coordinator Nodes are the in-charge of data management and distribution on Historical Nodes. They tell the historical nodes to load new data, drop outdated data, replicate data and move data to load balance. Coordinator nodes undergo leader-election process that determines a single node that runs the coordinator functionality. The remaining coordinator nodes act as redundant backups.

4. Storage Format

Data tables in Druid(called data sources) are collections of timestamped events and partitioned into a set of segments. Segments represent the fundamental storage unit in Druid and replication and distribution are done at segments. Druid creates additional lookup indices for string columns such that only those rows that pertain to a particular query filter are ever scanned. They also discuss how much storing of columns indices could help in maximising compression. Druid components allow storage engines such as the JVM heap or in memory-mapped structures(default).

5. Query API

Druid has its own query language and accepts queries as POST requests. Broker, historical, and real-time nodes all share the same query API. It also supports filter set. Druid supports many types of aggregations including sums on floating-point and integer types, minimums, maximums, and complex aggregations(cardinality estimation and others). The one main drawback of using Druid that it doesn’t support a Join query. They still haven’t done that yet and said that research is going on to resolve this.

6. Performance

The paper shows many insights(graphs and tables) of running druid in production that they shared in the paper. The results are:

Average query latency: 550 milliseconds, with 90% queries returning in less than 1 second, 95% queries in less than 2 second and 99% in less than 10 seconds.
For the most basic datasets cluster ingested data 800,000 events/second/core. Ingestion totally depends on the complexity of data source.

7. Druid in Production

Druid is often used to explore data and generate reports on data. Users tend to explore short time intervals of recent data.
Concurrent queries could be problematic which they solve by query prioritization to address the issues.

Assumes that many nodes failing at once are not that practical and left the capacity to completely reassign the data from 2 historical nodes.
In the case of Data Center outages, it relies on Deep Storage.
Operational Metrics on nodes are provisioned too and include system level data(CPU usage, available memory, JVM statistics, and disk capacity). The metrics used for performance and stability of the cluster and also for the aspects of data users.
Druid also paired with Stream Processor(Apache Storm) for both real-time and historical data. Storm handles the streaming data processing work, and the columnar storage used for responding to queries.
Segments are the essence of Druid and they are distributed. They can be exactly replicated over multiple data centers. Such setup may be desired if one data center is situated near to users.

8. Conclusions

They tried to provide every essential information about the Druid and if anyone wants to get started with it they could start with this paper. They gave references to many more papers which helps you understanding the OLAP databases, columnar storage vs row storage, distributed and real-time analytical store and much more in a single paper. It’s an essential read for people getting started with Druid.

How to document the learning on Internship?

Rushal Verma — Wed, 10 Jan 2018 03:37:37 +0000

I am a Data Engineering Intern at a startup in India. I am learning new things every day because of switching teams back n forth. How to document these learning day to day or week to week so that I will remember these stuffs after a year or two? Any ideas will help a lot :D

Twitter University test preparation

Rushal Verma — Tue, 08 Aug 2017 10:54:40 +0000

I applied for Twitter University Test. Is there anyone here from twitter or knows about the process?
What kind of questions should I look at / practice with for Twitter University - Test 2 specifically?

How do I know if I’m good at programming?

Rushal Verma — Sun, 23 Jul 2017 15:37:40 +0000

I am reading a article(http://www.danielslater.net/2017/07/how-do-i-know-if-im-good-at-programming.html) where a senior developer try to explain the answer of it. I also want to know more answers?
How you all know that you are good?

How to be a junior software developer? Tips for students to grab a good job, what was your story?

Rushal Verma — Mon, 10 Jul 2017 09:59:45 +0000

I am right now a student entering in my last year of engineering in computer science from India. I'll be glad to know some responses in order to grab a good job in the field of junior software development. I am right now learning about Docker and would like to know more about it too and if possible a job reference will be great :D

Publish your first image to Docker Hub

Rushal Verma — Sun, 09 Jul 2017 21:02:15 +0000

As you are familiar with Docker from my previous post. Let dive in to explore more.

Now you know how to run a container and pull an image, now we should publish our image for others too. Why you should have all the fun ;)

So what we need to publish our Docker Image?

A Dockerfile
Your App

Yeah, that’s it.

Why we need my app the Docker way?

Historically we have to our app(maybe python app) and we need python(or all dependencies) runtime environment on our machine. But then it creates a situation where the environment on your machine has to be just so in order for your app to run as expected and for your server too where you are running the server. With Docker, you don’t need anything(no environment). You can just grab a portable Python runtime as an image, no installation necessary. Then, your build can include the base Python image right alongside your app code, ensuring that your app, its dependencies, and the runtime, all travel together. These portable images are defined by something called a Dockerfile.

Dockerfile serves as the environment file inside the container. It helps in creating an isolated environment for your container, what ports will be exposed to outside world, what files you want to “copy in” to that environment. However, after doing that, you can expect that the build of your app defined in this Dockerfile will behave exactly the same wherever it runs.

So let's create a directory and make a Dockerfile.

FROM python:3.6
WORKDIR /app
ADD . /app
RUN pip install -r requirements.txt
EXPOSE 80
ENV NAME world
CMD [“python”, “app.py”]

So you have your Dockerfile. You can see the syntax is pretty easy and self-explanatory.

Now we need our app. Let's create one, a python app ;)

app.py

from flask import Flask
import os
import socket
app = Flask(__name__)
@app.route("/")
def hello():
    html = "<h3>Hello {name}!</h3>" \
           "<b>Hostname:</b> {hostname}<br/>"
    return html.format(name=os.getenv("NAME", "world"), hostname=socket.gethostname())
if __name__ == "__main__":
    app.run(host='0.0.0.0', port=80)

requirements.txt

Flask

Now you have all the things in order to proceed. Now just build your app.

Let's Build it

ls will now show you this

$ ls
app.py        requirements.txt        Dockerfile

Now create the image.

docker build -t imagebuildinginprocess .

Where is your image? It’s in your local image registry.

$ docker images
REPOSITORY                TAG                 IMAGE ID            CREATED             SIZE
imagebuildinginprocess    latest              4728a04a9d39        14 minutes ago      694MB

Lets Run it too

docker run -p 4000:80 imagebuildinginprocess

What we did here is mapping the port 4000 to the container exposed port 80. You should see a notice that Python is serving your app at http://0.0.0.0:80. But that message is coming from inside the container, which doesn’t know you mapped to port 80 of that container to 4000, making the URL http://localhost:4000. Go to that URL in a web browser to see the display content served up on a web page, including “Hello World” text and the container ID.

Let's Share it :D

we will be pushing our built image to the registry so that we can use it anywhere. The Docker CLI uses Docker’s public registry by default.

Log into the Docker public registry on your local machine.(If you don’t have account make it here cloud.docker.com)

docker login

Tag the image: It is more like naming the version of the image. It’s optional but it is recommended as it helps in maintaining the version(same like ubuntu:16.04 and ubuntu:17.04)

docker tag imagebuildinginprocess rusrushal13/get-started:part1

Publish the image: Upload your tagged image to the repository: Once complete, the results of this upload are publicly available. If you log into Docker Hub, you will see the new image there, with its pull command.

docker push rusrushal13/get-started:part1

Yeah, that's it, you are done. Now you can go to Docker hub and can check about it also ;). You published your first image.

I found out this GitHub repository really awesome. Have a look on it https://github.com/jessfraz/dockerfiles

Do give me feedbacks for improvement ;)

Hello Docker

Rushal Verma — Sat, 08 Jul 2017 05:18:30 +0000

Docker is a new craze which is quite popular nowadays. I don’t know much before about it but at my internship, I attended a workshop from Dieter Reuter(Docker Captain)and Niclas Mietz from bee42 solutions gmbh.

It was quite fun two days workshop with them and I got to know a lot about it. So let’s get started with Docker.
Before getting started with it, you should install Docker on your machine. Take a help from Google.

Okay now, your machine has Docker, so now you may be thinking what Docker really is?

Docker is an open platform for building, shipping, and running applications. With Docker, you can manage your infrastructure in the same ways you manage your applications. It’s easy, simple and quite powerful too.

Okay as everyone knows we always start with Hello-world

docker container run hello-world

In the output, you can see that we are doing many things(what Docker doing). But before that, I want to illustrate something regarding Docker Engine.

Docker Engine

Docker is same like your car, it has an engine which manages everything. It also consists a server known as Docker Daemon(the dockerd command) which manages everything for the client(Docker CLI). In between, the API handles the request from the client to the server. The server responds accordingly to the client by managing the images, containers, networks, and volumes.

Docker Daemon: The Docker daemon listens to API requests and manages Docker objects(images, containers, networks, and volumes). A daemon can also communicate with other daemons to manage Docker services(swarm mode).

Docker Client: The Docker client is the way that many Docker users interact with Docker. When you use commands such as docker run the client sends these commands to dockerd which carries them out. The docker command uses the Docker API. The Docker client can communicate with more than one daemon.

Docker Registries: A Docker registry stores Docker images. Docker Hub and Docker Cloud are public registries that anyone can use, and Docker is configured to look for images on Docker Hub by default. When you use the docker pull or docker run commands, the required images are pulled from your configured registry. When you use the docker push command, your image is pushed to your configured registry. You can upgrade the application by pulling the new version of the image and redeploying the containers.

Images: An image is a snapshot of a Docker container. Often, an image is based on another image, with some additional customization. For example, you may build an image which is based on the ubuntu image but installs the Nginx web server and your application, as well as the configuration details needed to make your application run. Images are lightweight, small, and fast when compared to other virtualization technologies.

Containers: A container is a runnable instance of an image. You can create, run, stop, move, or delete a container using the CLI. You can connect a container to one or more networks, attach storage to it, or even create a new image based on its current state. By default, a container is relatively well isolated from other containers and its host machine. You can control how isolated a container’s network, storage, or other underlying subsystems are from other containers or from the host machine. A container is defined by its image as well as any configuration options you provide to it when you create or run it. When a container stops, any changes to its state that are not stored in persistent storage disappears.

Services: It allows you to scale containers across multiple Docker daemons, which all work together as a swarm with multiple managers and workers. Each member of a swarm is a Docker daemon, and the daemons all communicate using the Docker API. A service allows you to define the desired state, such as the number of replicas of the service that must be available at any given time. By default, the service is load-balanced across all worker nodes. To the consumer, the Docker service appears to be a single application.

After some theory let’s see the output now,

Hello from Docker!

The docker client contacted to Docker daemon.
The Docker daemon pulled the “hello-world image from the Docker Hub.
The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading.
The Docker daemon streamed that output to the Docker client, which sent it to your terminal.

That’s it, pretty simple and easy. Enjoy the easiness of it.