Forem: Darsh Shukla

Intoduction To Danfo.js - Manipulating And Processing Data

Darsh Shukla — Wed, 26 Aug 2020 11:40:25 +0000

Danfo.js

An open-source, JavaScript library providing high-performance, intuitive, and easy-to-use data structures for
manipulating and processing structured data. It is heavily inspired by the Python's Pandas
library and provides a similar interface and API. Moreover, Danfo.js is fast and It is built on
Tensorflow.js and supports tensors out of the box.

Data science thrives in Python because of the ecosystem of open-source libraries - NumPy, Pandas, sklearn, and more.
It's great to see similar tools being developed by the JavaScript community. This could be the start of something big.
So let us see Danfo.js in action.

Installation

There are two ways to get danfo.js. To install it via npm, you can do the following:

npm install danfojs-node

We can also install and use it in the browsers by using the CDN below:

<script src="https://cdn.jsdelivr.net/npm/danfojs@0.1.1/dist/index.min.js"></script>

Creating a Series object by passing a list of values, letting danfo.js create a default integer index:

const dfd = require("danfojs-node")
s = new dfd.Series([1, 3, 5, undefined, 6, 8])
s.print()

	0
0	1
1	3
2	5
3	NaN
4	6
5	8

Reading JSON data and vector operations

const json_data = [{ A: 0.4612, B: 4.28283, C: -1.509, D: -1.1352 },
{ A: 0.5112, B: -0.22863, C: -3.39059, D: 1.1632 },
{ A: 0.6911, B: -0.82863, C: -1.5059, D: 2.1352 },
{ A: 0.4692, B: -1.28863, C: 4.5059, D: 4.1632 }]
df = new dfd.DataFrame(json_data)
// Adding to series object, can use sub, mul, div, and pow
df['A'].add(df['B']).print()
df['A'].pow(2).print()
// Maximum value of C
console.log(df['C'].max()) // 4.505899

Add A and B

	A
0	4.744029998779297
1	0.2825700044631958
2	-0.13752996921539307
3	-0.8194299936294556

A Square

	A
0	0.21270543336868286
1	0.2613254487514496
2	0.4776192009449005
3	0.22014862298965454

Reading CSV file from URL

dfd.read_csv("https://raw.githubusercontent.com/curran/data/gh-pages/jsLibraries/jsLibs.csv")
.then(df => {

//prints the first five columns
df.head().print()

}).catch(err => {
console.log(err);
})

	Library	Minified File Size (kb)	Github Stars
0	Knockout.js	17	5036
1	Angular.js	106	24580
2	Ember.js	71	10368
3	Can.js	82	928
4	React.js	123	7015

Calculate descriptive statistics for all numerical columns

df.describe().print()

	Minified File Size (kb)	Github Stars
count	7	7
mean	58.071426	9464.286133
std	49.75978	9038.434833
min	1	156
median	71	7015
max	123	24580
variance	2476.035714	81693304.23

The shape of the data, column names, and dtypes

console.log(df.shape);

console.log(df.column_names);

df.ctypes.print()

[ 7, 3 ]
[ 'Library', 'Minified File Size (kb)', 'Github Stars' ]

	0
Library	string
Minified File Size (kb)	float32
Github Stars	int32

dfd.read_csv("https://raw.githubusercontent.com/curran/data/gh-pages/jsLibraries/jsLibs.csv")
.then(df => {
df['Library'].print()
}).catch(err => {
console.log(err);
})

	Library
0	Knockout.js
1	Angular.js
2	Ember.js
3	Can.js
4	React.js
5	Backbone.js
6	Model.js

Selecting on a multi-axis by label, by slicing, and by query

dfd.read_csv("https://raw.githubusercontent.com/curran/data/gh-pages/jsLibraries/jsLibs.csv")
.then(df => {
// Selection by label
const sub_df = df.loc({ rows: [0, 1], columns: ["Library", "Github Stars"] })
sub_df.print()

// Selection by slicing
const slice_df = df.loc({ rows: ["0:4"], columns: ["Library", "Github Stars"] })
slice_df.print()

// Selection by query
const query_df = df.query({ "column": "Github Stars", "is": ">", "to": 10000 })
query_df.print()
}).catch(err => {
console.log(err);
})

Selection By Multi-Axis Label

	Library	Github Stars
0	Knockout.js	5036
1	Angular.js	24580

Selection By Slicing

	Library	Github Stars
0	Knockout.js	5036
1	Angular.js	24580
2	Ember.js	10368
3	Can.js	928

Selection By Query

	Library	Minified File Size (kb)	Github Stars
1	Angular.js	106	24580
2	Ember.js	71	10368
5	Backbone.js	6.5	18167

There are many mathematical operations we can perform over the dataframe object.

Danfo supports plotting

Danfo uses Plotly.js as backend for plotting. This gives us the ability to make interactive plots from DataFrame and Series. Plotting only works in the browser version of danfo.js, and requires an HTML div to show plots.

<!DOCTYPE html>
<html lang="en">

<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<!--danfojs CDN -->
<script src="https://cdn.jsdelivr.net/npm/danfojs@0.1.1/dist/index.min.js"></script>
<title>Document</title>
</head>

<body>

<div id="plot_div"></div>
<script>

dfd.read_csv("https://raw.githubusercontent.com/curran/data/gh-pages/jsLibraries/jsLibs.csv")
.then(df => {

var layout = {
title: 'JavaScript Libraries and Github Stars',
xaxis: {
title: 'Libraries',
},
yaxis: {
title: 'Stars',
}
}

new_df = df.set_index({ key: "Library" })
new_df.plot("plot_div").bar({ columns: ["Github Stars"], layout: layout })

}).catch(err => {
console.log(err);
})

</script>
</body>

</html>

[Python] Part 2 Generator: Python Coroutine

Darsh Shukla — Sun, 16 Aug 2020 16:02:14 +0000

If you don't know about What generator is ? Please have a look at Part1 post.

Let’s, deep dive!

Understanding Subroutine

When the logic for a function with complex behavior is divided into several small steps that are themselves functions, these functions are called helper functions or subroutines. Subroutines are called by the main function that is responsible for coordinating the use of several subroutines.

Understanding Coroutine

There is another approach for dividing or decomposing a complex program. In the above approach, we have only one entry point, the main function. Using coroutines we will have one entry point but multiple re-entry points. When using coroutines, there is no main function to coordinate results. Instead, coroutines themselves link together to form a pipeline. There may be a coroutine for consuming the incoming data and sending it to other coroutines. There may be coroutines that each do simple processing steps on data sent to them, and there may finally be another coroutine that outputs a final result.

Let’s explore how python supports building coroutines with the yield and send() statements.

So far we have seen that python generator function yield a value. But it can also consume a value using (yield). Apart from yield and next(), the generator object also have send() and close() functions.


>>> value = (yield)

PEP 342 explains the exact rules, which are that a yield-expression must always be parenthesized except when it occurs at the top-level expression on the right-hand side of an assignment.

With this statement, execution pauses until the object's send method is invoked with an argument


>>> coroutine.send(data)

Then, execution resumes, with the value being assigned to the value of data.

Let’s look at an example to clarify things up:

We will design two functions. The first function emit_word() will produce words from the given input string and emit/send our second function pattern() will consume the input word and will give the output if a pattern p is in the input word.

Consumer:

# Consumer
>>> def pattern(p):
        print(f"Searching for {p}")
        try:
            while True:
                s = (yield)
                if p in s:
                    print(s)
        except GeneratorExit:
            print("=== Completed ===")

Examining the consumer functioin:


>>> p = pattern("apple")
>>> next(p)
Seraching for apple
# The generator function will pause until send() is called.
>>> p.send("An apple a day keeps the doctor away")
An apple a day keeps the doctor away
>>> p.send("A strawberry a day keeps the doctor away")
>>> p.send("My mom gave me an apple")
My mom gave me an apple
>>> p.close()
===Completed===

Producer:

#Producer
>>> def emit_word(string, match):
        for word in string.split():
              match.send(word)
        match.close()

Examining the producer functioin:

For each word in a given string it emits/send the word to the consumer function

Joining all together:

>>> string = "Pen Pineapple apple pen"
>>> match = pattern("apple")
>>> next(match)
Seraching for apple
>>> emit_word(string, match)
Pineapple
apple
===Completed===

Key takeaways: generators

Generators produce values one-at-a-time as opposed to giving them all at once.
There are two ways to create generators: generator functions and generator expressions.
Generator functions yield, regular functions return.
Generator expressions need (), list comprehensions use [].
You can only use generator expressions once.
There are two ways to get values from generators: the next() function and a for loop. The for loop is often the preferred method.
We can use generators to read huge files to give us one line at a time. By not loading everything in memory at once.

[Python] Part 1 Generator: Introduction

Darsh Shukla — Sun, 16 Aug 2020 15:51:48 +0000

Generator

Before starting with what generators are, we’ll first discuss what iteration, iterables, and iterators are.

Iteration

Iteration is a term for taking each item of something, one after another. Any time you use a loop, explicit or implicit, to go over a group of items, that is iteration.

Iterable

Iterable is an object that is, well, iterable, which simply means that it can be used in iteration, e.g. with a for loop. How? By using an iterator. I'll explain it below.
Iterable objects also define __iter__ that returns an iterator and also have a __getitem__ method suitable for indexed lookup. Examples of iterables include lists, tuples, and strings - any such sequence that can be iterated over a loop.

Iterator

An iterator is an object that defines how to actually do the iteration, specifically what is the next element. That's why it must have the next() method. It remembers where it is during iteration.

Let’s look an example to clarify things up:

>>> # ranks is an iterable
>>> ranks = [1,2,3]

>>> # Returns True ranks has __iter__ method
>>> __iter__ in dir(ranks)
True
>>> # Returns True ranks has __getitem__ method
>>> __getitem__ in dir(ranks)
True

>>> # ranks is now an iterator
>>> ranks = iter(ranks)
>>> # Retuns True ranks has __next__ method
>>> __next__ in dir(ranks)
True
>>> next(ranks)
1
>>> next(ranks)
2
>>> next(ranks)
3
>>> # next() raises StopIteration, to signal that iteration is complete
>>> next(ranks)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

Whenever we use a for loop, or map, or a list comprehension, etc. in Python, the next method is called automatically to get each item from the iterator, thus going through the process of iteration.

With our knowledge of iterators, we can implement the evaluation rule of a for statement in terms of while, assignment, and try statements.

>>> i = ranks.__iter__() # or iter(ranks)
>>> try:
        while True:
            item = i.__next__() # or next(i)
            print(item)
    except StopIteration:
        pass
1
2
3

In the above example if we want to keep track of our iteration we need to introduce a new field current to keep track of progress through the sequence. With simple sequences like one shown above, this can be done easily. With complex sequences, however, it can be quite difficult for the next() function to save its place in the calculation. Hence, comes generators allowing us to define more complicated iterations.

What is a generator?

A generator is an iterator returned by a special class of function called a generator function. Generator functions are distinguished from regular functions in that rather than containing return statements in their body, they use yield statement to return elements of a series.


>>> def countdown(n): 
        while n > 0: 
            yield n 
            n -= 1 
>>> for i in countdown(5):
        print(i, end=' ')

5 4 3 2 1

Above, the behavior is different from normal functions. Calling a generator function creates a generator object.


>>> def countdown(n): 
        while n > 0: 
            yield n 
            n -= 1 
>>> c = countdown(5)
<generator object countdown at 0x0000017CD6788360>

The function only executes on next()


>>> next(c)
5
>> >next(c)
4
# and so on until next() raises StopIteration

A generator is kinda different from objects that support iteration. We can iterate over the generated data once, but if we want to do it again, we have to call the generator function again (resetting generator).

Like list comprehensions, generators also support concise notions for such operations. Using ‘()’

>>> gen = (expression for i in s if condition)

Note: In the above countdown example it doesn’t start running the function. It only computes when we request for data (The technical term for this behavior is lazy evaluation). Therefore not loading everything at once in in-memory is a useful perk of generators that we can exploit while dealing with a huge amount of data. We can model sequences with no definite end using generators

Below is an example of an `infinite` sequence of even numbers

>>> def infinte_range(step):
        i = 0
        while True:
            yield i
            i += step
>>> even_numbers = (x for x in infinte_range(2))
>>> next(even_numbers)
0
>>> next(even_numbers)
2
>>> next(even_numbers)
4
>>> next(even_numbers)
6

Data Lake - 5 Major Principles

Darsh Shukla — Tue, 11 Aug 2020 16:26:42 +0000

In the previous post we understood what is a data warehouse and it's architecture. In this post, we are going to explore Data Lake.

Data Lake

If you think data mart as a store of bottled water -
cleansed, packaged and structured for easy consumption -
the data lake is a large body of water in a more natural
state - James Dixon

A Data Lake is a central place where all company’s raw data flows, rapidly and provides a 360° view of the company’s data.

It is intended to be raw data - as close to the source as possible. It is still a good idea to capture the metadata and describe the data so that people can explore the lake and re-use what is available.

Data Lake comprises of these 5 major principles

Ingest

Ability to collect all data that businesses care about. Systems frequently ingest data through APIs and batch processing from various sources such as transactional systems, web scraping, relational databases, etc.

Store

Getting all data in one place and breaking down data silos. The storage should be scalable that supports storing structured, unstructured, and semistructured data.

Analyze

Matching correct data points and having correct systems and finding relations between all data gathered. Schema is written at the time of analysis (schema on read)

Surface

There needs to be a simple method to display all analysis. The data need to be understood here; the easier to see the result the easier to take actions.

Act

"Make Me More Money". A plan has to be put in place to take the results of data analysis and fit it into an operating business model.

The main challenge with a data lake architecture

Raw data is stored with no oversight of the contents. For a data lake to make data usable, it needs to have defined mechanisms to catalog, and secure data. Without these elements, data cannot be found, or trusted resulting in a “data swamp.”

For those who are thinking that how Data Warehouse is
different from a Data Lake, well to put it succinctly,
proving the value of data is the role of Data Lake and In
Data Warehouse one’s valuable data resides which they use
for reporting purposes.

For data discovery and transformation of data stored in the data lake, we can set up an execution engine(Spark/Hadoop/Dremio) layer over the top of our data lake for a better understanding of data.

Data Warehouse - The Minimal Architectural Approach

Darsh Shukla — Tue, 11 Aug 2020 16:20:32 +0000

Data Warehouse

A central repository of information that can be analyzed to make informed decisions. Data flows into a data warehouse from transactional systems, relational databases, and other sources.

The core architecture boils down to some

Pre-Processing

After collecting data from various sources, consistency in the format is essential before loading into a data warehouse.
In this step, our concern is with preparing data and validating data to maintain consistency in the format and we achieve it by using the following techniques.

Data Preparing

Dealing with missing values

We can straight away delete the record containing the missing value but that can lead to bias and also reduce sample size.
Another approach and mainly used is data imputation, we use column average, or can interpolate from nearby values, or can predict missing values. Some known techniques for data imputation are hot-deck imputation, mean substitution, and regression.

Dealing with outliers

Outliers can be genuine or erroneous data points. To identify outliers the mainly used techniques are finding distance from mean or distance from the fitted line.
For coping with outliers we can either drop, cap/floor, or set the data point to mean value.

Data Validation

Certain field's value lies between a specific range, checking whether this criterion is matched or not.
Check data type, some fields require specific kinds of data types like Boolean, Numeric, Date, or Timestamp. We have to validate it.
Check for fields having compulsory constraints i.e specific columns shouldn't be having an empty value.

Staging

This is a holding area, where we put data after pre-processing (but not always) — and store it transiently until it’s processed further down the line. This is the last point where the data should be found in its raw form. (Amazon s3, Google Cloud Storage, etc are useful cloud products for staging area). It is recommended to have a staging area before loading data into the data warehouse. If storage is the concern we can archive the data after a reasonable duration.

Master

The master area is where the incoming data takes some real shape. The master schema should contain correctly modeled tables, that are appropriately named. Data Cleaning is mostly done here.

Data Cleaning

Removing irrelevant data, remove data that doesn't fit the context of the problem or business KPIs.
Remove duplicate values.
Fix typos to have uniformity in data.
Convert data types and check for any inconsistency in date format and timezone.

Reporting

Business analysts, data scientists, and decision-makers access the data through business intelligence (BI) tools, SQL clients, and other analytics applications. Businesses use reports, dashboards, and analytics tools to extract insights from their data, monitor business performance, and support decision making. These reports, dashboards, and analytics tools are powered by data warehouses. (To analyze data Amazon Redshift, Google BigQuery, etc are useful big data cloud products)

Key Point

Data Warehouse consists of highly curated data that serves as the central version of the truth, the data schema is designed prior to DW implementation (schema-on-write). DW is used for batch reporting, business intelligence, and data visualization by the business analysts.