Forem: PyProDev

Mastering Logical Comparison, Control Flow, Filtering on Numpy Array and Pandas DataFrame

PyProDev — Sat, 17 Dec 2022 08:31:28 +0000

In this article, we will learn about different comparison operators, how to combine them with Boolean operators, and how to use Boolean outcomes in control structures. Boolean logic is the foundation of decision-making in Python programs. We'll also learn to filter data in pandas DataFrames using logic, a skill that a data scientist must have.

Comparison Operators

Comparison operators are operators that can tell how two values relate, and result in a boolean.

Numeric comparisons

In the simplest sense, we can use these operators on numbers. For example, if we want to check if 2 is smaller than 3, we type 2 less than sign 3.

print(2 < 3)

output:
True

Because 2 is less than 3, we get True. we can also check if two values are equal, with a double equals sign. From this call, we see that 5 equals 6 gives us False.

print(5 == 6)

output:
False

It makes sense because 5 is not equal to 6. We can also make a combination of equality and smaller than. Have a look at this command that checks if 5 is smaller than or equal to 6.

print(5 <= 6)

output:
True

It's TRUE, but also 6 smaller than or equal to 6 is True.

print(6 <= 6)

output:
True

Of course, we can also use comparison operators directly on variables that represent these integers.

x = 5
y = 6
print(x < y)

output:
True

Comparison between strings

All these operators also work for strings. Let's check if "abc" is smaller than "acd".

print("abc" < "acd")

output:
True

According to the alphabet order, "abc" comes before "acd", so the result is True.

Comparison between integer and string

Let's find out if comparing a string and an integer works. Here if the integer 2 is smaller than the string "abc".

print(2 < "abc")

output:
TypeError: '<' not supported between instances of 'int' and 'str'

We get an error (TypeError: '<' not supported between instances of 'int' and 'str'). Typically, Python can't tell how two objects with different types relate.

Comparison between integer and float

Different numeric types, such as floats and integers, are exceptions.

print(3 < 4.12)

output:
True

No error this time. In general, always make sure that we make comparisons between objects of the same type.

Compare on Numpy array

Another exception arises when we compare on NumPy array, lengths, with an integer, 22. This works perfectly.

import numpy as np
lengths = np.array([21.85, 20.97, 21.75, 24.74, 21.44])
print(type(lengths))
print(lengths > 22)

output:
<class 'numpy.ndarray'>
[False False False  True False]

NumPy figures out that we want to compare every element in lengths with 22, and returns corresponding booleans. Behind the scenes, NumPy builds a NumPy array of the same size filled with the number 22, and then performs an element-wise comparison. This is concise, very efficient code, which data scientists love!

We can also compare two NumPy arrays element-wise. house1 and house2 contain the areas for the kitchen, living room, bedroom and bathroom in the same order. Which areas in house1 are smaller than the ones in house2 like this?

house1 = np.array([18.0, 20.0, 10.75, 9.50])
house2 = np.array([14.0, 24.0, 14.25, 9.0])
print(house1 < house2)

output:
[False  True  True False]

It appears that the living room and bedroom in house1 are smaller than the corresponding areas in house2.

Comparators

Here is the table that summarizes all comparison operators.

Comparator	Meaning
<	less than
<=	less than or equal to
>	greater than
>=	greater than or equal to
==	equal to
!=	not equal to

We are already familiar with some of these. They're all pretty straightforward, except for the not equal !=. The exclamation mark followed by an equals sign stands for inequality. It's the opposite of equality.

Equality

To check if two Python values, or variables, are equal you can use ==. To check for inequality, you need !=. Have a look at the following examples that all result in True.

print(2 == (1 + 1))
print("PYTHON" != "python")
print(True != False)
print("Python" != "python")

output:
True
True
True
True

Write a code to see if True equals False.

# Comparison of booleans
print(True == False)

output:
False

Write Python code to check if -3 * 15 is not equal to 45.

# Comparison of integers
print(( -3 * 15 ) != 45)

output:
True

Ask Python whether the strings "python" and "Python" are equal.

# Comparison of strings
print("python" == "Python")

output:
False

Note that strings are case-sensitive. What happens if you compare booleans and integers? Write code to see if True and 1 are equal.

# Compare a boolean with an integer
print(True == 1)
print(True == 2)

output:
True
False

A boolean is a special kind of integer: True corresponds to 1, False corresponds to 0.

Greater and less than

We also talked about the less than and greater than signs, < and > in Python. We can combine them with an equals sign to get <= and >=. Note that =< and => are not valid. For examples.

print(3 < 4)
print(3 <= 4)
print("alpha" <= "beta")

output:
True
True
True

Remember that for string comparison, Python determines the relationship based on alphabetical order.

Check if x is greater than or equal to -13.

# Comparison of integers
x = -4 * 3
print(x >= -13)

output:
True

Check if True is greater than False.

# Comparison of booleans
print(True > False)

output:
True

Remember that True is 1 and False is 0 in value.

Boolean Operators

We can produce booleans by performing comparison operations. The next step is combining these booleans. We can use boolean operators for this. The three most common ones are

and,
or, and
not.

`and`

The and operator works just as we would expect. It takes two booleans and returns True only if both the booleans themselves are True.

Case1	Case2	Case1 `and` Case2
True	True	True
True	False	False
False	True	False
False	False	False

print(True and True)
print(True and False)
print(False and True)
print(False and False)

output:
True
False
False
False

Instead of using booleans, we can also use the results of comparisons. Suppose we have a variable x, equal to 8. To check if this variable is greater than 5 but less than 15, we can use x greater than 5 and x less than 15.

x = 8
print(x > 5 and x < 15)

output:
True

As we already learned, the first part will evaluate to True. The second part will also evaluate to True. So the result of this expression, True and True, is True. This makes sense, because 8 lies between 5 and 15.

`or`

The or operator works similarly, but the difference is that only at least one of the booleans should be True.

Case1	Case2	Case1 `or` Case2
True	True	True
True	False	True
False	True	True
False	False	False

print(True or True)
print(True or False)
print(False or True)
print(False or False)

output:
True
True
True
False

Also here we can make combinations with variables, like this example that checks if a variable y, which is equal to 3, is less than 5 or above 10.

y = 3
print(y < 5 or y > 10)

output:
True

3 less than 5 is True, 3 greater than 10 is False. The or operation thus returns True.

`not`

Finally, let's the not operator. It simply negates the boolean value we use it on. not True is False, not False is True. The not operation is typically useful if we're combining different boolean operations and then want to negate that result.

print(not True)
print(not False)

output:
False
True

Nested Boolean operators

Let's take the boolean operators to another level.

Note that not has a higher priority than and and or, it is executed first.

x = 8
y = 9
not(not(x < 3) and not(y < 8 or y > 14))

output:
False

Correct! x < 3 is False. y < 8 or y > 14 is False as well. If you continue working like this, simplifying from inside to outward, you'll end up with False.

Filtering on NumPy arrays

Now, for NumPy arrays, things are different. Retaking the lengths example, we can try to find out which lengths are higher than 21, but lower than 22. The output of lengths greater than 21 is easily found, so is the one for the lengths lower than 22.

print(lengths)
print(lengths > 21)
print(lengths < 22)

output:
[21.85 20.97 21.75 24.74 21.44]
[ True False  True  True  True]
[ True  True  True False  True]

Let's now try to combine those with the and operator we just learned.

print(lengths > 21 and lengths < 22)

output:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Oops, python return ValueError: The truth value of an array with more than one element is ambiguous. Clearly it doesn't like an array of booleans to work on.

Numpy provides these "array equivalents" of and, or and not functions,

logical_and,
logical_or and
logical_not.

To find out which lengths are between 21 and 22, we will use these functions. Again, as we expect from NumPy, the and operation is performed element-wise.

print(np.logical_and(lengths > 21, lengths < 22))

output:
[ True False  True False  True]

To select only these lengths are between 21 and 22, we can use the resulting array of booleans in square brackets.

print(lengths[np.logical_and(lengths > 21, lengths < 22)])

output:
[21.85 21.75 21.44]

Again, NumPy wins when it comes to writing short yet very expressive Python code. How about this on Pandas DataFrames, the de facto standard for dataset manipulation?

Boolean operators on NumPy Array

Before, the operational operators like < and >= worked with NumPy arrays out of the box. Unfortunately, this is not true for the boolean operators and, or, and not.

To use these operators with NumPy, we will need np.logical_and(), np.logical_or() and np.logical_not(). Here's an example on the house1 and house2 arrays.

Generate boolean arrays that answer the following questions:

Which areas in my_house are greater than 18.5 or smaller than 10?

# house1 greater than 18.5 or smaller than 10
print(np.logical_or(house1 > 18.5, house2 < 10))

output:
[False  True False  True]

Which areas are smaller than 11 in both house1 and house2?

# Both house1 and house2 smaller than 11
print(np.logical_and(house1 < 11, house2 < 11))

output:
[False False False  True]

Filtering on pandas DataFrames

The NumPy array can be useful to do comparison operations and boolean operations on an element-wise basis. Let's now use this knowledge on Pandas DataFrame. Click here to download the countries.csv file. First, let's import the countries dataset from the CSV file using pandas.

import pandas as pd
countries = pd.read_csv('countries.csv', index_col=0)
print(countries)

output:
       country    capital  population
IND      India  New Delhi  1393409030
MMR    Myanmar     Yangon    54806010
THA   Thailand    Bangkok    69950840
SGP  Singapore  Singapore     5453570
CHN      China    Beijing  1412360000

Suppose you now want to keep the countries, for which the population is greater than 100,000,000. There are three steps to this.

First of all, we want to get the population column from countries.
Next, we perform the comparison on this column and store its result.
Finally, we should use this result to do the appropriate selection on the DataFrame.

Step 1: Get the column

So the first step, getting the population column from countries. There are many different ways to do this. What's important here, is that we ideally get a Pandas Series, not a Pandas DataFrame. Let's do this with square brackets, like this.

print(type(countries['population']))
print(countries['population'])

output:
<class 'pandas.core.series.Series'>
IND    1393409030
MMR      54806010
THA      69950840
SGP       5453570
CHN    1412360000
Name: population, dtype: int64

This loc alternative and this iloc version would also work perfectly fine.

print(countries.loc[:, 'population'])

output:
IND    1393409030
MMR      54806010
THA      69950840
SGP       5453570
CHN    1412360000
Name: population, dtype: int64

print(countries.iloc[:, 2])

output:
IND    1393409030
MMR      54806010
THA      69950840
SGP       5453570
CHN    1412360000
Name: population, dtype: int64

Step 2: Compare

Next, we perform the comparison. To see which rows have a population greater than 100,000,000, we simply append greater than 100000000 to the code from before, like this.

print(countries['population'] > 100000000)

output:
IND     True
MMR    False
THA    False
SGP    False
CHN     True
Name: population, dtype: bool

Now we get a Series containing booleans. If you compare it to the population values, you can see that the population with a value over 100000000 corresponds to True, and the ones with a value under 100000000 correspond to False now. Let's store this Boolean Series as is_huge.

is_huge = countries['population'] > 100000000
print(is_huge)

output:
IND     True
MMR    False
THA    False
SGP    False
CHN     True
Name: population, dtype: bool

Step 3: Subset the DataFrame

The final step is using this boolean Series is_huge to subset the Pandas DataFrame. To do this, we put is_huge inside square brackets.

print(countries[is_huge])

output:
    country    capital  population
IND   India  New Delhi  1393409030
CHN   China    Beijing  1412360000

The result is exactly what we want: only the countries with an population greater than 100000000, namely India and China.

Summary

So let's summarize this: we selected the population column, performed a comparison on the population column and stored it as is_huge so that we can use it to index the countries DataFrame. These different commands do the trick. However, we can also write this in one line. simply put the code that defines is_huge directly in the square brackets.

print(countries[countries['population'] > 100000000])

output:
    country    capital  population
IND   India  New Delhi  1393409030
CHN   China    Beijing  1412360000

Great! Pandas help data scientists' life much easy.

Boolean operators on Pandas DataFrame

Now we haven't used boolean operators yet. Remember that we used this logical_and function from the NumPy package to do an element-wise boolean operation on NumPy arrays? Because Pandas is built on NumPy, we can also use that function here. Let's write the codes which keep the observations that have a population between 10,000,000 and 90,000,000.

print(countries)

output:
       country    capital  population
IND      India  New Delhi  1393409030
MMR    Myanmar     Yangon    54806010
THA   Thailand    Bangkok    69950840
SGP  Singapore  Singapore     5453570
CHN      China    Beijing  1412360000

print(np.logical_and(countries['population'] > 10000000, countries['population'] < 90000000))

output:
IND    False
MMR     True
THA     True
SGP    False
CHN    False
Name: population, dtype: bool

The only thing left to do is placing this code inside square brackets to subset countries appropriately. This time, only Myanmar and Thailand are included. Look how easy it is to filter DataFrames to get interesting results.

print(countries[np.logical_and(countries['population'] > 10000000, countries['population'] < 90000000)])

output:
      country  capital  population
MMR   Myanmar   Yangon    54806010
THA  Thailand  Bangkok    69950840

Now we know about comparison operators such as

<
<=
>
>=
==
!=

and we also know how to combine the boolean results, using boolean operators such as

and,
or and
not.

Control Flow

Things get interesting when we can use these concepts to change how our program behaves. Depending on the outcome of our comparisons, we might want our Python code to behave differently. we can do this with conditional statements in Python:

if,
else and
elif.

if

Suppose we have a variable x, equal to 4. If the value is even, we want to print out: "x is even".

x = 4
if x % 2 == 0:
    print('x is even.')

output:
x is even.

The modulo operator % with 2 will return 0 if x is even. Python checks if the condition holds. It's true, so the corresponding code is executed: "x is even" gets printed out.

Let's compare this to the general recipe for an if statement. It reads as follows: if the condition is True, execute the codes.

Notice the colon at the end, and the fact that we simply have to indent the Python code with four spaces (or a tab) to tell Python what to do in case the condition succeeds. To exit the if statement, simply continues with some Python code without indentation, and Python will know that it's not part of the if statement. It's perfectly possible to have more lines inside the if statement, like this for example.

x = 4
if x % 2 == 0:
    print('Cheching if x (', x, ') is divisible by 2...')
    print('x is even.')

output:
Cheching if x ( 4 ) is divisible by 2...
x is even.

The script now prints out two lines if we run it. If the condition does not pass, the expression is not executed. You can see this if we change x to be 3 and rerun the code.

x = 3
if x % 2 == 0:
    print('Cheching if x (', x, ') is divisible by 2...')
    print('x is even.')

output:

There's no output. Suppose now that we want to print out "x is odd" in this case. How to do this?

else

Well, we can simply use an else statement, like this.

x = 3
if x % 2 == 0:
    print('x is even.')
else:
    print('x is odd.')

output:
x is odd.

If we run it with x equal to 3, the condition is not true, so the expression for the else statement gets printed out. The general recipe looks like this: for the else statement, we don't need to specify a condition. The else corresponding expression gets run if the condition of the if statements don't hold True.

elif

We can think of cases where even more customized behavior is necessary. Say we want different printouts for numbers that are divisible by 2 and by 3. We can use some elif in there to get the job done. Here is an example.

x = 3
if x % 2 == 0: # False
    print('x is divisible by 2.')
elif x % 3 == 0: # True
    print('x is divisible by 3.')
else:
    print('x is not divisible by both 2 & 3.')

output:
x is divisible by 3

If x equals 3, the first condition is False, so it goes over to check the next condition. This condition holds True so the corresponding print statement is executed.

Suppose now that x equals 6. Both the if and elif conditions hold True in this case. Will two printouts occur?

x = 6
if x % 2 == 0: # True
    print('x is divisible by 2.')
elif x % 3 == 0: # never reach here
    print('x is divisible by 3.')
else:
    print('x is not divisible by both 2 & 3.')

output:
x is divisible by 2.

Nope. As soon as Python finds a true condition, it executes the corresponding code and then leaves the whole control structure after that. This means the second condition, corresponds to the elif, is never reached so there's no corresponding printout. Control flow can be extremely powerful when we're writing Python scripts.

In this article, we learned logical comparison, control flow, and filtering on Numpy Array and Pandas DataFrame.

Read the original article.

Connect & Discuss with us on LinkedIn

Mastering Python Dictionaries & Pandas DataFrames

PyProDev — Sun, 11 Dec 2022 02:00:00 +0000

Python dictionary and Pandas dataframe are the most frequent data structures used in dealing with data. The Pandas DataFrame, is a standard popular data structure to work with tabular data for advanced data analysis. In this article, we will get hands-on practice with how to

create,
manipulate,
- select,
- add,
- update,
- delete

data in dictionaries and dataframes.

List

First, let's talk about the basic Python data type: list. Imagine that we work for the World Bank and want to keep track of the population of each country.

Let's say we have 2021 population data of each country:

India(1,393,409,030),
Burma(54,806,010),
Thailand(69,950,840),
Singapore(5,453,570), and so on.

These data are based on Population Data | The World Bank.

To keep track about which population belongs to which country, we create 2 lists as follow, with the names of the countries in the same order as the populations.

# lists
countries = ['India', 'Burma', 'Thailand', 'Singapore']
populations = [1393409030, 54806010, 69950840, 5453570]

Now suppose that we want to get the population of Burma. First, we have to figure out where in the list Burma is, so that we can use this position to get the correct population. We will use the method index() to get the index.

burma_index = countries.index('Burma')
print(burma_index)

Output:
1

We get 1 as the index of 'Burma' because the index of python's list starts from 0. Now, we can use this index to subset the populations list, to get the population corresponding to Burma.

print(populations[burma_index])

output:
54806010

As expected, we get 54806010, the population of Burma.

Motivation for Dictionaries

So we have two lists, and used the index to connect corresponding elements in both lists. It worked, but it's a pretty terrible approach: it's not convenient and not intuitive. Wouldn't it be easier if we had a way to connect each country directly to its population, without using an index?

Dictionary

This is where the "dictionary" comes into play. Let's convert this population data to a dictionary. To create the dictionary, we need curly brackets. Next, inside the curly brackets, we have a bunch of what are called key:value pairs.

my_dict = {
   "key1":"value1",
   "key2":"value2",
}

In our case,

the keys are the country names, and
the values are the corresponding populations.

The first key is India, and its corresponding value is 1,393,409,030. Notice the colon that separates the key and value here. Let's do the same thing for the three other key-value pairs, and store the dictionary under the name country_population.

country_population = {'India':1393409030, 'Burma':54806010, 'Thailand':69950840, 'Singapore':5453570}

If we want to find the population for Burma, we simply type world_population, and then the string "Burma" inside square brackets.

print(country_population["Burma"])

output:
54806010

In other words, we pass the key in square brackets, and we get the corresponding value. This approach is not only intuitive, it's also very efficient, because Python can make the lookup of these keys very fast, even for huge dictionaries.

Create a Dictionary

We will create a dictionary of countries and capitals data where the country names are the keys and the capitals are the corresponding values.

With the strings in countries and capitals, create a dictionary called asia with 4 key:value pairs. Beware of capitalization! Strings in the code, are case-sensitive.
Print out asia to see if the result is what we expected.

# From string in countries and capitals, create dictionary called asia
asia = {'India':'New Delhi', 'Burma':'Yangon', 'Thailand':'Bangkok', 'Singapore':'Singapore'}

# Print 
print(asia)
# Print type of asia
print(type(asia))

output:
{'India': 'New Delhi', 'Burma': 'Yangon', 'Thailand': 'Bangkok', 'Singapore': 'Singapore'}
<class 'dict'>

Great! <class 'dict'> means that the class of asia is a dictionary. class is out of this article's scope and we will explain it in another article which focus on class. Now that we've built our first dictionary.

Manipulating a Dictionary

If the keys of a dictionary are chosen wisely, accessing the values in a dictionary is easy and intuitive. For example, to get the capital for India from asia we can use India as the key.

print(asia['India'])

output:
New Delhi

We can check out which keys are in asia by calling the keys() method on asia.

# Print out the keys in asia
print(asia.keys())

# Print out value that belongs to key 'Burma'
print(asia['Burma'])

output:
dict_keys(['India', 'Burma', 'Thailand', 'Singapore'])
Yangon

Next, we created the dictionary country_population, which basically is a set of key value pairs. we could easily access the population of Burma, by passing the key in square brackets, like this.

country_population = {'India':1393409030, 'Burma':54806010, 'Thailand':69950840, 'Singapore':5453570}
print(country_population['Burma'])

output:
54806010

Note: For this lookup to work properly, the keys in a dictionary should be unique.

If we try to add another key:value pair to country_population with the same key, Burma, for example,

country_population = {'India':1393409030, 'Burma':54806010, 'Thailand':69950840, 'Singapore':5453570, 'Burma':54800000,}

we'll see that the resulting country_population dictionary still contains four pairs. The last pair('Burma':54800000) that we specified in the curly brackets was kept in the resulting dictionary.

country_population = {'India':1393409030, 'Burma':54806010, 'Thailand':69950840, 'Singapore':5453570, 'Burma':54800000,}
print(country_population)

output:
{'India': 1393409030, 'Burma': 54800000, 'Thailand': 69950840, 'Singapore': 5453570}

let's see how we can add more data to a dictionary that already exists.

Add data to a Dictionary

Our country_population dictionary currently does not have china's data. We want to add "China":1412360000 to country_population.

# Before adding China data
country_population = {'India':1393409030, 'Burma':54806010, 'Thailand':69950840, 'Singapore':5453570}
print(country_population)

output:
{'India': 1393409030, 'Burma': 54806010, 'Thailand': 69950840, 'Singapore': 5453570}

To add this information, simply write the key "China" in square brackets and assign population 1412360000 to it with the equals sign.

# After adding China data
country_population["China"] = 1412360000
print(country_population)

output:
{'India': 1393409030, 'Burma': 54806010, 'Thailand': 69950840, 'Singapore': 5453570, 'China': 1412360000}

Now if you check out world_population again, indeed, China is in there. To check this with code, you can also write 'China' in country_population which gives us True if the key China is in there. Note that China is string type and case sensitive.

print('China' in country_population)

output:
True

Update data in a Dictionary

With the syntax dict_name[key]=value, we can also change values, for example, to update the population of China to 1412000000. Because each key in a dictionary is unique, Python knows that we're not trying to create a new pair, but want to update the pair that's already in there.

country_population["China"] = 1412000000
print(country_population)

output:
{'India': 1393409030, 'Burma': 54806010, 'Thailand': 69950840, 'Singapore': 5453570, 'China': 1412000000}

Delete data from a Dictionary

Suppose now that we want to remove it. We can do this with del, again pointing to China inside square brackets. If we print country_population again, China is no longer in our dictionary.

del(country_population['China'])
print(country_population)

output:
{'India': 1393409030, 'Burma': 54806010, 'Thailand': 69950840, 'Singapore': 5453570}

List vs Dictionary

Using lists and dictionaries, is pretty similar. We can select, update and remove values with square brackets.There are some big differences though. The list is a sequence of values that are indexed by a range of numbers. The dictionary, on the other hand, is indexed by unique keys.

	List	Dictionary
Select, update, remove	use `[]`	use `[]`
Indexed by	range of numbers	unique keys
Use	when a collection of values, order matters, selecting entire subsets	when lookup table with unique keys

When to use which one? Well, if we have a collection of values where the order matters, and we want to easily select entire subsets of data, we'll want to go with a list.

If, on the other hand, we need some sort of look up table, where looking for data should be fast and where we can specify unique keys, a dictionary is the preferred option.

Nested Dictionaries

Remember lists? They could contain anything, even other lists. Well, for dictionaries the same holds. Dictionaries can contain key:value pairs where the values are again dictionaries.

As an example, have a look at the code where another version of asia - the dictionary we've been working with all along. The keys are still the country names, but the values are dictionaries that contain more information than just the capital.

# Dictionary of dictionaries
asia = {'India': {'capital':'New Delhi', 'population':1393409030},
        'Burma': {'capital':'Yangon', 'population':54806010},
        'Thailand': {'capital':'Bangkok', 'population':69950840},
        'Singapore': {'capital':'Singapore', 'population':5453570},
        }

It's perfectly possible to chain square brackets to select elements. To fetch the population for Burma from asia,

print(asia['Burma']['population'])

output:
54806010

Use chained square brackets to select and print out the capital of Burma.

# Print out the capital of Burma
print(asia['Burma']['capital'])

output:
Yangon

Great! It's time to learn about a new data structure!

Tabular dataset examples

As a data scientist, we'll often be working with tons of data. The form of this data can vary greatly, but we can make it down to a tabular structure which is the form of a table like in a spreadsheet. Let's have a look at some examples.

Suppose we're working in a chemical factory and have a ton of temperature measurements to analyze. This data can come in the following form:

temperature	measured at	location
76	2021-03-01 12:00:01	chamber 1
86	2021-03-01 12:00:01	chamber 2
72	2021-03-01 12:00:01	chamber 1
88	2021-03-01 12:00:01	chamber 2

every row is a measurement, or an observation, and
columns are different variables.

For each measurement, there is the temperature, but also the date and time of the measurement, and the location.

Another example: we have information on India, Burma, Thailand and so on. We can again build a table with this data.

Country	Capital	Population
India	New Delhi	1393409030
Burma	Yangon	54806010
Thailand	Bangkok	69950840
Singapore	Singapore	5453570
China	Beijing	1412360000

Each row is an observation and represents a country. Each observation has the same variables: the country name, the capital and the population.

Datasets in Python

To start working on this data in Python, we'll need some kind of rectangular data structure. How about the 2D NumPy array? Well, it's an option, but not necessarily the best one. There are different data types and NumPy arrays are not great at handling these.

Datasets containing different data types

In the above data, the country and capital are string types while the population is float type. Our datasets will typically comprise different data types, so we need a tool that's better suited. To easily and efficiently handle this data, there's the Pandas package.

Pandas

Pandas is

an open source library,
built on the NumPy package,
easy-to-use data structures,
a high level data manipulation tool.

making it very interesting for data scientists all over the world. In pandas, we store the tabular data in an object called a DataFrame. Have a look at the Pandas DataFrame version of the data:

DataFrame

	Country	Capital	Population
IND	India	New Delhi	1393409030
MMR	Myanmar	Yangon	54806010
THA	Thailand	Bangkok	69950840
SGP	Singapore	Singapore	5453570
CHN	China	Beijing	1412360000

The rows represent the observations, and the columns represent the variables. Also notice that each row has a unique row label: IND for India, MMR for Myanmar, and so on. The columns, or variables, also have labels: country, capital, and so on. Notice that the values in the different columns have different types. But how can we create this DataFrame in the first place? Well, there are different ways.

Create a DataFrame from Dictionary

First of all, we can build it manually, starting from a dictionary. Using the distinctive curly brackets, we create key value pairs. The keys are the column labels, and the values are the corresponding columns, in list form.

asia_dict = {
    'country':['India', 'Myanmar', 'Thailand', 'Singapore', 'China'],
    'capital':['New Delhi', 'Yangon', 'Bangkok', 'Singapore', 'Beijing'],
    'population':[1393409030,54806010,69950840, 5453570, 1412360000]
}

After importing the pandas package as pd, we can create a DataFrame from the dictionary using pd.DataFrame.

import pandas as pd
asia_df = pd.DataFrame(asia_dict)
print(type(asia_df))
print(asia_df)

output:
<class 'pandas.core.frame.DataFrame'>
     country    capital  population
0      India  New Delhi  1393409030
1    Myanmar     Yangon    54806010
2   Thailand    Bangkok    69950840
3  Singapore  Singapore     5453570
4      China    Beijing  1412360000

If we check out asia_df now, we see that Pandas assigned some automatic row labels, 0 up to 4. To specify them manually, we can set the index attribute of asia_df to a list with the correct labels.

asia_df.index = ['IND', 'MMR', 'THA', 'SGP', 'CHN']
print(asia_df)

output:
       country    capital  population
IND      India  New Delhi  1393409030
MMR    Myanmar     Yangon    54806010
THA   Thailand    Bangkok    69950840
SGP  Singapore  Singapore     5453570
CHN      China    Beijing  1412360000

The resulting asia_df DataFrame is the same one as we saw before. Using a dictionary approach is fine, but what if we're working with tons of data, which is typically the case as a data scientist? Well, we won't build the DataFrame manually. Instead, we import data from an external file that contains all this data.

Create a DataFrame from CSV file

Suppose the countries' data that we used before comes in the form of a CSV file called countries.csv. CSV is short for comma separated values. The countries.csv file used in this article, can be downloaded at this link.

Let's try to import this data using Pandas read_csv function. We pass the path to the csv file as an argument.

countries = pd.read_csv('path\to\countries.csv')
print(countries)

output:
  Unnamed: 0    country    capital  population
0        IND      India  New Delhi  1393409030
1        MMR    Myanmar     Yangon    54806010
2        THA   Thailand    Bangkok    69950840
3        SGP  Singapore  Singapore     5453570
4        CHN      China    Beijing  1412360000

If we print countries, there's still something wrong. The row labels are seen as a column. To solve this, we'll have to tell the read_csv function that the first column contains the row indexes. We do this by setting the index_col argument, like this.

countries = pd.read_csv('path\to\countries.csv', index_col=0)
print(countries)

output:
       country    capital  population
IND      India  New Delhi  1393409030
MMR    Myanmar     Yangon    54806010
THA   Thailand    Bangkok    69950840
SGP  Singapore  Singapore     5453570
CHN      China    Beijing  1412360000

This time countries nicely contains the row and column labels. The read_csv function features many more arguments that allow us to customize our data importing. Check out its documentation for more details.

Indexing and selecting data in DataFrames

This is important to make accessing columns, rows and single elements in our DataFrame easy. There are numerous ways in which we can index and select data from DataFrames. We're going to see about how to use

square brackets [],
advanced data access methods,
- loc and
- iloc,

that make Pandas extra powerful.

Access data using square brackets [ ]

Suppose that we only want to select the country column from countries. How to do this with square brackets? Well, we type countries, and then the column label inside square brackets. Python prints out the entire column, together with the row labels.

print(countries['country'])

output:
IND        India
MMR      Myanmar
THA     Thailand
SGP    Singapore
CHN        China
Name: country, dtype: object

But there's something strange here. The last line says Name: country, dtype: object. We're clearly not dealing with a regular DataFrame here. Let's find out about the type of the object that gets returned, with the type function as follows.

print(type(countries['country']))

output:
<class 'pandas.core.series.Series'>

So we're dealing with a Pandas Series here. In a simplified sense, we can think of the Series as a 1-dimensional array that can be labeled, just like the DataFrame. If we put together a bunch of Series, we can create a DataFrame.

If we want to select the country column but keep the data in a DataFrame, we'll need double square brackets, like this.

print(countries[['country']])

output:
       country
IND      India
MMR    Myanmar
THA   Thailand
SGP  Singapore
CHN      China

If we check out the type of this result, we will see it is DataFrame type.

print(type(countries[['country']]))

output:
<class 'pandas.core.frame.DataFrame'>

Note that the single bracket version gives a Pandas Series, the double bracket version gives a Pandas DataFrame.

We can perfectly extend this call to select two columns, country and capital, for example. If we look at it from a different angle, we're actually putting a list with column labels inside another set of square brackets, and end up with a sub DataFrame, containing only the country and capital columns.

print(countries[['country', 'capital']])

output:
       country    capital
IND      India  New Delhi
MMR    Myanmar     Yangon
THA   Thailand    Bangkok
SGP  Singapore  Singapore
CHN      China    Beijing

You can also use the same square brackets to select rows from a DataFrame. The way to do it is by specifying a slice. To get the second and third rows of countries, we use the slice 1 colon 3. Remember that the end of the slice is exclusive and that the index starts at zero.

print(countries[1:3])

output:
      country  capital  population
MMR   Myanmar   Yangon    54806010
THA  Thailand  Bangkok    69950840

These square brackets work, but it only offers limited functionality. Ideally, we'd want something similar to 2D NumPy arrays.

To do a similar thing with Pandas, we have 2 ways.

loc is label-based, which means that we have to specify rows and columns based on their row and column labels.
iloc is integer index based, which we have to specify rows and columns by their integer index.

Let's start with loc first.

Access data using `loc`

Let's have another look at the countries DataFrame, and try to get the row for Myanmar. We put the label of the row of interest in square brackets after loc.

print(countries.loc['MMR'])

output:
country        Myanmar
capital         Yangon
population    54806010
Name: MMR, dtype: object

We get a Pandas Series, containing all the row's information, rather inconveniently shown on different lines.

To get a DataFrame, we have to put the 'MMR' string inside another pair of brackets.

print(countries.loc[['MMR']])

output:
     country capital  population
MMR  Myanmar  Yangon    54806010

Selecting Rows using `loc`

We can also select multiple rows at the same time. Suppose we want to also include India and Thailand. Simply add some more row labels to the list.

print(countries.loc[['MMR', 'IND', 'THA']])

output:
      country    capital  population
MMR   Myanmar     Yangon    54806010
IND     India  New Delhi  1393409030
THA  Thailand    Bangkok    69950840

This was only selecting entire rows, that's something you could also do with the basic square brackets. The difference here is that we can extend your selection with a comma and a specification of the columns of interest.

Selecting Rows & Columns using `loc`

Let's extend the previous call to only include the country and capital columns. We add a comma, and a list of column labels we want to keep.

print(countries.loc[['MMR', 'IND', 'THA'], ['country', 'capital']])

output:
      country    capital
MMR   Myanmar     Yangon
IND     India  New Delhi
THA  Thailand    Bangkok

The intersection gets returned.

Selecting Columns using `loc`

we can also use loc to select all rows but only a specific number of columns. Simply replace the first list that specifies the row labels with a colon, a slice going from beginning to end.

print(countries.loc[:, ['country', 'capital']])

output:
       country    capital
IND      India  New Delhi
MMR    Myanmar     Yangon
THA   Thailand    Bangkok
SGP  Singapore  Singapore
CHN      China    Beijing

This time, the result contains all rows, but only two columns.

So, let's take a step back. Simple square brackets countries[['country', 'capital']] work fine if we want to get columns. To get rows, we can use slicing countries[1:4].

row access: countries[1:4]
column access: countries[['country', 'capital']]

The loc function is more versatile: we can select rows, columns, but also rows and columns at the same time. When you use loc, subsetting becomes remarkable simple.

row access: countries.loc[['MMR', 'IND', 'THA']]
column access: countries.loc[:, ['country', 'capital']]
row and column access: countries.loc[['MMR', 'IND', 'THA'], ['country', 'capital']]

The only difference is that we use labels with loc, not the positions of the elements. If we want to subset Pandas DataFrames based on their position, or index, you'll need the iloc function.

Access data using `iloc`

In loc, you use the 'MMR' string in double square brackets, to get a DataFrame, like this.

print(countries.loc[['MMR']])

output:
     country capital  population
MMR  Myanmar  Yangon    54806010

In iloc, we use the index 1 instead of MMR. The results are exactly the same.

# return Series type
print(countries.iloc[1])

output:
country        Myanmar
capital         Yangon
population    54806010
Name: MMR, dtype: object

# return DataFrame type
print(countries.iloc[[1]])

output:
     country capital  population
MMR  Myanmar  Yangon    54806010

Selecting Rows using `iloc`

To get the rows for Myanmar, India and Thailand, the code is like this when using loc,

print(countries.loc[['MMR', 'IND', 'THA']])

output:
      country    capital  population
MMR   Myanmar     Yangon    54806010
IND     India  New Delhi  1393409030
THA  Thailand    Bangkok    69950840

We can now use a list with the index(in the order we want) to get the same result.

print(countries.iloc[[1,0,2]])

output:
      country    capital  population
MMR   Myanmar     Yangon    54806010
IND     India  New Delhi  1393409030
THA  Thailand    Bangkok    69950840

Selecting Rows & Columns using `iloc`

To only keep the country and capital column, which we did as follows with loc,

print(countries.loc[['IND', 'MMR', 'THA'],['country', 'capital']])

output:
      country    capital
IND     India  New Delhi
MMR   Myanmar     Yangon
THA  Thailand    Bangkok

we put the indexes 0 and 1 in a list after the comma, referring to the country and capital column when using iloc.

print(countries.iloc[[0,1,2,],[0,1]])

output:
      country    capital
IND     India  New Delhi
MMR   Myanmar     Yangon
THA  Thailand    Bangkok

Selecting Columns using `iloc`

Finally, you can keep all rows and keep only the country and capital column in a similar fashion. With loc, this is how it's done.

print(countries.loc[:,['country', 'capital']])

output:
       country    capital
IND      India  New Delhi
MMR    Myanmar     Yangon
THA   Thailand    Bangkok
SGP  Singapore  Singapore
CHN      China    Beijing

For iloc, it's like this.

print(countries.iloc[:,[0,1]])

output:
       country    capital
IND      India  New Delhi
MMR    Myanmar     Yangon
THA   Thailand    Bangkok
SGP  Singapore  Singapore
CHN      China    Beijing

loc and iloc are pretty similar, the only difference is how we refer to columns and rows. We aced indexing and selecting data from Pandas DataFrames!

Update data in a DataFrame

Updating data in dataframe is similar to selecting data from dataframe. First we select the data we want to update and assign it with new data. In the following we will try to update Country Name Myanmar to Myanmar(Burma). Note that we can do it using loc or iloc.

# Before updateing data
print(countries)

output:
       country    capital  population
IND      India  New Delhi  1393409030
MMR    Myanmar     Yangon    54806010
THA   Thailand    Bangkok    69950840
SGP  Singapore  Singapore     5453570
CHN      China    Beijing  1412360000

Change Myanmar to Myanmar(Burma)

# Update data using loc
countries.loc[['MMR'], ['country']] = 'Myanmar(Burma)'
print(countries)

output:
            country    capital  population
IND           India  New Delhi  1393409030
MMR  Myanmar(Burma)     Yangon    54806010
THA        Thailand    Bangkok    69950840
SGP       Singapore  Singapore     5453570
CHN           China    Beijing  1412360000

Change Myanmar(Burma) to Myanmar

# Update data using iloc
countries.iloc[[1], [0]] = 'Myanmar'
print(countries)

output:
       country    capital  popualation
IND      India  New Delhi   1393409030
MMR    Myanmar     Yangon     54806010
THA   Thailand    Bangkok     69950840
SGP  Singapore  Singapore      5453570
CHN      China    Beijing   1412360000

Delete data in DataFrame

During cleaning a dataset, we might want to remove some row of data from a dataframe. We can do it by using the drop method on the dataframe. Let's try to remove China row from dataframe.

# Before delete/drop data
print(countries)

output:
            country    capital  population
IND           India  New Delhi  1393409030
MMR  Myanmar(Burma)     Yangon    54806010
THA        Thailand    Bangkok    69950840
SGP       Singapore  Singapore     5453570
CHN           China    Beijing  1412360000

# we pass ['CHN'], telling we want to remove row/column related to 'CHN'
# axis=0 means,we want to drop row(s)
# inplace=True means dropping takes place on original data
countries.drop(['CHN'], axis=0, inplace=True)
print(countries)

output:
            country    capital  population
IND           India  New Delhi  1393409030
MMR  Myanmar(Burma)     Yangon    54806010
THA        Thailand    Bangkok    69950840
SGP       Singapore  Singapore     5453570

Printing countries shows that the data row we want to remove is no longer in the dataframe countries. Next let's try to remove a column population from dataframe.

# we pass ["population"], telling we want to remove row/column related to "population"
# axis=1 means,we want to drop column(s)
# inplace=True means dropping takes place on original data
countries.drop(["population"], axis=1, inplace=True)
print(countries)

output:
            country    capital
IND           India  New Delhi
MMR  Myanmar(Burma)     Yangon
THA        Thailand    Bangkok
SGP       Singapore  Singapore

As we expected, the column population is dropped from the dataframe.

Add data to DataFrame

What if we want to add data to a datafame. We can do it using square brackets[]. Let's try to add the popualation data we dropped in the previous one.

# before adding data
print(countries)

output:
            country    capital
IND           India  New Delhi
MMR  Myanmar(Burma)     Yangon
THA        Thailand    Bangkok
SGP       Singapore  Singapore

# Add population column data
# the length of column data need to be same as the number of the rows in dataframe
countries["population"] = [1393409030,54806010,69950840,5453570]
print(countries)

output:
            country    capital  population
IND           India  New Delhi  1393409030
MMR  Myanmar(Burma)     Yangon    54806010
THA        Thailand    Bangkok    69950840
SGP       Singapore  Singapore     5453570

Great! Do note that pandas does not know which population data belong to which country and will add the data in the order we give. Now, let's add our China data row back to the dataframe countries. Since our data having index label CHN, we need to add using loc.

countries.loc['CHN'] = ['China', 'Beijing', 1412360000]
print(countries)

output:
            country    capital  population
IND           India  New Delhi  1393409030
MMR  Myanmar(Burma)     Yangon    54806010
THA        Thailand    Bangkok    69950840
SGP       Singapore  Singapore     5453570
CHN           China    Beijing  1412360000

Super!! Now we mastered how to create, select, add, update, delete data in Python dictionaries and Pandas dataframes.

See the original article.

Connect & Discuss with us on LinkedIn

Create Plots using Matplotlib

PyProDev — Thu, 10 Nov 2022 06:18:11 +0000

Data visualization is a key skill for aspiring data scientists. Matplotlib makes it easy to create meaningful and insightful plots.

In this article, we’ll see how to build line plots, scatter plots, histograms and customize them to be more visually appealing.

Data used in this article, are created based on the data which is available at World Development Indicators | DataBank.

Line plot

With matplotlib, we can create a bunch of different plots in Python. The most basic plot is the line plot.

Example:

import matplotlib.pyplot as plt
plt.plot(x,y)
plt.show()

We have to import matplotlib.pyplot as plt. pyplot is a sub-package of matplotlib.
Use plt.plot() to build a line plot. years will be mapped on the horizontal axis, world_population_billion on the vertical axis.
Then plt.show() function to actually display the plot.

years = [ '2011','2012','2013','2014','2015','2016','2017','2018','2019','2020','2021']
world_population_billion = [7, 7.09, 7.17, 7.26, 7.35, 7.43, 7.52, 7.6, 7.68, 7.76, 7.84]
# Import matplotlib.pyplot as plt
import matplotlib.pyplot as plt
print('World pouplation at', years[-1], 'is', world_population_billion[-1])
# Make a line plot: years on the x-axis, world_population_billion on the y-axis
plt.plot(years, world_population_billion)

# Display the plot with plt.show()
plt.show()

Now that we've built our first line plot, let's work on the next data.

life_expectancy which contains the life expectancy(2020) for each country and
gdp_per_capita, which contains the GDP per capita(2020) for each country expressed in US Dollars.

GDP stands for Gross Domestic Product. It basically represents the size of the economy of a country. Divide this by the population and you get the GDP per capita.

life_expectancy = [65.173, 78.686, 77.063, 61.487, 77.146, 76.813, 75.224, 76.434, 83.2, 81.19268293, 73.123, 74.053, 77.419, 72.868, 79.308, 74.22682927, 80.79512195, 74.754, 62.077, 72.08, 71.771, 77.545, 69.793, 76.084, 75.998, 73.60731707, 61.981, 61.916, 73.166, 70.054, 59.626, 81.74878049, 53.679, 54.505, 80.329, 77.097, 77.46, 64.525, 60.971, 64.804, 80.465, 58.104, 77.72439024, 78.892, 81.135, 78.22682927, 81.55121951, 67.49, 74.257, 77.216, 72.15, 73.533, 59.057, 78.34634146, 60.721, 66.953, 67.561, 82.13170732, 82.17560976, 77.836, 66.69, 62.383, 73.919, 80.94146341, 64.347, 81.08780488, 72.426, 80.277, 74.529, 61.962, 58.634, 70.023, 64.315, 75.448, 85.38780488, 75.61707317, 83.06585366, 69.887, 71.908, 76.87, 70.748, 82.20487805, 82.34390244, 74.586, 84.61560976, 74.655, 71.37, 66.991, 68.611, 83.42682927, 75.586, 71.8, 68.219, 75.38780488, 79.004, 54.836, 64.423, 73.082, 74.92926829, 81.74146341, 84.37, 67.39, 64.694, 76.306, 79.208, 59.692, 82.65365854, 65.129, 74.17707317, 75.131, 68.002, 72.006, 70.056, 75.93170732, 76.901, 61.387, 67.363, 64.045, 71.067, 81.4097561, 77.77117073, 82.05609756, 74.697, 62.792, 55.018, 75.69317073, 83.2097561, 78.078, 67.428, 78.68, 64.725, 74.363, 76.947, 71.36, 76.6, 80.97560976, 80.08746341, 80.363, 74.35365854, 71.33878049, 69.329, 73.45, 70.583, 75.28, 68.213, 55.066, 83.74390244, 76.86585366, 80.53170732, 73.132, 57.697, 64.379, 82.33414634, 77.144, 76.343, 72.658, 65.53, 71.802, 82.40731707, 83.1, 71.301, 65.815, 77.344, 69.712, 61.34, 71.018, 73.628, 76.891, 77.928, 63.713, 71.18512195, 78.12, 80.90243902, 77.2804878, 78.056, 71.848, 70.623, 75.493, 79.8195122, 66.181, 64.194, 61.738]
gdp_per_capita = [20116137326.0, 15131866271.0, 145000000000.0, 53619071176.0, 1370281481.0, 390000000000.0, 12641209802.0, 2496648045.0, 1330000000000.0, 433000000000.0, 42693000000.0, 9699500000.0, 34723357447.0, 374000000000.0, 4689534962.0, 61489588895.0, 522000000000.0, 1585631670.0, 15651545332.0, 2315436303.0, 36629843806.0, 19955120005.0, 14930072799.0, 1450000000000.0, 12005825770.0, 69889347433.0, 17933606353.0, 2780510625.0, 1703698677.0, 25872798012.0, 40804449726.0, 1650000000000.0, 2326720921.0, 10715396135.0, 253000000000.0, 14700000000000.0, 270000000000.0, 1223876065.0, 48716960860.0, 10483151094.0, 62158002233.0, 61348579465.0, 57203783203.0, 107000000000.0, 24692095945.0, 245000000000.0, 356000000000.0, 3181071154.0, 78844702329.0, 99291124000.0, 365000000000.0, 24638720000.0, 10099157270.0, 30650285472.0, 3984840580.0, 108000000000.0, 4574413823.0, 272000000000.0, 2630000000000.0, 5669368717.0, 15316824039.0, 1830413000.0, 15842922533.0, 3850000000000.0, 70043199814.0, 189000000000.0, 1043411111.0, 5844000000.0, 77604632621.0, 14169626010.0, 1431758243.0, 5471256595.0, 14508218017.0, 23827840810.0, 345000000000.0, 157000000000.0, 21718075725.0, 2670000000000.0, 1060000000000.0, 232000000000.0, 184000000000.0, 426000000000.0, 1890000000000.0, 13812425037.0, 5040000000000.0, 43697659296.0, 171000000000.0, 101000000000.0, 180911843.6, 1640000000000.0, 106000000000.0, 7780874537.0, 18981800705.0, 33645460617.0, 25948915861.0, 2250717718.0, 3039982540.0, 52320215472.0, 56546957475.0, 73353132794.0, 25586111076.0, 13225591804.0, 12172128470.0, 337000000000.0, 3742769967.0, 17465392779.0, 14917038504.0, 7915985514.0, 10926820603.0, 1090000000000.0, 407515928.0, 11859730544.0, 13312981595.0, 4780722122.0, 115000000000.0, 14028811072.0, 78930257227.0, 10562637376.0, 33433670512.0, 914000000000.0, 9435529927.0, 212000000000.0, 12586941393.0, 13741378450.0, 432000000000.0, 12116981815.0, 362000000000.0, 73971391417.0, 300000000000.0, 53977037000.0, 24667052023.0, 35432178068.0, 202000000000.0, 362000000000.0, 597000000000.0, 229000000000.0, 103000000000.0, 144000000000.0, 250000000000.0, 1490000000000.0, 10184345442.0, 807147527.6, 472914469.9, 703000000000.0, 24493157583.0, 4063289450.0, 345000000000.0, 105000000000.0, 53589609581.0, 1545888426.0, 6965285325.0, 335000000000.0, 1280000000000.0, 80969683537.0, 1616772748.0, 872134546.8, 26987563444.0, 2884248048.0, 541000000000.0, 752000000000.0, 8133996648.0, 62409709111.0, 500000000000.0, 1902156800.0, 7574636979.0, 488829964.1, 21392536138.0, 42514151614.0, 720000000000.0, 37600368181.0, 157000000000.0, 359000000000.0, 2760000000000.0, 20900000000000.0, 53560755047.0, 59894305353.0, 896827873.1, 343000000000.0, 4204000000.0, 18840511908.0, 18110631358.0, 18051170799.0]

Build a line chart, with life_expectancy on the x-axis, and gdp_per_capita on the y-axis. Does it make sense to plot this data on a line plot?

# Make a line plot, gdp_per_capita on the x-axis, life_expectancy on the y-axis
plt.plot(gdp_per_capita, life_expectancy)

# Display the plot
plt.show()

Well, this doesn't look right. Let's build a plot that makes more sense.

Scatter Plot

When we have a time scale along the horizontal axis, the line plot is good to go most of the time. But in many other cases, when we're trying to assess if there's a correlation between two variables, for example, the scatter plot is the better choice.

Example:

import matplotlib.pyplot as plt
plt.scatter(x,y)
plt.show()

Let's continue with the gdp_per_capita versus life_expectancy plot, the GDP and life expectancy data for different countries in 2020. Let's see if a scatter plot is a better alternative?

# Use a scatter plot
plt.scatter(gdp_per_capita, life_expectancy)

# Show plot
plt.show()

Great! The scatter plot looks much better than line plot in this case.

A correlation will become clear when we display the GDP per capita on a logarithmic scale using plt.xscale('log').

# Use a scatter plot
plt.scatter(gdp_per_capita, life_expectancy)

# Put the x-axis on a logarithmic scale
plt.xscale('log')

# Show plot
plt.show()

It looks like the higher GDP usually corresponds to a higher life expectancy. Is there a relationship between population and life expectancy of a country?

population = [38928341.0, 2837849.0, 43851043.0, 32866268.0, 97928.0, 45376763.0, 2963234.0, 106766.0, 25693267.0, 8916864.0, 10093121.0, 393248.0, 1701583.0, 164689383.0, 287371.0, 9379952.0, 11544241.0, 397621.0, 12123198.0, 771612.0, 11673029.0, 3280815.0, 2351625.0, 212559409.0, 437483.0, 6934015.0, 20903278.0, 11890781.0, 555988.0, 16718971.0, 26545864.0, 38037204.0, 4829764.0, 16425859.0, 19116209.0, 1411100000.0, 50882884.0, 869595.0, 89561404.0, 5518092.0, 5094114.0, 26378275.0, 4047680.0, 11326616.0, 1207361.0, 10697858.0, 5831404.0, 988002.0, 10847904.0, 17643060.0, 102334403.0, 6486201.0, 1402985.0, 1329479.0, 1160164.0, 114963583.0, 896444.0, 5529543.0, 67379908.0, 280904.0, 2225728.0, 2416664.0, 3722716.0, 83160871.0, 31072945.0, 10700556.0, 112519.0, 168783.0, 16858333.0, 13132792.0, 1967998.0, 786559.0, 11402533.0, 9904608.0, 7481000.0, 9750149.0, 366463.0, 1380004385.0, 273523621.0, 83992953.0, 40222503.0, 4985674.0, 59449527.0, 2961161.0, 126261000.0, 10203140.0, 18755666.0, 53771300.0, 119446.0, 51836239.0, 4270563.0, 6579900.0, 7275556.0, 1900449.0, 6825442.0, 2142252.0, 5057677.0, 6871287.0, 2794885.0, 630419.0, 649342.0, 27691019.0, 19129955.0, 32365998.0, 540542.0, 20250834.0, 515332.0, 4649660.0, 1265740.0, 128932753.0, 115021.0, 2620495.0, 3278292.0, 621306.0, 36910558.0, 31255435.0, 54409794.0, 2540916.0, 29136808.0, 17441500.0, 271960.0, 5090200.0, 6624554.0, 24206636.0, 206139587.0, 2072531.0, 5379475.0, 5106622.0, 220892331.0, 4314768.0, 8947027.0, 7132530.0, 32971846.0, 109581085.0, 37899070.0, 10297081.0, 3281538.0, 2881060.0, 19257520.0, 144073139.0, 12952209.0, 198410.0, 219161.0, 34813867.0, 16743930.0, 7976985.0, 5685807.0, 5458827.0, 2102419.0, 686878.0, 15893219.0, 59308690.0, 47363419.0, 21919000.0, 183629.0, 110947.0, 43849269.0, 586634.0, 10353442.0, 8636561.0, 9537642.0, 59734213.0, 69799978.0, 1318442.0, 8278737.0, 105697.0, 1399491.0, 11818618.0, 84339067.0, 45741000.0, 44132049.0, 9890400.0, 67081000.0, 331501080.0, 3473727.0, 34232050.0, 307150.0, 97338583.0, 106290.0, 29825968.0, 18383956.0, 14862927.0]

# Build Scatter plot
plt.scatter(population, life_expectancy)

# Show plot
plt.show()

Let's see the plot with x axis in a logarithmic scale.

# Build Scatter plot
plt.scatter(population, life_expectancy)

# Put the x-axis on a logarithmic scale
plt.xscale('log')

# Show plot
plt.show()

There's no clear relationship between population and life expectancy, which makes perfect sense.

Histogram

The histogram is a type of visualization that's very useful to explore the data. It can help use to get an idea about the distribution of our variables.

To see how life expectancy in different countries is distributed, let's create a histogram of life_expectancy using plt.hist().

# Create histogram of life_expectancy data
plt.hist(life_expectancy)

# Display histogram
plt.show()

In the above code, we didn't specify the number of bins. By default, Python sets the number of bins to 10 in that case. The number of bins is pretty important.

Too few bins will oversimplify reality and won't show you the details.
Too many bins will overcomplicate reality and won't show the bigger picture.

To control the number of bins to divide your data in, you can set the bins argument.

We'll create two plots specifying bins.

Build a histogram of life_expectancy, with 5 bins. Can you tell which bin contains the most observations?

# Build histogram with 5 bins
plt.hist(life_expectancy, bins=5)

# Show and clean up plot
plt.show()

Build another histogram of life_expectancy, this time with 20 bins. Is this better?

# Build histogram with 20 bins
plt.hist(life_expectancy, bins=20)

# Show and clean up again
plt.show()

Compare using histograms

Histograms are helpful in doing comparisons. life_expectancy contains life expectancy data for different countries in 2020. life_expectancy_1960, containing similar data for 1960. Let's make a histogram for both datasets.

Build a histogram of life_expectancy with 15 bins.
Build a histogram of life_expectancy_1960, also with 15 bins.

and let's see is there any pattern?

life_expectancy_1960 = [32.446, 62.283, 46.141, 37.524, 61.968, 65.055, 65.972, 65.662, 70.81707317, 68.58560976, 61.034, 64.74, 51.869, 45.379, 64.472, 67.70809756, 69.70195122, 59.981, 37.271, 34.526, 41.82, 60.353, 49.179, 54.143, 54.81, 69.24756098, 34.432, 41.281, 48.461, 41.242, 41.785, 71.13317073, 36.249, 38.02, 57.219, 43.725, 57.269, 41.447, 41.098, 45.721, 60.381, 36.095, 64.60865854, 63.834, 69.618, 70.34878049, 72.17658537, 44.038, 51.602, 52.982, 48.042, 49.95, 36.535, 67.90290244, 43.572, 38.419, 60.811, 68.8197561, 69.86829268, 56.282, 39.694, 32.054, 63.651, 69.31002439, 45.843, 68.16390244, 62.231, 60.97, 46.702, 34.89, 37.478, 60.26, 41.762, 46.274, 66.96168293, 68.00317073, 73.42317073, 41.422, 46.664, 44.947, 48.022, 69.7965122, 69.12390244, 64.77, 67.66609756, 52.651, 58.36758537, 46.76, 47.061, 55.41553659, 59.343, 56.12807317, 43.204, 69.78682927, 63.267, 47.919, 34.264, 42.609, 69.84731707, 68.44639024, 64.828, 39.962, 36.672, 59.991, 37.343, 28.199, 69.4332439, 44.432, 58.74521951, 57.077, 54.513, 61.995, 48.392, 63.70560976, 48.458, 39.439, 42.381, 46.483, 35.583, 73.39268293, 58.63902439, 71.23658537, 46.998, 35.053, 36.976, 60.62280488, 73.5497561, 42.672, 45.299, 60.864, 38.935, 63.881, 48.012, 61.105, 67.6804878, 63.27290244, 68.71960976, 61.094, 65.64243902, 66.05529268, 42.616, 56.902, 50.378, 45.638, 38.223, 31.566, 65.65982927, 69.92365854, 68.97804878, 48.123, 36.976, 48.406, 69.10926829, 59.369, 56.739, 59.26, 48.194, 59.682, 73.00560976, 71.31341463, 50.613, 43.6, 54.701, 33.729, 40.297, 59.885, 62.222, 42.021, 45.369, 44.359, 68.29953659, 51.537, 71.12682927, 69.77073171, 67.783, 58.835, 48.975, 59.039, 66.22485366, 29.919, 46.687, 53.019]

# Histogram of life_expectancy_1960, 15 bins
plt.hist(life_expectancy_1960, bins=15)

# Show and clear plot
plt.show()

# Histogram of life_expectancy, 15 bins
plt.hist(life_expectancy, bins=15)

# Show and clear plot
plt.show()

By comparing 2 histograms, we can see that most of life expectancies in 1960 are lower compared to most of life expectancies in 2020.

Does life expectancies become higer as the world become more advanced in health care system?

Customizing Plots

Creating a plot is great. Making the correct plot, that makes the message very clear, is the real challenge.

For each visualization, we have many options. First of all, there are the different plot types. And for each plot, you can do an infinite number of customizations.

You can change

colors,
shapes,
labels,
axes, and so on.

The choice depends on

the data, and
the story you want to tell with the data.

There are so many possible customizations. Let's see the code script which we wrote to build a simple line plot of world population from 2011 to 2021.

# Make a line plot: years on the x-axis, world_population_billion on the y-axis
plt.plot(years, world_population_billion)

# Display the plot with plt.show()
plt.show()

We already get a pretty nice plot.But some things can be improved. It should be clear which data we are displaying, especially to people who are seeing the graph for the first time.

Label the axes

The first thing you always need to do is label your axes.

Let's do this by adding the xlabel and ylabel functions. As inputs, we pass strings that should be placed alongside the axes. We have to call these functions before calling the show function, otherwise our customizations will not be displayed.

# Make a line plot: years on the x-axis, world_population_billion on the y-axis
plt.plot(years, world_population_billion)

plt.xlabel('Year')
plt.ylabel('Population')

# Display the plot with plt.show()
plt.show()

Add Title

We're also going to add a title to our plot, with the title function. We pass the actual title, 'World Population Plot', as an argument.

# Make a line plot: years on the x-axis, world_population_billion on the y-axis
plt.plot(years, world_population_billion)

plt.xlabel('Year')
plt.ylabel('Population')
plt.title('World Population Plot')

# Display the plot with plt.show()
plt.show()

Now we can give readers more information about the data on the plot, telling what the plot is about.

Ticks

We can also customize the y-axis.We can do this with the yticks function. First specify the list of ytick locations.

# Make a line plot: years on the x-axis, world_population_billion on the y-axis
plt.plot(years, world_population_billion)

plt.xlabel('Year')
plt.ylabel('Population')
plt.title('World Population Plot')

plt.yticks(ticks=[7,7.1,7.2,7.3,7.4,7.5,7.6,7.7,7.8,7.9,],)

# Display the plot with plt.show()
plt.show()

The reslut showing ticks exactly at the positons we specified.

We want to make it clear we're talking about billions, we can add a second argument to the yticks function, which is a list with the display names of the ticks. This list(labels) should have the same length as the first list(ticks).

# Make a line plot: years on the x-axis, world_population_billion on the y-axis
plt.plot(years, world_population_billion)

plt.xlabel('Year')
plt.ylabel('Population')
plt.title('World Population Plot')

plt.yticks(ticks=[7,7.1,7.2,7.3,7.4,7.5,7.6,7.7,7.8,7.9, ],
           labels=['7 B','7.1 B','7.2 B','7.3 B','7.4 B','7.5 B','7.6 B','7.7 B','7.8 B','7.9 B', ])

# Display the plot with plt.show()
plt.show()

The labels changed accordingly. Awesome!

Sizes

We have seen that the scatter plot is just a cloud of blue dots, indistinguishable from each other. Wouldn't it be nice if we can set the size of the dots corresponds to the population? We can do that by using the argument s, for size. dot_size_list is a list containing size of each point scaled to each country's population.

dot_size_list = [item*2/1000000 for item in population]

plt.scatter(gdp_per_capita, life_expectancy, s = dot_size_list)

# plt.xscale('log') 
plt.xlabel('GDP per Capita in USD')
plt.ylabel('Life Expectancy in years')
plt.title('World Development in 2020')

# Display the plot
plt.show()

Now the dots have their own sizes but the plot is still difficult to observe.

Color

Next we will make the plot more colorful!
dot_color is a list with a color for each corresponding country, depending on the continent the country is part of. dot_color is already created for each country using the following continent_colors mapping.

continent_colors = {
    'Asia':'red',
    'Europe':'green',
    'Africa':'blue',
    'Americas':'yellow',
    'Oceania':'black'
}

Add c = dot_color to the arguments of the plt.scatter() function.
Change the opacity of the bubbles by setting the alpha argument to 0.8 inside plt.scatter(). Alpha can be set from zero to one, where zero is totally transparent, and one is not at all transparent.

dot_color = ['red', 'green', 'blue', 'blue', 'yellow', 'yellow', 'red', 'yellow', 'black', 'green', 'red', 'yellow', 'red', 'red', 'yellow', 'green', 'green', 'yellow', 'blue', 'red', 'yellow', 'green', 'blue', 'yellow', 'red', 'green', 'blue', 'blue', 'blue', 'red', 'blue', 'yellow', 'blue', 'blue', 'yellow', 'red', 'yellow', 'blue', 'blue', 'blue', 'yellow', 'blue', 'green', 'yellow', 'red', 'green', 'green', 'blue', 'yellow', 'yellow', 'blue', 'yellow', 'blue', 'green', 'blue', 'blue', 'black', 'green', 'green', 'black', 'blue', 'blue', 'red', 'green', 'blue', 'green', 'yellow', 'black', 'yellow', 'blue', 'blue', 'yellow', 'yellow', 'yellow', 'red', 'green', 'green', 'red', 'red', 'red', 'red', 'green', 'green', 'yellow', 'red', 'red', 'red', 'blue', 'black', 'red', 'red', 'red', 'red', 'green', 'red', 'blue', 'blue', 'blue', 'green', 'green', 'red', 'blue', 'blue', 'red', 'red', 'blue', 'green', 'blue', 'blue', 'yellow', 'black', 'green', 'red', 'green', 'blue', 'blue', 'red', 'blue', 'red', 'green', 'black', 'black', 'yellow', 'blue', 'blue', 'green', 'green', 'red', 'red', 'yellow', 'black', 'yellow', 'yellow', 'red', 'green', 'green', 'yellow', 'red', 'green', 'green', 'blue', 'black', 'blue', 'red', 'blue', 'blue', 'red', 'green', 'green', 'black', 'blue', 'blue', 'green', 'red', 'yellow', 'yellow', 'blue', 'yellow', 'green', 'green', 'red', 'blue', 'red', 'red', 'blue', 'black', 'yellow', 'blue', 'red', 'blue', 'green', 'red', 'green', 'yellow', 'yellow', 'red', 'black', 'red', 'yellow', 'red', 'blue', 'blue']

# Specify c and alpha inside plt.scatter()
plt.scatter(gdp_per_capita, life_expectancy, s = dot_size_list, c=dot_color, alpha=0.8)

# Previous customizations
plt.xlabel('GDP per Capita in USD')
plt.ylabel('Life Expectancy in years')
plt.title('World Development in 2020')

# Show the plot
plt.show()

let's scale the x axis into log to get another point of view.

# Specify c and alpha inside plt.scatter()
plt.scatter(gdp_per_capita, life_expectancy, s = dot_size_list, c=dot_color, alpha=0.8)

# Previous customizations
plt.xscale('log') 
plt.xlabel('GDP per Capita in USD(log scaled)')
plt.ylabel('Life Expectancy in years')
plt.title('World Development in 2020')

# Show the plot
plt.show()

Interesting. We can see the plot more clearly in this plot with log-scaled x-axis.

Display text on the plot

Now we will display text on the plot by specifying x, y positions as arguments using text function.

# Specify c and alpha inside plt.scatter()
plt.scatter(gdp_per_capita, life_expectancy, s = dot_size_list, c=dot_color, alpha=0.8)

# Previous customizations
plt.xscale('log') 
plt.xlabel('GDP per Capita in USD(log scaled)')
plt.ylabel('Life Expectancy in years')
plt.title('World Development in 2020')

# China
plt.text(1.470000e+13, 77.097, 'CHN')
# United state
plt.text(2.090000e+13, 77.280488, 'USA')
# Myanmar
plt.text(7.893026e+10, 67.363, 'MMR')

# Show the plot
plt.show()

Looks like we need to customize the size of the figure to get a better view.

Figure size

We can do it by specifying figsize=[Width inches, height inches] in plt.figure().

plt.figure(figsize=[14,6])
# Specify c and alpha inside plt.scatter()
plt.scatter(gdp_per_capita, life_expectancy, s = dot_size_list, c=dot_color, alpha=0.8)

# Previous customizations
plt.xscale('log') 
plt.xlabel('GDP per Capita in USD(log scaled)')
plt.ylabel('Life Expectancy in years')
plt.title('World Development in 2020')

# China
plt.text(1.470000e+13, 77.097, 'CHN')
# United state
plt.text(2.090000e+13, 77.280488, 'USA')
# Myanmar
plt.text(7.893026e+10, 67.363, 'MMR')

# Show the plot
plt.show()

Here we notice that the countries in blue, located in Africa, have both low life expectancy and a low GDP per capita.

Beautiful! A visualization only makes sense if we can interpret it properly.

In this article we learned how to plot line, scatter, histogram using matplotlib and how to customize them.

See the original article.

Connect & Discuss with us on LinkedIn

What are Jupyter Notebooks?

PyProDev — Thu, 03 Mar 2022 14:57:39 +0000

Welcome to this section using Jupyter notebooks. The notebook is a web application that allows you to combine explanatory text, math equations, code, and visualizations all in one easily sharable document. You can view a few more tutorial notebooks at Gravitational Wave Open Science Center homepage.
Notebooks have quickly become an essential tool when working with data.

You'll find them being used for data cleaning and exploration, visualization, machine learning, and big data analysis.

Typically you'd be doing this work in a terminal, either the normal Python shell or with IPython. Your visualizations would be in separate windows, any documentation would be in separate documents, along with various scripts for functions and classes. However, with notebooks, all of these are in one place and easily read together.

Notebooks are also rendered automatically on GitHub. It’s a great feature that lets you easily share your work. There is also http://nbviewer.jupyter.org/ that renders the notebooks from your GitHub repo or from notebooks stored elsewhere.

Literate Programming

Notebooks are a form of literate programming proposed by Donald Knuth in 1984. With literate programming, the documentation is written as a narrative alongside the code instead of sitting off by its own. In Donald Knuth's words,

Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do.

After all, code is written for humans, not for computers. Notebooks provide exactly this capability. You are able to write documentation as narrative text, along with code. This is not only useful for the people reading your notebooks, but for your future self coming back to the analysis.

Just a small aside: recently, this idea of literate programming has been extended to a whole programming language, Eve.

Forem: PyProDev

Mastering Logical Comparison, Control Flow, Filtering on Numpy Array and Pandas DataFrame

Comparison Operators

Numeric comparisons

Comparison between strings

Comparison between integer and string

Comparison between integer and float

Compare on Numpy array

Comparators

Equality

Greater and less than

Boolean Operators

and

or

not

Nested Boolean operators

Filtering on NumPy arrays

Boolean operators on NumPy Array

Filtering on pandas DataFrames

Step 1: Get the column

Step 2: Compare

Step 3: Subset the DataFrame

Summary

Boolean operators on Pandas DataFrame

Control Flow

if

else

elif

Mastering Python Dictionaries & Pandas DataFrames

List

Motivation for Dictionaries

Dictionary

Create a Dictionary

Manipulating a Dictionary

Add data to a Dictionary

Update data in a Dictionary

Delete data from a Dictionary

List vs Dictionary

Nested Dictionaries

Tabular dataset examples

Datasets in Python

Datasets containing different data types

Pandas

DataFrame

Create a DataFrame from Dictionary

Create a DataFrame from CSV file

Indexing and selecting data in DataFrames

Access data using square brackets [ ]

Access data using loc

Selecting Rows using loc

Selecting Rows & Columns using loc

Selecting Columns using loc

Access data using iloc

Selecting Rows using iloc

Selecting Rows & Columns using iloc

Selecting Columns using iloc

Update data in a DataFrame

Delete data in DataFrame

Add data to DataFrame

Create Plots using Matplotlib

Line plot

Scatter Plot

Histogram

Compare using histograms

Customizing Plots

Label the axes

Add Title

Ticks

Sizes

Color

Display text on the plot

Figure size

What are Jupyter Notebooks?

Literate Programming

`and`

`or`

`not`

Access data using `loc`

Selecting Rows using `loc`

Selecting Rows & Columns using `loc`

Selecting Columns using `loc`

Access data using `iloc`

Selecting Rows using `iloc`

Selecting Rows & Columns using `iloc`

Selecting Columns using `iloc`