Forem: Cezar Peixeiro

Why Python lists and AND / OR operators return "stranger things"?

Cezar Peixeiro — Thu, 06 Feb 2020 04:33:53 +0000

Let's suppose that you're working on a Python project and decide to use the Boolean operators and and or to compare some lists. After that, you have a heart attack when you get these outputs:

lst1 = [False, False, False]
lst2 = [True, False, True]

print(lst1 and lst2)
>>> [True, False, True]

You begin to doubt of your sanity, because you're expecting at least a [False, False, False] list (the output if both lists were compared by indexes) and begin to look for a psychiatrist!!

HEY! Take it easy!!! This is not the way that and and or work with lists. Let me explain you this secret:

When you compare lists with these operators you're not comparing the values but if the list really has values inside it. It sounds a bit confusing, so let's see some examples:

Let's create five lists: 2 of them with mixed values, 2 with same values and one empty and then run the bool() function to check the Boolean value of them.

lst1 = [True, False, True]
lst2 = [False, True, False]
lst3 = [False, False, False]
lst4 = [True, True, True]
lst5 = []

print(bool(lst1), bool(lst2), bool(lst3), bool(lst4), bool(lst5))
>>>True True True True False

We can see that all lists return True except the empty one!
Now let's understand how and and or work:

and: when using it, the operator will evaluate the first element of the comparison. If it's True than is returned the second element, if False than is returned the first element itself.

#all integers are considered True except zero
x = 1
y = 2
z = False

print(x and y)
>>> 2

print(y and x)
>>> 1

print(z and y)
>>>

or: when using it, the operator will first evaluate the first element of the comparison. If it's False than is returned the second element, if True than is returned the first element itself.

#all integers are considered True except zero
x = 1
y = 2
z = False

print(x and y)
>>> 1

print(y and x)
>>> 2

print(z and y)
>>> 2

Now you can have an idea about what happened in the first code. The first list was evaluated and returned True, so the operation returned the second element (second list)!

You must be thinking right now, "Well, just False keyword and an empty string are considered False values in Python?"

Answer: NOOO!!!
According to the official reference are considered False values:

"False, None, numeric zero of all types, and empty strings and containers (including strings, tuples, lists, dictionaries, sets and frozensets)"

The docs also tell us an example that this behavior can be useful:

"if s is a string that should be replaced by a default value if it is empty, the expression s or 'foo' yields the desired value. Because not has to create a new value, it returns a boolean value regardless of the type of its argument (for example, not 'foo' produces False rather than ''.)"

"What if I want to compare the pairs of elements by their positions?"

To do this, you can use & (and) and | (or) operators. These operators work bitwise:

lst1 = [False, True, False, True, True]
lst2 = [True, True, False, False, True]

print(lst1 & lst2)
>>> [False, True, False, False, True]

print(lst1 | lst2)
>>> [True, True, False, True, True]

Hope these tips can be useful! See ya!

Pandas 1.0.0 - jan/2020 - What's new?

Cezar Peixeiro — Sun, 02 Feb 2020 05:16:28 +0000

Pandas is a famous Python library for data wrangling and in January-2020 its development team made a major release! YES, big changes were made in this lib, what made it jump from 0.25 to 1.0.0 version.

I just read the official release dosc and wrote above what important changes were made:

IMPROVEMENTS

1. DataFrame.apply() method can use Numba engine!

Numba is a JIT compiler project that can translate a subset of Python code in optimized machine code using the LLVM compiler library. In other words, Numba creates faster Python codes. (You can apply Numba in your general code importing and embedding it as a decorator. Read more in the official docs)

The official release says:
"Using the Numba engine can yield significant performance gains if the apply function can operate on numpy arrays and the data set is larger (1 million rows or greater)"

When using DataFrame.rolling.apply() or DataFrame.expanding.apply() until now, the processing cost was huge. Now, we can pass engine = "numba" to the apply method and have an increase in the performance as follows:

import pandas as pd #pandas 1.0.0 version

data = pd.Series(range(1_000_000))
roll = data.rolling(10)

def f(x):
    return np.sum(x) + 5

# Running in Jupyter Notebook
# Run the first time, compilation time will affect performance
In [4]: %%timeit -r 1 -n 1 
        roll.apply(f, engine='numba', raw=True)
Out [4]: 1.23 s ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)

# Function is cached and performance will improve
In [5]: %%timeit 
        roll.apply(f, engine='numba', raw=True)
Out [5]: 188 ms ± 1.93 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [6]: %%timeit 
        roll.apply(f, engine='cython', raw=True)
Out [6]: 3.92 s ± 59 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

DataFrame.to_markdown()

This new method is just an easy way to output dataframes in markdown format. To illustrate this I created a simple dataframe below and applied the method.

df = pd.DataFrame(data={"col_1": ["a", "b"], "col_2": ["c", "d"]})

print(df.to_markdown())
>>> |    | col_1 | col_2 |
    |---:|:------|:------|
    |  0 | a     | c     |
    |  1 | b     | d     |

EXPERIMENTAL NEW FEATURES

This kind of changes has its use optional and is available for the community evaluation. Besides this, some experimental features are really cool and useful...and can make some work and learning easier. Let's see:

pandas.NA value

When dealing with missing values in our dataframes is really common the usage of numpy.nan to represent them. Pandas now has its own missing value data type. Creating a pandas series with a None value with pandas 1.0.0, we'll see something like this:

import pandas as pd #pandas 1.0.0 version
s = pd.Series([1, 2, None], dtype="Int64")

print(s)

>>> 0       1
    1       2
    2    <NA>    
    Length: 3, dtype: Int64

and if we print the type of s[2], we'll get the following answer:

print(type(s[2]))
>>> pandas._libs.missing.NAType

Comparisons

Working with the comparison operators ==, >, >=, <, <=, between numpy.nan and numbers the result is always False, but when using !=, Python returns True
With the new pandas.NA, the value will be propagated with all the comparison operators. So, all operations with return

Logical

Working with logical operators with numpy.nan is not supported.
To pandas.NA some different outputs will occur:

import pandas as pd #pandas 1.0.0 version

print(pd.NA & False)
>>> False

print(pd.NA & True)
>>> <NA>

print(pd.NA | False)
>>> <NA>

print(pd.NA | True)
>>> True

String and Boolean Data Types

When analyzing a pandas dataframe dtype with strings the usual result is an OBJECT TYPE. But an object type column can hold more than one data type and make the analysis confusing. Now the string dtype is defined and a string column can hold only strings

import pandas as pd #pandas 1.0.0 version 
s = pd.Series(['abc', None, 'def'], dtype="string")

print(s)
>>> 0     abc
    1    <NA>
    2     def
    Length: 3, dtype: string

Different from strings a bool dtype already exists, but the columns with boolean values doesn't support missing values. This inconvenient was solved with the boolean dtype

import pandas as pd #pandas 1.0.0 version 
s = pd.Series([True, False, None], dtype="boolean")

print(s)
>>> 0     True
    1     False
    2     <NA>
    Length: 3, dtype: boolean

Well, that's all! New improvements were made in some existing functions and I really encourage you to take a look there!

Pandas #2: Opening files and using them as dataframes.

Cezar Peixeiro — Wed, 11 Dec 2019 19:49:30 +0000

(Previous post about Pandas: Pandas #1: Working with data in Python and its main structures)

Previously we saw the basic Pandas objects and an introduction about how creating them with lists and dictionaries and access singles values. But, if you want to open files (like spreadsheets) to be manipulated in dataframes. Are we able to do this?!

Of course!!! Let's see:

Opening files

Pandas has many methods that can save us from spending a lot of time coding alternatives to open files and transform them in dataframes to be manipulated. There are so many of them and you can see in Pandas' webpage a list of names of all methods to open files. Here I'll write about the two most used of them:

Open a csv file

import pandas as pd

x = pd.read_csv('my_file.csv')

This is the simplest way to do this, but if needed we have some parameters that we can pass to it to make our life easier. Let's suppose you have a tsv (tab-separated values) instead of an csv (comma-separated values). To solve this, you can pass sep to the method and declare the separator used in the file (comma is the default value):

import pandas as pd

x = pd.read_csv('my_file.csv', sep="\t") #\t means tab

If you import your file just like the cells above, the default behavior of read_csv is infer that the first line of the file refers to the name os the columns. You can pass header = None if you don't have columns names. If you want to put your own columns names, you can use the parameter names and pass the list of your names.

import pandas as pd

x = pd.read_csv('my_file.csv', header = None, names=['col1','col2,'col3'])

You can also pass a column that will be used as index in your dataframe. It's passed using index_col:

import pandas as pd

x = pd.read_csv('my_file.csv', header = None, names=['col1','col2,'col3'], 
index_col='col2')

There are many other parameters that you can use to open a csv file, but these ones are enough to start your analysis.

Open an excel file

Microsoft Windows is the most popular OS and Microsoft Office a well-known package (dragging excel with it). So, it's important to know how to open these files. Let's see:

import pandas as pd

x = pd.read_excel('my_file.xls', sheet_name='sheet2')

The most important parameters are the same as in csv version, but this time we'll include one more, the sheet_name. Here you can specify which sheet to open in your file if you have many of them. If they're not named, you can pass a integer to call it (remember that it starts in 0).

Well, that's it for now! My plan is use this DEV blog as a notebook, saving things from the most simplest and elementary to the most elaborate. My intention is to keep information easy to access.

See ya!

Pandas #1: Working with data in Python and its main structures

Cezar Peixeiro — Fri, 29 Nov 2019 21:19:52 +0000

We currently know that Python and R are the mainstream languages when talking about Data Science and also that one of the major characteristics of both languages is their huge communities and the willing of help each other. In the “Pythonic World” the common sense is the existence of a basic stack of libraries that are mandatory to anyone who want to manipulate data with speed, elegance and efficiency.

One of the most important is Pandas:

Pandas is being developed since 2008 as an open source project and for years has been saving working times of programmers and data analysts / scientists. But what makes Pandas so incredible? The heart of Pandas is its capabilities in manipulating data structures and provide tools to complex data analysis. You can find instructions to install Pandas here (obs: Pandas is dependent of Numpy).

Here you have an useful overview if you want to begin working with Pandas:

Basic Structures

Pandas has two main objects: Series and Dataframe.

SERIES

Series is an one-dimensional labeled array and holds any data type you want (int, float, list, …). The labels are called indexes and are useful to explicitly find the position of any data and manipulate it. I’ll create a simple list and use it to create an example of Pandas series:

import pandas as pd
s = pd.Series(["a", "b", "c", "d"])
print(s)

>>> 0  a
    1  b
    2  c
    3  d
    dtype: object

As you can see, using the Series() method I created a series with a 0 to 3 indexed items, but I know what you’re thinking: “Big deal..this is exactly what a traditional Python list does!”. Yes, but if you want more intuitive names to the items? Well, you can change the indexes passing a list of names to the series while creating effectively it:

s = pd.Series(["a", "b", "c", "d"], index=["first", "second", "third", "fourth"])

or calling the .index attribute

s.index = ["first", "second", "third", "fourth"]
print(s)

>>> first  a
    second b
    third  c
    fourth d
    dtype: object

Another way to create customized indexes is passing a dictionary to Series() method instead of a list. The keys will be considered as the indexes of the series:

s = pd.Series({"first": "a", "second": "b", "third": "c", "fourth": "d"})
print(s)

>>> first  a
    second b
    third  c
    fourth d
    dtype: object

And the data can be accessed as in a dictionary:

print(s.second)
>>> b

print(s['second'])
>>> b

Changing the indexes to names that are easy to remember is convenient, but that’s all. If you want to keep accessing by the original order you still being able to do it:

print(s)
>>> first  a
    second b
    third  c
    fourth d
    dtype: object
print(s[0])
>>> a

DATAFRAME

Dataframe is a two-dimensional labeled data structure with columns and rows like a spreadsheet or a table. Even as the series the dataframes can hold any kind of data. An important thing to highlight is that all columns in a dataframe are Pandas series. So, a dataframe is an association of many series as columns! I’ll create a simple dataframe using the DataFrame() method.

import numpy as np
import pandas as pd

df = pd.DataFrame([[1,2,3],[4,5,6],[7,8,9]])print(df)
>>>    0  1  2
    0  1  2  3
    1  4  5  6
    2  7  8  9

Note that, to create a dataframe is passed to the method a list of lists where each secondary list became a row and just like a series, each row and column has its indexes. But we can rename it in the same way:

df = pd.DataFrame([[1,2,3],[4,5,6],[7,8,9]],index=["first","second","third"], columns=["col1","col2","col4"])
print(df)

>>>       col1 col2 col3
    first    1    2    3
    second   4    5    6
    third    7    8    9

Access anything in a dataframe is only allowed using an index directly if you don’t give names to the columns. If you try to do this with the named dataframe above the interpreter will raise an error:

print(df[0])
>>> KeyError: 0

Directly using its name, you can access any column:

print(df["col2"])

>>> first     2
    second    5
    third     8
    Name: col2, dtype: int64

Well…and if you want to access rows or even one specific value? There are two different easy ways to access rows and columns and this is using the attributes loc and iloc.

How loc works?

With loc you can pass the row x columns coordinates with the respectively names to access a value:

print(df.loc["third","col4"])

>>> 9

Sometimes you have unnamed rows. In these cases the rows are automatically indexes with numbers and you can use them inside loc.

df = pd.DataFrame([[1,2,3],[4,5,6],[7,8,9]], columns=["col1", "col2", "col4"])print(df)

>>>  col1 col2 col3
    0   1    2    3
    1   4    5    6
    2   7    8    9

print(df.loc[2,"col3"])

>>> 9

Previously I said that you couldn’t use indexes directly to access data, but iloc helps us in this case!!

How _iloc_ works?

This attribute works exactly the same way as loc except that you pass indexes instead of names inside it:

df = pd.DataFrame([[1,2,3],[4,5,6],[7,8,9]], columns=["col1", "col2", "col4"])
print(df)

>>>  col1 col2 col3
    0   1    2    3
    1   4    5    6
    2   7    8    9

print(df.iloc[0, 0])

>>> 1

Alternative!!!

In addition to loc and iloc, there is another attribute called at. This atribute works exactly as loc, but it really access only one item, one data position while loc can acess a range of rows and columns.

df = pd.DataFrame([[1,2,3],[4,5,6],[7,8,9]], columns=["col1", "col2", "col4"])
print(df)

>>>  col1 col2 col3
    0   1    2    3
    1   4    5    6
    2   7    8    9

print(df.at[2,"col3"])

>>> 9

This access of many rows and columns (called slicing) will be explained in a coming post and this post will be updated below with the link when done!

Welcome to Pandas world!