Forem: Ogheneyoma Okobiah

Email Analysis with Python 3 (Part II)

Ogheneyoma Okobiah — Tue, 29 Jun 2021 18:36:32 +0000

Welcome back! This is a sequel to Part I where we covered making changes to our Gmail account, getting the subject of the email and its sender and visualising some of the email data.
The emphasis of this part is on getting the body of the emails.

If you read the first part and tried it out, I hope it was without hitches. If you did not but you are interested in learning how to get the body of the email, you can follow from here. Let's jump right in.

Prerequisites

Python 3
Pandas
A gmail account

Getting The Data

This was covered in Part I as it involves making changes to your Gmail account in order for IMAPLib to work with it.

Step 1: Importing the required libraries to get the email data

Here we import the libraries we need which are imaplib, email, getpass and pandas. You may want to install pandas using pip install pandas if you do not have it.

import imaplib
import email
import getpass
import pandas as pd

Step 2: Gaining access to the email server

Here we log into the email server with our credentials.

username =  input("Enter the email address: ")
password = getpass.getpass("Enter password: ")
mail = imaplib.IMAP4_SSL('imap.gmail.com')
mail.login(username, password)

Step 3: Specifying the mailbox to get data from.

Here we print out the mail list to see the available mailboxes and we select one.

print(mail.list())
mail.select("inbox")

Step 4: Searching and Fetching the data

The block of code below searches the selected mailbox with the given criteria, fetches the emails and stores it to the variable messages.
Here I am searching for emails from FreeCodeCamp.

result, numbers = mail.search(None, '(FROM "quincy@freecodecamp.org")')
uids = numbers[0].split()
uids = [id.decode("utf-8") for id in uids ]
result, messages = mail.fetch(','.join(uids) ,'(RFC822)')

Step 5: Preparing the data to be exported

The block of code below loops through the fetched emails, gets the date it was received, who sent it, the subject of the mail and the body of the mail.

We use the walk() method in the email library to get the parts and subparts of the message.
We use the get_content_type() method to get the email body maintype/subtype.
We use the get_payload() to get a string or message instance of the part.

body_list =[]
date_list = []
from_list = [] 
subject_list = []
for _, message in messages[::2]:
  email_message = email.message_from_bytes(message)
  email_subject = email.header.decode_header(email_message['Subject'])[0]
  for part in email_message.walk():
    if part.get_content_type() == "text/plain" :
        body = part.get_payload(decode=True)
        body = body.decode("utf-8")
        body_list.append(body)
    else:
        continue
    if isinstance(email_subject[0],bytes):
      decoded = email_subject.decode(errors="ignore")
      subject_list.append(decoded)
    else:
      subject_list.append(email_subject[0])
  date_list.append(email_message.get('date'))
  fromlist = email_message.get('From')
  fromlist = fromlist.split("<")[0].replace('"', '')
  from_list.append(fromlist)

Here we convert the objects in date_list to datetime objects using the to_datetime() method, because the time has its UTC format attached, we sliced off the UTC format.
The retrieved information is then converted to a pandas DataFrame and exported to a CSV file.

date_list = pd.to_datetime(date_list)
date_list = [item.isoformat(' ')[:-6]for item in date_list]
data = pd.DataFrame(data={'Date':date_list,'Sender':from_list,'Subject':subject_list, 'Body':body_list})
data.to_csv('emails.csv',index=False)

Data Cleaning

Now we are going to view the data and clean it where necessary to make it readable. First we read in the csv file and view it:

data = pd.read_csv("\emails.csv")
data.head()

The output can be seen below, going through there are escape characters in the Body column:

The function below removes the escape characters in the text. In this case "\r\n" as seen in the screenshot above. This makes the text more readable.

def clean_data(data, column, i):
    data.loc[i, column] = data.loc[i, column].split("\r\n")
    new_string = " ".join(data.loc[i, column])
    new_string = new_string.split("'',")
    data[column][i:i+1] = pd.DataFrame(data = new_string)
    return data

The code below is using the function above to clean every email body in the file.

for n in range(len(data)):
    new_data = clean_data(data, column = "Body", i = n)

The output can be seen below:

You are advised to rewrite a new function according to the escape characters you may find in the Subject or Body of the email you retrieved.

Conclusion

I encountered ERRORS while writing this, the most recurring one was being unable to sign in even after following the instructions on the Google help page. This problem was encountered because I have more than one Gmail account signed in, and I was not using my default email. In case you encounter the same, the solution is outlined below:

The instruction said, "If the tips above didn't help, visit https://www.google.com/accounts/DisplayUnlockCaptcha and follow the steps on the page." This opens on a new tab.
The link on the new tab was "https://accounts.google.com/b/0/DisplayUnlockCaptcha" where the digit 0 is for the default account logged in.
Check your accounts in the order in which they are listed and change the digit accordingly (e.g.,”1" is the next email and so on).

You can find the full code on GitHub below:

yomaokobiah / email_analysis

Email data analysis

email_analysis

Email data analysis

View on GitHub

Thank you for getting to the end of this article, I hope you had fun trying it out.

References

Email Analysis Using Python 3 (Part I)

Ogheneyoma Okobiah — Sat, 19 Jun 2021 19:34:19 +0000

There is a lot of data out there, mostly unstructured. Emails are a great source of communication data as such there is no limit to what we can harness from it.
At the end of this tutorial you would be able to get email data for insights.

Prerequisites

Python 3
Pandas
Matplotlib
Seaborn
Wordcloud
A gmail account

Getting The Data

There are several ways to go achieve the aim of this article; find below, how I did mine.

Here a Gmail account is being used; for the imaplib script to work I made the following changes to my account; enabled IMAP and turned on less secured apps.

First, open Gmail, click on the settings ⚙️ icon and click See all settings to enable IMAP.
On the next page, click on the Forwarding and POP/IMAP tab.
In the IMAP Access section, select Enable IMAP. Then click Save changes. If you need more help, kindly visit this Gmail help page.
To turn on less secured apps, navigate to your Google dashboard by clicking on your account avatar in the upper right-hand corner of your screen and then click My Account or navigate to myaccount.google.com.
Then choose Sign-in & security, scroll down until you see the option Allow less secure apps, and turn the access on.

If you still can’t log in after doing the above, kindly visit here for official Google help support.

Step 1: Importing the required libraries to get the email data

imaplib is an Internet Message Access Protocol (IMAP) library
email is a python library that parses, handles and generates email messages.
getpass is a python library that contains utilities to get password or current username
pandas is a python library for data manipulation and analysis.

import imaplib
import email
import getpass
import pandas as pd

Step 2: Gaining access to the email address

username is the email address.
password is the password to the email address when prompted. [If you don't want to use the getpass package, you can enter your password as a string.]
mail is the email server we're going to connect to and it varies, for this tutorial we're using gmail.
mail.login logs into the server using the provided credentials.

username =  input("Enter the email address: ")
password = getpass.getpass("Enter password: ")
mail = imaplib.IMAP4_SSL('imap.gmail.com')
mail.login(username, password)

Step 3: Specifying the mailbox to get data from.

mail.list() is a method that gives a list of the mailboxes - i.e inbox, draft and so on in the email address.
mail.select() is a method that takes an argument of the mailbox you want to get data from.

print(mail.list())
mail.select("inbox")

Step 4: Searching and Fetching the data

Line 1: mail.uid() is a method whose first argument is the command you want to execute, in this case the command is "search". The rest of the arguments are used for the search. (Search gives from oldest to recent)
Line 1: result is an exit code of the command while numbers is a list that contains an object of type byte.
Line 2: is a list of every section in numbers.
Line 3: is a list of decoded bytes
Line 4: is a slice of the recent 100 items (recall that search orders it from oldest to recent).
Line 5: the command we want to execute is "fetch" and store it in messages. We're fetching the subject of the messages in this case.

result, numbers = mail.uid('search', None, "ALL")
uids = numbers[0].split()
uids = [id.decode("utf-8") for id in uids ]
uids = uids[-1:-101:-1]
result, messages = mail.uid('fetch', ','.join(uids), '(BODY[HEADER.FIELDS (SUBJECT FROM DATE)])')

Step 5: Preparing the data to be exported

Line 1-3: empty lists for the data we specified in messages.
Line 4: looping through the content of the message we fetched. Using a step of two because it returned a tuple of two items.
Line 5: parsing the bytes email to message object.
Line 6-11: msg is in bytes, in order to use it it had to be decoded to a format we can read.
Line 12: adding the dates to date_list.
Line 13-15: getting the sender detail, it's in the format "Sender name" hence the split and replace methods are used to get only the "Sender name".
Line 16-19: converting the objects in date_list to datetime objects, because the time has it's UTC format attached, a new list was created and the UTC format was sliced off from each object in the list.
Line 20-22: checking the length of created lists, because arrays have to be the same length.
Line 23-25: converting the lists to a dictionary and then a pandas dataframe, viewing it and saving it for download.

date_list = []
from_list = [] 
subject_text = []
for i, message in messages[::2]:
    msg = email.message_from_bytes(message)
    decode = email.header.decode_header(msg['Subject'])[0]
    if isinstance(decode[0],bytes):
        decoded = decode[0].decode()
        subject_text.append(decoded)
    else:
        subject_text.append(decode[0])
    date_list.append(msg.get('date'))
    fromlist = msg.get('From')
    fromlist = fromlist.split("<")[0].replace('"', '')
    from_list1.append(fromlist)
date_list = pd.to_datetime(date_list)
date_list1 = []
for item in date_list:
    date_list1.append(item.isoformat(' ')[:-6])
print(len(subject_text))
print(len(from_list))
print(len(date_list1))
df = pd.DataFrame(data={'Date':date_list1, 'Sender':from_list, 'Subject':subject_text})
print(df.head())
df.to_csv('inbox_email.csv',index=False)

Visualisation

Now that we have a the email data in CSV format, we can read the data using pandas, and visualise it.
There are several Python data visualisation libraries, but here I used Wordcloud, Matplotlib and Seaborn. I wanted to see an infographic on the most used words in the subjects of my emails and here is how I did it.

Step 1: Reading and viewing the CSV

Step 2: Getting statistical data

I used the the describe method to get the statistical data, unique values and all to get insight on the what's in the data.

Step 3: Creating new variables

I created two variables; Time and SinceMid. SinceMid is the number of hours after midnight.

(Note: The time can be removed from the date column completely)

from datetime import datetime
FMT = '%H:%M:%S'
emails['Time'] = emails['Date'].apply(lambda x: datetime.strptime(x, '%Y-%m-%-d%H:%M:%S').strftime(FMT))
emails['SinceMid'] = emails['Time'].apply(lambda x: (datetime.strptime(x, FMT) - datetime.strptime("00:00:00", FMT)).seconds) / 60 / 60

Step 4: The plots

I created a wordcloud image of the most used words in the subjects of my mails. In this example there are no stopwords, stopwords are usually filtered out as most times they're not informative.

from wordcloud import WordCloud
import matplotlib.pyplot as plt


# Create a list of words
text = ""
for item in emails["Subject"]:
    if isinstance(item,str):
        text += " " + item
    text.replace("'", "")
    text.replace(",","")
    text.replace('"','')


# Create the wordcloud object
wordcloud = WordCloud(width=800, height=800, background_color="white")

# Display the generated image:
wordcloud.generate(text)
plt.figure(figsize=(8,8))
plt.imshow(wordcloud, interpolation="bilinear")
plt.axis("off")
plt.margins(x=0, y=0)
plt.title("Most Used Subject Words", fontsize=20,ha="center", pad=20)
plt.show()

Here's the output:

I created a histogram of the hours after midnight using seaborn.

import seaborn as sns
sns.distplot(emails["SinceMid"],bins=20)
plt.title("Hours since midnight")

Here is the histogram:

You can check out python gallery for more possible visualisations.

Conclusion

I had fun writing this, I hope you did too while reading it. This goes without saying, I encountered ERRORS while doing this [some of them I had never seen before]. When you get error messages, a good starting point is using the print statement to get insight and then googling the error message.
The Part II will also be published on this blog, it would focus on getting the body of the mail and not the subject as this one.

The full code can be found below:

yomaokobiah / email_analysis

Email data analysis

email_analysis

Email data analysis

View on GitHub

Thank you for reading up to this point.

Disclaimer: I encourage you to experiment outside what's written here, if you encounter bugs and you feel like getting me involved [after Googling], send me a DM on Twitter I'd be happy to learn something new. Thank you in anticipation.

References/Credits

Decluttering Mailbox Using Python

Ogheneyoma Okobiah — Sat, 29 May 2021 09:32:29 +0000

I've had so many unread emails in my inbox because they were promotional content, or I was just too lazy to read when they came in. Now imagine opening your mailbox and seeing over a thousand emails — screams in 1917!! How can I declutter as fast as possible?

Now I know Gmail has a delete feature but I believe we're meant to "automate the boring stuff with python" and you may learn something new or not.

In this article, I will show you how to set up a script to send unread/unwanted messages from your inbox to trash using IMAPlib and how to delete unread/unwanted messages using IMAPClient permanently.

Let's jump right into it!

Prerequisites

Python
Gmail account
ImapClient [optional]

Setting up your Gmail account

IMAP stands for Internet Mail Access protocol. According to Wikipedia, it is "an Internet standard protocol used by email clients to retrieve email messages from a mail server over a TCP/IP connection.”

To get started, we need to set up a Gmail account for the process to be successful. I made the following changes to my Gmail account, which enabled IMAP and turned on less secured apps.

First, open Gmail, click on the settings ⚙️ icon and click See all settings to enable IMAP.
On the next page, click on the Forwarding and POP/IMAP tab.
In the IMAP Access section, select Enable IMAP. Then click Save changes. If you need more help, kindly visit this Gmail help page.
To turn on less secured apps, navigate to your Google dashboard by clicking on your account avatar in the upper right-hand corner of your screen and then click My Account or navigate to myaccount.google.com.
Then choose Sign-in & security, scroll down until you see the option Allow less secure apps, and turn the access on.

If you still can’t log in after doing the above, kindly visit here for official Google help support.

How to trash your emails using IMAPlib

The IMAP library is an inbuilt python library used for accessing and manipulating emails over IMAP. The code below logs you into the desired email address. After logging in, we select the desired mailbox to see a list of all available mailboxes to select from using print(mail.list()).

Starter code for IMAPlib

The criterion for emails to be sent to the trash in this section are:

Unread emails.
Emails based on a specific subject.
Emails based on a specific sender.

import imaplib
import getpass
username =  input("Enter the email address: ")
password = getpass.getpass("Enter password: ")
mail = imaplib.IMAP4_SSL('imap.gmail.com')
mail.login(username, password)
mail.select("INBOX")

Trashing all unread emails

The code below sends all unread emails to the trash. You never can tell if you need any email, so you have thirty (30) days before Gmail removes it from the trash. First, we search through the emails setting the criteria to "UNSEEN" (which is the flag for unread messages). The variable result is an exit code of the command, while messages are a list that contains an object of type byte. Finally, we loop through the emails in the list and put them in the trash folder.

results, messages = mail.search(None, 'UNSEEN')
messages = messages[0].split()
for x in messages:
  result, message = mail.store(x, '+X-GM-LABELS', '\\Trash')
mail.close()
mail.logout()

Trashing all emails with a particular subject

Here, we search the selected mailbox using our desired criteria. You should replace "Title" with your desired subject in the code below:

results, messages = mail.search(None, '(SUBJECT "Title")')
messages = messages[0].split()
for x in messages:
  result, message = mail.store(x, '+X-GM-LABELS', '\\Trash')
mail.close()
mail.logout()

Trashing all emails from a specific sender

Here, we search the mailbox using our desired criteria. You should replace "abc@xyz.com" with your preferred email address in the code below:

results, messages = mail.search(None, '(FROM "abc@xyz.com")')
messages = messages[0].split()
for x in messages:
  result, message = mail.store(x, '+X-GM-LABELS', '\\Trash')
mail.close()
mail.logout()

Here I'll briefly talk about the methods used.

search() is a method used to look through the selected mailbox, it takes two arguments; a charset and a criterion. The charset can be set to None.
store() is a method used to change the flag or label of a message; it takes three arguments; the message, the command, and the flag.
close() is a method used to close the selected mailbox.
logout() is a method used to close the connection to the server.
'+X-GM-LABELS' a label is treated as a folder so that changes can be made to messages using IMAP commands.

How to permanently delete your emails using IMAPClient

While researching this article, I came across the IMAPClient library. From its official documentation, I discovered that: "although IMAPClient actually uses the imaplib module from the Python standard library under the hood, it provides a different API."

Using IMAPClient to delete emails will not send the emails to the trash folder; they will be permanently deleted. You should BE SURE you want to permanently delete your emails before using any code snippets here.

Starter Code for IMAPClient

To use IMAPClient, you have to install the package using the command below:

pip install imapclient

Next, we import the library and set the server of our email address.

from imapclient import IMAPClient
import getpass
server = IMAPClient('imap.gmail.com', use_uid=True)
username =  input("Enter the email address: ")
password = getpass.getpass("Enter password: ")
server.login(username, password)
select_info = server.select_folder("INBOX")

The criterion for emails to be sent to the trash in this section are:

Unread emails.
Emails based on a specific subject.
Emails based on a specific sender.

Permanently deleting all unread emails

The code below permanently deletes all unread emails.

messages = server.search("UNSEEN")
for x in messages:
  server.delete_messages(x)
server.logout()

Permanently deleting all emails with a particular subject

We search the selected mailbox using our desired criteria. You should replace "Title" with your desired subject in the code below:

messages = server.search(['SUBJECT', 'Title'])
for x in messages:
  server.delete_messages(x)
server.logout()

Permanently deleting all emails from a specific sender

We search the mailbox using our desired criteria. You should replace "abc@xyz.com" with your preferred email address in the code below:

messages = server.search(['FROM', 'abc@xyz.com'])
for x in messages:
  server.delete_messages(x)
server.logout()

Note

Logging out of the server is important
To access all emails while using IMAPlib, use mail.select('"[Gmail]/All Mail"').
Ensure you turn off less secure apps.
A lot of emails were harmed in the process of making this tutorial 😂.

Conclusion

I had fun writing this, and I hope you did too while reading it. I also encountered ERRORS while writing the script. The most recurring one was being unable to sign in even after following the instructions on the Google help page. This problem was encountered because I have more than one Gmail account signed in, and I was not using my default email. In case you encounter the same, the solution is outlined below:

The instruction said, "If the tips above didn't help, visit https://www.google.com/accounts/DisplayUnlockCaptcha and follow the steps on the page." This opens on a new tab.
The link on the new tab was "https://accounts.google.com/b/0/DisplayUnlockCaptcha" where the digit 0 is for the default account logged in.
Check your accounts in the order in which they are listed and change the digit accordingly (e.g.,”1" is the next email and so on).

You can find the complete script on GitHub.

Thank you for reading this tutorial!