Forem: Dylan

⚡⚡⚡ 3 Tips and 1 Gist For Using Python and The Twitter Search API

Dylan — Thu, 23 Apr 2020 20:10:57 +0000

These past days I've been playing around with the Twitter Search API. Here are a few things that you may find useful if you are getting started with it:

Use Tweepy. It's much better documented and easier to work with compared to python-twitter
If you are only reading public data, use Application-only authentication. You can do up to 450 requests in a 15-min window. The Application-user authentication gives you only 180 requests in the same time window.
Read the best practices for Standard Search. There's a lot of things you can do in the queries (e.g., filter by tweets' sentiment)

Finally, here's a simple gist that allows you to get the last 100 tweets for a given query and inserts them in a SQLite DB every 30 seconds: https://gist.github.com/dylanjcastillo/99a79d53a3d73d8eebf0a5c0b81531d1

Use Google Sheets, S3, and Python to Build a Website Quickly

Dylan — Thu, 16 Apr 2020 19:15:19 +0000

This post is a follow-up to How to Build a Site Quickly If You Are Not a Web Developer, where I only talked about the stack.

For this tutorial, we will set up a script that reads data from Google Sheets, generates a static site using a predefined template, and deploys it to an S3 bucket. This article is meant for programmers with little knowledge of web development that want to get something running quickly.

There are five sections in the tutorial: Requirements, Review Code and Jinja Template, Using the Google Sheets API, and Build and Deploy Your Site.

Requirements

These you'll need to set up or review on your own. I added some links for that purpose.

Python >= 3.7
Google account
Google Cloud Platform (GCP) account
Amazon AWS account
AWS CLI (You can use brew if you have a Mac)
A bit of HTML and CSS

Code and Jinja Template

First, create a directory called my_pokemon_stats and open a terminal from there. Then, create a virtual environment and install the required packages as follows:

python3 -m venv venv
source venv/bin/activate
pip3 install boto3 gspread jinja2 oauth2client

Next, download and save these two files there: template.html and site_builder.py. These are the building blocks for generating the site.

template.html is the Jinja template we will use for building the site. It's an HTML-like file where you can add logic that will be processed in Python and generate the definitive site. This file looks as follows:

<!doctype html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport"
          content="width=device-width, user-scalable=no, initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <link href="https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css" rel="stylesheet"
          integrity="sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh" crossorigin="anonymous">
    <title>My Pokemon Stats</title>
</head>
<body>

<header id="header">
    <div class="container text-center">
        <h1 class="pt-5 pb-1 font-weight-bold">
            My Pokemon Stats
        </h1>
        <hr>
        <p class="pt-2">
            This is a site I use to store the stats of all my Pokemon.
        </p>
    </div>
</header>

<section id="pokemon_table">
    <div class="container py-4">
        <div class="table-responsive">
            <table class="table table-hover">
                <thead class="thead-dark">
                <tr>
                    <th scope="col">Name</th>
                    <th scope="col">Type 1</th>
                    <th scope="col">Type 2</th>
                    <th scope="col">Total</th>
                    <th scope="col">HP</th>
                    <th scope="col">Attack</th>
                    <th scope="col">Defense</th>
                </tr>
                </thead>
                <tbody>
                {% for row in data %}
                <tr>
                    <td>{{ row["Name"] }}</td>
                    <td>{{ row["Type 1"] }}</td>
                    <td>{{ row["Type 2"] }}</td>
                    <td>{{ row["Total"] }}</td>
                    <td>{{ row["HP"] }}</td>
                    <td>{{ row["Attack"] }}</td>
                    <td>{{ row["Defense"] }}</td>
                </tr>
                {% endfor %}
                </tbody>
            </table>
        </div>
    </div>
</section>
</body>
</html>

Let's break it down:

You can safely ignore most of what's inside the <head> tag. It's standard HTML5 code that you'll see in most pages. However, there are just two interesting tags that we'll take a closer a look at: <link> and <title>.
In this case, the <link> tag is used to import the Bootstrap component library. We will use it to define simple styles for the different sections of the page and make it look good without much effort. <title> defines the title of the page (what you see in the browser's tab) and it is useful for SEO and social media sharing.
Next, there's the <header> section inside the <body> tag. This is where we define the text that will appear in the page. It will look like the image below. I used standard styling from Bootstrap to center the text and added a bit of padding.

Finally, we have the <section id="pokemon_table">. The <div> and <table> tags provide some basic styling for building a table. Next, we define the header of the table inside the <thead> tags. Inside the <tbody> tag is where Jinja does its magic
The {% for row in data %} is a loop that goes through each row of the Pokemon's data. In each of the <td>{{ row["..."] }}</td> we get the information per row from field (e.g. Name, Type 1, Type 2). This generates something that will look as follows:

Next, we have the site_builder.py file. This script downloads the Pokemon's data from Google Sheets, processes the data and the template.html file, and then uploads the resulting file to an S3 bucket.

import csv

import boto3
import gspread
import jinja2
from oauth2client.service_account import ServiceAccountCredentials

AWS_PROFILE = "INSERT-AWS-PROFILE-HERE"
BUCKET = "INSERT-BUCKET-NAME-HERE"
WORKBOOK = "INSERT-WORKBOOK-NAME-HERE"


def download_data():
    """Download data using the Google Sheets API"""
    scope = [
        "https://spreadsheets.google.com/feeds",
        "https://www.googleapis.com/auth/drive",
    ]
    credentials = ServiceAccountCredentials.from_json_keyfile_name(
        "credentials.json", scope
    )
    client = gspread.authorize(credentials)

    worksheet = client.open(WORKBOOK).get_worksheet(0)
    sheet_values = worksheet.get_all_values()

    print(f"Downloading: {worksheet.title}")
    with open("my_pokemon_stats.csv", "w") as f:
        writer = csv.writer(f)
        writer.writerows(sheet_values)


def generate_site():
    """Generate site in local directory"""
    print("Process data and build site")

    template_loader = jinja2.FileSystemLoader(searchpath="./")
    template_env = jinja2.Environment(loader=template_loader)
    template = template_env.get_template("template.html")

    with open("my_pokemon_stats.csv") as csv_file:
        csv_reader = csv.DictReader(csv_file)
        data = [row for row in csv_reader]

    output = template.render(data=data)

    with open("index.html", "w") as f:
        f.write(output)


def deploy_site():
    """Deploy site S3 bucket"""
    print("Upload data to S3")
    session = boto3.Session(profile_name=AWS_PROFILE)
    s3 = session.resource("s3")
    s3.Bucket(BUCKET).upload_file(
        Filename="index.html", Key="index.html", ExtraArgs={"ContentType": "text/html"}
    )


if __name__ == "__main__":
    download_data()
    generate_site()
    deploy_site()

The code is structured in 3 functions: download_sheets, generate_site, and deploy_site. We will fill the details for accessing AWS and the Google Sheets API in the next sections.

Using the Google Sheets API

Follow these steps to download the Pokemon's data using the Google Sheets API:

Create a Workbook in Google Sheets (you can copy mine: My Pokemon Stats)
Go to the Google APIs Console
Create a new project called MyPokemonStats.
Click on Enable API and Services. Search for and enable the Google Sheets API.
Go back to the Google APIs Console and click on Enable API and Services again. Now search for and enable the Google Drive API.
Click on Create credentials.*For the next 4 questions select:*Google Drive API, Web Server (e.g. node.js, Tomcat), *Application data, *and *No, I'm not using them. *
Click on What credentials do I need? *Select a name for the service account (e.g. get-data) grant it a *Project Role of Editor. Select the JSON option for Key type
A dialog box will open. Save the JSON file, copy it to the my_pokemon_stats directory, and rename it to credentials.json.
Open the credentials.json file. Find a key called client_email, copy its value (e.g., get-data@iam....). Go back to your Workbook in Google Sheets, click the Share button in the top right, and paste the client email into the People field to give it edit rights. Hit Send.
Go to the site_builder.py script, and set the WORKBOOK variable to whatever name you gave to your workbook on step 1.

Setting up an S3 bucket and AWS-related configurations

Now, let's create the S3 bucket and configure our code to access AWS programmatically:

Go to the Amazon S3 Console
Create an S3 bucket
Once in the bucket, click on Properties and then on Static website hosting
Select the option Use this bucket to host a website
Under Index document and Error document put index.html
Save the URL from Endpoint. You'll use that URL for connecting to your site.
Go to Permissions and click Edit
Clear Block all public access, choose Save, and confirm. When you change this anyone on the internet will have access to the contents of this bucket. That's what you want when you are publishing a site, however don't put anything private there!
Now go to Bucket Policy, replace the bucket name in the following policy, paste it there, and click Save.

{
  "Version":"2012-10-17",
  "Statement":[{
    "Sid":"PublicReadGetObject",
        "Effect":"Allow",
      "Principal": "*",
      "Action":["s3:GetObject"],
      "Resource":["arn:aws:s3:::BUCKET-NAME-HERE/*"
      ]
    }
  ]
}

Go to the site_builder.py script. Set the variable AWS_PROFILE variable to the profile name you use for accessing AWS (in a UNIX sytem it should be one of the profiles in ~/.aws/credentials).

Build and Deploy Your Site

Finally, you should be able to run python site_builder.py from the root folder of the project to generate the site. This will download the data from Google Sheets, process the template.html file using Jinja and upload the site to the S3 bucket.

If you want to check the site, go to the endpoint URL (step 6 from the previous section).

Closing Words

This method is by no means perfect but will help you ship something quickly. I used this strategy to build stayhomeandlearn.org and it worked quite well. From April 1st until April 16th, the site got over fifteen thousand visitors, which exceeded my most optimistic expectations.

The site is slowly walking towards its death now. However, this process taught me how important it is to focus on shipping instead of wasting time looking for the perfect tools. I built the site quickly, people liked it, and after the first day it already had more traffic than any of the side projects I've done so far. That thing about perfect being the enemy of good is real.

In my case, I had to add more functionality to the script for styling and deploying purposes. If you are interested, you can take a look at the code in my GitHub repository. Or check other articles in my site.

Finally, there are a couple of things you can do now to make your site more interesting:

References

[1] G. Bauges, Google Spreadsheets and Python (2017)

[2] V. Drake, Hosting your static site with AWS S3, Route 53, and CloudFront (2017)

[3] A. Barradas, Pokemon With Stats (2016)

How to Build a Site Quickly If You Are Not a Web Developer: AWS + Google Sheets + Python + Bootstrap ⚡⚡

Dylan — Fri, 10 Apr 2020 09:45:25 +0000

A week ago, I was looking for courses that were free or discounted due to the COVID-19. I thought that maybe others were doing the same, so I decided to compile resources and publish them online.

I started compiling the courses using Google Sheets and was planning on sharing the sheet after I had enough resources. However, something was bothering me. Opening sheets on a mobile suck and most people use their phones for browsing the internet. I thought I could do better.

The issue is that I'm a Data Scientist, not a Web Developer. I hardly have any experience building and deploying web pages on my own. So I decided to build a simple static site that met the following requirements:

Look good on mobiles (or at least better than a Google Sheet)
Should allow me to easily add or remove resources
Use Python as much as possible (and as little as possible of HTML/CSS/JS)
Should take me only a few hours to develop and launch
Maintenance should be very very close to zero 😛

So here's what I did to solve these issues:

Used Bootstrap to make it responsive and look good without a lot of effort from my side
Used Google Sheets as a basic Content Management System. Got content into the site programmatically using the API
Used Jinja2 as a templating engine
Hosted the site on AWS S3 + Cloudfront (for SSL certificate). Bought the domain using Route 53.
So far, I've only invested ~15€ on the site. I don't foresee that I will need to spend more than a few cents for its maintenance, given what AWS offers in its Free Tier.

Finally, after I had the first version, I published it on reddit and LinkedIn. It got some traction and was shared in different sites. During the first two days, it got over 3k visitors. The number of visitors has been steadily decreasing since then, totaling around 12.5k right now.

Building the site was an entertaining learning exercise for me and, based on how people reacted, it was useful for others. I was not really expecting to have more than a few visitors, so it was a positive surprise seeing how many people were interested.

I wanted to share the process and code in case others may find them useful.

Finally, here's the Github repository: https://github.com/dylanjcastillo/stayhomeandlearn.org

Also, here's the site: https://stayhomeandlearn.org

Fast & Asynchronous In Python: Accelerate Your Requests Using asyncio

Dylan — Sun, 01 Mar 2020 16:50:06 +0000

A few weeks ago, I was working on a Python script to extract books' metadata for a content-based recommender. After a couple of hours, I realized that I needed to make thousands of requests to the Google Books API to get the data. So I thought there had to be a way of speeding up the process.

As I enjoy learning, especially when it's also a chance of procrastinating on my goals, I decided to build a project using asyncio. Afterward, feeling guilty for the time wasted, I decided to write this tutorial with what I learned in the process.

This article aims to provide the basics of how to use asyncio for making asynchronous requests to an API. I focus mostly on the actual code and skip most of the theory (besides the short introduction below). However, if you are looking for a more in-depth introduction to asyncio, check the recommendations in the references^1.

asyncio in 30 Seconds or Less

asyncio is a Python library that allows you to execute some tasks in a seemingly concurrent^2 manner. It is commonly used in web-servers and database connections. It is also useful for speeding up IO-bound tasks, like services that require making many requests or do lots of waiting for external APIs^3.

The essence of asyncio is that it allows the program to continue executing other instructions while waiting for specific processes to finish (e.g., a request to an API). In this tutorial, we will see how to use *asyncio*for accelerating a program that makes multiple requests to an API.

Sequential vs. Asynchronous

So let's get down to business. To get the most out of this tutorial, try running the code yourself. You can copy-and-paste the code in a Jupyter Notebook and run it without modifications. Just remember to first install the required libraries (requests and aiohttp.)

We'll build a sequential and an asynchronous version of a small program and compare their results and structure. Both programs will do the same:

Read a list of ISBNs (international identifier of books)
Request the books' metadata to the Google Books API
Parse the results from the requests
Print the results to the screen.

The algorithm would look something like the diagram below.
We'll now compare two possible approaches for building this algorithm. First, Option A, which executes the requests sequentially. Then, Option B, which uses asyncio to run requests asynchronously.

Option A: Sequential Algorithm

A sequential version of that algorithm could look as follows:

import os
import requests  
from requests.exceptions import HTTPError


GOOGLE_BOOKS_URL = "https://www.googleapis.com/books/v1/volumes?q=isbn:"
LIST_ISBN = [
    '9780002005883',
    '9780002238304',
    '9780002261982',
    '9780006163831',
    '9780006178736',
    '9780006280897',
    '9780006280934',
    '9780006353287',
    '9780006380832',
    '9780006470229',
]


def extract_fields_from_response(item):
    """Extract fields from API's response"""
    volume_info = item.get("volumeInfo", {})
    title = volume_info.get("title", None)
    subtitle = volume_info.get("subtitle", None)
    description = volume_info.get("description", None)
    published_date = volume_info.get("publishedDate", None)
    return (
        title,
        subtitle,
        description,
        published_date,
    )


def get_book_details_seq(isbn, session):
    """Get book details using Google Books API (sequentially)"""
    url = GOOGLE_BOOKS_URL + isbn
    response = None
    try:
        response = session.get(url)
        response.raise_for_status()
        print(f"Response status ({url}): {response.status_code}")
    except HTTPError as http_err:
        print(f"HTTP error occurred: {http_err}")
    except Exception as err:
        print(f"An error ocurred: {err}")
    response_json = response.json()
    items = response_json.get("items", [{}])[0]
    return items


with requests.Session() as session:
    for isbn in LIST_ISBN:
        try:
            response = get_book_details_seq(isbn, session)
            parsed_response = extract_fields_from_response(response)
            print(f"Response: {json.dumps(parsed_response, indent=2)}")
        except Exception as err:
            print(f"Exception occured: {err}")
            pass

Now, let's breakdown the code to understand what's going on.

As usual, we start by importing the required libraries. Then, we define two variables:

GOOGLE_BOOKS_URL for specifying the URL of the Google's API we'll use for the requests. This is how a request to the Google Books API looks like: https://www.googleapis.com/books/v1/volumes?q=isbn:9780002005883
LIST_ISBN, which is a sample list of ISBNs for testing the program.

    import os
    import requests
    from requests.exceptions import HTTPError

    GOOGLE_BOOKS_URL = "https://www.googleapis.com/books/v1/volumes?q=isbn:"
    LIST_ISBN = [
        '9780002005883',
        '9780002238304',
        '9780002261982',
        '9780006163831',
        '9780006178736',
        '9780006280897',
        '9780006280934',
        '9780006353287',
        '9780006380832',
        '9780006470229',
    ]

Next, we define the extract_fields_from_responsefunction. This function takes as input the response from the API and extracts the fields we're interested in.

    def extract_fields_from_response(response):
        """Extract fields from API's response"""
        item = response.get("items", [{}])[0]
        volume_info = item.get("volumeInfo", {})
        title = volume_info.get("title", None)
        subtitle = volume_info.get("subtitle", None)
        description = volume_info.get("description", None)
        published_date = volume_info.get("publishedDate", None)
        return (
            title,
            subtitle,
            description,
            published_date,
        )

The parsing process in extract_fields_from_response is based on the response's structure from the Google Books API, which looks as follows:

    {
     "kind": "books#volumes",
     "totalItems": 1,
     "items": [
      {
       "kind": "books#volume",
       "id": "3Mx4QgAACAAJ",
       "etag": "FWJF/JY16xg",
       "selfLink": "https://www.googleapis.com/books/v1/volumes/3Mx4QgAACAAJ",
       "volumeInfo": {
        "title": "Mapping the Big Picture",
        "subtitle": "Integrating Curriculum and Assessment, K-12",
        ...

Finally, we go into the most relevant parts of the program: how we make requests to the Google Books API.

    def get_book_details_seq(isbn, session):
        """Get book details using Google Books API (sequentially)"""
        url = GOOGLE_BOOKS_URL + isbn
        response = None
        try:
            response = session.get(url)
            response.raise_for_status()
            print(f"Response status ({url}): {response.status_code}")
        except HTTPError as http_err:
            print(f"HTTP error occurred: {http_err}")
        except Exception as err:
            print(f"An error ocurred: {err}")
        response_json = response.json()
        items = response_json.get("items", [{}])[0]
        return items


    with requests.Session() as session:
        for isbn in LIST_ISBN:
            try:
                response = get_book_details_seq(isbn, session)
                parsed_response = extract_fields_from_response(response)
                print(f"Response: {json.dumps(parsed_response, indent=2)}")
            except Exception as err:
                print(f"Exception occured: {err}")
                pass

There are two major pieces here:

get_book_details_seq, which is the function that executes the requests. It takes as input an ISBN and a session object^4 and returns the response from the API as a JSON structure. It also handles possible errors, like providing a wrong URL or going over your daily quota of requests.
The code block under with requests.Session() as session, is where the full pipeline is orchestrated. It iterates through the list of ISBNs, gets the books' details, parses them, and finally prints the details to the screen.

For me, executing this process takes ranges from 4 to 6 seconds. If you only need to do this a couple of times, you will not find much benefit from using asyncio. However, if instead of 10 requests, you need to do 10,000, having some concurrency in your program pays out. In the next section, we'll see how to make this algorithm faster using asyncio.

Option B: Asynchronous Algorithm

An asynchronous version of the same algorithm may look something as follows:

    import aiohttp
    import asyncio
    import os

    from aiohttp import ClientSession


    GOOGLE_BOOKS_URL = "https://www.googleapis.com/books/v1/volumes?q=isbn:"
    LIST_ISBN = [
        '9780002005883',
        '9780002238304',
        '9780002261982',
        '9780006163831',
        '9780006178736',
        '9780006280897',
        '9780006280934',
        '9780006353287',
        '9780006380832',
        '9780006470229',
    ]


    def extract_fields_from_response(response):
        """Extract fields from API's response"""
        item = response.get("items", [{}])[0]
        volume_info = item.get("volumeInfo", {})
        title = volume_info.get("title", None)
        subtitle = volume_info.get("subtitle", None)
        description = volume_info.get("description", None)
        published_date = volume_info.get("publishedDate", None)
        return (
            title,
            subtitle,
            description,
            published_date,
        )


    async def get_book_details_async(isbn, session):
        """Get book details using Google Books API (asynchronously)"""
        url = GOOGLE_BOOKS_URL + isbn
        try:
            response = await session.request(method='GET', url=url)
            response.raise_for_status()
            print(f"Response status ({url}): {response.status}")
        except HTTPError as http_err:
            print(f"HTTP error occurred: {http_err}")
        except Exception as err:
            print(f"An error ocurred: {err}")
        response_json = await response.json()
        return response_json


    async def run_program(isbn, session):
        """Wrapper for running program in an asynchronous manner"""
        try:
            response = await get_book_details_async(isbn, session)
            parsed_response = extract_fields_from_response(response)
            print(f"Response: {json.dumps(parsed_response, indent=2)}")
        except Exception as err:
            print(f"Exception occured: {err}")
            pass

    async with ClientSession() as session:
        await asyncio.gather(*[run_program(isbn, session) for isbn in LIST_ISBN])

First, check the get_book_details_async function. An async keyword prepends it. This keyword tells Python that your function is a coroutine. Then, in the function's body, there are two await keywords. These tell that coroutine to suspend execution and give back control to the event loop, while the operation it is awaiting finishes.

A coroutine is a type of generator function in Python that, instead of producing values, consumes values^5. The interesting thing about it is that its execution pauses while waiting for new data being sent to it. In our case, this allows the execution of other parts of the program to continue in a seemingly concurrent manner.

In this case, the execution of get_book_details_async is suspended while the request is being performed: await session.request(method='GET', url=url). It is suspended again, while the request response is being parsed into a JSON structure: await response.json().

Next, we have the run_program coroutine. This one is simply a wrapper around the pipeline of getting a response from the API, parsing it, and printing the results in the screen. It awaits the execution of the get_book_details_async coroutine.

Finally, we have the code block under async with ClientSession() as session:. Using the asyncio.gather syntax, we tell the program to schedule all the tasks based on the list of coroutines we provided. This is what allows us to execute tasks concurrently.

For me, running this process takes around 800-1000 milliseconds.

Results

Comparing both versions, we see that the asynchronous one is around 4 to 7.5 times faster than the sequential version. If we increase the number of requests, you'll likely get an even higher speedup. Besides, the version using asyncio is not much more complicated than the sequential version, which makes using asyncio an excellent option for the kind of task we reviewed in this tutorial.

Additional recommendations

Here are some tips I gathered while working with asyncio:

asyncio keeps changing all the time, so be wary of old Stack Overflow answers. Many of them are not up to date with the current best practices
Real-life APIs will not allow you to run unlimited concurrent requests. To overcome that, take a look at asyncio'sSemaphore. It will enable you to limit the concurrency of your application. Check my Google Books Crawler project, if you want to see an example on how to implement it.
Not all programs can be speedup with asyncio. Research the type of issue you are facing before doing any substantial modification of code. Other alternatives might work for you (e.g., threading,multiprocessing)
I made a complete version of the program we went through in this tutorial for getting the metadata of almost 7 thousand books. Here's a link to it, in case you may find it useful: Google Books Crawler.

Notes and References

[1] Real Python has a two of amazing articles introducing asyncio: Async IO in Python and Speed Up Your Python Program With Concurrency

[2] It is not strictly concurrent execution. But in practical terms, it looks like it is.

[3] S. Buczyński, What Is the use case of coroutines and asyncio in Python 3.6? (2017)

[4] The session object is a functionality from the requests library that allows you to persist certain parameters across sessions. This usually results in requests with lower latency. Read more here.

[5] D. Beasly, A Curious Course on Couroutines and Concurrency (2009

[6] Cover image by Marc-Olivier Jodoin