Forem: Frank Corso

Creating Your First Chart Using Seaborn

Frank Corso — Mon, 04 Sep 2023 14:54:33 +0000

When working with data in Python, there are many times when you will need to analyze or visualize the data. For example, maybe you are quickly looking for outliers or are creating a few charts to be shared.

There are many libraries for creating graphs. The most common is matplotlib. However, that can be a lot lower level than you need most of the time and has a steep learning curve.

This is where seaborn comes in. Seaborn is a library built on top of matplotlib and makes creating charts from data sets simple.

Seaborn Basics

Seaborn is tightly integrated with Pandas, so you can pass almost any data frame as the data set. Additionally, you can use a variety of other data structures too. Let’s take a look at the minimum steps for using Seaborn.

When creating a chart with Seaborn, there are only a few required steps as shown in the code below.

# Import Seaborn, usually aliased as sns.
import seaborn as sns

# Create a sample dictionary of x,y values.
data = {
    'a': 1,
    'b': 2,
    'c': 3,
    'd': 4
}

# Create a basic line chart.
sns.lineplot(data=data)

This will create a line chart using all the default options assuming the dictionary pairs are x, y values, as shown in the screenshot below.

Seaborn includes a variety of methods for built-in chart types, including lineplot, countplot, relplot, and boxplot. You can view all of the available methods in the Seaborn API reference.

Creating a Line Chart Using Pandas Dataframes

In many cases, you will start with a Pandas data frame with multiple columns. In this case, it will work almost the same, except we can pass the x and y names.

Let’s start with a basic dataframe that contains a few rows for the total revenue a store earned per day. We can then pass that dataframe to seaborn as we did before, but this time, we will specify the x and y axes.

# Import Pandas and Seaborn
import pandas as pd
import seaborn as sns

# Create our dataframe.
df = pd.read_csv('example-df.csv')

# Create a basic line chart.
sns.lineplot(data=df, x='date', y='total_revenue')

Improving Your Chart

While the default graphs might work for just quickly visualizing a data set, you will often need to share these charts. With Seaborn, you have many ways to alter a chart, such as using a Seaborn helper method or using a matplotlib function on the return chart instance.

Adding Labels

First, let’s add the title. To do this, we can chain the set method directly on the chart call.

sns.lineplot(data=df, x='date', y='total_revenue').set(title='Our revenue continues to rise')

We can also use this to set the axis labels by setting the xlabel and ylabel values.

sns.lineplot(data=df, x='date', y='total_revenue').set(title='Our revenue continues to rise', xlabel='', ylabel='')

This works well if we are just adding some text to the chart. But, what if we want to adjust size or positioning? We could use global Seaborn values, such as font scale, to alter all font sizes.

sns.set(font_scale=2)

However, you will often need to adjust specific labels or titles. To do this, we can access the underlying matplotlib values as shown in this code:

# Import matplotlib so we can use the plot methods.
import matplotlib.pyplot as plt

# Add title and axis labels
plt.title('Our revenue continues to rise', fontsize=18)
plt.xlabel('Day', fontsize=12)
plt.ylabel('Revenue', fontsize=12)

Once we start using the matplotlib methods, we have a little more control over our text as well. For example, for the title, it might be better to left-align it which, we can do by using the loc parameter:

plt.title('Our revenue continues to rise', fontsize=18, loc='left')

If we need to, we can also adjust the font size of the tick labels (the labels along each axis) using a similar method:

plt.tick_params(axis='both', labelsize=8)

This gets our graph to look like this:

Adjusting Styles

Now that we have some labels in place, we should start adjusting the graph itself. The first thing we may want to do is adjust our starting values along one or both axes. We can do this by calling matplotlib's ylim method. If we needed to, there is also a xlim method.

# Set the y axis to start at 0.
plt.ylim(bottom=0)

Seaborn also comes with a variety of default themes and styles as well as methods to help us control different aspects of the graph. You can start with setting the set_style method to one of five themes: darkgrid, whitegrid, dark, white, and ticks.

I normally start with white which we can set with this code before we start creating our charts:

sns.set_style('white')

From there, I sometimes remove the border not along the labeled axes. Seaborn has a method that takes care of this for us:

sns.despine()

The despine method can also accept values for specific sides and offsets. I usually start with the defaults and then fine-tune, if needed.

So, if we use these with our graph, our code looks like this:

import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns

# Create our dataframe.
df = pd.read_csv('example-df.csv')

# Set the default theme.
sns.set_style('white')

# Create our chart.
sns.lineplot(data=df, x='date', y='total_revenue')

# Add our text.
plt.title('Our revenue continues to rise', fontsize=18, loc='left')
plt.xlabel('Day', fontsize=12)
plt.ylabel('Revenue', fontsize=12)
plt.tick_params(axis='both', labelsize=8)

# Set the y axis to start at 0.
plt.ylim(bottom=0)

# Remove extra borders.
sns.despine()

This results in a graph like the following image.

While there is a lot more we could do with this graph using Seaborn's functionality, this is already a chart that can be used in your analysis or sharing with colleagues. Great job learning the basics of Seaborn!

Next Steps

Now that you know how to create a basic graph using Seaborn, go ahead and try to create a few charts. From there, you will want to try adjusting the styling more as well as creating other chart types. Here are a few ideas to get you started:

Try to change the color of the title to a color that matches a brand you like.
Try to change the color and width of the line.
Try creating a bar graph using Seaborn.

Be sure to follow to get notified when my next article is published!

AWS Lambda 101: How To Create an Endpoint for a Lambda Function Using API Gateway

Frank Corso — Sat, 01 Jan 2022 19:47:31 +0000

In my previous articles in this series, I looked at setting up an AWS Lambda function and setting it up on a recurring schedule. Another common use of a serverless function is connecting it to an endpoint to be able to send data to it or receive data from it.

For example, maybe your server needs to fire off a request to a Lambda to trigger a new calculation. Or, maybe your web app needs to pull in data that is processed by a Lambda.

In this article, I will set up a basic Lambda function that accepts two numbers, multiplies them together, and then returns the results. We will then connect this to an endpoint that we can send some JSON data to. If a user sends JSON with keys of a and b with both being a number, they will get back the result of the two multiplied. To do this, we will be using AWS's API Gateway to handle the endpoints. Let's get started!

Writing the Code for our Lambda function

To get started, let's write a very basic lambda handler:

def lambda_handler(event, context):
    return 'Something'

In order to use the numbers being sent to our Lambda function, we will need to use the event parameter. This parameter contains any data sent to the Lambda function. If we were triggering the Lambda function manually, we would use the event variable as an exact representation of the data.

For example, if we had passed only the number 16 to the Lambda function, event would be equal to 16.

However, with the API Gateway endpoint, the event variable will include a variety of other data with the JSON sent being included within the body key. If we were to print our the event variable for a Lambda triggered by an endpoint, it would look like:

{
    "version": "2.0",
    "routeKey": "GET /testFunction",
    "rawPath": "/testFunction",
    "rawQueryString": "",
    "headers": {},
    "requestContext": {},
    "body": "{\"exampleJSONKey\": \"Random text\"}",
    "isBase64Encoded": false
}

So, in order to get the JSON sent with the request, we will use event['body'] but we will need to decode the JSON. This will look like this:

import json

def lambda_handler(event, context):
    body = json.loads(event['body'])
    return 'Something'

For this example project, the Lambda function will be expecting the user to send in JSON with keys of a and b to be multiplied together. So, we can add then use these in the return like this:

import json

def lambda_handler(event, context):
    body = json.loads(event['body'])
    return body['a'] * body['b']

However, if someone sends in values that are not numbers, this will result in an error or unexpected results. So, we should create a small conditional to ensure both are valid numbers.

import json

def lambda_handler(event, context):
    body = json.loads(event['body'])
    if not is_numerical(body['a']) or not is_numerical(body['b']):
        return 'One or both of the values are not a number'
    return body['a'] * body['b']


def is_numerical(value):
    """"Checks if value is an int or float."""
    return isinstance(value, int) or isinstance(value, float)

Since the user is sending JSON to the endpoint, they would probably expect JSON as the response. Luckily, Lambda will automatically take care of this process if we return Python objects. So, we can update our returns:

import json

def lambda_handler(event, context):
    body = json.loads(event['body'])
    if not is_numerical(body['a']) or not is_numerical(body['b']):
        return {
            'error': 'One or both of the values are not a number'
        }
    return {
        'result': body['a'] * body['b']
    }


def is_numerical(value):
    """"Checks if value is an int or float."""
    return isinstance(value, int) or isinstance(value, float)

Lastly, the user would encounter a Lambda error if they are not submitting both a and b. So, let's add one last conditional to check that:

import json

def lambda_handler(event, context):
    body = json.loads(event['body'])
    if 'a' not in body or 'b' not in body:
        return {
            'error': 'You must provide keys for both "a" and "b"'
        }
    if not is_numerical(body['a']) or not is_numerical(body['b']):
        return {
            'error': 'One or both of the values are not a number'
        }
    return {
        'result': body['a'] * body['b']
    }


def is_numerical(value):
    """"Checks if value is an int or float."""
    return isinstance(value, int) or isinstance(value, float)

Great! Our code is looking good so let's get this into a Lambda function.

Setting Up Our Lambda Function

First, let's go ahead and create a new Lambda function. If you've never set up one before, check out my "How To Create Your First Python AWS Lambda Function" article.

Inside the Lambda function admin area, paste the code above into the "Code" tab as shown below:

Click the white "Deploy" button right above the code to finish setting it up.

Great, our Lambda function is now ready!

Setting Up Our API Gateway Endpoint

In API Gateway, there are many different types of APIs that you can create. Since we are creating a single endpoint, we will use HTTP API. However, there are many other types you could consider such as REST and WebSocket.

Additionally, you can have each endpoint accept different HTTP request types, such as GET and POST, and you can also use "stages" to have different versions of each for development. To keep our endpoint simple, we will only use the default "stage" and accept all HTTP request types.

To get started, in your AWS console, you can either click on "Services" to go to "Networking & Content Delivery"->"API Gateway" or use their search bar to search for it.

If you have not created any APIs yet in API Gateway, you will be directed to the page to select which type of API you want to build. If you already have created APIs click the APIs link in the left navigation menu and then click "build". In either case, select the "HTTP API" option.

Once on the "Create an API" page, add an integration and select Lambda. Once selected, you will be prompted to select which Lambda function. Underneath, name your API. I like to use something descriptive that is similar to the Lambda function's name.

If you wanted to limit the HTTP request types allowed on this endpoint, the path for the endpoint, or the "stages" you can click "Next" to set those up. For this endpoint, I'll keep the defaults and click "Review and Create". On this page, we can verify the name, integrations, routes, and stages.

Click the "Create" button to finish creating this endpoint. Now, you will see the domain of this new API.

We can use this domain with our route to craft our final endpoint. For example, for the endpoint I created, the full URL is https://j485uzw9v0.execute-api.us-east-1.amazonaws.com/exampleEndpointFunction. If you do not remember what the route was, you can click the "Routes" link in the left navigation to see all routes on the API.

Great, our endpoint is set up!

Testing Our Endpoint

Now, let's see it in action. Open your favorite API tester, such as Postman. Create a new request with any HTTP method and paste in the full URL to your API route.

Then, inside the body, create a new JSON object with keys of "a" and "b" and click send. You should see something like this:

Then, try passing something instead of numbers or omit one of the fields to see one of the error messages like this:

Great! Everything looks like it's working correctly. We have now successfully created a Lambda function that is accessible from a URL.

Next Steps

Now that you have set up a Lambda function and connected it to an API, you are on your way to creating new serverless endpoints that can accept and return data. In future articles, I'll look at safely passing credentials to it and adding other Python libraries to it.

Be sure to follow to get notified when my next article is published!

AWS Lambda 101: Setting Up a Recurring AWS Lambda Function Using AWS EventBridge

Frank Corso — Tue, 02 Nov 2021 14:01:10 +0000

There are many ways for triggering a Lambda function. You could manually run it. You can have it run automatically when a new object is added to an s3 bucket. You can have it run when someone accesses a specific endpoint.

One common need is to have the Lambda function run on a recurring schedule. We can achieve this using AWS's EventBridge.

Our AWS Lambda Function

Before we set up our schedule, let's quickly create our Lambda function. If you've never set up one before, check out my "How To Create Your First Python AWS Lambda Function" article.

For this example, I will use a very basic Lambda function that simply prints out the time the event triggered the Lambda function. When EventBridge triggers a Lambda function, it will pass a variety of data to the event object including a time value.

def handler(event, context):
    '''Gets called by the Lambda environment.'''
    print(event['time'])

Now, I'll create my Lambda function and copy in this code as shown in the screenshot below.

If you are copying in that same sample code, remember to update the "handler" setting in the runtime settings. Then, click the "Deploy" button.

Setting the AWS Lambda Function on a Recurring Schedule

Now that I have the sample function in place, let's go to AWS's EventBridge.

To get started, you will need to sign in to your AWS account.

Once in your AWS console, you can either click on "Services" to go to "Application Integration"->"Amazon EventBridge" or use their search bar to search for Amazon EventBridge.

On the EventBridge page, you will see some navigation along the left. Click on "Rules".

On the rules page, click the orange "Create rule" button along the right.

On the rule creation page, you will need to fill in a few fields. First, enter in a name. I recommend something that includes what it does and how often. For example, something such as "cleanTweetDataHourly". Next, fill in a description.

In the "Define pattern" section, click on "Schedule". Now, you will have two options. First, you can enter a fixed rate, such as every 1 hour. Or, you can use a cron expression.

To demonstrate the Lambda working, I am going to go with a fixed rate of once every 1 minute. If you are following along, I strongly recommend another rate or making sure you disable this so you don't encounter a large bill at the end of the month.

You can skip the event bus section for scheduled rules. If you were using rules for something beyond scheduled Lambda functions, you can set up different event buses to act as different event pipelines. Since that isn't supported on scheduled rules, we can skip it.

In the "Select targets" section, select "Lambda function" for the target and then select your Lamdba function in the "Function" dropdown. For most functions, you will keep the default settings for the function here.

If you have a lot of rules, you can set up some tags to organize them. I'll skip that here.

Lastly, click the "Create" button.

Great job! Your Lambda is now set on a recurring schedule. We can verify this by going back to our Lambda function and clicking the "Monitor" tab.

Here, we can see when the function was invoked, how long it ran, and other metrics. We see our function being invoked on a recurring schedule. We can also click the "Logs" tab to view logs from our function. If we click into one, we will be able to see where it printed out the time from the event sent by EventBridge.

If you are following along and are finished testing your rule, you can disable or delete the rule by going back to the "Rules" page. If you select your rule, you can click the "Disable" button to stop the function from running or the "Delete" button to remove the rule entirely.

Next Steps

Now that you have set up a Lambda function and set it up on a recurring schedule, there is so much you can do such as having a Twitter bot tweet on a recurring schedule or having a script to clean your data that gets run every hour. In future articles, I'll look at safely passing credentials to it and add other Python libraries to it.

Be sure to follow to get notified when my next article is published!

AWS Lambda 101: Creating Your First Python AWS Lambda Function

Frank Corso — Mon, 06 Sep 2021 13:46:13 +0000

When creating a small Python script, the question that eventually comes up is where the code will be hosted and run. For some small scripts, you might be able just to run the code locally. But, what if you want the code to be run on a regular schedule, even when your laptop is off? Or, what if you want the script to be part of an endpoint that other scripts can access?

One option would be to set up a server and running the code there. You could spin up a droplet on Digital Ocean, set up the Python environment, and run your scripts there. But, what if, you do not want to have to set up and manage servers?

This is where serverless functions can help! Serverless functions are scripts that you can set up without needing to manage the underlying infrastructure. Additionally, these systems auto-scale so you can easily run the function a few times a year or thousands of times per day without you needing to figure out how to scale.

AWS's Lambda is a great serverless function system that tightly integrates with the rest of the AWS ecosystem. AWS Lambda allows you to create serverless functions in various environments, including Python, NodeJS, and Go. Even better, the AWS free tier includes 1 million Lambda requests per month!

Writing Your First Python Lambda Function

For the most part, Python code within AWS Lambda is almost the same as writing normal Python scripts. However, within Lambda, you will need to specify a handler function which is then called by the environment. In most of my Lambda functions, I call this something similar to handler or lambda_handler. This looks like this:

def handler(event, context):
    '''Gets called by the Lambda environment.'''
    return true

For the handler function, we have two parameters: event and context.

The "event" parameter is an object that contains data that was passed to the Lambda function. For example, if the Lambda was attached to an endpoint, then we could access the data sent to the endpoint with something like event['body'].

The "context" parameter is an object that contains data about the Lambda function itself and its runtime environment.

Let's go ahead and write a very basic Lambda function that will accept a value and then return its square root.

import math

def handler(event, context):
    '''Gets called by the Lambda environment.'''
    return math.sqrt(event)

Creating Your Lambda Function

To get started, you will need to sign in to your AWS account.

Once in your AWS console, you can either click on "Services" to go to "Compute"->"Lambda" or use their search bar to search for Lambda.

Once inside Lambda, you will find your Lambda dashboard as shown here:

Click on the "Functions" page from within the menu to get to your functions.

Next, click on the "Create function" button in the top right.

Use the "Author from scratch" option. Name your function anything you want. For the runtime, select a version of Python.

If this function were interacting with other parts of the AWS ecosystem, we'd want to set up permissions or an AWS role. But, for this example, we can keep the defaults.

Next, click the final "Create function" button.

Once created, you will be taken to the function's overview and code. Here you can review any triggers or layers attached to the function. For this one, we will not be using these.

Scroll down to the code section and open the default lambda_function.py file. Copy and paste our basic Python code from above into the file. Then, click the "Deploy" button.

Next, scroll down to the "Runtime settings" section.

We want to edit the "Handler" field. This is how Python will try to import your function. So, it should be the file name followed by the function name. In this case, it should be updated to lambda_function.handler because we kept the default file but used our function called handler. (Note: We could have made this easier by just naming our function the default Lambda one but then we wouldn't have been able to look at this setting in this tutorial 😉)

Now, scroll back up to the top and switch to the "Test" tab. Replace the example JSON with any number, such as 16. Next, click the "Test" button.

A box will appear with the execution results, which will include the square root value. This area will also display some information about the request, such as duration time.

Congratulations! Your very first Lambda function has been created.

Next Steps

While it's useful to explore how the function works, you probably want to do a lot more with your Lambda function. In the next article in this series, I look at setting up a Lambda function on a recurring schedule. In future articles, I'll look at safely passing credentials to it, connect it to an endpoint, and add other Python libraries to it.

Be sure to follow to get notified when my next article is published!

SQL 101: How to Insert and Update Data From Databases Using SQL

Frank Corso — Wed, 18 Aug 2021 12:53:44 +0000

In my previous posts in this series, we explored how tables are set up and how to select data from them. But, what if you need to add or edit the data in the tables? That is what we are exploring in this article.

The INSERT Statement

In order to insert data into a table, you will use the INSERT INTO and VALUES statements. The SQL will look like this:

INSERT INTO table_name (column_1)
VALUES ("some_value");

To begin, we need to specify which table we are inserting data into. From there, we need to state which columns you are adding data for by listing them in a comma-separated list enclosed in parenthesis. Lastly, we then set the data for each of those columns using the VALUES statement, in a similar list as the column names.

Let's look at a table we explored in an earlier article in this series:

In this table, we have a list of pet owners. If we wanted to add a new owner, such as Jessica, we would use this SQL:

INSERT INTO Owners (name, email, phone)
VALUES ("Jess", "fake@fake.com", "555-1234");

You may notice that we are not setting the value for the id column. In most tables, the primary key is set to auto-increment so by not specifying the value, it will automatically be set to the next number. Similarly, the table you are working on may have other columns that have default values in which you could omit those in your SQL unless you are setting the values to something other than the default.

So, our table will now look like this:

In addition to our table of pet owners, we had also looked at our pets table, shown below.

Now, let's say Jessica has two pets: Brownie and Mr. Whiskers. To insert multiple rows, you can include as many value lists after the VALUES keyword as you need. We can use the same SQL syntax as before but we can pass two sets of values as shown below.

INSERT INTO Pets (owner_id, name, birthday, type)
VALUES (4, "Brownie", "02/23/2021", "dog"),
(4, "Mr. Whiskers", "12/04/2020", "cat");

This will get us an updated table that looks like this:

The UPDATE Statement

Once you have data in your tables, there will be many times where you need to update that data. To do this, we use the UPDATE statement. The syntax looks like this:

UPDATE table_name
SET column_name = "some value"
WHERE column_name = "some other value";

Notice the WHERE statement in this SQL. If your update SQL does not have a WHERE condition, it will update the value for all the records in the table. It is important to remember to almost always use the WHERE condition.

Let's say that when we entered the information for "Mr. Whiskers" that we accidentally put in the wrong birthday. To update his birthday, we can use this SQL:

UPDATE Pets
SET birthday = "12/14/2020"
WHERE id = 5;

With our WHERE statement, we are only updating the value for the birthday column where the id is 5, which is the row for Mr. Whiskers.

We can update as many columns as we want during this SQL. Let's say that we accidentally entered the wrong birthday and misspelled Brownie's name when we entered her into the table. We can update her row using this SQL:

UPDATE Pets
SET birthday = "02/13/2021", name = "Brownie"
WHERE id = 4;

Like with the SELECT statement, we can use as many conditions in the WHERE statement as needed. So, if we had accidentally set both Mr. Whiskers and Brownie as belonging to Chris instead of Jessica, we could update both of their owner_id values using this SQL:

UPDATE Pets
SET owner_id = 4
WHERE id = 4 OR id = 5;

* A quick side note: We can simplify this condition using the IN statement with SQL like this: WHERE id IN (4, 5). We'll look more at advanced statements and conditions in a future article in this series but this is a great example of when you could use the IN statement so wanted to mention it.

Next Steps

Now that you can select, insert, and update data within a database, you are on your way to becoming an SQL expert! To continue on your training, you will want to:

Look into how to delete data using the DELETE statement
Start learning how to join multiple tables together

Be sure to follow to get notified when the next entry in this SQL 101 series (and other articles on SQL + data) is published!

SQL 101: How To Select Data From Databases Using SQL

Frank Corso — Mon, 29 Mar 2021 13:30:05 +0000

In my previous article, I wrote about how databases and tables are structured. In this article, I will explore how to query those tables to get data from them.

We use a language called SQL for querying databases. The implementation for SQL will differ slightly between different database types, such as MySQL vs. Postgres, but the majority of the language will be consistent across all database engines.

For most queries, you will have a series of statements -- these are usually all uppercase for readability -- and ending with a semi-colon. For many database engines, the semi-colon is optional, but some require it. Additionally, if we are running multiple queries in a row within the same action, you will need the semi-colon to tell the engine when each statement ends. So, I tend to always use a semi-colon.

For most database engines, white space does not matter, so you could write a query like:

SELECT name FROM Customer WHERE LTV > 25;

SELECT name
FROM Customer
WHERE LTV > 25;

For readability, I tend to have each main statement on its own line unless I am writing the query inside some code, such as in a Python script.

Now that we covered the basics, let's look at what these statements are and how to use them.

The SELECT and FROM Statements

In order to begin selecting data, we need to learn the two most common SQL statements: SELECT and FROM. In the most basic usage, these statements work like this:

Using the SELECT statement, we specify exactly what we want to retrieve from our database.
Using the FROM statement, we specify what table we are retrieving data from.

Let's look at a very basic example. We will use our Pets table that we outlined in my previous article. It looks like this:

If we wanted to get all the data from this table, we would use this query:

SELECT *
FROM Pets;

In a SELECT statement, the * is the same as "get all columns". If we use this command in our database with the Pets table, we would see all the columns and rows of the table.

But, maybe the table has a lot of columns, and we only need 2 or 3. Instead of using *, we can list the columns we want to retrieve. This is almost always the preferred way as you are only searching for and retrieving the data you actually need.

Let's get only the name and animal type of each pet. The SQL query will look like this:

SELECT name, type
FROM Pets;

This will give us something that looks like this:

The WHERE Statement

For the majority of queries, it is unlikely that you will want to select all rows within a table. Instead, you will probably be only looking at specific rows. This is when we use the WHERE statement.

The WHERE statement takes a condition (or multiple conditions) and returns rows only when those conditions are true, such as if a column equals a certain value. If the column type is numerical, such as INT, you can use comparison operators, such as > or <.

Let's say we want to only get the name and birthday for only pets that our cats. We would only want rows where type is equal to cat. The SQL would look like this:

SELECT name, birthday
FROM Pets
WHERE type = 'cat';

This will give us something that looks like this:

You can also have multiple conditions in your WHERE statement using either AND or OR. Let's say we wanted to get all dogs named "Spot". The SQL query would be:

SELECT name, birthday
FROM Pets
WHERE type = 'dog' AND name = 'Spot';

which would return this:

The LIMIT Statement

For many queries, you may only want to look at the first set of rows that match your WHERE condition. This is where the LIMIT statement comes in.

Using the LIMIT statement, we can specify how many rows to return. More importantly, the database will stop searching once it reaches this limit. If your table has two million rows, searching all the rows can take a while. Limiting the results to just the first matching rows can significantly speed up how fast the query can return results.

Let's say we only want the name and birthday of the first dog in our Pets table. Our SQL would be:

SELECT name, birthday
FROM Pets
WHERE type = 'dog'
LIMIT 1;

Next Steps

You are now ready to get out and start querying some data from databases. But, we have only looked at the most basic versions of each of these statements, and there is much more that you can do.

You will want to look into aggregation functions, such as COUNT() and SUM(), as well as more advanced statements, such as ORDER BY, GROUP BY, and HAVING. Lastly, you will want to learn about joining tables together using JOIN.

Be sure to follow to get notified when the next entry in this SQL 101 series (and other articles on SQL + data) is published!

SQL 101: How Databases and Tables Are Structured

Frank Corso — Thu, 04 Feb 2021 15:28:10 +0000

If you are just starting to learn SQL (or even if you know it!), it is essential to understand how databases and tables are structured.

Each site or project will have its own database. The databases will contain most or all of the data for the project.

There are a lot of different database technologies and platforms including MySQL, PostgreSQL, MariaDB, and more. But, before you jump into any of them, let's take a high-level overview of how the databases are structured.

To do so, we will explore structuring a very basic database for a pet groomer, which will include all the pet owners, each of their pets, and which toys the pets like.

I'll be creating something called an "entity-relationship diagram" to demonstrate how databases are structured. While these are usually fairly complex yet useful, I am creating a very simplified version for learning purposes. However, I encourage you to learn more about them as you continue learning SQL as they can be valuable in creating models and representations for databases you are working with.

What Are Tables?

Within the database will be a series of tables. Each of these tables will represent a main concept or thing, referred to as an "entity." For example, one of our tables will contain all of our pet owners. Another table will include all of their pets.

These tables look a lot like spreadsheets with rows and columns. However, while represented that way, they are actually very different in how they work and how the database stores the data.

Setting Up Our Pet Owners Table

We want to store many things about each pet owner in our table, including their name, email, and phone number. These are called our pet owner's "attributes."

Here is a very basic diagram of our pet owners, displayed as an "entity-relationship diagram":

These different attributes will become the "columns" of our table, where each attribute is its own column, as shown in the table.

Now, you may have noticed there was an additional column we didn't have listed as an attribute. The "id" column is our "primary key." The primary key is what represents this data elsewhere in the database as well as throughout any application or site that uses the database.

Let's add a few people into our pet owners table.

As you can see, each entry we add into the table becomes a new "row" in the table. And, each entry gets its own primary key assigned to it.

Setting Up Our Pets Table Using One-to-Many Relationship

Our pets will also have some data about them including their name, type of animal, and their birthday.

The tables for our pet owners and their pets will be related. We want to know who owns which pet. This will be a "relationship," which is what we call the way two tables are connected.

There are many types of relationships that you will come across as you learn SQL. For now, we will stick with one of the most common, called the "One to Many" relationship. In this connection, a row in one table can be connected to many rows in another table.

For example, in our database, one pet owner can own many pets, but each pet will only have one owner.

Let's add a few pets into a pets table.

Now, you might have noticed another column that wasn't an attribute: the "owner_id" column. This is called a "foreign key." A foreign key is an identifier that matches a row's primary key from another table. This column tells us which pet owner owns this animal.

For example, if we look for "Frank" in our pet owners table, we see his id is 1. If we look in the pets table at the pet_owner_id column and find the 1, we can see that he has two pets: Bambino and Skittles.

Setting Up Our Toys Table Using Many-to-Many Relationship

Let's add one more thing to our pet groomer's database. Maybe our pet groomer likes to surprise the pets with a toy they like. So, we need to know which toys which pets like. The toys may have a few attributes like name, color, and cost.

Next, let's consider what type of relationship this table will have with our pets table. Each pet might like multiple toys, like our one-to-many relationship for pet owners and pets. However, each toy could be liked by multiple pets too.

This is where the 2nd most common relationship comes in: the "Many to Many" relationship.

In a many-to-many relationship, each row in each table can be attached to multiple rows in the other table. For example, the pet Bambino might like both the ball and the mouse. But, the ball might be liked by the pets Bambino and Spot.

In the one-to-many relationship before, we added a foreign key to represent which one row the pets were attached to. However, since Bambino likes two toys, we couldn't add both toys into a single column for the pet. Likewise, since the ball is liked by two pets, we couldn't add both pets into a single column for the toy either.

Instead, in a many-to-many relationship, we'll create a separate table to represent this relationship.

So, we will create a 3rd table for toys but then create a 4th table too. In this 4th table, which we'll call "pet_toy," we will keep a record of which pets like which toys. You may notice that pet_id and toy_id look similar to the owner_id from before, and you would be right. Each of these columns will be a foreign key that represents the pet or toy from their respective table.

So, if we want to say Bambino likes the ball and mouse toys, we would add two new rows as shown.

Then, to add the toys that Spot likes, we will add a few more rows.

With our four tables set up, our database for the pet groomer is now ready!

Next Steps

Now that you have a high-level overview of how databases are structured, you are ready to start learning how to get data out of the database using SQL. Or, if you want to learn more about the structure first, I would encourage you to learn about the different types of columns, including INT, VARCHAR, and TEXT.

Be sure to follow to be alerted when I post my next articles in my SQL 101 series!

How To Save Data From Help Scout’s API to CSV in Python

Frank Corso — Fri, 29 Jan 2021 17:14:27 +0000

There are many times where you need to download data from an API and store it into a CSV. Perhaps, you want to analyze some data. Or, you want to combine the data with other data sets.

In this article, we are going to explore how to do this while using Help Scout's API. Help Scout is a help desk solution that teams use when communicating with their customers.

We will retrieve the "conversations" from one of our Help Scout mailboxes and store that data into a CSV. These conversations will include who sent the email, the subject, dates, which of our team members were assigned the conversation, and when it was closed.

Getting Your App ID and Secret

Before we can begin getting data from the Help Scout API, we need to generate an app ID and secret. First, log into HelpScout and go to "Your Profile."

Click the "My Apps" in the left menu.

Click the "Create My App" button and fill in a name. Since we won't have users authenticating with this app, you can enter any URL for the redirect. I usually just put the URL for the company's site.

Click the "Create" button. Once created, you will see your app's ID and Secret.

While in Help Scout, open the mailbox you want to retrieve the conversation for. In the URL, you will find the mailbox's ID, which we will need later:

Getting Your Access Token

Since we won't be authenticating users, we will use Help Scout's client credentials flow for authentication. To do so, we first need to get an access token.

First, we need to import any packages we need. I almost always use the "requests" package when working with APIs. Additionally, since we have the created time and closed time, I like to go ahead and calculate resolution time here. To do so, I will use Python's "datetime" library.

import csv
import datetime
import requests

Next, we need to send our app's ID and secret to their token endpoint. We can do this by making a request and getting the access_token key from the JSON they return.

# The token endpoint.
auth_endpoint = 'https://api.helpscout.net/v2/oauth2/token'

# Preparing our POST data.
post_data = ({
    'grant_type': 'client_credentials',
    'client_id': YOUR_HELPSCOUT_APP_ID,
    'client_secret': YOUR_HELPSCOUT_APP_SECRET
})

# Send the data.
r = requests.post(auth_endpoint, data=post_data)

# Save our token.
token = r.json()['access_token']

We need way to keep going through our loop until we have downloaded all the conversations. The Help Scout api contains a totalPages key that we can check against.

There are a few different ways we could handle this but I like to just set a self-explanatory flag to false.

all_conversations = False
page = 1

Then, we need to prepare our authorization headers.

# Prepare our headers for all endpoints using token.
endpoint_headers = {
    'Authorization': 'Bearer {}'.format(token)
}

Preparing the CSV File

Now that we have our token, we can set up our CSV file. If you haven't worked with CSV's in Python before, check out my "Reading From and Writing to CSV Files in Python" article.

First, open our CSV file and write the header:

# Creates our file or rewrites it if one is present.
with open('conversations.csv', mode="w", newline='', encoding='utf-8') as fh:
    # Define our columns.
    columns = ['ID', 'Customer Name', 'Customer email addresses', 'Assignee', 'Status', 'Subject', 'Created At',
               'Closed At', 'Closed By', 'Resolution Time (seconds)']  
    csv_writer = csv.DictWriter(fh, fieldnames=columns) # Create our writer object.
    csv_writer.writeheader() # Write our header row.

Downloading Our Conversations and Putting Them Into the CSV

Now that we have our access token and our CSV opened, it's time to retrieve the conversations. Help Scout's API sends over the conversations in a paginated format.

Remember to keep all of this indented inside the with open() statement so we can write to our CSV.

Inside a while loop, we will get our first page of conversations:

while not all_conversations:
    # Prepare conversations endpoint with the status of conversations we want and the mailbox.
    conversations_endpoint = 'https://api.helpscout.net/v2/conversations?status=all&mailbox=YOUR_MAILBOX_ID&page={}'.format(
        page
    )
    r = requests.get(conversations_endpoint, headers=endpoint_headers)
    conversations = r.json()

Next, we will cycle over each of the conversations within this page to save them to the CSV. Since there are many keys in the JSON that might be missing depending on the conversation, we need to clean and process the data just enough to save it.

# Cycle over conversations in response.
for conversation in conversations['_embedded']['conversations']:

    # If the email is missing, we won't keep this conversation.
    # Depending on what you will be using this data for,
    # You might omit this.
    if 'email' not in conversation['primaryCustomer']:
        print('Missing email for {}'.format(customer_name))
        continue

    # Prepare customer name.
    customer_name = '{} {}'.format(
        conversation['primaryCustomer']['first'],
        conversation['primaryCustomer']['last']
    )

    # Prepare assignee, subject, and closed date if they exist.
    assignee = '{} {}'.format(conversation['assignee']['first'], conversation['assignee']['last']) \
        if 'assignee' in conversation else ''
    subject = conversation['subject'] if 'subject' in conversation else 'No subject'
    closed_at = conversation['closedAt'] if 'closedAt' in conversation else ''

    # If the conversation has been closed, let's get the resolution time and who closed it.
    closed_by = ''
    resolution_time = 0
    if 'closedByUser' in conversation and conversation['closedByUser']['id'] != 0:
        closed_by = '{} {}'.format(
            conversation['closedByUser']['first'], conversation['closedByUser']['last']
        )
        createdDateTime = datetime.datetime.strptime(conversation['createdAt'], "%Y-%m-%dT%H:%M:%S%z")
        closedDateTime = datetime.datetime.strptime(conversation['closedAt'], "%Y-%m-%dT%H:%M:%S%z")
        resolution_time = (closedDateTime - createdDateTime).total_seconds()

Then, write our results to the CSV:

# Write row to CSV
csv_writer.writerow({
    'ID': conversation['id'],
    'Customer Name': customer_name,
    'Customer email addresses': conversation['primaryCustomer']['email'],
    'Assignee': assignee,
    'Status': conversation['status'],
    'Subject': subject,
    'Created At': conversation['createdAt'],
    'Closed At': closed_at,
    'Closed By': closed_by,
    'Resolution Time (seconds)': resolution_time
})

Lastly, check their 'totalPages' key against the page we are on to see if we have retrieved all the conversations. If not, increment the current page and then we'll start the process over again.

if page == conversations['page']['totalPages']:
    all_conversations = True
    continue
else:
    page += 1

Great! We now have our script ready to run. You can see all this code together in my GitHub gist here.

Next Steps

Go ahead and run your script to retrieve all the conversations from your Help Scout mailbox.

If you were looking to just get a backup, then you are good to go. If you wanted the data to perform data analysis, you can now open the file in Excel or use pandas with it. Or, if this was the first step of your data pipeline, you could now start working on the next step in the pipeline.

Python 101: Reading From and Writing to CSV Files

Frank Corso — Tue, 12 Jan 2021 13:11:06 +0000

One of the most common things you will do in Python is working with CSV files. Whether it is to create a CSV file with data from an API or perform some analysis on an existing CSV, working with CSV files is an essential step in learning Python.

Opening Files in Python

When working with files, your first step is to open the file. In Python, this looks like this:

fh = open('somefile.csv')

The open function the file and return a file object, usually set to an fh variable for "File Handler". By default, Python's open function will open in reading mode. You can change this with the mode argument. If you wanted to open a file, delete everything in it, and then start writing to it, you would use:

fh = open('somefile.csv' mode='w')

Or, to open for reading and writing, you would use:

fh = open('somefile.csv' mode='r+')

Lastly, if you wanted to open the file to write to but keep the existing data, you use a for "append" like this:

fh = open('somefile.csv' mode='a')

Once you open the file, you need to close it using the close() method:

fh.close()

Instead of using the close() method, you can also use the with statement. Since I often forget to close the file (resulting in many issues), this is my preferred approach when I only need to open one file.

fh = open('somefile.csv' mode='w')

# Do stuff

fh.close()

is the same as

with open('somefile.csv' mode='w') as fh:
    # Do stuff

Opening CSV Files in Python

While the above works with a lot of CSV files, there are two more arguments to the open() function you may need to use.

Newlines

The first is the newlines argument. Depending on how the CSV was created and where it may be used, the newlines may be saved a few different ways. So, we want to turn on "universal newlines mode" but return the actual unaltered line endings by setting the newline argument like this:

with open('somefile.csv', newline='') as fh:
    # Do stuff

Encoding

Depending on the encoding of the file, you may also need to specify the encoding argument. This would look like this:

with open('somefile.csv', newline='', encoding='utf-8') as fh:
    # Do stuff

Reading and Writing Your First CSV File

To get started, let's do a simple exercise of writing a row to a new file and then reading that row.

To work with CSV files, we first need to import the csv module.

import csv

Then, we will use it's writer() method to create a writer object that will perform the writing for us. We will then use its writerow() method to write a row in our CSV.

with open('example_one.csv', mode='w', newline='') as fh: # Open our file.
    csv_writer = csv.writer(fh) # Create our writer object.
    csv_writer.writerow(['Frank', 'Corso']) # Write our row of data.

Each item in the list given to writerow() will be in a different column within the CSV.

Now that we have our CSV created, we can now read it to get the first row. This time, we will use csv's reader() method to create a reader object which can be iterated over.

with open('example_one.csv', newline='') as fh: # Open our file.
    csv_reader = csv.reader(fh) # Create our reader object.
    for row in csv_reader: # Iterate over each row.
        print(row) # Print entire row
        print(row[0], row[1]) # Print only certain columns

['Frank', 'Corso']
Frank Corso

Each row in this example would be one row within the CSV file with each item corresponding to a column in the row.

Reading and Writing CSV Files With Headers

Instead of using writer() and reader(), we can use DictReader() and DictWriter() when working with headers in our CSV file.

If we wanted to write the same data from above but set the columns as "FirstName" and "LastName", our code would look very similar. The main difference is that DictWriter() requires fieldnames to know the names of the columns.

with open('example_two.csv', mode='w', newline='') as fh: # Open our file.
    columns = ['FirstName', 'LastName'] # Define our columns.
    csv_writer = csv.DictWriter(fh, fieldnames=columns) # Create our writer object.
    csv_writer.writeheader() # Write our header row.
    csv_writer.writerow({'FirstName': 'Frank', 'LastName': 'Corso'}) # Write our row of data.

If the CSV you are reading from has headers, you can use the DictReader() method instead to not only skip the header row but also create dictionaries for the rows instead of just lists.

We will use similar code to before with only a slight change:

with open('example_two.csv', newline='') as fh:
    csv_reader = csv.DictReader(fh) # Create our reader object, automatically using the first row as the column names.
    for row in csv_reader: # Iterate over each row.
        print(row) # Print entire row
        print(row['FirstName'], row['LastName']) # Print only certain columns

{'FirstName': 'Frank', 'LastName': 'Corso'}
Frank Corso

Next Steps

Now that you can read and write CSV files, what will you do with them? You can:

download data from an API and save it to a CSV
clean data from a raw CSV and create a new "processed" CSV
analyze data from a CSV
and much more!

If you are analyzing data from CSV's, your next step may be learning pandas, the data analysis module, which can load CSV files and Excel files.

Python 101: Setting Up Your Environment With Venv and requirements.txt

Frank Corso — Mon, 14 Dec 2020 13:21:12 +0000

By default, all the Python packages you install on your computer are used within all of your projects. But, what if one project requires version 1 of a package and another project requires version 2?

Or, if you have multiple people working on a project, how do you tell them which dependencies are needed and make sure everyone is using the same versions?

This is where using requirements.txt and virtual environments come in.

What Is requirements.txt?

When you usually install a package, you probably do something like:

pip install pandas

But, imagine if you have someone else working on the project and they also install pandas. Or, maybe you are deploying your project to Netlify or are using Docker. How do you make sure the right dependencies get installed and the correct version?

With requirements.txt, you can list all the required packages for your project and what version is needed. If you are familiar with NPM or Composer, you may have seen a similar concept in their package.json or composer.json files.

Creating Your requirements.txt

In your project, you can create a requirements.txt file. Inside, you can list each package that is needed.

matplotlib
numpy
openpyxl
pandas

To make it easy to scan and read, the list is normally alphabetical. If you want a specific version, you could add an equal sign like this:

matplotlib==3.3.3
numpy==1.19.3
openpyxl==3.0.5
pandas==1.0.5

Now, it would be challenging to manually write out all your dependencies and remember to change the version number when you upgrade. Luckily, pip can help with this.

If you ever want to see what packages you have installed, you can run pip list. This will output all the packages installed and their version numbers.

Package         Version
--------------- ---------
matplotlib      3.3.3
numpy           1.19.3
openpyxl        3.0.5
pandas          1.0.5

Even better, you can use pip freeze > requirements.txt to automatically take this list and store it in your requirements.txt file.

As you add, update, and remove packages, you can run that command again to update your requirements.txt file.

Installing From Your requirements.txt

Let's say you just cloned a git repo. This repo contains a requirements.txt file. What do you do next?

Once again, pip is here to help. We can run pip install -r requirements.txt to have pip automatically install all dependencies listed in the requirements.txt file.

Using Virtual Environments

Now, if you only have one Python project on your computer, this may be fine as-is. But, what if you start cloning or creating several projects all with their own requirements.txt files? You can quickly end up with package versions that are not compatible with each other.

This is where virtual environments come in. You can set up your project to exist within its own environment and packages only installed within that environment. Each project will have its own environment and its own packages installed within it.

To create your virtual environment, go into your project and run:

python -m venv .venv

The last parameter, .venv, is the name of the directory to install the virtual environment into. You can name this whatever you would like, but I like to stick with .venv as it's easy to tell what it is and it doesn't conflict with directory names I use with other systems.

Once the command is finished, your virtual environment will be ready. Next, you can "activate" it by running the activation script.

If you are on Windows, you will use .venv\Scripts\activate.bat.

On other OSes, you will use source .venv/bin/activate.

Once activated, you will see the name of the environment within the terminal.

(.venv) fpcorso:~$ pip install -r requirements.txt

Now, you will be able to install packages and run Python within the environment without interfering with packages installed globally.

Once you are finished, just use the deactivate command to exit the virtual environment.

Next Steps

Going forward, you will want to install a virtual environment within all of your projects and install packages within. Then, use the pip freeze > requirements.txt command to generate your requirements.txt file to keep everyone working on the project in-sync.

Using Jinja for HTML Email Templates in Python

Frank Corso — Mon, 07 Dec 2020 15:55:20 +0000

When first creating emails in a new project in Python, I usually start by creating the email text within the Python code itself. This is quick and easy and gets the job done.

For example, one of my original email functions in my SAAS app, SiteAlert, looked like this:

def send_new_site_user_email(user, site_url):
    """Sends an email to the user when they add a site."""
    subj = 'You added a site!'
    body = '<p>Hey {}!</p>'.format(user.first_name if user.first_name != '' else 'there')
    body += '<p>You just added a new site to your SiteAlert account: {}</p>'.format(site_url)
    body += """<p>Now, you can sit back and relax as SiteAlert begins monitoring your site for:</p>
       <ul>
          <li>Broken links</li>
          <li>Broken images</li>
          <li>Blacklists</li>
          <li>Accessibility</li>
          <li>And more!</li>
       </ul>
     <p>This may take some time, so feel free to go about your day. We will email if we find an issue.</p>
     <p>As always, feel free to reach out if you ever need anything!</p>
     <p>~Frank Corso</p>
     <p><a href="https://sitealert.io">SiteAlert</a></p>"""
    send_email(user.email, subj, body)

The send_email function I defined elsewhere and is only a wrapper for sending emails through Postmark.

However, hardcoding the HTML into the Python code has some limitations. If you have anyone who wants to make changes to the emails but doesn't know Python, such as marketing or customer success, they cannot edit them here. Additionally, any time you make a change to the email, you risk breaking the code.

Instead, we can break these out into HTML files using Jinja! Jinja is a powerful templating engine that allows you to have dynamic content through variables, loops, blocks, and more. You can even extend templates to set up some "base" templates that all other templates build off of.

Setting Up Jinja and the Templates Directory

First, we want to set up a directory that will hold the HTML templates for your emails inside your project. I created an email_templates directory, but you can name it anything that works for you.

Next, make sure Jinja is installed in your environment using:

pip install Jinja2

Creating Your Main Template Function

Now that you have the directory set up, you will need some code to find the templates and render them.

First, import the relevant modules:

from jinja2 import Environment, PackageLoader, select_autoescape

Then, we will use Jinja's Environment to tell it where the template files are. In my case, the email templates directory is within our project module. For security, it is recommended to always have Jinja autoescape the templates for you.

env = Environment(
    loader=PackageLoader('project', 'email_templates'),
    autoescape=select_autoescape(['html', 'xml'])
)

Once we have our env variable, we can use that to create our template using:

template = env.get_template('your-template.html')

To create the final HTML, you can use the render method:

html = template.render()

Since these templates will be dynamic, Jinja will let you pass kwargs to render which will get passed to the template. For example, we use the user's name in the email, so it looks like this:

html = template.render(first_name="Steve")

To make this re-usable and easier to implement, I wrap all of this in a function within my utils.py file. Here is the final code for this part:

from jinja2 import Environment, PackageLoader, select_autoescape

def send_template_email(template, to, subj, **kwargs):
    """Sends an email using a template."""
    env = Environment(
        loader=PackageLoader('project', 'email_templates'),
        autoescape=select_autoescape(['html', 'xml'])
    )
    template = env.get_template(template)
    send_email(to, subj, template.render(**kwargs))

def send_email(to, subj, body):
    """Sends an email."""
    # Send the finalized email here.

Creating Your First Template

Now that we have our function created, let's create our first template so we can see how this works out. I will convert the "new site" email from the top of this post into our first template.

First, we can create a 'new_site.html' file in our email_templates directory. Inside our template, we want to say hey to the user and use either their first name or say "there", like in the original function. To do this, we use {{ }} syntax for the Jinja variables, and we will use a Jinja filter for setting the default. The default() built-in filter takes a second parameter where, if set to true, will use the default when the variable evaluates to false, such as empty strings. This will look like this:

{{ name|default('there', true) }}

From there, we also list the URL, but that is easier as we are not applying any filters to it:

{{ url }}

Our final template file looks like this:

<p>Hey {{ name|default('there', true) }}!</p>
<p>You just added a new site to your SiteAlert account: {{ url }}</p>
<p>Now, you can sit back and relax as SiteAlert begins monitoring your site for:</p>
<ul>
    <li>Pagespeed</li>
    <li>Broken links</li>
    <li>Broken images</li>
    <li>Blacklists</li>
    <li>Accessibility</li>
    <li>And more!</li>
</ul>
<p>This may take some time, so feel free to go about your day. We will email if we find an issue.</p>
<p>As always, feel free to reach out if you ever need anything!</p>
<p>~Frank Corso</p>
<p><a href="https://sitealert.io">SiteAlert</a></p>

As you can see, it will be much easier for someone who only knows HTML to hop in and make changes to the email.

Putting It All Together

Now that we have everything created, let's try to use our new template.

The function we defined earlier takes template, to, subj, **kwargs as arguments. The kwargs are then passed to the render() method.

So, within our code for adding the new site, we will call this function using this:

send_template_email(
    template='new_site.html',
    to=user.email,
    subj='You added a site!',
    name=user.first_name,
    url=site.url
)

Our user and site objects are from our ORM (peewee) but you can pass any strings or variables to the kwargs.

We have now set up our email to be sent using our HTML template!

Next Steps

Now that you have the basic template created, try to convert some more of our emails into the HTML templates. If you want to take it a step further, look into how to create a base template which you can then extend for the individual emails.

Deploying Your Pelican Static Site To Netlify

Frank Corso — Fri, 30 Oct 2020 20:55:58 +0000

This was originally published over on frankcorso.dev.

In my previous article, I wrote about how to create a Pelican site. But, creating a site is only half the setup. You need to host the site somewhere in order for people actually to see it.

There are many different ways to host a site including: hosting on GitHub Pages, AWS, and Digital Ocean. But, one of the simplest solutions for a static site is using Netlify. Even better, they have a free plan!

Add A Requirements File

First, we want to make sure we have our requirements.txt file added to our project. This is a standard file in Python projects to ensure that the environment the script is run in contains all the dependencies needed.

Netlify will install the dependencies (specifically Pelican) when deploying, so it's important we have this file.

The file itself only needs to have pelican, but you may need to add others if you have added other packages for running your site.

pelican[Markdown]

Setting Up Your GitHub Repo

In order to continuously deploy your site over to Netlify, you will want to set up a repo on GitHub that contains your site. This repo can be public or private.

Since we will be building the files when deploying, you can ignore the output directory if you want to do so.

Creating Your Netlify Account

If you haven't already, you will need to create a new Netlify account. You can go over to their signup page and sign in with GitHub:

Once signed up, you will need to grant Netlify access to your GitHub repos. You can either grant Netlify access to all repositories or just grant access to your specific repo for the site.

Setting Up Deployment

Next, if Netlify did not start the "Create a new site" wizard automatically (or if you already had a Netlify account), you will need to click the "New site from Git" button.

On the next screen, you may need to click the GitHub button. From there, you should see a place to select your site's repo.

For this site, I will select my fpcorso/example-pelican-site repo. Once selected, select which branch to deploy from. In most cases, you will want to leave this the default branch, usually master or trunk.

Scroll down to the "Basic build settings" section. For "Build command" enter in pelican content -s publishconf.py. This will build your output files using the publish configuration.

In the "Publish directory" setting, enter in output/.

Click the "Show advanced" button.

Since the newer versions of Pelican require Python 3.7+, we need to make sure Netlify uses a recent version of Python. By default, it currently doesn't, which will cause errors when building your site, causing the deployment to fail.

Netlify uses environment variables to allow you to switch versions of its available platforms. Right now, the newest Python version available in Netlify is 3.7 so we will use that. Review Netlify's manage dependencies docs if you want to know more.

Click the "New variable" button and enter in PYTHON_VERSION and set it to 3.7.

Now that we have the settings entered, click the "Deploy site" button.

You should see your new site in Netlify, and the deployment should be starting up.

Depending on the size of your site, the build may take a few minutes. Once finished, it should say "Published" and a URL should be available.

Your site is now deployed!

Any time you push a change to the master branch (or whichever one you specified), you will see a new build appear in the site overview. Once built, you will automatically see your changes go live.

Next Steps

There are a lot of things you can do from here, but a few steps I recommend next would include:

Using the "Domain settings" to point a custom domain to your site instead of using the autogenerated one. (See Netlify's docs on domain management)
Explore moving your build settings and environment variables to a netlify.toml file so you can keep all configuration inside the code. (See Netlify's docs on file-based configuration)
Set up deploy notifications so you receive an alert if a deployment fails. (See Netlify's docs on deploy notifications)

Now, go share your deployed site with the world!