Forem: Dan Keefe

Harry Potter and the Accessing of APIs

Dan Keefe — Sun, 31 May 2020 10:10:38 +0000

Header image by Artem Maltsev on Unsplash

APIs are everywhere. They're a core element of modern software/services, and an incredibly powerful tool for developers. Learning how to use and access APIs unlocks an incredible number of possibilities with code.

This post explains what an API is and how to connect to one using Python, aiming to give an accessible introduction to a topic that can be bewildering to explore. It's designed for people who already have some experience with Python and are now looking to expand their skillset. Both for my own amusement, and because it's helpful to have an example to make concepts concrete, the post is structured around the Harry Potter API.

❗In the time since I wrote this tutorial, the Harry Potter API has been deactivated, a casualty of ongoing controversy. This means that the code examples will no longer run, but I'm leaving the article itself up in the hope that the explanations themselves are still of benefit to someone.

Code

This tutorial was written using using Jupyter notebooks & Python 3.7.5; things might behave slightly differently if you're in a different IDE or using different versions of the language.

You can find a complete copy of the code for this tutorial on Github.

What is an API?

An API (Application Programming Interface) is a service that provides data when asked for it. There are more specific and complex definitions, but that one is sufficient most of the time. APIs are designed to allow different machines and programs to speak to each other through code; there doesn't need to be a human in between. Most modern APIs use the HTTP(S) protocol to communicate.

APIs are everywhere - you can get weather APIs, which allow you to get data on the weather. There are Canadian football APIs, which provide data on Canadian football. There are APIs that will give you love or hate or placeholder images of kittens.

Many APIs charge for access, but many allow either totally free access, or have several usage tiers so that you can experiment for free, but would be charged if - like many new tech companies - you wanted to build an entire business on top of existing APIs.

Everyone who uses the internet interacts with APIs every day - any time you see an interactive map, or see a list of products on a website, an API is probably being used in the background. Many modern organisations and companies are built on top of APIs provided by other organisations, and may provide APIs themselves. In short, APIs are everywhere.

By connecting to different APIs, you can dramatically increase the power and scope of the software that you build; you don't need to independently map the world, or obsessively track the weather: using APIs, you can connect to services that are already doing that, building your awesome idea on top of existing structures.

Accessing an API

There are two main pieces of information you need when attempting to use an API:

Where to find the API (the endpoint)
How to make the request

API endpoints

In order to request data from an API, you need to know where to send the request. An address that an API provides for people to make requests is called an endpoint. Some APIs have just one endpoint, responding to only one type of request. More usually, an API will have several different endpoints, each one allowing you to request different information.

The Harry Potter API has a base address - https://www.potterapi.com/v1/ and then several endpoints that extend from there. If you want, for example, to get a random Hogwarts house, you can use the sorting hat endpoint - sortinghat to make that request. The full address for the sorting hat endpoint is https://www.potterapi.com/v1/sortinghat.

Because APIs communicate using HTTPS, the endpoint is a valid web address - or URL - that you can access. Visiting that address will show you a randomly-chosen Hogwarts house.

Access the Harry Potter API sorting hat endpoint.

API calls

In its simplest form, an API request is just the address of an API endpoint. Making an API call is when you access - or "hit" - an endpoint with a request.

The link above lets you hit the sortinghat endpoint manually. From a human point of view, you click the link and visit another site. What's actually happening is that - when you click the link - your browser makes a request to the API endpoint, which gives back the data in JSON format, which your browser then displays for you. Most of the time, we don't need to think about the requests going back and forth across the internet, but it's helpful to be aware of them when talking about APIs.

Not all requests are so simple. Some API endpoints listen out for extra information in requests, and return different data depending on the parameters you provide. Some endpoints require authentication, and will only return data to requests which contain a secret API key. We'll look at both of those further on.

API documentation

Although many APIs work in very similar ways, you'll always need to do a bit of research to work out exactly what the endpoints for a particular API are, or how requests should be formatted. Luckily, most APIs come with detailed documentation, including example requests and the data they would return.

The Harry Potter API's documentation explains what each endpoint is for, and how to use it.

Accessing an API using Python

While it is possible to just visit API endpoints as a human user, it's not really what they're for. APIs are designed to be accessed using code, from within programs. We'll look now at how to use Python to access the sortinghat api endpoint.

Importing libraries

We only need one library to access the Harry Potter API. requests allows us to make HTTPS requests through Python.

import requests  # Make calls to web API endpoints

Creating the URL

The next step is to craft the URL - the actual address to request data from.

Although we could just store the URL as one string for this request, it's both good practice, and useful for later, to first create the different bits of the URL and then connect them together. This makes it easier to edit this URL and create new ones in the future.

# Create the URL components

base_url = "https://www.potterapi.com/v1/"

endpoint_url = "sortinghat"

# Join the pieces together

url = base_url + endpoint_url

# View the url

print(url)

https://www.potterapi.com/v1/sortinghat

Making the request

Once the URL has been created, we can use the .get() method in the requests library to ask the API for the data. There are other types of requests that we could use, but mostly, when dealing with APIs, you'll use GET requests: the HTTPS request that asks for information.

# Make a request - accio data

response = requests.get(url)

In response to our request, we get (appropriately enough) a response object. This not only contains our data, but also key information about the request and how it was received.

All response objects have an HTTP status code. This is a 3-digit number that tells you if the request was successful and, if it wasn't successful, what went wrong. You are probably already familiar with some status codes, such as 404: the status code for when a requested resource could not be found. There are many different codes, each one with a different meaning.

The status code for a successful request with no problems is 200.

# Check the status code of the response object

print(response.status_code)

200

Getting the data

Lastly, we need to actually extract the data from the response object. As already mentioned, the API returns data in JSON format. JSON stands for Java Script Object Notation, and it is one of the most popular data formats in the world. It's a relatively lightweight format, it's human-readable, and it's easy to work with using most programming languages.

The response object has a built-in method, .json(), that extracts the data from JSON format and returns it as the most appropriate Python data structure. In our current case, that's just a str.

# Access the data

data = response.json()

# View the data

print(data)

Slytherin

Authentication

Many APIs require you to provide user credentials to access their data. This allows them to manage traffic, control server costs, charge for access, and understand how the API is actually being used.

As a general rule, these credentials take the form of an API key - a long string of letters and numbers that identifies a request as coming from a particular user. When making the request, you attach your API key.

Getting an API key

The Harry Potter API doesn't require authorisation to access the sortinghat endpoint, but it is required for any of the other endpoints. You can get a free key from the API by creating an account with a valid email.

Once you've created an account, you'll be given your unique key.

API keys should be kept private and secure - don't share your keys with anyone.

In the cell below, I've replaced my actual key with a placeholder string.

# Store the API key as a variable.

HP_API_KEY = "XXXXXXXXXXXXXXXXXX"

Making a request with an API key

We can use the API key to make requests to a different endpoint - the spells endpoint.

The first step here is similar to our earlier API call: we construct the URL from a base component and and endpoint component.

# Construct the required URL pieces
# base_url already exits

endpoint_url = "spells"

# Construct the URL

url = base_url + endpoint_url

# View the URL

print(url)

https://www.potterapi.com/v1/spells

If you attempt to visit the URL at the moment though, you'll get this response:

"error": "Must pass API key for request"

We need to add the API key onto the request in order to authenticate it. However, it's not as simple as just sticking it onto the end - the API will think it's part of the actual URL and we'll end up requesting data from an endpoint that doesn't exist. Instead, we need to add it as a parameter.

Parameters

Parameters - also called "query parameters" - are the extra pieces of pieces of information that the API is listening for. Each parameter has a name and a value, and they're attached onto the URL with a special syntax so that the API knows how to interpret them.

Later on, we'll use several different parameters at once, but for this call, we just need one: the key parameter.

# Add the parameter onto the URL

url = url + "?key=" + HP_API_KEY

# Display the URL

print(url)

https://www.potterapi.com/v1/spells?key=XXXXXXXXXXXXXXXXXX

Because I've replaced my API key with a placeholder, the link above won't work. With a real API key though, it would give you the data.

The "?" tells the API to stop reading the URL as an address from that point on, and start looking for parameters. Next comes the name of the parameter - key - followed by an equals sign and then the actual value. In this case, I've replaced the real value with a fake one, for security reasons.

When the API receives this request, it will start by identifying the address part of the URL, and directing it towards the right endpoint. Then it will extract any parameters it finds, matching the names of parameters it will accept to the names in the URL.

# Send the request

response = requests.get(url)

# Check the response code

print(response.status_code)

A response of 200 means that the server has accepted the API request and that the API key is valid.

Extracting the data

The spells endpoint is a bit more complex than sortinghat and returns more data. In order to extract meaningful information, we'll have to go through a few more steps.

The start is the same - we can access the data using .json().

# Access the data

data = response.json()

# Check the type of the data

print(type(data))

The response this time has given us a list - we'll need to loop through it to get at the spell details.

Let's start by looking at just the first item in the list.

# Check the type of the first list item

print(type(data[0]))

# Print out the first item

print(data[0])

{'_id': '5b74ebd5fb6fc0739646754c', 'spell': 'Aberto', 'type': 'Charm', 'effect': 'opens objects'}

Each item in the list is a dictionary of key:value pairs. Now that we understand the structure of the data, we can actually access the data we requested.

Because .json() converts everything into Python objects, anything that you would normally do with Python is an option. With just a little more code, we can get the total number of spells:

# Print the number of spells

print(len(data))

151

We could also extract the spell names from the list.

# Loop through the first five items of the list, printing out the name of each spell.

for item in data[:5]:
    print(item["spell"])

Aberto
Accio
Age Line
Aguamenti
Alarte Ascendare

And - in a slightly more complex example - we can count spells by type.

# Count up each type of spell
spell_counts = {}

for item in data:
    if item["type"] not in spell_counts:
        spell_counts[item["type"]] = 1
    else:
        spell_counts[item["type"]] += 1

for key in spell_counts:
    print(key + ":", spell_counts[key])

Charm: 40
Enchantment: 1
Spell: 92
Hex: 1
Curse: 15
Jinx: 2

The API calls get the data into your program; you can then do whatever you want with it.

Parameters

key is a required parameter for all the endpoints except sortinghat, but it's not the only one available. By consulting the documentation, you can learn which API endpoints accept which query parameters.

You can use parameters to filter the data, returning only a subset of the available data. To explore this, we'll use the characters endpoint, which accepts several different parameters.

Accessing characters

# Construct the url

endpoint_url = "characters"

url = base_url + endpoint_url + "?key=" + HP_API_KEY

# Request all character data

response = requests.get(url)

# Check the response status

print(response.status_code)

200

# Extract the data

data = response.json()

# Count the number of characters

print(len(data))

195

Adding more parameters

In order to add more parameters, filtering the data, we add them onto the end of the URL. "&" is used to connect the different parameters together.

# Add another parameter onto the url

url = url + "&deathEater=True"

# Request character data on characters who are Death Eaters

response = requests.get(url)

# Check the response status

print(response.status_code)

200

# Extract the data

data = response.json()

# Count the Death Eaters

print(len(data))

24

You can combine parameters in any way you want, filtering the data to whatever degree you need. The query below, for example, requests information on all the pure-blood wizards who work at the Ministry of Magic .

# Craft the URL

url = base_url + endpoint_url + \
      "?key=" + HP_API_KEY + \
      "&bloodStatus=pure-blood&ministryOfMagic=True"

# Hit the endpoint

response = requests.get(url)

# Count the wizards

print(len(response.json()))

6

Conclusions

There's an awful lot more complexity to APIs that is worth exploring; hopefully this post has made some of the key ideas clear and given you a springboard from which to investigate further.

One of the best ways to build your skills & understanding is to find an API you're interested in and just start playing around with it. Different APIs will have their own rules and documentation, but the broad principles are very similar: hit an endpoint to make a request, include parameters to be more specific. As APIs want you to use them, the documentation is normally quite clear and accessible.

This GitHub repository holds a large list of publicly-accessible APIs for you to play with. Go explore & experiment, and if you have any questions or find something interesting, please do let me know.

A world of dreams and adventure awaits you.

Dungeons & Data

Dan Keefe — Thu, 14 May 2020 11:55:23 +0000

I recently came across a Dungeons and Dragons (D&D) API, designed to support people playing D&D by giving them a quick way to check spell details etc. Not being a D&D player myself, but being interested in both fantasy monsters and data mining, I thought it would be an interesting data source to play around with.

To that end, I extracted as much information about monsters as the API would give me, and started sifting through the data to see if anything interesting jumped out.

Background

Dungeons & Dragons is a tabletop roleplaying game published by Wizards of the Coast. Arguably/probably, it's the most well-known and influential one ever, casting a long shadow over much of the fantasy genre for the past several decades. Of course, older fantasy such as The Lord of the Rings and Beyond the Fields we Know cast an even longer shadow over D&D itself, but it's still very important within its scope.

It's a collaborative roleplaying game, with some players taking on the roles of adventurers - with all sorts of different classes/abilities - and one player taking the role of "dungeon master" (DM), creating the world and animating the monsters.

D&D is famous for, amongst other things, the number and variety of its various monsters. Monsters like the beholder and the owlbear originate from D&D, but have taken on a wider cultural significance. Other D&D monsters were adapted from earlier legendary/folklore sources, but their portrayal within the game has had a definite effect on the way they are widely thought of.

As the game's name suggests, much of D&D revolves around noble heroes delving deep into dungeons and fighting monstrous creatures. This post describes my delving into monster data in a way that is both similar to, and much less brave, than those adventurers' expeditions.

Disclaimer

I don't know much about the specifics of D&D; I have a general cultural knowledge of how it works, etc., but I might say something dumb at any point because I don't know the rules; please view me as a well-intentioned but naive tourist, and forgive any irritating ignorance.

This post is data-mining D&D from an outside perspective, and some of the things I find interesting might be less so to you if you have prior knowledge. I'm not claiming any definitive analysis, or special authority here, but it's fair to say that charging in without requisite prior knowledge is a time-honoured dungeon-crawling tradition.

Aims

My analysis was driven primarily by curiosity, rather than any set aim, but there were a couple of questions I was interested in answering:

Are there neglected monster niches?

People create "homebrew" D&D content all of the time, developing new quests and classes and monsters to put in their games. If there are particular kinds or styles of monster that are relatively uncommon, it could be useful to identify them, so that people creating new content can target the most untrodden ground.

Are the raw stats (such as strength and charisma) strongly linked to challenge rating (a monster's canonical difficulty to overcome)?

Challenge rating (CR) seems - from an outside perspective - to be a bit of an arbitrary figure. Surely so much of the game's difficulty will depend on player choice, how intelligently the DM plays the undead wizard, etc. I'd like to look at how closely linked the CR of a monster is to its other numeric stats. My initial assumption was that there would be a noticeable trend, but that there would be a lot of noise caused by abilities and such that affected the monster's difficulty alongside its raw scores.

Motivation

I was motivated to carry out this analysis by four factors, listed in ascending order of importance:

I'm currently doing the Udacity Data Scientist Nanodegree, and projects are required.
There's a very active modding/homebrew community for D&D, and this sort of analysis might conceivably be useful to someone looking to create new monsters, etc.
It was an opportunity to practice various data-related techniques.
I found the API and thought it would be fun to play with.

Code

All the code written during this project can be found on Github. I wrote it using Python 3, inside a Jupyter notebook. The linked repository contains a full list of libraries used and commented code explaining what I did to generate the various visualisations explained below.

Data Source

The data for this project was pulled from the Dungeons and Dragons 5th edition API. It's a really great API to work with - the documentation is very clear, and there are no fiddly authorisation problems.

I made a total of 323 calls to the API - one to get a full list of available monsters, and 322 to pull full details for each of those listed monsters. I stored the whole thing in a Pandas dataframe, and then was good to go. Creating the full dataframe took around a minute.

I'd like to stress that this analysis is only really scratching the surface of what can be done with this data; this is very much the kill n common mobs of data-mining D&D. It could absolutely be taken further and used to do even more exciting things, probably by someone who is both a better data scientist than me and actually a player of the game.

Data cleaning & processing

One of my absolute favourite things about APIs is that - if they're well-designed and reasonably maintained, almost no data cleaning is required. I dropped a few columns that were mostly empty (very few monsters have legendary actions, for example), but otherwise the data was automatically in a fit state. It was lovely.

In terms of processing the data, I dropped several more columns that were very granular, such as the actions column containing specific details of the various actions each monster was capable of. I wanted to keep the analysis relatively high-level, so I kept only those columns that could be reduced to a single figure/detail for each separate monster.

In order to extract key details - such as which monsters could swim - I had to do a small amount of data manipulation to pull data out dicts into a usable format, but mostly this was a very easy dataset to work with.

Data exploration

In total, the API provided me with data on 322 different monsters. For each one, I had information about its name, size, type (zombies are the "undead" type, for example), moral leanings (more on that later) and various numeric stats, ranging from how strong it was to how difficult it would be to defeat.

Type and Size

I started just by counting up the monsters for each type. "Beast" was by far the most common type of monster, followed by "dragon". Dragon being one of the most popular types makes sense - given the name of the game - but I was disappointed to discover that that's mostly because there are a whole bunch of different sub-types of dragon (red vs. black vs. green, and so on) and that each type appears several times at various ages. This makes dragons seem more dominant in the data than I feel is strictly fair; I get that it is important to have separate stat blocks for a creature that grows so dramatically in power, but it still seems like all the many and varied humanoids get a raw deal.

I did spot an almost immediate imbalance though, linking back to the first question I wanted to answer. There are 23 separate "fiend" type monsters (basically demons), but only six "celestials" (basically angels). This violates the time-honoured principle of "as above, so below", and suggests that, if you are a dungeon master looking to brew up some new monsters, you're a lot more likely to come up with an original heavenly concept than you are a hellish one.

The next thing I looked at was monster size category. This seemed to show a reasonable spread: the majority of monsters were human-sized (medium) or one step larger, which makes sense both in terms of drama and providing sufficient challenge to players. A small number of extremely large monsters exist in the "gargantuan" category to provide challenge and terror.

Stat distributions

I plotted the distribution of each of the main numeric stats across the whole set of monsters. This, again, was in aid of seeing if there were any obvious under-represented monsters. There was a lot of variation present, showing that for every stat, there are monsters at both ends of the scale. A small number of monsters had truly gargantuan (see what I did there) amounts of hit points, but the vast majority of monsters fell into a smaller range.

One interesting thing did jump out to me here though - when you compare strength and intelligence, the distributions are very different. Discounting hit points because it clearly scales by other rules, you can see that the peak distribution for strength is around 16 (later than any other stat), while intelligence peaks close to zero (earlier than any other). I can see an ecological justification for it - many of the monsters are "beasts", and cannot be expected to have great minds - but it does suggest that the majority of monsters rely on physical, rather than intellectual prowess. Wisdom, the other cerebral stat, has a similar distribution to intelligence.

"Dumb brute" seems to be a more common monster archetype than "scheming mastermind", and I suggest that - if anyone is looking to create new, original monsters - smart ones would be more easily differentiated. Plus, they're more fun, in my opinion; out of the three kidnappers in The Princess Bride, Vizzini is the most tense confrontation.

Senses and movement

Well over half of monsters can see in the dark. A respectable number have "blindsight" - they perceive the world through other senses, such as echolocation. A very small number have "truesight", which lets them see through illusions to unvarnished reality. This seems like a reasonable progression to me, with not too many monsters having an advantage in the dark, and the more exotic abilities being more rare.

I do think that "tremorsense" deserves more related monsters though; detecting vibrations is something that a lot of real animals, such as moles, do, and some of the most iconic movie monsters ever have used it to great effect.

The movement types suggest a similar neglected niche; while almost everything can walk, and flying and swimming have reasonable monster counts, very few monsters burrow. I'd like to champion the cause of blind things gnawing through the earth, because I find them extremely terrifying and also because they seem like they'd present new & exciting challenges compared to yet another goblinoid clan.

Alignment

D&D has a moral alignment system based on two axes, one between good & evil, the other between law & chaos. Canonically, characters & monsters are given a two-word alignment description, such as "Chaotic evil" or "Lawful neutral".

I know enough to know that there's a whole discourse around this in terms of how consistent/meaningful alignment is, how strictly it should be considered, and whether or not it's just a bad idea for the game/society generally. I have no plan of getting embroiled in that.

However, I was curious about the representation of each alignment amongst monsters. It turns out, somewhat disappointingly, that a lot of monsters - 128 out of 322 - are "unaligned": they have no moral stance at all. A further few have awkward alignments, such as the Cloud Giant, which is "neutral good (50%) or neutral evil (50%)". After removing all the unaligned and awkward though, I was able to plot the monster counts for each of the nine normal alignments.

Unsurprisingly, the most common alignments for monsters were evil ones, most commonly chaotic evil. Neutral alignments on either axis were quite unpopular, though "True Neutral" - the middle category - had quite a few occupants. Off-hand, I'm going to blithely assert that that's probably because it's a bit of a catch-all category. If there is a neglected niche here, it's monsters that aren't especially monstrous, ones that lean towards good or sit on the fence a lot.

Neglected monster niches

My first aim was to examine neglected monster niches - particular types of monster that might be less than evenly-represented in the bestiary. Based on my exploration so far, I venture to suggest that there are some monster niches that are, if not under-occupied, at least less-occupied than others. D&D will always have a disproportionate number of dragons, but if you're looking for new monsters, there are some areas that would be more original than others.

By my reckoning, the most original and exciting monster of all would be a well-intentioned burrowing celestial mastermind. If you're looking for further inspiration, allow me to point you towards the most celestial of all moles.

Stats and challenge rating

My other aim was to see how closely related monster stats are to challenge rating. To explore this, I calculated the correlations between CR and each other stat.

It's worth mentioning at this point that this analysis rests on shaky ground, mathematically-speaking; while CR looks like a number, it's actually a set of ordered and inconsistently-spaced categories. For that reason, plus the limited data available, I chose not to do any more complicated modelling, because it wouldn't be very valid. Correlations suffer from the same issue, but can still be indicative as long as the above caveat is borne in mind; the correlations used here shouldn't be taken as precise metrics, but solely as a rough way of comparing two variables.

Stat	CR correlation
Strength	0.72
Dexterity	-0.02
Constitution	0.86
Intelligence	0.64
Wisdom	0.55
Charisma	0.69
Hit points	0.94
Armour class	0.76

The majority of the stats showed a clear positive correlation; this is to be expected, as the point of those stats is to quantify a creatures attributes, and a creature that is stronger/faster/smarter than another will present a more significant challenge.

Hit points, followed by constitution (which is linked to hit points and affects their growth) were the most highly correlated. This suggests that one way of increasing a monster's CR is just to bump its health up a bit so it takes longer to kill and has more time to fight back.

The more physical stats - strength and constitution - are more correlated with CR than the cerebral ones, which, again, makes sense; I'm more terrified of intelligent enemies over the longterm, but when wrestling in a cave, I'd be more confident grappling a wizard than a cave troll.

The most interesting correlation though, was with dexterity. Unlike the other stats, which plotted against CR in broadly similar neat lines, dexterity was all over the place. There were several high-CR monsters that have relatively poor scores for dexterity, and a couple of otherwise pitiful creatures (by CR) with high dexterity; the most anomalous is the will-o'-the-wisp, with a CR of 2 but a dexterity score of 28, significantly higher than any other monster.

The D&D wiki defines dexterity as measuring "agility, reflexes, and balance", but it seems to be used as a proxy for size in some cases, with dexterity being assumed to belong more with dainty creatures than hefty ones.To some extent, this makes sense, but it does feel slightly out of the spirit of the stat. The highest dexterity score amongst gargantuan creatures is 14, but the majority of the behemoths can fly, demonstrating that they aren't simply lumbering beasts. The kraken has many weaving tentacles (the description in the wiki uses verbs such as "twining" when describing it) which again matches my layman's understanding of dexterity, but is not reflected in the stat.

I'm aware that I sound rather defensive of the monsters here (I have always had a soft spot for krakens; I am unsure why), and that this is probably one of those things that would make more intuitive sense if I was more familiar with the game in practice. I think possibly the stat is thinking of whole body dexterity (how easily can you fling yourself out of the path of an arrow), whereas I'm picturing the kraken's fine control of a single tentacle: the difference is one of nimbleness vs. precision.

I do still think it's interesting that dexterity does not match the pattern of the other stats, suggesting that this stat is - either perceived to be or actually - less important than the others. This is definitely something that I lack the necessary domain knowledge to untangle further; if any reader can shed more light on this, I'd be very interested.

I was, overall, a little disappointed by how high the correlations were between stats and CR. I expected it to be noticeable, but not quite to the extent that it was (particularly with hit points). Perhaps naively, I wanted to find that actually the scores were less important than the fine details of the monster.

Still, there is definitely sufficient variation amongst monsters and stats to show that all those abilities and attributes do matter somewhere, albeit not to the extent that I wished. It must always be borne in mind, however, that CR itself is a crude and highly-subjective metric, heavily dependent on how the monster is played, rather than how it is written.

Final thoughts

I don't have anything earth-shattering to wrap up with here; no big conclusions or stunning reversals. I got to explore an unfamiliar API and mine data that is a lot more interesting than most datasets floating around on the web.

I'd like to experiment further with the API in the future, perhaps trying to generate new spells based on the descriptions of existing ones; unfortunately (because generating monsters would be the most fun of all), the descriptions of monsters are not available through the API.

I enjoyed playing around with this data, and I found sufficient things to amuse me in it; it is my fervent hope that anyone who read this far down the post found it at least a fraction as interesting to read as it was to write.

If anyone has suggestions for further avenues of analysis, I'd love to hear them. If anyone would like to clarify or explain anything that confused me (like the dexterity correlation), I'd be extremely interested in that as well. Please do get in touch.

Wordclouds in Python

Dan Keefe — Sun, 05 Apr 2020 15:06:07 +0000

Wordclouds are a quick, engaging way to visualise text data. In Python, the simplest and most effective way to generate wordclouds is through the use of the Wordcloud library. In this tutorial, I'll explain how to generate wordclouds using the Wordcloud library, showing how to customise and improve your visualisations.

This tutorial was written using using Jupyter notebooks, Python 3.7.5 and Wordcloud 1.6.0; things might behave slightly differently if you're in a different IDE or using different versions of the language/library.

You can find a complete copy of the code for this tutorial on Github, along with the text data and images used throughout.

Disclaimer

Every statistician I have ever met requires me to inform you that wordclouds are not useful for analysis because they are simplistic and often misleading; I recommend this excellent article which goes into more detail on the problems with the form.

The criticisms of wordclouds are absolutely valid, but that's not to say that wordclouds are pointless. While I wouldn't recommend them as a method of analysis or information extraction, they're a useful tool for presentation - people seem to find them fun and engaging, and they're a good thing to have on initial pages of presentations, etc. As long as you focus on using them for aesthetic, rather than analytic, reasons, they have a place.

Data source

In order to demonstrate the possibilities of wordclouds, some text data is required. For the purposes of this tutorial, I've chosen to use the text of Addie's Husband or Through Clouds to Sunshine, a novel by Mrs. Gordon Smythies, one of the most popular and most forgotten of Victorian novelists.

You can find a copy of Addie's Husband on Amazon or on Project Gutenberg. It's the affecting story of Adelaide, a young woman with bad lungs but a good heart. I accessed the text from Project Gutenberg, using the excellent Gutenberg Python library.

In order that this tutorial can focus almost entirely on wordclouds, rather than text cleaning, I've already processed the novel's full text into a more consistent form. To be specific, the following steps have been carried out:

Removal of all metadata, including chapter headings, leaving just the prose
Conversion of all text to lowercase
Removal of all punctuation, special characters, and numeric values
Removal of stopwords and all words with fewer than 3 letters
Lemmatisation (converting each word to its dictionary form)
Removal of all proper nouns I found on a cursory search

The end result of this process is a text file - adelaide.txt - containing a standardised and simplified form of Addie's Husband.

Required libraries

In order to work with the Wordcloud library effectively, you require several imports. Obviously the Wordcloud library itself, but also it helps to have libraries to deal with text processing and images.

from collections import Counter  # Count the frequency of distinct strings
from wordcloud import WordCloud, ImageColorGenerator  # Generate wordclouds
from PIL import Image  # Load images from files
import numpy as np  # Convert images to numbers

Loading and preparing the data

As already mentioned, the data for this tutorial is stored in a .txt file. The first step is simply to load the file's data.

# Load the data from a file
with open("./resources/adelaide.txt", "r") as file:
    text = file.read()

# View the first 200 characters of the text

text[:200]

'soldier sailor tinker tailor policeman plowboy gentleman lift lovely head dear marry gentleman miss absorbed enjoyment ruddy ribstone pippin turn blooming freckle face speaker answer pleasantly though'

Our current text is perfectly suitable for wordcloud generation - given raw text, the Wordcloud library will automatically process it and generate a wordcloud.

However, text can also be provided in the form of a frequency dictionary, in which the key:value pairs have the form word:frequency. I find this conceptually neater, as it allows you more explicit control over the processing steps before generating the cloud.

# Split the text into a list of individual words

tokens = text.split(" ")

# Count the frequency of each word

word_counts = Counter(tokens)

# Display the count for a single word.

word_counts["love"]

133

Basic wordclouds

In order to generate and display the most basic of wordclouds, very little is required. You need a WordCloud object, and then to call .generate() on it, passing a string as an argument.

Finally, to display the cloud, .to_image().

fog_machine = WordCloud()  # Create a wordcloud generator 

fog_machine.generate(text)  # Generate the cloud using raw text

fog_machine.to_image()

Our basic wordcloud is rather small, and will require some customisation to improve.

In the code below, I've generated a slightly better one, by using the following parameters:

width and height to increase the size of the cloud's canvas area, and thus image quality
min_font_size to ensure that no words are too small to read
background_color to demonstrate that it doesn't have to be black if you don't want it to
colormap to choose a specific colour palette. Any valid Matplotlib colormap name is acceptable.

I've also chosen to generate the cloud using the frequency dictionary this time, not the raw text; that doesn't really have an effect on the style, it's just to demonstrate how you would do it.


# Create a wordcloud generator with some better defaults

fog_machine = WordCloud(width=800,
                        height=600,
                        min_font_size=14,
                        background_color="#333333",
                        colormap="spring")

# Generate the cloud using a frequency dictionary

fog_machine.generate_from_frequencies(word_counts)

fog_machine.to_image()  # Display the cloud

It's not the best thing ever - I don't really have much of a talent for visual design - but I think it's clearly an improvement.

Shaped wordclouds

Once you can create wordclouds with whatever colours you like, the next step is to shape the wordcloud, changing it from a boring rectangle into something appropriate to the content.

This is done by using an image as a mask; you provide the WordCloud object a numerical representation of an image with a white background, and the wordcloud will only draw words in positions where the image is not white.

As Addie's Husband is a romance, it seems appropriate that the mask we use should be a heart. The image I'm using was sourced from Template Trove.

image = Image.open("./resources/heart.jpg")  # Load the image from a file

image  # Display the image

Once the image is loaded in, we need to convert it into a numeric form, in which each pixel is represented as an array of three integers; the WordCloud object will then only draw words on top of pixels in the image which are not equal to [255, 255, 255].

mask = np.array(image)  # Convert the image to a numeric representation (a 3D array)

mask[0][0]  # Display the top left pixel of the mask, which is white

array([255, 255, 255], dtype=uint8)

After creating the mask, you create the wordcloud almost exactly as before, just passing in the mask parameter.

One key point to note is that masked wordclouds ignore the width and height parameters, instead sizing themselves based on the dimensions of the mask, so there is no need to include them.


# Create a wordcloud generator with a mask

fog_machine = WordCloud(mask=mask,
                        min_font_size=14,
                        colormap="Reds")  

# Generate the cloud using a frequency dictionary

fog_machine.generate_from_frequencies(word_counts)

fog_machine.to_image()  # Display the cloud

Shaping wordclouds like this works best with simple images, as otherwise the core shape is not clear. By default, it also only works with images that have a fully white background - transparent backgrounds, for example, are treated as non-masked, and the wordcloud occupies the full rectangle again.

It is possible to convert any image so that it has the correct structure for wordclouds, but doing so is rather beyond the scope of this tutorial; we're focused on wordclouds today, not image processing.

Shaped & Coloured wordclouds

We can both shape and colour a wordcloud based on an image, so that the words not only form the rough shape, but that each word is coloured as the image is in the same place. For this task, we'll use another heart, this one rainbow-coloured, and sourced from PNGfuel.

It's important to note here that this is a relatively crude fitting - a word that spans across several colour changes on the image will still only be in one colour, and you're not going to get lines as crisp and clear as the image itself. Again, this works best with simple images, ideally with clearly contrasting colours.

The first step is to load in an image, as before.

image = Image.open("./resources/rainbow_heart.jpeg")  # Load the image from a file

mask = np.array(image)  # Convert the image to a numeric representation

image  # Display the image

Once you've created the mask, you can then use it as an input to the ImageColorGenerator class, which is also part of the WordCloud library. This generates colours based on an image, and then can be used to colour words to match.

image_colours = ImageColorGenerator(mask)

You can then pass the ImageColorGenerator to a WordCloud object using the color_func parameter.

In order to ensure that our wordcloud roughly matches the colours of the image, I've set the max_words argument to 2000. This means that the WordCloud object will use more words than the default of 200, which will result in smaller words and - hopefully - clearer bands of colour.


# Create a wordcloud generator with a mask

fog_machine = WordCloud(mask=mask,
                        max_words=2000,
                        color_func=image_colours)

# Generate the cloud using a frequency dictionary

fog_machine.generate_from_frequencies(word_counts)

fog_machine.to_image()  # Display the cloud

Custom colour functions

It is sometimes useful to have full control over the colours of words, so that you can highlight particular words or groups of word; you might, for example, wish to show positive words in one colour and negative words in another.

We can define a custom colour function to do this, passing it to the color_func parameter just as for mask colours. In the code below, I've defined a very simple one - return gold for words with the letter "o" in, grey for every other word - but you can customise this function to do whatever you want.

When the WordCloud object calls the function, it passes it a lot of information; this means that - when defining your custom function - it's better to have the parameters as *args, *kwargs, as then you can pick and choose which arguments to later care about. The word itself is always passed as the first argument, or args[0].

# The custom function

def custom_colours(*args, **kwargs):
    word = args[0]  # First argument is the word itself
    if "o" in word: 
        return "#FFD700"
    return "#CCCCCC"

# Create a wordcloud generator with a custom color_func

fog_machine = WordCloud(width=800,
                        height=600,
                        min_font_size=14,
                        color_func=custom_colours)

# Generate the cloud using a frequency dictionary

fog_machine.generate_from_frequencies(word_counts)

fog_machine.to_image()  # Display the cloud

Saving wordclouds

You can save a wordcloud to a file with a single line of code.

fog_machine.to_file("wordcloud.png")

The header image

The header image for this tutorial was generated using Python and the techniques we've so far discussed. The code to create that wordcloud is below.

# One more wordcloud for Addie - the book does have a happy ending.
# The image is sourced from freepik.com.

image = Image.open("./resources/romance.jpeg")  # Load the image from a file

mask = np.array(image)  # Convert the image to a numeric representation

image_colors = ImageColorGenerator(mask)

# Create a wordcloud generator with a mask & colour function

fog_machine = WordCloud(mask=mask,
                        background_color="white",
                        max_font_size=28,
                        max_words=6000,
                        color_func=image_colors)

# Generate the cloud using a frequency dictionary

fog_machine.generate_from_frequencies(word_counts)

fog_machine.to_image()

Conclusions

The final wordcloud we've created is a lot more complex than the previous ones; with a max of 6000 words (almost as many as there are unique words in Addie's Husband), it takes a lot longer to generate, and the text is noticeably smaller. However, the end result is (in my opinion) worth the effort. Though the words themselves are now somewhat hard to read (always an issue with big wordclouds), it's an image that expresses one of the main ideas in Addie's Husband, using words from Addie's Husband, and I think that's neat.

The Wordcloud library is stuffed with extra tweakable parameters, and there's a lot more you can do to refine things than is covered in this tutorial. However, hopefully this gives some idea of the possibilities and the code to move towards them. The module documentation has many more examples.

There are a couple of caveats with wordclouds that should always be borne in mind. As already mentioned, it's easy to make confusing or misleading word clouds, because the focus on frequency ignores any information from the text that was contained in more than one word; "not happy" and "happy" are both going to make the word "happy" appear larger.

Even if your wordclouds manage not to be misleading, that still doesn't make them meaningful; again, the focus on frequency ignores context and significance. Even with stopwords removed, it's clear from the wordclouds generated in this tutorial that not all words are equally informative about what's actually happening in Addie's Husband. The word "say" is frequent, but would be equally so in almost any long narrative text. Arguably, the elaborate style of Victorian prose means that you'll get more frequent-but-not-significant words than a comparable modern text, but unless you do heavy cleaning, wordclouds will always contain some functional words that get in the way a bit.

With the above said though, I still think wordclouds are a useful thing to be able to generate; as long as you bear the caveats in mind, and don't use them too heavily as actual analysis tools, they're an accessible and appealing visualisation that people respond to well.

You can find a complete copy of the code for this tutorial on Github, along with the text data and images used throughout.

If you create any wordclouds using this tutorial, I'd love to see them. Send them to me via the links in my author bio, or tweet them to me at @peritract.