Forem: Felix Hildén

Multi-app web server in 30 terminal commands

Felix Hildén — Thu, 03 Nov 2022 23:48:34 +0000

"Spices and Stuff" by Benson Kua is licensed with CC BY-NC-SA 2.0.

In a short while you'll be able to set up your own server for hosting multiple web applications from scratch. The server will:

Serve a static website and a Python application using Docker
Follow best practices of web security and server management
Be easy to reinstall and develop further with e.g. Node or a database
Keep you in control of everything

Estimates to:

read: 10 min
follow along: 30 min to several hours depending on experience level and potential issues, ~$10 / month for domain and server

This tutorial is aimed at you if you want to elegantly manage something more than a static website, like a full web application, your own cloud storage, a game server etc. or a combination of all of them! Or if you're just interested in figuring out cool stuff.

We will go fast, so don't feel bad if you have to pause, think and search for information. This took me a long while to figure out too. I don't aim to explain everything, but instead provide a good template to start building on.

Prerequisites

A domain name and control of the DNS records, most likely on your provider's website.
A Debian server with root access. A virtual private server is probably your best bet.
A local Unix machine (Linux / Mac) with common tools like git, ssh and scp installed. On Windows, you can use Windows Subsystem for Linux.

Setting up a domain and a server should cost approximately $5 to $20 per month long-term, depending on the domain and caliber of the server. For this tutorial, even as little as 10 GB of disk space and 2 GB of RAM should be safe. Processor power won't be an issue.

DNS configuration

To point the abstract domain to your physical server, we'll set up two DNS records. A root address record and a wildcard for any subdomains. If you want to be strict, you can also set each subdomain separately. Substitute the IP address below with your server. If the address wasn't provided to you, you can determine it inside your server.

@ 1800 IN A 1.1.1.1
* 1800 IN A 1.1.1.1

After configuring DNS you should be able to resolve your domain. Propagating the records could take hours, so to avoid checking constantly we'll watch the status. Continue the tutorial when the address is resolved successfully. Note that the request will still result in error, but the server IP should be found.

curl -v DOMAIN              # check if domain can be resolved
watch -n 10 curl -v DOMAIN  # check every 10s

Template repository

Unfortunately there's no way to keep this tutorial short without some help. We'll use a template repository that contains build scripts, configuration and dummy website content. Consider forking the repository or using it as a template to manage your own future project!

git clone https://github.com/felix-hilden/server-template.git
cd server-template

Server configuration

Let's get to the meat of the issue and start configuring our server. The setup steps below are laid out on two indentation levels, the first being on your local machine, and the second with a remote connection to the server. Please read about the commands being used and look at the scripts that are executed to understand what is being done.

First, we avoid root access by creating a new admin user on the server. Please substitute DOMAIN for your domain name and USER for your desired user name. The commands below assume you have SSH access immediately, but if you're using other login methods you can run the commands in user-setup manually.

cd server

scp user-setup root@DOMAIN:~   # copy setup script to server
ssh root@DOMAIN                # login as root
    cat user-setup             # peek at the script
    chmod +x user-setup        # make it executable
    ./user-setup USER          # setup new sudo user
    rm user-setup              # remove setup script

The next step will configure SSH and a firewall along with a few important applications for our system. Before closing the root connection, continue the setup in another terminal or make sure you can access the server another way if something goes wrong and you are logged out without a way to enter. The configuration will disable root access and password logins through SSH for security.

cat setup-ssh            # peek at the script
chmod +x setup-ssh       # make it executable
./setup-ssh USER DOMAIN  # generate and send SSH keys

scp files/* DOMAIN:~     # copy build files to server
ssh DOMAIN               # login as new user
    cat build            # peek at the script
    chmod +x build       # make it executable
    ./build USER         # build server
    rm *                 # remove build files

For additional configuration, this nixCraft article has great tips on Linux security and it seems to be kept up to date: https://www.cyberciti.biz/tips/linux-security.html

Applications

From now on, we'll only be working inside of the server. So let's clone the template repository on the server too. Before setting up a proper deployment process, the git project will do fine for synchronising changes across our local machine and the server.

git clone https://github.com/felix-hilden/server-template.git
cd server-template

At this point it is appropriate to explain the basics of our tech stack.

Docker: provides light virtual machines for us to isolate services with. Instead of configuring our host server further, which quickly becomes unmanageable, we'll build and run containers defined in our git project, where all the relevant information is stored in one place.
nginx: a staple web server. We'll use it for both serving a static website in its container and as a proxy to route traffic to our Python application in another container.
Python: using the trending FastAPI framework, we'll serve a simple site with an API.

To get the website itself up and running, we'll only need to follow a few steps. First, replace all occurences of "example.com" in the template files with your domain using the magic script below. Trust me. Or this guy.

# Linux
grep -RIl 'example.com' | xargs sed -i 's/example.com/DOMAIN/g'

# Mac OS
grep -RIl 'example.com' | xargs sed -i '' 's/example.com/DOMAIN/g'

git diff  # inspect the result

Then, we'll issue SSL certificates for your domain to be able to use secure connections.

make certify  # bring up a dummy website to issue certificates

Assuming that the certificates were issued successfully, we can start up the real applications.

make up-prod  # serve applications on HTTPS
make logs     # start following logs to see if initialisation was successful

Behind the scenes our Makefile is orchestrating a set of containers to run while setting up networking and shared filesystems to find our configuration. We should now be able to visit the domain in an ordinary browser! Or get the home page with curl.

curl DOMAIN      # static site
curl app.DOMAIN  # Python app

Developing the site

The repository includes configuration to run the site locally to facilitate development. On your local machine, install Docker Desktop and run:

make  # bring up development website & inspect logs
curl localhost  # get home page

You can now modify the contents of website/site and app/src to start building your applications. The sites will be updated with your latest modifications while they're running. If something goes wrong, the default make target can be used to restart.

Farewell

I hope you've found this terse tutorial useful. Thank you for reading. The template is lacking in many ways, particularly when it comes to actual software development. However, we won't set up a full development environment for the website or Python here. That, along with automating the release process, is left as an exercise to you, my dear reader.

Enhance your code examples with links to reference docs

Felix Hildén — Tue, 05 Oct 2021 20:17:13 +0000

"Like the Sphinx" by Benson Kua is licensed with CC BY-SA 2.0.

Why is it not everywhere?

Some time ago when browsing Matplotlib's documentation, I was awestruck: I could click their code examples to get taken to the reference documentation of whatever class or function was used. What a great feature! It makes navigating documentation much easier.

So I tried to track down where those links were inserted. Turns out it was an extension called Sphinx-Gallery, which converts Python files to RST while capturing the script output. Awesome! But it didn't work for ordinary RST documentation, and it executes the code - two assumptions I didn't want to make with my own documentation. After searching around and slowly realising that it really did not exist, I set out to create a tool of my own! Enter sphinx-codeautolink.

What it does

As advertised, it matches code example contents to definitions documented elsewhere with Sphinx:

Clicking the link would take you to numpy.linspace's documentation. Other key features include:

Backreference tables: a table of examples where a particular definition is used
Autodoc integration: insert backreference tables to all definitions documented with autodoc
Intersphinx integration: as seen in the image above, linking to docs of external libraries is super simple
Python console example support (>>> prefixed code)
Invisible imports, example concatenation, simple type hint resolving and more!

How to use it

Easy. I wanted to make it work out of the box for most documentation, so if your examples are valid Python you're most likely already there. I had to add some import statements to my examples. Other than that, it's only a matter of installing the package and adding it to the list of Sphinx extensions!

$ pip install sphinx-codeautolink

# conf.py
extensions = [
    ...,
    "sphinx_codeautolink",
]

Our examples have a more comprehensive explanation of all the use cases and RST syntax.

How it works

The principle is quite simple: all code blocks are parsed and the names contained in them are traced back to definitions documented with Sphinx elsewhere in the documentation. Sphinx does its thing and after the HTML is written, links are injected to the correct places.

Naturally I made a few assumptions, some different than the folks at Sphinx-Gallery: it is only usable with HTML documentation and I had to cut some corners with the parsing implementation. I didn't aim to emulate the Python runtime, just provide a correct result for the 99 % case. None of the shortcomings should cause problems with reasonable example code. But Sphinx-Gallery does perform better in some situations. For example where we'd have to analyse complex type hints, Sphinx-Gallery can simply look at the variable's type.

I've poured my last month on this and really enjoyed the deep dive into Sphinx and docutils. I hope you'll get some use out of it!

Links

Documentation, PyPI, GitHub

Smart playlist shuffle using Travelling Salesman

Felix Hildén — Sun, 06 Sep 2020 11:15:46 +0000

What if your playlists were ordered by track similarity? No Taylor Swift followed by death metal, we can rearrange tracks in terms of loudness, mood, tempo and other factors. Naturally, we could also create many playlists that each contain only one type of music, but where's the fun in that? Let's get down to some data analysis with Python and Spotify. Their Web API contains lots of neat features, including the ability to analyse songs and manipulate a user's playlists.

First, to get us up and running, we'll need access to the API. This is done by authorising our application to have access to our own Spotify account. If you want to follow along, we are using a client library for the API, Tekore, to make interacting with it a lot easier. For the sake of brevity, I won't explain details about the API here, but more information can be found in Tekore's documentation.

import tekore as tk

# API credentials
client_id = 'your_client_id'
client_secret = 'your_client_secret'
redirect_uri = 'https://example.com/callback'
conf = (client_id, client_secret, redirect_uri)

token = tk.prompt_for_user_token(*conf, scope=tk.scope.every)

Once we've retrieved the token, we are free to use the API. So let's start by pulling tracks of a playlist and analysing them. You can swap in your own playlists of course. The ID of a playlist can be retrieved by right-clicking a playlist and choosing "share > Spotify URI" and deleting everything else before the actual ID.

playlist = '37i9dQZEVXbMDoHDwVN2tF'  # Global Top 50
spotify = tk.Spotify(token, max_limits_on=True, chunked_on=True)

items = spotify.playlist_items(playlist)
tracks = [t.track for t in spotify.all_items(items)]
analysis = spotify.tracks_audio_features([t.id for t in tracks])

analysis contains audio features for each track in the playlist. They look like this:

AudioFeatures with fields:
  acousticness = 0.0194
  danceability = 0.935
  duration_ms = 187541
  energy = 0.454
  instrumentalness = 0
  key = 1
  liveness = 0.0824
  loudness = -7.509
  mode = 1
  speechiness = 0.375
  tempo = 133.073
  time_signature = 4
  valence = 0.357

Now we can get to the good stuff. Let's first choose the most important aspects of songs to be considered similar. You can choose your own and see how the results change. For me, duration, key, liveness, mode and time signature are second-class citizens when compared to the other features. Let's select everything else and convert the features into a NumPy array for easy numerical operations.

import numpy as np

cols = [
    'acousticness', 'danceability', 'energy', 'instrumentalness',
    'loudness', 'speechiness', 'tempo', 'valence'
]
features = [tuple(getattr(a, col) for col in cols) for a in analysis]
data = np.array(features)

It is important to note that while other features chosen above are between 0 and 1, loudness and tempo have much larger scales. Normalising them keeps things balanced when choosing the song ordering. We'll also use Principal Component Analysis to bring the problem dimensionality down to two for easy visualisations.

from sklearn.decomposition import PCA

for i, col in enumerate(cols):
    if col in ('loudness', 'tempo'):
        minimum = data[:, i].min()
        data[:, i] = (data[:, i] - minimum) / (data[:, i].max() - minimum)

pca = PCA(n_components=2)
data = pca.fit_transform(data)

So... How can we represent and solve the problem of transitioning from song to another smoothly? Well, it's in the title already, so I'll cut to the chase. Since we have this data available, we can represent each track with its distance to each other track. And how can we most efficiently transition through each track? By minimising the total distance "travelled" once each song is played. Sounds a lot like the Travelling Salesman Problem. It's a hard problem, but for small numbers of tracks we can do it quite effectively by using an approximation algorithm, two-opt for example. I'll provide my implementation here, though figuring it out is a good excercise.

def two_opt(
    distances: np.ndarray,
    tolerance: float = 0,
    verbose: bool = False
) -> np.ndarray:
    n_nodes = distances.shape[0]
    current_route = np.concatenate((np.arange(n_nodes, dtype=int), [0]))
    candidate = np.zeros(n_nodes + 1, dtype=int)
    best_cost = np.sum(distances[current_route[:-1], current_route[1:]])
    cost = np.inf

    while True:
        for i in range(1, n_nodes):
            for k in range(i + 1, n_nodes + 1):
                candidate[:i] = current_route[:i]
                candidate[i:k] = current_route[k-1:i-1:-1]
                candidate[k:] = current_route[k:]
                cost = np.sum(distances[candidate[:-1], candidate[1:]])
                if cost + tolerance < best_cost:
                    break
            if cost + tolerance < best_cost:
                current_route[:] = candidate
                best_cost = cost
                if verbose:
                    print(f'2-opt: node={i}/{n_nodes}, cost={best_cost}')
                break
        else:
            break
    return current_route

Now we can use it to solve the optimal route, or at least our best guess at the optimum. First we'll construct the distance matrix, then feed it to the algorithm. If you have a large playlist, you can set the threshold argument to reject the smallest improvements to the route to hopefully speed things up a bit.

from scipy.spatial.distance import pdist, squareform

distances = squareform(pdist(data))
route = two_opt(distances, verbose=True)

Visualising results is important. Let's see how our algorithm performed. We'll plot the PCA-reduced data and the chosen path along with song names.

from matplotlib import pyplot as plt

plt.figure()
plt.plot(data[route, 0], data[route, 1], 'r')
plt.scatter(data[:, 0], data[:, 1], s=10, c='k')
for i, t in enumerate(tracks):
    plt.text(data[i, 0], data[i, 1], t.name, fontsize=6)
plt.xticks([])
plt.yticks([])
plt.show()

And it seems indeed that the playlist was ordered successfully:

I don't know much about these songs, but running the algorithm for my own playlists convinces me that it is doing the right thing! Finally, we'd of course want to listen to this version of the playlist. Let's create one that has the new song order.

# Leave the longest distance between the last and first songs
route_dist = distances[route[1:], route[:-1]]
worst = np.argmax(route_dist)
new_route = np.concatenate((route[worst + 1:], route[1:worst + 1]))

# Reorder tracks
reordered = [tracks[i].id for i in new_route]
uris = [tk.to_uri('track', i) for i in reordered]

# Create playlist
me = spotify.current_user()
pl = spotify.playlist_create(me.id, 'Reordered playlist')
spotify.playlist_add(pl.id, uris)

That's it, I hope you enjoyed following along and had some fun!