Forem: Paul Cochrane 🇪🇺

Changing a hostname in Debian seems harder than necessary

Paul Cochrane 🇪🇺 — Wed, 18 Feb 2026 23:00:00 +0000

Changing a host’s name in Debian¹ is a two-step process, and there doesn’t appear to be a more direct method. This seems odd to me, somehow. Because I stumbled upon this issue and a confusing error while relying solely on sudo, I thought I’d write up how I worked around it.

It feels like I missed something obvious here. I mean, changing the hostname should be a one-liner, right? Perhaps someone knows of a better solution.

System setup stumbling-block

Ok, so here’s the situation: I wanted to create a new Vagrant VM and run some basic provisioning steps. Since the use case wasn’t very involved, I didn’t want to use something like Ansible: a simple set of bash commands was sufficient.

One step changes the hostname from the default to something more descriptive. Sounds simple, right? Technically, it is simple. The thing is, two steps are required, which I found strange. And if you’re using sudo, you’ll likely be surprised by a name resolution error:

sudo: unable to resolve host <new-hostname>: Temporary failure in name resolution

What gives?

The name resolution failure appears because the new hostname doesn’t match the entry in /etc/hosts. Thus, any sudo call thereafter is unable to find the host it’s running on. The hostname lookup within sudo times out, and we get the name resolution failure message I mentioned above.

I must admit, this tripped me up. I don’t remember having had this stumbling block previously. I ended up wondering how I’d done this before. Wasn’t this a single command in the past? And what’s with name resolution being needed as part of sudo?² Once upon a time, I’d have opened a root session and hence wouldn’t have run into this problem, but those days are long over. I now almost exclusively use sudo for admin tasks.

More recently, I use Ansible when provisioning new hosts, and hence haven’t spotted this issue.³ At least, not for a while. Maybe I’ve just forgotten? Dunno.

Anyhow, back to the story. The problem manifests itself if you run these commands:

# either use the "old" way to set a hostname
$ sudo hostname <new-hostname>
# ... or use the new-fangled, strangely repetitive way to set a hostname
$ sudo hostnamectl set-hostname <new-hostname>
# now open /etc/hosts to fix hostname entry in there
$ sudo vim /etc/hosts
# ... wait until the name resolution times out ...
# get the name resolution error message
sudo: unable to resolve host <new-hostname>: Temporary failure in name resolution
# now vim will open /etc/hosts

All online resources I looked at–such as nixCraft, which is usually a good source of info–recommend this two-step process. And to do it within a root session.

But what if you don’t want to do that as root directly? What if you want to script this and are restricted to using sudo? That’s the issue I was trying to solve, and here’s my solution.

Wonky workflow workaround

The approach first adds the new hostname to /etc/hosts. That way, the name is able to be resolved after changing the hostname. Then I change the hostname in the usual way. Afterwards, I delete the old /etc/hosts entry to remove unnecessary cruft.

Putting this all together as a shell script in a Vagrantfile provisioning step, we get (by way of example):⁴

    bookworm_new.vm.provision "shell", privileged: false, inline: <<-SHELL
      sudo DEBIAN_FRONTEND=noninteractive apt update
      sudo DEBIAN_FRONTEND=noninteractive apt -y upgrade
      sudo DEBIAN_FRONTEND=noninteractive apt install -y vim
      # add new hostname to /etc/hosts; keep old one for now
      # avoids `sudo: unable to resolve host <hostname>: Temporary failure in name resolution`
      # errors when running `sudo` after changing the hostname
      sudo sed -i 's/bookworm/bookworm bookworm-new/g' /etc/hosts
      sudo hostnamectl set-hostname bookworm-new
      # remove the old hostname
      sudo sed -i 's/bookworm //g' /etc/hosts
    SHELL

The provisioning commands appear in a here doc called SHELL.⁵ The first few lines are standard update commands I use when setting up a new VM. I find it’s a good idea to ensure everything’s up to date before proceeding, even if the base image is recent. You’ll note that I install vim as well. Basically, my muscle memory won’t think of another editor, and this saves friction for me when modifying config files or viewing logs.

The lines after that are the important ones for the hostname updating process:

# add new hostname; keep old one temporarily
# avoids `sudo: unable to resolve host <hostname>: Temporary failure in name resolution`
# errors when running `sudo` after changing the hostname
sudo sed -i 's/bookworm/bookworm bookworm-new/g' /etc/hosts
sudo hostnamectl set-hostname bookworm-new
# remove the old hostname
sudo sed -i 's/bookworm //g' /etc/hosts

The first non-comment line uses sed to edit /etc/hosts in-place (hence the -i option). The example system used here is Debian bookworm, hence the default hostname is set to bookworm. I want to set the new hostname to be (unimaginatively) bookworm-new, so the sed command replaces bookworm with the text bookworm bookworm-new. Now the new hostname is available, and the old hostname is still there, but with a space after it, which will come in handy when removing that entry later.

Next, I use hostnamectl to set the hostname. The old-fashioned hostname command still works, but hostnamectl seems to be what all the cool kids use.

Now it’s just a matter of removing the old hostname from /etc/hosts. Here I use sed with the -i option again, and replace the string 'bookworm ' (note the space after the name)⁶ with the empty string, thus removing the old hostname from /etc/hosts.

Now the change is complete, and there aren’t any “Temporary failure in name resolution” errors. Yay! 🎉

Wrapping up

For some reason, I still expect the hostname and /etc/hosts changes to be coupled so that they occur as part of a single command. Oh well. Ya get that, I guess.

Anyway, if you run across the Temporary failure in name resolution error when running sudo in the future, now you know why and can work around it. Enjoy!

… and probably Linux in general, but I didn’t check. ↩
The reason sudo looks up the host’s name is that /etc/sudoers is designed to be distributed across multiple servers. Hence, sudo needs to know which host it’s on to apply any host-specific rules. ↩
Although, to be honest, the ansible.builtin.hostname module docs do mention that the module doesn’t update /etc/hosts and that one has to do this it oneself. Perhaps I haven’t done this in a while and have forgotten the full story. ↩
I have a suspicion that my woes could be solved by using privileged: true and removing sudo from the script. However, I realised this after having written everything up, and yet still found my explanation useful. ↩
Of course, one could extract the script code into a file and run that via sudo. But where’s the fun in that? ↩
Yes, I could have used a better regex here, e.g. something like bookworm\s\+, but then I’d need to use extended regular expressions and, well, this did the job. ↩

It’s not DNS; it was DNS

Paul Cochrane 🇪🇺 — Wed, 11 Feb 2026 23:00:00 +0000

Logging in to a VPN shouldn’t cause DNS to stop working, right? Well, you’d like to think that, wouldn’t ya? And the problem wasn’t PEBKAC this time, either. Finding the right setting to fix the problem did take some time, but I got there in the end.

A computer by any other name

While doing some work for a customer¹ recently, I needed to log in to their VPN to check on some stuff. I’d not used their VPN for about a year, so I expected some hiccups along the way. I mean, there’s always something, right? Network settings are changed, systems get updated, and the slow plod of time always causes some problem or other. But I wasn’t expecting DNS not to work when connected to their VPN.

Here was the situation. I was happily using the internet (as you do) and needed to access one of the customer’s services that is only available within their network. Thus, I needed to log in to their VPN. I entered my login details into their VPN client and voila! I was in their network. All as expected, right? Wrong. Any address I entered into my web browser gave me the same error: server not found. And yes, I was using the correct address. This situation made it rather difficult to access the services I needed to use.

Bring out the machine that goes ping

One of my first instincts when confronted with such a situation is to try to ping something. A habit (and probably a bad one at that) is to ping www.heise.de because, well, it’s always up. That resulted in this error:

$ ping -4 www.heise.de
ping: www.heise.de: Name or service not known

Ok, not good. How about an address closer to home? For instance, the customer’s main website? It turned out that pinging that address also wouldn’t resolve a name. Crikey! What to do now?

The hard part here was debugging the problem as an outside user. What else could I check? The IP address that the VPN client software had given me looked ok:

$ ip addr
<snip>
10: cscotun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1390 qdisc
pfifo_fast state UNKNOWN group default qlen 500
    link/none
    inet x.x.x.x/22 brd x.x.x.255 scope global cscotun0
       valid_lft forever preferred_lft forever
    inet6 fe80:<la-la-la>/126 scope link noprefixroute
       valid_lft forever preferred_lft forever

The list of routes also looked good:

$ ip route list
default via 192.168.2.1 dev wlp2s0 proto dhcp metric 600
x.x.x.x dev cscotun0 proto unspec scope link
<snip>

Logging out of the VPN, I contacted the IT staff at the customer’s site and told them of my sorrows. Unfortunately, as far as the network engineers on the other side could tell, everything looked good. Nothing in their DNS had changed. After all, the DNS servers were still the same as they’d always been. These servers also appeared in my route list. Hrm.

Persistence pays off

Some time passed, and after some more discussion, we (the IT staff and I) still weren’t any closer to resolving the issue. Thus, I decided to dig a bit further on my side to see if there was anything I could do to fix things. After all, it wouldn’t be the first time that I’d set some configuration value on my laptop and then forgotten that I’d set it, and it ended up provoking some weird problem a long time later. Lots of digging ensued, but there wasn’t anything obvious that seemed to be causing the issue. Through stubborn perseverance, I eventually honed in on the NetworkManager service’s configuration as a possible origin of my woes.

One thing with DNS on a Linux box is that (historically) this is configured in /etc/resolv.conf. Having a look at this file with the VPN off showed something like this:

$ cat /etc/resolv.conf
# Generated by NetworkManager
search internet-provider.com
nameserver 192.168.2.1
nameserver fe80::1%wlp2s0

Then, when logged in to the VPN, NetworkManager changed this file’s contents to:

$ cat /etc/resolv.conf
# Generated by NetworkManager
search internet-provider.com customer.com
nameserver 192.168.2.1
nameserver fe80::1%wlp2s0

Ok, so the only change is to add the customer’s domain to the resolver’s search settings. This shouldn’t affect name resolution. In fact, I would expect it to help with name resolution.

Using a bit of old-fashioned prompt-engineering (i.e. googling), I stumbled upon NetworkManager’s dns configuration parameter. It controls the DNS processing mode and is set in the [main] configuration section of the /etc/NetworkManager/NetworkManager.conf file.

The comment in /etc/resolv.conf:

# Generated by NetworkManager

somehow gave me the idea that maybe NetworkManager didn’t need to update this file after all. And that maybe some other software component, which was also trying to update /etc/resolv.conf, would get the chance to set things properly. Thus, I tried setting dns=none to stop NetworkManager modifying resolv.conf.

I.e., I changed /etc/NetworkManager/NetworkManager.conf to look like this in the [main] section:

[main]
... other settings
dns=none

Restarting the service to re-read the config,

$ sudo service NetworkManager restart

and sure enough, something was changing /etc/resolv.conf when in the VPN and was changing it back after leaving the VPN. My guess, retrospectively, is that this was the VPN client software. Since, in the default configuration, NetworkManager thinks that it should have full control over this file, it kept changing /etc/resolv.conf back to what it thought was the right config, which then screwed up DNS for me.

Here’s what /etc/resolv.conf looked like for me now after connecting to the VPN:

$ cat /etc/resolv.conf
domain customer.com
nameserver <dns-ip-1>
nameserver <dns-ip-2>
search customer.com internet-provider.com

Something about this change told me I was onto something. Especially the new nameserver values signalled an improvement.

Importantly, I could now ping www.heise.de while in the VPN:

$ ping -4 www.heise.de
PING (193.99.144.85) 56(84) bytes of data.
64 bytes from www.heise.de (193.99.144.85): icmp_seq=1 ttl=249 time=9.13 ms
64 bytes from www.heise.de (193.99.144.85): icmp_seq=2 ttl=249 time=10.1 ms
^C
--- ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 9.134/9.607/10.080/0.473 ms

Brilliant!

Now I could access the services I needed to use and continued merrily on my way.

Everything works until it doesn’t

… until …

Time did that thing again that it always does: change stuff. sigh

For other network-related reasons, I needed to update some packages on my (admittedly very outdated) dev system. While watching the progress of an apt upgrade command whiz past, a package name caught my attention: resolvconf. Oh dear.

Guess what? DNS no longer worked. It even turned out that when I wasn’t connected to the VPN that dns=none needed to be removed for name resolution to work. But dns=none needed to be set (and NetworkManager restarted) for DNS to work inside the VPN. Bugger.

Man, this was so not a solution. I can’t be changing the NetworkManager config and restarting it each time I change a network connection!

Read the fine manual

Thus, lots more reading ensued, including more of the NetworkManager.conf man page. One of the big hints in the section about the dns option is the comment that:

If the key is unspecified, default is used, unless /etc/resolv.conf is a symlink to /run/systemd/resolve/stub-resolv.conf, /run/systemd/resolve/resolv.conf, /lib/systemd/resolv.conf or /usr/lib/systemd/resolv.conf. In that case, systemd-resolved is chosen automatically.

An idea formed in my mind. What if I used systemd-resolved? Let’s give it a go! YOLO and all that jazz.

It turned out that the systemd-resolved package wasn’t installed on my laptop. No worries, that’s easily fixed:

$ sudo apt install systemd-resolved

Next, I changed the dns setting in NetworkManager.conf to use systemd-resolved:

[main]
... other settings ...
dns=systemd-resolved

After restarting NetworkManager:

$ sudo service NetworkManager restart

… everything worked! By that I mean I could change the network I was using (i.e. to the VPN and back again) and still have name resolution. Phew!

So, it turned out that it was a local setting causing problems. It wasn’t, however, a setting that I’d mucked about with before all these problems appeared, so I couldn’t blame myself (at least, not completely), as is often the case.

The solution turned out not to be to set dns=none and let the VPN software (or whatever) work out what to do with DNS resolution and whatnot in /etc/resolv.conf. The better, more sustainable solution turned out to be to use systemd-resolved to handle things for me. Great! Now I can get back to work!

It’s not DNS

As it turns out, “it’s not DNS” is a meme, at least among sysadmins. There are even t-shirts and haikus and stuff!

It’s not DNS

There’s no way it’s DNS

It was DNS

Things ya learn!

Persistence still pays off

So what did we learn? Computers are hard. Networks are also hard. The fact that the internet, with so many computers connected to it, functions at all is something short of a miracle. It’s also a PITA when things that usually “just work”–like DNS–don’t. But, stubbornness, persistence, and liberal use of RTFM saved the day in the end.

If you need a software developer who is stubborn, persistent, and thorough, give me a yell! I’m available for freelance Python/Perl backend development and maintenance work. Contact me at paul@peateasea.de and let’s discuss how I can help solve your business’s hairiest problems. ↩

An unhinged carriage problem

Paul Cochrane 🇪🇺 — Thu, 08 Jan 2026 23:00:00 +0000

As part of tutoring high school students in maths and physics, I sometimes run across problems that I find interesting. Here’s a deep dive into one involving an unhinged railway carriage which, at first glance, seems to lack sufficient information.

I find it interesting that some solutions become almost embarrassingly obvious given the right perspective. I also like that there’s usually more than one way to solve a problem. This is one of those situations. Here’s the problem:

A train is moving straight and horizontally at a constant speed. The last carriage is uncoupled from the train and slows with a constant deceleration. The train continues to move at its previous speed. The carriage moves straight and horizontally, eventually coming to a stop after 200 m. Determine the distance the train travelled from the time the carriage was uncoupled to the moment the carriage stopped.

Write the answer in metres (m)

When I first saw this, my instinct was that there wasn’t enough information. The solution requires a very specific answer in metres, yet we’ve not been told the carriage’s deceleration, nor the train’s speed. That struck me as odd.

It turns out that it is possible to calculate a definite value for how far the train travelled before the carriage stopped; it just wasn’t obvious to me from the get-go. This leads to one piece of advice:

Have a go at solving the problem, even if you’re unsure.

There’s another piece of advice that I usually dish out to my students:

Try solving the problem in more than one way. If you get the same answer by a different path, then the answer is more likely to be correct. You might also learn something along the way.

But I’m getting ahead of myself.

Let’s have a go at solving the problem. We’ll first use a geometric argument by drawing the speed versus time graphs for the train and the unhinged carriage. Later, we’ll look at other paths to a solution.

Visualising speed versus time

I’m a big fan of drawing pictures and diagrams to get a feel for a given situation. Interestingly enough, my gut feeling this time wasn’t to draw a train moving along a track (which would have been my usual approach) but to draw graphs of speed versus time.

The question mentions constant speed a couple of times, as well as the uncoupled carriage’s constant deceleration. Thus, my physics-addled mind’s eye straight away saw lines on a speed versus time graph, and I started doodling these on a piece of paper. What follows is a more neatly-drawn¹ version interspersed with explanations.

First, we need some axes and to label them:

The carriage’s initial speed is the same as the train’s speed when it is uncoupled. Let’s call this $v_t$ and mention it on the graph, so that we can refer to it again later.

Since we’re only interested in elapsed time, the time that the carriage is uncoupled from the train happens at zero seconds. The time that the carriage stops, we’ll call $t_1$ . We now label the graph with this information:

We know that the carriage slows with constant deceleration and that deceleration is simply negative acceleration, hence we’ll denote it by $- a$ . Deceleration is a line of negative slope $(- a)$ in the speed versus time graph and connects the time-speed coordinate $0, v_t)$ with $t_1, 0)$ . Adding this to the graph, we have:

We also know that the area under a speed versus time graph is the distance travelled. Therefore, we shade this area to highlight it and make a note on the graph that this is equal to $m200\ \mathrm{m}$ .

That’s cool. We’ve described, in a graphical form, the uncoupled carriage’s behaviour from when it separated from the train up until it stopped moving.

So what happened to the train over this time? Well, we know that it kept going at the same speed. And we know that constant speed on a speed versus time graph is also a straight line, but with zero slope. In other words, it’s a horizontal line from the time-speed coordinate $0, v_t)$ to $t_1, v_t)$ . Let’s add that to our graph:

As with the carriage, the area under its curve is how far the train travelled in this time. We can shade in this area as well:

It’s clear from the graph that the area under the line representing the train’s speed versus time behaviour is enclosed by a rectangle. It’s also clear that the area under the carriage’s curve is enclosed by a triangle, and that the triangle’s hypotenuse bisects the rectangle. In such a case, the triangle’s area is half the rectangle’s area. Since we know that the triangle’s area is equal to $m200\ \mathrm{m}$ , we can say that the rectangle’s area must be equal to double that, i.e. $m400\ \mathrm{m}$ . Thus, the train travelled $m400\ \mathrm{m}$ in the time it took the uncoupled carriage to come to a halt.

There we have it! That’s the answer!

Now we can see that it wasn’t important how fast the carriage decelerated, nor was it important how fast the train was travelling. Thus, my initial feeling about missing information was unfounded. Hence, we see why it was a good idea to attempt to solve the problem to see if we could get anywhere. Also, the answer sort of “pops out” at us as soon as we draw the speed versus time graphs: we only needed a bit of geometry and to know the physical meaning of the area under the curve to find the answer. It’s almost blindingly obvious in the end.

We now put our fingers in our ears and go “la! la! la!” loudly, to try and ignore the fact that we know the answer. Then, we’ll solve the problem algebraically without using any pictures or graphs.

Why do this “the hard way”? Why not? Come on, let’s dig in and see what happens!

An algebraic approach

As with the graphical approach above, we collect what we know from the problem description and see if we can make some headway by combining things logically.

We know that the train travels at a constant speed the entire time, so let’s call that $v_t$ . We’re told that the last carriage is unhinged (i.e. uncoupled) from the train at some time; thereafter, it decelerates at a constant rate. Since deceleration is negative acceleration, let’s call this quantity $- a$ . The carriage starts with the same speed as the train, finally coming to a stop at some later time. Let’s set the initial time to zero and call the time the carriage stops $t_1$ . With this information, we can write an equation to describe the carriage’s speed between these two times. Let’s call the carriage’s speed $v_c$ , which we can write as:

\begin{equation} v_c = v_t - a t, \end{equation}

where $t$ is the elapsed time.

Note that this equation has the same form as the equation for a straight line in the plane:

y = m x + c,

where $y = v_c$ , $m = - a$ , $x = t$ and $c = v_t$ .

We can see that (1) behaves the way we want by considering the start and end points.

At the uncoupling event, $t = 0$ , hence when we substitute this into (1), we find that the speed of the carriage when it gets unhinged is:

v_c(0) = v_t - a \cdot 0 = v_t,

which is the $y$ intercept. That makes sense, because we know that the carriage’s initial speed is the same as the train’s speed. Cool, that means we’re probably on the right track.

The $- a t$ term in (1) linearly decreases the value of $v_c$ , tracing out a line which then intersects the $x$ -axis at $t_1$ . In other words, the speed is equal to zero at $t_1$ . This is the behaviour we want at the other end of the time interval. Good, we’re still on the right track.

We know that the carriage stops at $t_1$ , thus we have the relationship:

v_c(t_1) = v_t - a t_1 = 0.

That might not look that useful, but we can rearrange it to give an expression for the time when the carriage stops, and that could come in handy later:

\begin{equation} v_t - a t_1 = 0, \end{equation}

\begin{equation} \Rightarrow a t_1 = v_t, \end{equation}

\begin{equation} \Rightarrow t_1 = \frac{v_t}{a}. \end{equation}

What else do we know? Well, we know that the train travels at constant speed between $t = 0$ and $t = t_1$ . That means we know how far the train went in that time: this is its speed multiplied by the elapsed time. In other words, the distance the train travelled (which we’ll call $d_t$ ) is:

d_t = v_t t_1.

Since we have an expression for $t_1$ (Equation (4)), we can rewrite the expression for $d_t$ to see if it might be useful:

\begin{equation} d_t = v_t t_1, \end{equation}

\begin{equation} = v_t \frac{v_t}{a}, \end{equation}

\begin{equation} \Rightarrow d_t = \frac{v_t^2}{a}. \end{equation}

Hrm, that doesn’t look like it’s going to be of much help. Oh well. Let’s add it to the growing pile of things that we know anyway and see what else we know and/or can deduce.

What about the distance the last carriage travels? We know that its value is $m200\ \mathrm{m}$ . Can we write an expression for it in terms of the other things we know? Yes, we can! We know that the carriage’s speed is described by:

v_c = v_t - a t.

We also know that distance is the integral of the speed between two points in time.² Thus, to find the distance travelled by the carriage, we integrate Equation (1) with respect to time between the start and end points, $t = 0$ and $t = t_1$ . To be consistent with our other notation, let’s call the distance travelled by the carriage $d_c$ . Putting this all together, we have:

d_c = \int_{0}^{t_1} v_t - a t\ dt,

\left[v_t t - \frac{1}{2} a t^2 \right]_{0}^{t_1},

\Rightarrow d_c = v_t t_1 - \frac{1}{2} a t_1^2.

Squinting at this result, one gets the feeling that it’d be nice to remove at least one of the variables. Since we already have an expression for $t_1$ (Equation (4)), we can try substituting that into the equation for $d_c$ to see what happens:

d_c = v_t \frac{v_t}{a} - \frac{1}{2} a \left( \frac{v_t}{a} \right)^2,

\frac{v_t^2}{a} - \frac{1}{2} a \frac{v_t^2}{a^2},

\frac{v_t^2}{a} - \frac{1}{2} \frac{v_t^2}{a},

\frac{1}{2} \frac{v_t^2}{a}.

Hrm, that $vt2a\frac{v_t^2}{a}$ bit looks familiar. Hang on! That’s the same as the expression for the distance the train travelled from earlier (Equation (7)). In other words,

d_c = \frac{1}{2} d_t,

\Rightarrow d_t = 2 d_c.

This means that the carriage travelled half the distance that the train travelled. Which is the same as saying that the train travelled twice the distance that the carriage travelled. Wow, that turned out to be a very simple relationship in the end!

Now, we know that $md_c = 200\ \mathrm{m}$ , as that was given to us in the question. Thus, the distance the train travelled in the time it took for the carriage to come to a stop is $md_t = 400\ \mathrm{m}$ .

And that’s it! We solved the problem. Yay! And we got the same result as the graphical approach outlined earlier. Cool! 🎉 I love it when a calculation comes together. 😉

There’s something else one can learn here:

What looks like a dead end in a calculation might come in useful anyway.

After all, calculating the distance travelled by the train in terms of its velocity and the carriage’s deceleration looked like it wasn’t very helpful. But in the end, it gave us a useful, simplifying hint which led to the answer.

An average solution

There’s another way of looking at this problem. We can consider average speeds.

The fundamental insight with this approach is that, for a situation involving constant acceleration, the distance travelled is proportional to the average speed.

Let’s first consider the case for the train. Its speed at both the beginning ( $t = 0$ ) and at the end ( $t = t_1$ ) is equal to $v_t$ . We know that the distance it travels is its speed multiplied by the elapsed time:

d_t = v_t (t_1 - 0) = v_t t_1.

Because the train doesn’t change speed, that means the train’s average speed, $vˉt\bar{v}_t$ , is the same as the train’s speed. I.e.

\bar{v}_t = \frac{v_{ t,0} + v_{t,1}}{2} = \frac{v_t + v_t}{2} = \frac{2 v_t}{2} = v_t,

where $v_{t,0}$ and $v_{t,1}$ are the speeds of the train at the beginning and end of the time interval, respectively.

We can now write the distance the train travelled in terms of its average speed:

d_t = v_t t_1 = \bar{v}_t t_1.

This can be formulated more loosely as:

d_t \sim \bar{v}_t.

Because we’re considering the same time interval for both the distance travelled by the train and by the unhinged carriage, the elapsed time (which is equal to $t_1$ ) ends up being just a scaling constant. Hence, the distance travelled is proportional to the average speed.

Of course, the carriage does change speed, and over the same time interval. We also have a constant acceleration³ for the unhinged carriage, so we have the same kind of proportionality relationship between the distance and average speed, i.e.:

d_c \sim \bar{v}_c.

The average speed for the carriage is:

\bar{v}_c = \frac{v_{c,0} + v_{c,1}}{2},

where $v_{c,0}$ and $v_{c,1}$ are the speeds of the carriage at the beginning and end of the time interval, respectively.

Because the carriage’s speed at the beginning is equal to the train’s speed, and the carriage’s speed at the end is equal to zero, we have $v_{c,0} = v_t$ and $v_{c,1} = 0$ . Thus, the carriage’s average speed ends up being:

\bar{v}_c = \frac{v_t + 0}{2} = \frac{v_t}{2}.

I’m sure you can see where this is going. But let’s continue anyway to nail everything down more firmly.

Since the distances are proportional to the average speeds, the ratios of the distances will be equal to the ratios of the respective average speeds. In other words, we can write:

\frac{d_t}{d_c} = \frac{\bar{v}_t}{\bar{v}_c}.

Since we know what $d_c$ , $vˉt\bar{v}_t$ and $vˉc\bar{v}_c$ are, we can rearrange this to give us our only unknown, $d_t$ :

d_t = d_c \frac{\bar{v}_t}{\bar{v}_c},

d_c \frac{v_t}{\frac{v_t}{2}},

d_c v_t \frac{2}{v_t},

2 d_c.

And since we know that $md_c = 200\ \mathrm{m}$ , we have the distance the train travels to be

d_t = 2 d_c = 2 \cdot 200\ \mathrm{m} = 400\ \mathrm{m},

which is our result from before. Yay!

Wrapping up

So there we have it. Although the solution wasn’t very mind-blowing, the journey turned out to be much more instructive than the destination. And that’s much more valuable from a problem-solving perspective.

Why, yes, I did create these graphs with TikZ. Why do you ask? ↩
Yes, I’m slyly re-using the idea of the distance being the area under the curve here. ↩
Well, it’s a deceleration, but that’s just a negative acceleration. I might have mentioned this before. 😉 ↩

A detailed description of the pull request process

Paul Cochrane 🇪🇺 — Wed, 17 Dec 2025 23:00:00 +0000

A while ago, when sharing some code with friends, I asked for feedback, when possible, as a pull request to the repository I’d shared. One friend replied along the lines of “can do, if you show me how to make a pull request first”. This is my attempt to address that remark.

What is a pull request?

Before we can get to the details of an explanation, what is a pull request, and why would anyone want to make one?

A pull request (also known as a merge request) is a way to contribute to a software-related project.¹ This could be by providing bug fixes or new features to code, by updating or extending documentation, or by providing some other kind of improvement to the project. The pull request workflow is one way (of many) to contribute to a project. Even so, it has become a very common way to work on both open- and closed-source projects. For instance, Linux kernel development uses a generalised version of this model. It has gained enormously in popularity through GitHub, which automated the workflow and integrated it into the service’s code contribution process. The pull request model process for contributing to a project is now so common that many (most?) other Git service providers (such as GitLab, BitBucket, Gitea, Codeberg, etc.) provide almost identical workflows to those that one can use on GitHub.

Before Git became popular as a version control system, it was common to send code changes to Open-Source Software projects as patches. These are files describing the changes to make to the source code so that it would, after the patches have been applied, contain the proposed new behaviour. With the advent of Git and distributed version control systems, it became possible to share the online location of a source code contribution. This was much simpler than emailing around files of those contributions. The online location is, for example, a publicly-accessible Git branch which the maintainer can pull to their computer. Thus, the act of telling the maintainer that a patch or set of patches was available in a Git repo became known as a “pull request”. GitHub then standardised and automated much of this process, effectively wrapping it into a simple web form. Consequently, GitHub is now strongly associated with the concept of pull requests.

High level overview

Let’s look at the process at a high level. We’ll see how everything works in more detail with a worked example later.

Let’s say you’ve been working on a software project. You’ve found it useful for the things you need to get done, and you think it’d be handy for others to use as well. You decide to make the code publicly available so that others can download it and run it on their computers. We’ll assume that you’ve kept the code in a Git repository on your local computer. This could be your laptop, a desktop PC, or maybe a Raspberry Pi on your home network. Whatever. The point is that the code is where you are, hence the idea that it is somehow “local” to you.

You have an account on an online Git service (such as GitHub, GitLab, and friends) and use that service to create an empty repository, ready to contain the code you wish to share. You then push your code into this empty upstream repository, making it available for anyone to download and use.

I’ve used the term “upstream” here to refer to the online copy of your repository. This is because it becomes the central point where everyone (including you) can get the project’s code. Anyone fetching code from this location is thus “downstream” from it, which makes it “upstream” for everyone.

Time passes. Your project has been available for a while now, and people have been happily using it. One of your users happens to be a software developer and has spotted a bug. They find the root cause in the code and fix it. But how can they send this fix to you? They might not know you personally and hence won’t be able to send you fixes directly, say via email. If there’s no previous connection between the author and contributor, how can complete strangers collaborate on an Open-Source Software project? One possibility (of many) is to use an intermediary of some kind, such as an online Git service.

Fortunately, they also use the online Git service that you’re using. They fork your repository, allowing them to make and publish code changes to a copy of your code.

A fork is a simple copy of your online repository. But instead of being owned by you, it’s owned by them, hence they can push code changes to it. They clone their fork of your upstream repository to their computer. This way, they can make changes to the code.

There, they fix the bug in a branch and push that branch back upstream to their fork.²

The fix is now available online for you to access, review, and (if so desired) incorporate into your code.

The thing is, you don’t know about it yet. But the Git service has this cool feature: through a simple web form, the other user submits their contribution to the Git service, which in turn notifies you about the bug fix. This notification is a pull request.

You can pull this new branch to your local Git repository, study the code and verify that it does in fact fix a bug. If you’re happy with the code changes, you can merge these into your main branch.³ Of course, if you’re not happy with the changes, you can ask (via a web form on the online Git service) the other person to make certain updates or ask for feedback/explanations about the submission. Usually, things get ironed out after a while, and you merge the fix from their branch into your main branch.

You then push the latest version of your main branch up to the online repository of your project. Now, anyone using your program can take advantage of the updated and fixed code. Yay!

The Git service even provides a way for you to review and merge the code into your upstream repository without having to explicitly pull the code to your local computer. Either way, it gets merged into main, and everyone wins. Cool!

That’s it. Through millions of such interactions daily, much of the software we use every day becomes better, bit by bit. Well, that’s the theory, anyway. And for the most part, that’s the case.

A detailed worked example

Let’s take these general concepts and overall process and make them more concrete via a detailed worked example.

Let’s imagine two people. One–whom we’ll call Eck–has written a program and wants to share it with the world. The other–called Alice–uses the program and wants to contribute changes back to the original project. We’ll now look over their respective shoulders as they go through the motions of the pull request process.

Setting the stage

Eck has written a simple fortune cookie application. The program prints a randomly-selected phrase from a list of fortune-cookie-like messages. Here’s the code:

# -*- coding: utf-8 -*-

import random

def fortune():
    messages = [
        "The early bird can kiss my backside",
        "Don't give up on your dreams: stay asleep",
        "All good ends must come to a thing",
        "People who live on trampolines should not throw bowling balls",
        "The second mouse gets the cheese",
        "Do not eat prunes when you are famished",
    ]

    print(random.choice(messages))

if __name__ == " __main__":
    fortune()

# vim: expandtab shiftwidth=4 softtabstop=4

You can verify that this code works by saving it into a file called fortune-cookie.py and running it with the python3 interpreter. For example:

$ python3 fortune-cookie.py
Don't give up on your dreams: stay asleep

Eck has been using this program daily and has found it to be really useful. So useful, in fact, that he thinks other people might also want to use it. Thus, he decides to make the source code available online for others to download and run on their computers. He has an account on GitLab (a common cloud-based Git service with free access for Open-Source Software) and decides to make his program available there. From previous experience, he knows that he can share Git repositories (and hence code) with other people by publishing them on GitLab. But first, he needs to create a local Git repository to hold his program’s source code.

Storing code in a local Git repository

He puts his program into its own directory by creating a folder called fortune-cookie. From the command-line, he did something like this:

$ mkdir fortune-cookie
$ mv $HOME/fortune-cookie.py fortune-cookie/ # everyone dumps their stuff in $HOME, right? Right?

Since Eck knows a bit of Git and has made repositories before, he enters the fortune-cookie directory and initialises a Git repository there:

$ cd fortune-cookie
$ git init .
Initialized empty Git repository in /home/eckzampill/fortune-cookie/.git/

With the local repository created, he adds the fortune-cookie.py source code file to it and commits that change:

$ git add fortune-cookie.py
$ git commit

At this point, Git opens an editor for Eck to write a commit message. Knowing that documentation is a love letter written to his future self, Eck takes a bit of time to compose an informative commit message. He ends up writing this:

Initial import of fortune-cookie code

... which is a program to randomly display a proverb or message similar
to that which one gets in a fortune cookie.

This text is better than the “Initial commit” string often found in first commit messages.

Saving this text and quitting the editor commits the newly added file to the repository:

[main (root-commit) 9771905] Initial import of fortune-cookie code
 1 file changed, 23 insertions(+)
 create mode 100644 fortune-cookie.py

Eck can now push his repository to GitLab. Before he can do that, he has to create a GitLab project.

Creating an empty online Git repository

With the code safely stored in his local Git repository, it’s time for Eck to create an empty repository on GitLab.⁴ There, he can publish a copy of his repository so that others can access it.

He logs in to GitLab and is presented with his home dashboard.

Within the GitLab parlance, if one wants to create a new online repository, one first has to create a project. To do that, Eck clicks on the “Projects” link in the navigation sidebar on the left-hand side of the page.

Now he clicks on the “New Project” button at the top right-hand side of the page. This shows him a selection of ways to create a new project.

He then selects the option “Create blank project”. Doing so, he doesn’t fall into the trap of using the “Import project” option, although it might sound like what one might want to do. One would think the task is to import the repository from your local computer into GitLab. That’s not what the “Import project” option does, though: it imports projects from other online Git service providers. That’s why Eck uses the “Create blank project” in this case. Clicking on the button for this option, he’s presented with a form asking for details about the project to create:

Now he fills out the form details.⁵

In the “Project name” field, Eck adds the text “Fortune cookie”. Doing so creates the project slug with the correct name of fortune-cookie. This is handy, because Eck wants it called fortune-cookie to match the directory name on his computer.

For the “Project URL” field, he selects the name eckzampill under the “Users” section of the drop-down menu.

The next option is to set a “Project deployment target”. Right now, Eck is only interested in getting his code online. Hence, he isn’t worried about setting a deployment target and leaves this option unset.

However, he notices that the “Visibility Level” is set to “Private” by default and thus changes its value to “Public”; he wants to share his program with the rest of the world after all!

The last thing he needs to do is to uncheck the “Initialize repository with a README” option under the “Project Configuration” heading. As the text for this option mentions:

Skip this if you plan to push up an existing repository.

Since Eck already has an existing repository, he doesn’t want to initialise one on GitLab.

The form now looks like this:

Once that’s all done, he scrolls down to the bottom of the page and clicks on the “Create project” button.

GitLab presents him with the initial project page for his fortune cookie program. This page helpfully includes instructions in the “Command line instructions” section on how to push new code to this project.

For those following along at home: Note that you might be asked to set up SSH keys if you haven’t done so already. I won’t go into setting up SSH keys in this worked example because that would be too much of a tangent. So, authentication with the upstream Git service (in the rest of this story) will take place over HTTPS instead. Just so you know. 🙂

The project’s initial page mentions that:

The repository for this project is empty

To get started, clone the repository or upload some files.

GitLab has provided Eck with instructions on how to do this. He scrolls down the page a bit to the “Add files” section.

He notes that the default option is to use SSH for communication between his computer and the upstream Git service. Since he hasn’t set up SSH with GitLab yet, he instead clicks on the “HTTPS” tab to show the instructions for that authentication mechanism.

Pushing the local repository upstream

Since Eck already has a repository to push, he needs to use the “Push an existing Git repository” instructions. He scrolls further down the page to see them in full.

Reviewing the instructions, Eck notices that they are very generic and include steps that he doesn’t need to take in his case. So, he follows an appropriate subset of steps to publish his code online. Let’s see those steps now.

After having made his first commit earlier, Eck’s shell session is still in the fortune-cookie directory on his computer. Thus, the step that GitLab recommends to change into the appropriate repository folder isn’t necessary. Also, because this is a completely new repository, Eck doesn’t have a pre-existing remote origin,⁶ and can ignore the step to rename the remote origin.

However, Eck does have to specify a new remote origin for his local repository. This is how Git knows where to push changes to. Thus, he calls his upstream (“remote”) repository origin and specifies its location as a URL as part of the git remote command:

$ git remote add origin https://gitlab.com/eckzampill/fortune-cookie.git

One can understand this command like so: add the remote repository called origin located at the URL https://gitlab.com/eckzampill/fortune-cookie.git.

Now he’s ready to push his local repository to GitLab, which he does via git push:

$ git push --set-upstream origin --all
Username for 'https://gitlab.com': eckzampill
Password for 'https://eckzampill@gitlab.com':
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Delta compression using up to 4 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 634 bytes | 634.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
To https://gitlab.com/eckzampill/fortune-cookie.git
 * [new branch] main -> main
 Branch 'main' set up to track remote branch 'main' from 'origin'.

Note that he had to specify his username and password because he’s using HTTPS rather than SSH.

Since this is a completely new repository, it doesn’t have any tags. Thus, Eck realises that the step to push the tags in GitLab’s documentation isn’t necessary in this case.

Since that was the last step, he reloads the GitLab project page in his browser and can now see the content he pushed.

Great! Now the world has access to Eck’s fortune cookie program!

Forking before contributing

Time passes, and Alice, a person using the fortune-cookie program, thinks that the it would be even more useful if there were installation and usage instructions in a README file. This way, new users can get up to speed quickly, and they can get their fortune cookies more easily.

Alice is a software developer and knows that to submit changes or new code to another project on GitLab, she has to fork it first. To do that, she visits the GitLab page of the fortune cookie project and clicks on the “Fork” button in the top right-hand corner of the page.⁷

(Notice that this image looks almost the same as the one Eck saw after the initial push. The main difference is Alice’s user avatar in the top right hand corner of the page.)

After clicking on the “Fork” button, Alice sees a form where she can specify the fork’s details in her GitLab environment. Note that this behaviour is different to GitHub. There, forks are added to the user’s main namespace, and one doesn’t have the option of a group to fork code into. GitLab’s process has more flexibility, but that flexibility means Alice needs to make a choice here. She decides to fork the project into her user’s main namespace (alliceellis).

Alice peruses the default project metadata settings, and everything seems in order. The project name looks good, the project slug is also correct, she wants to copy all branches from the original project, and to make her fork public as well. Looking good! She clicks on the “Fork project” button at the bottom of the page to complete the fork process.

Because of the text

Forked from Eck Zampill / Fortune cookie

on her fortune cookie project page, Alice can see that it is a fork of another repository. Also, GitLab helpfully tells her what state her fork is in relative to the original repository:

Up to date with the upstream repository.

Because she’s just forked the project, it comes as no surprise that her fork is up to date with the original repository. Eck’s repository is–relative to hers–“upstream”, hence the phrasing used in the text here.

Cloning the fork

For Alice to work on her fork of the code, she needs to clone it to her laptop. To find out the URL to use when cloning, she clicks on the “Clone” button on her fork’s project page. This opens a pop-up window with various options.

Again, we don’t want to set up SSH for this worked example, hence we’ll use the (short-term) easier HTTPS option.

Alice copies the URL shown under the heading “Clone with HTTPS”. She uses this in her git clone command to clone her fork of the fortune cookie project repository to her laptop:⁸

$ git clone https://gitlab.com/aliceellis/fortune-cookie.git
Cloning into 'fortune-cookie'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
Receiving objects: 100% (3/3), done.

Alice didn’t need to authenticate because her fork is publicly available. She’ll need to authenticate when pushing changes back up to her fork, though. We’ll see this in action later.

Adding a README to the project

Alice enters her freshly cloned repository:

$ cd fortune-cookie

and creates a Markdown-formatted README file explaining what the project is, how to set it up, and how to run the program:

# Fortune cookie

A program to print a randomly-chosen fortune cookie message from a list of
available messages.

Requires [Python version 3](https://www.python.org/) to be installed to run.

## Installation

Download the file with `wget` into your `~/bin/` directory:

```shell
$ wget https://gitlab.com/eckzampill/fortune-cookie/-/blob/main/fortune-cookie.py?ref_type=heads -O $HOME/bin/fortune-cookie.py
```

## Usage

Call the program with `python3`:

```shell
$ python3 ~/bin/fortune-cookie.py
```

She saves this text into a file called README.md.

Before adding and committing this file, she runs git status to see the state of her repository:

$ git status
On branch main
Your branch is up to date with 'origin/main'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        README.md

nothing added to commit but untracked files present (use "git add" to track)

Oops! She’s still on the main branch!

Alice notices the text On branch main after the git status command. She knows that to create a pull request to another project, her changes need to be committed to their own branch. Thus, she creates a new branch and changes to it in one step with git switch:

$ git switch -c add-project-readme
Switched to a new branch 'add-project-readme'

The -c option creates the branch before switching to it. Equivalently, Alice could have used git checkout -b add-project-readme to achieve the same effect. This older way of doing things is still a common sight in online documentation.

Alice now adds the new file to her local repository and commits the change:

$ git add README.md
$ git commit
[add-project-readme 7b8b638] Add an initial project README
 1 file changed, 22 insertions(+)
 create mode 100644 README.md

She took the time to write a clear commit message, like this:

Add an initial project README

A README file helps introduce the project to new users and describes
what the main program does.

This change adds a README file to the project describing briefly what
the program does and showing potential users how to install and run the
`fortune-cookie` program.

Alice checks her working directory status to make sure she’s not missed anything:

$ git status
On branch add-project-readme
nothing to commit, working tree clean

Looking good!

Pushing proposed changes to the upstream fork

Now she’s ready to push her changes up to her upstream fork of the project on GitLab. To do this, she uses git push:

$ git push
Username for 'https://gitlab.com': aliceellis
Password for 'https://aliceellis@gitlab.com':
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 4 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 740 bytes | 370.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
remote:
remote: To create a merge request for add-project-readme, visit:
remote: https://gitlab.com/aliceellis/fortune-cookie/-/merge_requests/new?merge_request%5Bsource_branch%5D=add-project-readme
remote:
To https://gitlab.com/aliceellis/fortune-cookie.git
 * [new branch] add-project-readme -> add-project-readme

The message shown as part of the git push operation (on the lines prefixed with remote:) mentions the URL Alice could use to create a merge request. Note that GitLab uses the phrase “merge request” for the same concept as what GitHub calls a “pull request”.

Visiting the provided merge request link is one way for Alice to open a merge request. Another way is for Alice to view her fork’s project page on GitLab. Let’s take that particular path now.

Creating a merge request

Reloading the project page of her GitLab fork of the fortune cookie project, she sees an invitation to create a merge request.

Alice has seen this before when creating other merge requests, and this time she decides to click on the “Create merge request” button. Alternatively, she could copy-paste the URL from the git push message into her browser to achieve the same goal.

She’s presented with a form to specify the metadata and options of her merge request.

One can see in the form that the subject line of her commit message is used as the merge request’s title. Its description is the body text from her commit message. This has all been automatically filled out by GitLab to streamline the merge request workflow.

Alice sees at the top of the page that her add-project-readme branch is to be merged into the main branch of the original project. This confirms to her that the source and destination locations of the proposed changes are correct.

Note that for single-commit changes, the commit subject line and text are used for the merge request title and description (respectively) in GitLab and GitHub. For multi-commit changes, GitLab pre-fills the form using the commit message from the first commit in the change set. However, this is often insufficient to describe work spanning multiple commits. So, for multi-commit changes, one should delete the automatically-created text in the web form and take the time to write a clear overview of the changes being submitted. Since Alice has submitted a single-commit change, she can use the title and description as they are, extending them if she sees fit.

When Alice scrolls down on the “New merge request” form, she can see the commit included as part of this merge request.

Since Alice is aware that Eck doesn’t know her personally, she extends the description text to mention that if Eck wants anything changed, she’ll be happy to make any required updates. From previous experience, Alice has found that being nice and offering assistance to project maintainers is not only plain good manners but also good netiquette when contributing to Open-Source Software projects. After all, she wants to change the look and behaviour of someone else’s project, so it makes sense to be civil and offer help where one can.

Alice is now satisfied that her merge request is configured correctly and that the changes are sufficiently well described. She now clicks on the “Create merge request” button to submit her merge request.

After submitting her merge request, she’s presented with an overview of it as it appears in Eck Zampill’s project environment:

This is also what Eck will see when he views the merge request online.

Let’s now switch back to Eck’s perspective.

Reviewing proposed changes in a merge request

As it turns out, Eck only had the project notification settings in GitLab set to “Participate”. Hence, he hadn’t received any email notifying him that Alice had submitted a merge request. Had he known about this, he could have changed this default notification setting to “Watch”. This way, he would have been informed of most activity on his projects, but that hadn’t happened in this case.

What should Eck use for his notification settings in general? As with many things, that depends. For instance, when one only has a few projects, it’s probably a good idea to set the notification setting as high as possible to not miss anything. However, involvement in many projects often means a flood of messages. Hence, in that case, it’d be necessary to throttle things back if the notification load becomes too high.⁹

Although he didn’t hear about Alice’s merge request immediately, after a while, Eck happens to notice that there’s a number next to the “Merge requests” item in the sidebar on his fortune cookie project. It’s not obvious, so I’ve highlighted the area with a red box.

This is fortunate for Alice, because otherwise she could have been waiting a while before getting a reaction.¹⁰

Eck wants to see the merge request and its details, so he clicks on the “Merge requests” item in the project sidebar. He’s taken to an overview of all open merge requests for the project.

Clicking on the single merge request item in the list, he’s taken to the details page for the one that Alice submitted.

Clicking on the “Commits” tab, he can see the commits that make up the merge request.

Clicking on the link for the commit, he can see the commit message as well as the diff, displaying what was changed as part of the commit.

Because Alice’s submission adds a single new file, there have only been lines added in this commit, hence all the text is shown in green. Were this a more involved change, it would likely also have had line deletions as well as additions. In such a case, deletions would be shown in red.

Explicitly pulling proposed changes by direct fetch (optional)

Now, Eck could merge this merge request from the GitLab interface by clicking on the “Overview” tab and then clicking on the “Merge” button. This is the common, streamlined workflow that both GitLab and GitHub support. However, since this process derives from the more general notion of a pull request using command-line tools, let’s see how a “pull” is involved. Ironically, we’ll see that git pull isn’t part of this process. Yet the idea of “pulling in changes from upstream” remains, even if the Git command verbiage doesn’t involve pull.

To do this on GitLab requires jumping through some hoops. By contrast, it seems to be easier to do on GitHub. Let’s try the harder path to see what it looks like.

Eck notices the link to Alice’s branch at the top of the merge request overview page:

Alice Ellis requested to merge aliceellis/fortune-cookie:add-project-readme into main

He copies this link (which in this case is https://gitlab.com/aliceellis/fortune-cookie/-/tree/add-project-readme) and uses it to create a git fetch command on his local computer. Yes, you read that correctly. One way to access the contents of an upstream branch is to fetch it to your local repository rather than pulling it, as the “pull request” name would suggest. Git terminology can be inconsistent at times. To be honest, Eck is still “pull”-ing the code to his computer. The subtlety is that he has to use the git fetch command to do it.

Here’s the command he used:

$ git fetch https://gitlab.com/aliceellis/fortune-cookie add-project-readme:add-project-readme
warning: redirecting to https://gitlab.com/aliceellis/fortune-cookie.git/
remote: Enumerating objects: 4, done.
remote: Counting objects: 100% (4/4), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
Unpacking objects: 100% (3/3), 718 bytes | 359.00 KiB/s, done.
From https://gitlab.com/aliceellis/fortune-cookie
 * [new branch] add-project-readme -> add-project-readme

where the warning highlights that Eck should have appended .git to the URL to point it to Alice’s upstream repository correctly. Fortunately, GitLab knew what he meant and appended the .git part for him.

Note that Eck specifies Alice’s repository in the first argument to git fetch, not the branch. In other words, he changed the copied link to point to Alice’s repository (https://gitlab.com/aliceellis/fortune-cookie). Then, he used the syntax <their-branch>:<my-branch> to fetch the add-project-readme branch from Alice’s remote repository to the add-project-readme branch in his local repository. Running the fetch command created a new branch which he can switch to. There, he can inspect the changes that Alice is submitting.

For instance,

$ git switch add-project-readme
Switched to branch 'add-project-readme'
$ ls
fortune-cookie.py README.md

where he can see the new README file in the directory listing.

He can also see the commit that Alice made by running git show:

$ git show
commit 7b8b638a524ffc52b7eabdec52ca8540514cf7fb (HEAD -> add-project-readme)
Author: Alice Ellis 
Date: Thu Nov 13 15:46:43 2025 +0100

    Add an initial project README

    A README file helps introduce the project to new users and describes
    what the main program does.

    This change adds a README file to the project describing briefly what
    the program does and showing potential users how to install and run the
    `fortune-cookie` program.

diff --git a/README.md b/README.md
new file mode 100644
index 0000000..4114c67
--- /dev/null
+++ b/README.md
@@ -0,0 +1,22 @@
+# Fortune cookie
+
+A program to print a randomly-chosen fortune cookie message from a list of
+available messages.
+
+Requires [Python version 3](https://www.python.org/) to be installed to run.
+
+## Installation
+
+Download the file with `wget` into your `~/bin/` directory:
+
+```shell
+$ wget https://gitlab.com/eckzampill/fortune-cookie/-/blob/main/fortune-cookie.py?ref_type=heads -O $HOME/bin/fortune-cookie.py
+```
+
+## Usage
+
+Call the program with `python3`:
+
+```shell
+$ python3 ~/bin/fortune-cookie.py
+```

We can see why the web interface is so comfortable: it wraps these Git commands and displays their output in a visually pleasing way. What’s great about Git is that if one does want to use only the command-line interface, it’s possible to do so; one isn’t restricted to a single interface for a given task.

Explicitly pulling proposed changes by adding an extra remote repo (optional)

There’s another option for Eck to access Alice’s proposed changes. For this, he needs to add Alice’s repository as a remote repository and then fetch the changes. This involves more steps than the process outlined above; however, it could be useful if Alice is a frequent contributor. This way, he won’t have to look up Alice’s repository URL for each contribution she makes. Let’s run through this variant to see what it looks like.

Eck adds Alice’s upstream repository as a “remote” of his local repository:

$ git remote add alice https://gitlab.com/aliceellis/fortune-cookie

and then fetches it to his computer:

$ git fetch alice
warning: redirecting to https://gitlab.com/aliceellis/fortune-cookie.git/
From https://gitlab.com/aliceellis/fortune-cookie
 * [new branch] add-project-readme -> alice/add-project-readme
 * [new branch] main -> alice/main

As before, he can switch to the branch that Alice created:

$ git switch add-project-readme
Branch 'add-project-readme' set up to track remote branch 'add-project-readme' from 'alice'.
Switched to a new branch 'add-project-readme'

and list the contents of the directory with ls:

$ ls
fortune-cookie.py README.md

Also, he can see what Alice committed by running git show:

$ git show
commit 7b8b638a524ffc52b7eabdec52ca8540514cf7fb (HEAD -> add-project-readme)
Author: Alice Ellis <alice@peateasea.de>
Date: Thu Nov 13 15:46:43 2025 +0100

    Add an initial project README

    A README file helps introduce the project to new users and describes
    what the main program does.

    This change adds a README file to the project describing briefly what
    the program does and showing potential users how to install and run the
    `fortune-cookie` program.

<snip> (same output as above)

Merging merge request changes

Eck was happy with Alice’s contribution from the view he had on GitLab. Thus, he didn’t need to go through the rigmarole of fetching Alice’s changes via the command-line.

To merge Alice’s changes, he went to the GitLab overview of Alice’s merge request:

and clicked on the “Merge” button. This changed the merge request’s state to “merged”. The steps that GitLab carried out on Eck’s behalf were documented in the “Activity” section on the merge request’s overview page.

Eck scrolls down to the comment box at the bottom of the page and thanks Alice for her contribution.

Because Alice is participating in this discussion (she submitted the merge request), she gets an email notification that the merge request was merged, and another with Eck’s thanks.

Updating the original local Git repository after an upstream merge

There’s one step that Eck needs to perform before he’s finished: he needs to pull the merged changes in his upstream repository down to the local repository on his computer.

Assuming that he’s on the main branch, he only needs to use git pull:

$ git pull
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
Unpacking objects: 100% (1/1), 273 bytes | 273.00 KiB/s, done.
From https://gitlab.com/eckzampill/fortune-cookie
   9771905..15007da main -> origin/main
Updating 9771905..15007da
Fast-forward
 README.md | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
 create mode 100644 README.md

Now his local repo is up to date with his upstream repo, and he can continue developing his fortune cookie program. Yay! 🎉

Updating the fork after an upstream merge

How do things now look from Alice’s perspective? Let’s change back to Alice’s view of the world and see what’s changed.

After receiving the notification that her merge request has been merged, Alice reloads her GitLab project page.

She sees that her fork on GitLab is now two commits behind the original repository (it is considered to be “upstream” from her online fork).

Why is her fork two commits behind? After all, she only submitted one commit as part of her merge request. The reason for the extra commit is that Eck configured his project to add a merge commit when merging changes from another branch.

A merge commit can be useful if one wants to have provenance of where a given commit or set of commits came from. For instance, the merge commit generated by GitLab when Eck accepted Alice’s merge request contains this commit message:

Merge branch 'add-project-readme' into 'main'

Add an initial project README

See merge request eckzampill/fortune-cookie!1

Here, we can see the original branch name, the merge request’s title, and a link to the merge request page on GitLab (eckzampill/fortune-cookie!1). Depending upon one’s taste, this information can be useful to have. Opinions vary somewhat concerning what the “correct” strategy is.

There are a few merge options to choose from on both GitLab and GitHub. It’s a good idea to have a look at what’s available and think about how you want merges to be handled in your situation.

But for now, let’s get back to Alice. She wants to update her repository to the current state of Eck’s original repository. How can she do this? One option is for her to click on the “Update fork” button in GitLab. This will automatically pull in the changes from the original repository into her upstream repository. Then she can update her local repository by pulling the state of her fork from GitLab (e.g. by running git pull).

There is, however, another way. It involves only the command-line and is common in Open-Source Software development: she can make the original repository an extra remote repository of her local repository. This is in addition to her fork on GitLab, which is her “origin” remote repository.

Why might Alice want to do this? Well, she might be planning further contributions to Eck’s fortune cookie project and having his upstream remote as one of her remotes makes the command-line-based workflow smoother. In that situation, there’s no need for her to log in to the GitLab interface to pull changes from Eck’s repository. Updating via GitLab would add an extra step to her workflow: pulling changes from Eck’s repository to her upstream repository means she would then have to pull those changes from her upstream repo to her local one. Instead, she can simply pull the changes directly from Eck’s upstream repository. Much easier!

Let’s look at how Alice uses this workflow now.

To get the latest changes from Eck’s online public repository, Alice adds it as an extra remote to her local repository. To do this, she uses the following git remote command:

$ git remote add upstream https://gitlab.com/eckzampill/fortune-cookie.git

As when Eck set up his upstream repository as his remote origin earlier, we read this command like so: add a remote repository and call it upstream, specifying its online location with the URL https://gitlab.com/eckzampill/fortune-cookie.git.

Alice now has two remotes configured in her local repository. These she can check with the git remote command:

$ git remote
origin
upstream

The entry called “origin” is her GitLab fork of Eck’s repository. When she pushes to (and pulls from) GitLab, this is the default repository that Git will use. The “upstream” repo points directly to Eck’s GitLab repository. If she wants to pull from Eck’s upstream repository, she will have to refer to it explicitly, e.g. as part of a git fetch command.

She can see more details about the remotes she has configured by using the --verbose option to git remote:

$ git remote --verbose
origin https://gitlab.com/aliceellis/fortune-cookie.git (fetch)
origin https://gitlab.com/aliceellis/fortune-cookie.git (push)
upstream https://gitlab.com/eckzampill/fortune-cookie.git (fetch)
upstream https://gitlab.com/eckzampill/fortune-cookie.git (push)

Now it’s much clearer that “origin” is her repo on GitLab and “upstream” is Eck’s repo on GitLab.

To get the latest changes from Eck’s GitLab repo, she fetches them to her local repository:

$ git fetch upstream main
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
Unpacking objects: 100% (1/1), 273 bytes | 273.00 KiB/s, done.
From https://gitlab.com/eckzampill/fortune-cookie
 * branch main -> FETCH_HEAD
 * [new branch] main -> upstream/main

where she tells git fetch to get the changes from the main branch on the repository called upstream. Had she not specified the repository name explicitly, git fetch would have used her “origin” repository by default. In this case, Git wouldn’t have fetched anything, as her local repo was up to date with that on GitLab.

The fetch operation added a new branch to her local repository: upstream/main (see the output from git fetch above). This is a copy of Eck’s main branch on GitLab, but with a different name in Alice’s repository. Git (and Alice) can thus tell it apart from her own main branch. Because Alice called Eck’s GitLab repository “upstream”, her local copies of his branches are prefixed with the string upstream/.

Alice can see a list of her local and remote branches by using the -a flag on the git branch command:

$ git branch -a
  add-project-readme
* main
  remotes/origin/HEAD -> origin/main
  remotes/origin/add-project-readme
  remotes/origin/main
  remotes/upstream/main

The branch with the star (*) next to it is her local main branch; the star signifies that she’s currently on that branch. Alice can also see the main branch in her “origin” remote repository as well as the main branch of the “upstream” remote repository (i.e. Eck’s repository on GitLab). There are thus three things called main here which, although related, aren’t exactly equal. This is because they can all be in different states at any given time.

This is something that can make Git seem so complex: names given to branches and repositories are often context-dependent. Alice’s main is not strictly the same as Eck’s main, although they may be very strongly related. Alice’s remote origin is “upstream” from her local repository, but Eck’s remote origin is “upstream” from Alice’s “upstream”. Some of these naming conventions have evolved organically, and concepts have been reused in different situations.¹¹ It’s not surprising people get confused.

Back to our example. Alice now wants to update her own main branch to the state of Eck’s main branch. To do this, she merges the changes from upstream/main into her main branch with git merge:

$ git switch main # make sure we're on the local main branch
$ git merge upstream/main
Updating 9771905..15007da
Fast-forward
 README.md | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
 create mode 100644 README.md

Running git log, she sees the same commits on her local main branch as those that exist on Eck’s GitLab repository.

Theoretically, she’s up to date and could start work on another merge request if she wanted to. There’s no real need for her to update her fork on GitLab if she doesn’t want to; all she’d change is the main branch. However, Alice likes things to be nice and tidy, so she decides to push the current state of her main branch up to her fork:

$ git push
Username for 'https://gitlab.com': alice@peateasea.de
Password for 'https://alice@peateasea.de@gitlab.com':
Enumerating objects: 1, done.
Counting objects: 100% (1/1), done.
Writing objects: 100% (1/1), 295 bytes | 295.00 KiB/s, done.
Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
To https://gitlab.com/aliceellis/fortune-cookie.git
   9771905..15007da main -> main

Now everything should be up to date. She returns to her web browser, and the landing page of her fork of the fortune cookie project, and reloads the page. She sees that her fork is back on par with the original project:

Great! All back up to date!

Also, she can see that since the project now has a README, GitLab shows its contents (the introductory text she added) on the project’s overview page. This information should help make the project more accessible for new users. Since that was her intention with the merge request, she achieved he goal. Awesome!

Trimming unneeded branches

There’s still one thing to tidy up. Now that the add-project-readme branch has been merged into the original project and Alice’s local, origin and upstream main branches are all in sync, she no longer needs the add-project-readme branch hanging around. Remember what git branch -a showed us earlier?

$ git branch -a
  add-project-readme
* main
  remotes/origin/HEAD -> origin/main
  remotes/origin/add-project-readme
  remotes/origin/main
  remotes/upstream/main

That list is unwieldy enough, however, imagine what things would look like with tens or even hundreds of dangling, already-merged branches. It would become very hard to see the wood for the trees! That’s why Alice now deletes her local add-project-readme branch. Since the commit on that branch is already in main, she doesn’t need the extra branch anymore:

$ git branch -d add-project-readme
Deleted branch add-project-readme (was 7b8b638).

Passing the -d option with a branch name to the git branch command deletes the named branch.

Also, she doesn’t need the add-project-readme branch on GitLab. This is for the same reason that she no longer needs the local version. Using the GitLab web interface, she could delete this branch by clicking on the “Branches” link under the “Project information” heading on the right-hand side of her fortune cookie project page:

(Equivalently, she could have clicked on the “Code” item in the sidebar on the left-hand side, and subsequently on the “Branches” sub-item.)

This will take her to the branches overview page for this project:

On this page, she can click on the three dots at the right on the line for the add-project-readme branch. Doing so will show a pop-up menu with the option to delete the branch:

Clicking on that option brings up a confirmation dialog box asking her if she is absolutely sure she wants to delete the branch.

Were she now to click on the “Yes, delete branch” button, GitLab would delete the add-project-readme branch in her fork of the fortune cookie project.

Of course, Alice knows of a much quicker and easier way by using the command-line. She can delete the branch on GitLab by pushing to it with the --delete option:

$ git push --delete origin add-project-readme
Username for 'https://gitlab.com': alice@peateasea.de
Password for 'https://alice@peateasea.de@gitlab.com':
To https://gitlab.com/aliceellis/fortune-cookie.git
 - [deleted] add-project-readme

This command tells Git to push the --delete operation on the add-project-readme branch of the origin remote repository. That’s just a roundabout way of saying “delete add-project-readme on the origin repo”.

Checking the list of all branches again, Alice sees that only those branches she expects to be present are in the command output:

$ git branch -a
* main
  remotes/origin/HEAD -> origin/main
  remotes/origin/main
  remotes/upstream/main

Yay! All cleaned up! Time for a celebratory hot beverage before digging into the next task.

Lather, rinse, repeat

Where to from here? Well, there are a few things one could do to improve the application. I’ve made a quick list of ideas below; each would be its own pull/merge request or possibly several.

Add new fortune cookie messages.
Extract messages into a data file (separate configuration from program).
Fix an incorrect message (if present).
Translate messages into another language.
Add documentation.

Of course, because it’s the internet, someone will notice that most of the phrases are proverbs and not fortunes and will want the name of the project changed (or similar). But ya get that.

A pull request, short and sweet

My friend had asked for a “concise” explanation¹² of how to make a pull request. Did I deliver? Well, no. It wasn’t concise in the slightest. Yet, to try and achieve that aim nonetheless, here’s the shortest version I could come up with.

Assume that you have an account on GitLab and have set up SSH. You fork the repository that you want to contribute to from the web interface. You make a note of your fork’s URL from the “Code” button on your fork’s main page in GitLab. Clone this URL to your laptop:

$ git clone git@gitlab.com:aliceellis/fortune-cookie.git

Change into the directory that Git created when cloning:

$ cd fortune-cookie

Imagine you want to add to the list of fortune cookie messages. Create a branch for the change you wish to make:

$ git switch -b add-new-fortune-messages

Now make the change and check that everything works as expected. Commit the change:

$ git commit fortune-cookie.py

Enter a well-written commit message in the editor that opens. When you have finished writing the message, save the file and quit the editor. The changes will now be committed to the branch. Push the branch to your fork:

$ git push

Note the URL mentioned in the output from the git push command and open it in a web browser. On the page that appears in your web browser, click on “Create merge request”.

Done!

Wrapping up

In the end, this wasn’t the concise explanation that my friend was wishing for; it turned into a rather detailed description instead. Nevertheless, I hope that it was helpful anyway!

To be honest, this is a simplification. Not all projects where one can contribute via pull requests are software-related. However, the vast majority are, hence my simpler statement. ↩
This is why they need to own a copy of your project. They can make changes (such as pushing to branches to their fork) that they wouldn’t be able to do with the original upstream repository. ↩
Historically, the main branch was called the master branch, but this usage is becoming less common. ↩
Actually called a bare repository, but that’s not important right now. ↩
Ever notice that, in English, filling in a form is the same as filling out a form? English can be weird at times. ↩
The “remote origin” mentioned here is a Git setting which points to Eck’s upstream repository on GitLab. The repository is “remote” because it’s not on Eck’s local computer. By convention, it’s called origin, hence why it’s referred to as the “remote origin”. ↩
If you hover over the button, you’ll see the help text “Create new fork”. ↩
An interesting point here is that there are now four copies of the repository: the original project repo on Eck’s local computer, Eck’s project repo on GitLab, Alice’s fork of the project repo on GitLab, and the local clone of the fork on Alice’s laptop. This duplication highlights the distributed nature of the Git version control model. Such duplication isn’t waste or a bug: it’s a feature. I’ve worked on projects where the upstream Git service had data corruption problems, and the fact that several developers each had a copy of the repository turned out to be very useful in recovering the data and getting the project back up and running again. ↩
While developing this worked example, I found this to be an odd default notification setting on GitLab. I would have expected that project owners want to be notified of merge requests in most cases, but it seems that’s not the case. My experience from GitHub is that, as a project owner or maintainer, notifications are enabled by default. Thus, when a new pull request comes in, an email is not far away. Hence, my surprise that Eck didn’t receive a message after Alice submitted her merge request. ↩
It has sometimes taken years for some of my pull requests to be seen and merged by maintainers. I’ve long forgotten that I submitted them, but sometimes an email appears with a “Hey, thanks!” message. I’m always really stoked to get such messages, even if I have to wait a long time for them 🙂 ↩
Just to be really annoying, sometimes Git terminology uses different names for the same thing. For instance, the staging area can also be referred to as the cache and as the index. ↩
The original German was “kurz und bündig”, which translates roughly to “concise”. ↩

Shrinking a VirtualBox disk

Paul Cochrane 🇪🇺 — Wed, 10 Dec 2025 23:00:00 +0000

Can you shrink VirtualBox disk images? It turns out you can, with the right setup. A recent cleanup operation led me to one way to get this particular set of ducks all in line. Here’s what I found out.

Copious collected cruft

Disk drives will do what they always tend to do: fill up with cruft over time. It’s the same with housework. To paraphrase Joan Rivers: you’ve just autocleaned the apt cache, deleted duplicate files, and purged dangling Docker containers, and six months later you have to start all over again.

While freeing up some space on my hard drive recently, I stumbled across a horrifying reality. A VirtualBox disk, created for what should have been a small Vagrant virtual machine (VM), was using up 20 GB of space!

$ cd ~/VirtualBox VMs/<vm-name>_1754671699254_15289
$ ls -ltrh
-rw------- 1 cochrane cochrane 20G Nov 26 18:41 box.vmdk

OMG. 😳 That’s rather a large chunk of the ~430 GB I’ve got on my host system’s main partition. How did it come to this?

The short answer to that question is: Docker.

The longer answer is that this VM was a machine I’d set up to test deployment of an application I’d been developing for a customer recently.¹ I’d Dockerised everything nicely, and each time I’d deployed a new version of the application, the deployment pulled in a new Docker image. The thing is, I wasn’t cleaning up old images, hence many dangling images began to collect on the system without me noticing. Oopsie.

Dynamic disk dilemma

Now, Vagrant has this nice thing where it creates a dynamic filesystem image when setting up a new virtual machine. Dynamic, in this sense, means that the filesystem image grows with the disk space demands of the guest system. This is as opposed to using all the allocated filesystem size from the get-go. I use the basic Vagrant setup, hence the VM provider is VirtualBox, which in turn uses the VMDK disk image format by default.

Anyway, since the filesystem image is dynamic, if you give a guest system a total of 100 GB to play with, the filesystem image on the host is only as large as the space that the guest actually uses. Thus, if the VM uses 20 GB of its available 100 GB, then the disk image on the host will also use 20 GB.

But what happens if you clean up the space inside the VM? Does that reduce the size of the disk image on the host? Unfortunately not. At least, it doesn’t in the default configuration. If one sets up things like auto-discard in fstab and uses a shrinkable disk image format (such as VDI), then yes, it’s possible. But since that’s not the standard setup, you’ll have the exploding disk image issue that I ran up against.

In my case here, a quick docker image prune purged many gigabytes of space inside my VM, but didn’t help the situation outside the VM. What to do?

A quick search online revealed a superuser.com answer with a nice set of instructions on how to compact a VDI disk image. What follows is my more detailed exposition of those instructions.

Copy, clone, convert

The first issue to handle is that I didn’t have the right disk image type to support shrinking or compacting. As mentioned earlier, I was using what VirtualBox via Vagrant gives me by default, the VMDK format. What I needed was the VDI format, which is native to VirtualBox and supports filesystem resizing.

Thus, we need to convert the VMDK disk image to VDI format. Fortunately, I’ve already covered this process in an article about resizing Vagrant VM disks.

Before converting the disk image format, be sure to shut down the VM:

$ vagrant halt <vm-name>
==> <vm-name>: Attempting graceful shutdown of VM...

Now is probably a good time to do a backup of the host system. Off you go! I’ll be here waiting for you. 🙂

Ah, you’re back. Great, now I’ll continue with the story.

Enter the directory containing the VirtualBox VM:

$ cd $HOME/VirtualBox\ VMs/<vm-name>_1754671699254_15289

and, assuming that there is enough space on the host system, make a copy of the VMDK file for safe-keeping:

$ cp box.vmdk box.vmdk.keep

You might also want to copy this file somewhere else (such as a completely different computer) just in case things go horribly wrong. You never know!

Now convert the VMDK image to VDI format by cloning it:²

$ vboxmanage clonehd box.vmdk box.vdi --format VDI
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Clone medium created in format 'VDI'. UUID: 9e7e5ea1-56d2-469e-8f64-44578964f6f4

You’ll find that the VDI image (like the VMDK image it’s based upon) is also ~20 GB in size.

$ ls -ltrh
-rw------- 1 cochrane cochrane 20G Nov 27 16:41 box.vdi

That makes sense; it’s essentially a 1-to-1 copy of the original. It was fortunate that I’d freed up enough space on my host system’s drive before adding all these large temporary files!

Swap similar storage

Ok, now that we’ve got our VDI image, we can attach this new storage medium to the VM using the configuration options mentioned in the “How to compact a VDI file” superuser.com answer. These options will let us trim the filesystem, which will then reduce the VDI disk image size.

But I’m getting a bit ahead of myself. To attach this new storage medium, we need to remove the old one first. To detach a storage medium from a VirtualBox-provided system, one uses the storageattach command:³

$ vboxmanage storageattach <vm-name>_1754671699254_15289 --storagectl 'SATA Controller' \
    --port 0 --medium none

The VMDK image is thus no longer associated with our VM. We then attach the VDI disk image like so:

$ vboxmanage storageattach <vm-name>_1754671699254_15289 \
    --storagectl 'SATA Controller' --port 0 --type hdd --nonrotational=on --discard=on \
    --medium $HOME/VirtualBox\ VMs/<vm-name>_1754671699254_15289/box.vdi

where the important extra options are --nonrotational=on and --discard=on.

The --nonrotational option tells the VM that the disk is an SSD, which then changes the IO behaviour to be better for SSDs. The VirtualBox manual explains this well:

--nonrotational=on | off

Enables you to specify that the virtual hard disk is non-rotational. Some guest OSes, such as Windows 7 or later, treat such disks as solid state drives (SSDs) and do not perform disk fragmentation on them.

The --discard option enables TRIM support on the disk. Again, the VirtualBox documentation is illuminating:

--discard=on | off

Specifies whether to enable the auto-discard feature for a virtual hard disk. When set to on, a VDI image is shrunk in response to a trim command from the guest OS.

The virtual hard disk must meet the following requirements:

The disk format must be VDI.

The size of the cleared area of the disk must be at least 1 MB.

Ensure that the space being trimmed is at least a 1 MB contiguous block at a 1 MB boundary.

You can use other methods to issue trim commands. The Linux fstrim command is part of the util-linux package. Earlier solutions required you to zero out unused areas by using the zerofree or a similar command, and then to compact the disk. You can only perform these steps when the VM is offline.

I’d checked for this support before converting the disk image and found that the output of

$ hdparm -I /dev/sda | grep TRIM

from within the VM gave no output. In other words, before adding the --discard=on option, TRIM support wasn’t available. Since the guest filesystem is ext4, this should be possible, hence why we’re adding it here via the --discard option.

For information about the other options in the storageattach command above, check out my earlier post about resizing Vagrant VM disks.

Remove redundant residue

Now that we’ve attached the cloned VDI disk image to the VM, we can start it again:

$ vagrant up <vm-name>
Bringing machine '<vm-name>' up with 'virtualbox' provider...

and enter it:

$ vagrant ssh <vm-name>

Now we can check again to see if the drive supports TRIM:

vagrant@<vm-name>:~$ sudo hdparm -I /dev/sda | grep TRIM
           * Data Set Management TRIM supported (limit unknown)

Looking good!

In contrast to the description in the How to compact a VDI file superuser.com answer, we’re not setting up auto-discard on the disk here, hence we don’t need to update /etc/fstab.

But we can use fstrim directly as the superuser.com answer describes:

vagrant@<vm-name>:~$ sudo fstrim /

The disk usage within the guest system (Vagrant VM) won’t change, hence df -h won’t show anything different.

Before:

vagrant@<vm-name>:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 207M 0 207M 0% /dev
tmpfs 46M 476K 46M 2% /run
/dev/sda1 98G 4.4G 89G 5% /
tmpfs 229M 0 229M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
vagrant 437G 380G 58G 87% /vagrant
tmpfs 46M 0 46M 0% /run/user/1000

After:

vagrant@<vm-name>:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 207M 0 207M 0% /dev
tmpfs 46M 476K 46M 2% /run
/dev/sda1 98G 4.4G 89G 5% /
tmpfs 229M 0 229M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
vagrant 437G 380G 58G 87% /vagrant
tmpfs 46M 0 46M 0% /run/user/1000

(Even though the output is identical, yes, I really did check this.)

Although the guest system won’t show any changes, the VDI image on the host system should now be much smaller.

$ ls -ltrh
-rw------- 1 cochrane cochrane 5.8G Nov 27 17:34 box.vdi

This reclaims 15 GB of space on the host system. Awesome!

Purge persisting PC parts

Since we’ve got a running system with a shrinkable filesystem, it’s now safe to remove the old VMDK image. The best way to do this is with VBoxManage’s closemedium command:

$ vboxmanage closemedium disk \
    $HOME/VirtualBox\ VMs/<vm-name>_1754671699254_15289/box.vmdk --delete

You may be wondering why I use a VBoxManage command rather than simply deleting the file with rm. It turns out that attaching a storage medium to a VirtualBox VM also registers that storage medium with it. If we only delete the disk image file, it remains registered with the VM even though the file no longer exists. Such a dangling registration will cause errors if we want to create a VMDK image with the same filename for this VM in the future. I found out about this subtlety while testing the clonehd process I described above.

Wrapping up

I definitely learned something new here: it wasn’t clear to me that one can trim a filesystem of unused blocks. It’s cool that this also reduces a guest system’s disk usage. It was also new to me that TRIM functionality is something beneficial for solid-state media.

Being able to use this functionality to reduce the size of a virtual machine’s disk was a bit of a lifesaver. It allowed me to free up space on my host system while retaining all the data I had in the guest system. It would have been painful to have to recreate everything in the guest system just to cut down on the host system disk usage.

Also, good, detailed documentation rocks. Thank you to whoever put the time and effort into creating the VBoxManage docs!

Addendum: clean up thoroughly before cloning again

Not everything always goes to plan. And so it was while writing this up. Because I wanted to test converting the VMDK image to VDI, I ran the clonehd command that I used above more than once.

In one test run, I removed the .vdi file with rm and ran clonehd a second time. That gave me this error:

$ rm box.vdi
$ vboxmanage clonehd box.vmdk box.vdi --format VDI
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...NS_ERROR_INVALID_ARG
VBoxManage: error: Failed to clone medium
VBoxManage: error: Cannot register the hard disk '$HOME/VirtualBox VMs/<vm-name>_1754671699254_15289/box.vdi' {5e0c4418-9c06-4b88-ab08-298ba834356e} because a hard disk '$HOME/VirtualBox VMs/<vm-name>_1754671699254_15289/box.vdi' with UUID {9e7e5ea1-56d2-469e-8f64-44578964f6f4} already exists
VBoxManage: error: Details: code NS_ERROR_INVALID_ARG (0x80070057), component VirtualBoxWrap, interface IVirtualBox
VBoxManage: error: Context: "RTEXITCODE handleCloneMedium(HandlerArg*)" at line 1208 of file VBoxManageDisk.cpp

Whoops! That doesn’t look good.

What’s going on here? Well, it turns out that VirtualBox registers a storage medium with the VM when cloning. I would guess that the original VMDK image points to its VM, and this pointer is recreated when making the cloned VDI image. Thus, cloning automatically registers the VDI image with the VM.

So, to really remove the VDI file before recreating it, we have to use the closemedium command to remove its entry from the registry:

$ vboxmanage closemedium disk 9e7e5ea1-56d2-469e-8f64-44578964f6f4 --delete
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%

where in this case I used the disk’s UUID instead of its full path.

Now it’s possible to run the format conversion again:

$ vboxmanage clonehd box.vmdk box.vdi --format VDI
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Clone medium created in format 'VDI'. UUID: 52d0c913-d7bc-490e-a056-96ac5cdca6aa

Yay! Something else learned!

If you need a persistent, thorough, and experienced software developer with Linux and DevOps experience, give me a yell! I’m available for freelance Python/Perl backend development and maintenance work. Contact me at paul@peateasea.de and let’s discuss how I can help solve your business’s hairiest problems. ↩
See the resizing Vagrant VM disks article for more details and background information. ↩
Think of it as attaching “none” to the VM rather than detaching the disk from the VM. ↩

Become-ing a user securely in Ansible

Paul Cochrane 🇪🇺 — Wed, 03 Dec 2025 23:00:00 +0000

Using become_user in Ansible requires the acl package to be installed on the remote host. This is a note to my future self, reminding me of the error, the root cause, and the solution. Maybe it helps someone else as well. 🙂

Running into a familiar problem

Once upon a time, one could use become_user in an Ansible task, and things would “just work”™. It turns out that Ansible wasn’t doing this as securely as it could have. These days, Ansible sets (or tries to set) the appropriate access controls on uploaded files and directories. Legacy Ansible tasks will therefore run into errors if the acl package isn’t installed on the remote host.

So why is this interesting, and how did I get here? Well, while updating some old Ansible playbooks for a client recently,¹ I ran across a problem that I’d seen before, but had never written up properly. This time, I decided to make notes for my future self and anyone else who might need to know about this problem and its solution.

Ansible is a great tool for provisioning and system configuration. The fact that I can define a system’s configuration state with something akin to code, which I can then keep in Git, is brilliant. It dovetails well with my needs as a programmer and sysadmin.

I’ve been using Ansible for a few years now, and sometimes it shows. Some of my playbooks don’t use current best practices, which is odd for me, because I’m usually really picky about adhering to best practices. Such things help teams work together more easily because there are fewer “surprises” in code.

But if you don’t touch things very often, stuff gets outdated. Thus, playbooks and tasks that once ran flawlessly no longer work. Here is the instance that struck me most recently:

- name: install base requirements
  ansible.builtin.pip:
      requirements: "{{ project_dir }}/base.txt"
      virtualenv: "{{ venv_path }}"
      virtualenv_command: "virtualenv --python=/usr/bin/python3"
  become_user: <functional_user>

This task uses pip to install the base requirements for a Python application and ensures that a virtual environment has been set up in advance.² The task also does this using a functional user account. A functional user is a user account that you create for the sole purpose of running the application. We’re not in the 90s anymore, and hence we don’t want to run our services as root or some other privileged user. Having a functional user account is nice because it encapsulates an application’s code and configuration into one location. Of course, this all assumes that the application isn’t containerised, something that one runs across often when dealing with legacy software.

Running the playbook containing this task, I got the following error message:

fatal: [old-fashioned-test-system]: FAILED! => {"msg": "Failed to set
permissions on the temporary files Ansible needs to create when becoming an
unprivileged user (rc: 1, err: chown: changing ownership of
'/var/tmp/ansible-tmp-1763739956.7741241-2042672-91787264687983/': Operation
not permitted\nchown: changing ownership of
'/var/tmp/ansible-tmp-1763739956.7741241-2042672-91787264687983/AnsiballZ_pip.py':
Operation not permitted\n}). For information on working around this, see
https://docs.ansible.com/ansible/become.html#becoming-an-unprivileged-user"}

I’d seen similar messages a couple of times in the past and had re-solved the underlying problem each time. So, it was high time I documented everything more thoroughly.

Understanding the root cause

So what does this error message mean, and what should we do to fix the situation? An obvious thing to try would be to look up the documentation link mentioned in the error message. Unfortunately, that link location no longer exists. Fortunately, the Internet Archive has our back and has kept a copy for us. The bit that we’re interested in is the Becoming an Unprivileged User section.

The gist of the problem is that, for modules where pipelining isn’t possible, Ansible needs to set the access control flags on the temporary file that it sends to the remote host. After all, we don’t want every Tom, Dick and Harry being able to see what files Ansible is shuffling around. By installing the acl package, we ensure that the setfacl command exists on the remote host. Ansible then uses setfacl to set file-level access controls. This is more secure than the previous behaviour.

For me, a big signal here is that I need to upgrade my Ansible installation. This will, in turn, require me to upgrade my Debian installation, which requires planning and a lot of time. Hence, upgrading Ansible isn’t going to happen in a hurry. At least, not right now. :-/ But, as fellow Kiwi Rachel Hunter once said, “It won’t happen overnight, but it will happen”. 🙂

So yeah, I have to confess, I’m currently using an ancient Ansible version:

$ ansible --version
ansible 2.10.17
  config file = /<project_path>/playbooks/ansible.cfg
  configured module search path = ['/<home_path>/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.9.2 (default, Feb 28 2021, 17:03:44) [GCC 10.2.1 20210110]

And yes, I’m still on Debian bullseye. sigh. It can be hard to upgrade an operating system while also getting things done, all right?

A solution and its nuances

Ok, back to the story at hand. As mentioned a couple of times, the solution to the problem above is to install the acl package. That turns out to be really easy. Install the package with the appropriate OS package manager module.³ You’ll need a task that looks something like this:

- name: install general packages
  ansible.builtin.apt:
    name: [
      'acl',
      .... other OS packages
    ]

Simply installing acl isn’t necessarily sufficient, however. After having installed the setfacl command, you might find that you run into the next security-related problem: a missing temporary directory on the remote host that has the correct permissions for the functional user. In other words, you might get a warning like this one:

[WARNING]: Module remote_tmp /home/<functional_user>/.ansible/tmp did not exist and was created with a mode of
0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with
the correct permissions manually

The solution to this part of the problem is to create the appropriate remote temporary directory for the functional user account. The best place to put this task is straight after that which created the functional user, e.g.:

# avoid warnings tmp dir being created with incorrect permissions
- name: create the Ansible temporary dir for the <functional> user
  ansible.builtin.file:
    path: /home/<functional>/.ansible/tmp
    owner: <functional>
    group: <functional>
    mode: '0700'
    state: directory

where I’ve left the value <functional> as a placeholder for the actual functional user account name.

Note that the task requiring become_user will also need become: True so that the become process works as expected.⁴ Thus, you might need to update the task to include this parameter as well as the other changes mentioned here.

So, the original task to install the app’s base requirements with pip, erm, becomes:⁵

- name: install base requirements
  ansible.builtin.pip:
      requirements: "{{ project_dir }}/base.txt"
      virtualenv: "{{ venv_path }}"
      virtualenv_command: "{{ ansible_python.executable }} -m venv"
  become: True
  become_user: <functional_user>

where I’ve fixed the virtualenv_command to replace virtualenv with (effectively) python3 -m venv to create the virtual environment.

Important: On Debian/Ubuntu, it’s also necessary to install the python3-venv package for the venv module to be available. Thus, you’ll need a task which runs beforehand that looks like this one:

- name: install dependencies for <app>
  ansible.builtin.apt:
      name: [
          'python3-pip',
          'python3-venv',
          ... other packages ...
      ]
      update_cache: yes
      cache_valid_time: 3600

I think that’s got most of the t’s dotted and i’s crossed!

Time-travelling love letters

This blog post is a love letter to my future self so that I find the solution more easily should I stumble across this issue again. Hullo, future me! 👋

As a side note, it was pretty cool to find that I’d put lots of detail into commit messages in the repo containing my Ansible configuration. For example:

    Create an Ansible tmp dir for <functional> user

    Some Ansible processes copy files to the remote system and if they use
    `become_user` then a new temporary directory is created to handle the
    files in that case. When this happens, the `remote_tmp` module gives a
    warning that a `tmp` dir is created with particular permissions and this
    might be a Bad Thing (TM) in certain conditions and recommends creating
    the temporary directory explicitly.

This (and other related commit messages) made piecing together the ideas outlined in this article much less onerous. Having described things now all in one place should make fixing everything in the future much easier.

Thank you, past me, for taking the time to do that!

If you need a persistent, thorough, and experienced software developer with Linux and DevOps experience, give me a yell! I’m available for freelance Python/Perl backend development and maintenance work. Contact me at paul@peateasea.de and let’s discuss how I can help solve your business’ hairiest problems. ↩
Come to think of it, the fact that I use virtualenv here rather than the venv module from Python’s standard library tells me that there’s more here that I need to modernise. ↩
Since I’m on Debian, I’m using apt. If you’re on another Linux distribution, you’ll have to use an appropriate other package manager. ↩
ansible-lint, for example, will tell you this. ↩
Future me might be wondering why I used ansible_python.executable when the pip module documentation clearly states that one should use {{ ansible_python_interpreter }}. It turns out that only the {{ ansible_python }} dict was available in my configuration (i.e. in the Ansible facts), hence the need to use {{ ansible_python.executable }}. This variable pointed to /usr/bin/python3, which I didn’t want to hard-code in the task. Hopefully, future me will understand! ↩

Private areas in Jekyll with Apache

Paul Cochrane 🇪🇺 — Wed, 16 Jul 2025 22:00:00 +0000

Jekyll-built static sites are public by default. However, have you ever wanted to create a private area where you can upload articles for review and keep them from the public eye until they’re ready? That was my use case recently. Here’s how I solved this particular puzzle.

My blog is a static site built with Jekyll and the Minimal Mistakes theme. The site is hosted in a Hetzner webspace, which means that the pages are served via Apache.¹ These are the assumptions I made when writing up this post. So, if you use a provider which doesn’t use Apache, or the files are different because of a different Jekyll theme, then this solution likely won’t work for you. Nevertheless, I hope it gives someone a helping hand when trying to create their own private area of a Jekyll-built site!

The two main ingredients we need are Jekyll collections and Apache Authentication and Authorization. Let’s see how we combine them to achieve our goal.

Create a “subscribers” area

The first step is to create a new path within which to put articles for reviewers or subscribers. If you’re using Jekyll in its standard configuration, posts will always have their path based at the root domain. For example, if you host your blog on https://example.com,² and /:title/ as your permalink setting (as I do), articles will appear under their own path. Hopefully, a concrete example will clear what I mean. Imagine you have an article called “Reading books for fun and profit”. Jekyll will make the article available under the URL https://example.com/reading-books-for-fun-and-profit/. The path I’m talking about is the reading-books-for-fun-and-profit/ bit.

Jekyll has lots of flexibility in how to set up permalinks for your posts. If you’ve used the default option, then you’ll see a URL with category, date and post title. Continuing with the example just mentioned, you might see something like this: https://example.com/books/2028/02/29/reading-books-for-fun-and-profit.html. In this case, the path will be books/2028/02/29/.

Anyway, I hope you get my point about the URL paths that Jekyll creates for blog posts. What we want to do here is to create the path subscribers/ at the domain root. Then, our “subscribers” can access preview articles under https://example.com/subscribers/, instead of whatever structure Jekyll uses for posts. To achieve this goal, we’ll use Jekyll collections.

As the docs say,

Collections are a great way to group related content like members of a team or talks at a conference.

That’s basically what we want to do; the only difference being we don’t want our collection to be publicly available. URL path protection is a separate step, so we’ll get to that later.

To create a collection, we use the collections setting in the _config.yml file:

collections:
  subscribers:
    output: true

With this configuration in place, Jekyll will know to look for Markdown files within a _subscribers/ directory located in the base directory (i.e. where _config.yml and friends live). Note that the leading underscore in the directory name is important: the configuration value is subscribers, but the directory name is called _subscribers/.

The output: true option tells Jekyll to process all files in the _subscribers/ directory and produce a rendered page for each file. Since each file we want to process in this directory is going to be an article for our “subscribers”, this is the behaviour we want.

Now, to start creating some content. Create the _subscribers/ directory in the base project directory:

$ mkdir _subscribers/

Then, create an article within that directory by opening a file called _subscribers/reading-books-for-fun-and-profit.md in your favourite editor. Fill it with this text:

---
title: Reading books for fun and profit
categories:
  - Books
---

Books rock! You should read more of them. Did you know that libraries are quite simply *packed* with them? Mind = Blown. :exploding_head:

Building your site in development mode, you should now see your article appear under your new, shiny subscribers/ URL path:

http://localhost:4000/subscribers/reading-books-for-fun-and-profit/

That’s cool! But it could be better. You’ll likely find that there isn’t any styling, so you’ll see only plain text at the top of the page. This is not something you’d like to show off to your mates! To get the styling right, we need to extend the defaults: setting in the Jekyll config. Open your _config.yml file and append the following code to your defaults: configuration:

# Defaults
defaults:

  <snip>

  # subscribers' area page styling
  - scope:
      path: ""
      type: subscribers
    values:
      layout: single
      author_profile: true
      read_time: true

Now, if you rebuild your site, you should see the article at http://localhost:4000/subscribers/reading-books-for-fun-and-profit/ styled nicely with a better layout and styling. This setup also shows your author bio information (author_profile: true) and how long it will take to read the article (read_time: true). How much those settings are only related to the Minimal Mistakes theme, I don’t know, so YMMV. In my case, this made the “subscribers” area consistent with the rest of my site.

Creating a “subscribers” landing page

With the subscribers/ path created and an article in it, it’d also be nice for “subscribers” to see an overview of the available articles in the “Subscribers’ Area”. I.e. if someone navigates to https://example.com/subscribers/, then they won’t be presented with a 404 page or similar; they’ll see a list of articles to read at their leisure.³

To make such a landing page, we need to create a file not in the _subscribers/ directory, but in the base project directory. This was something that surprised me, so hopefully you’ll now be spared that confusion. I’d expected everything associated with the new collection to be kept within that directory, but alas, I was wrong. Oh well.

Jekyll expects this file to consist of the collection name followed by a .html extension, i.e. in our case that is, subscribers.html. This file’s content is a mixture of Jekyll frontmatter, Markdown and HTML, which is a bit weird, but that’s the way this particular cookie crumbles, I guess.

Open your favourite editor and create a file called subscribers.html with this content:

---
layout: archive
title: Subscribers' Area
author_profile: true
---

{% assign entries_layout = page.entries_layout | default: 'list' %}
<div class="entries-{{ entries_layout }}">
  {% for post in site.subscribers %}
    {% include archive-single.html type=entries_layout %}
  {% endfor %}
</div>

I gleaned this code from perusing the Minimal Mistakes theme code, so hopefully this also works for you. 🤞

I’m fairly sure I understand what this code is doing, so here’s my explanation.

We base our layout on the archive layout as shown in the frontmatter at the beginning of the file. This seems to be sensible as it is what’s used for the main “Recent Posts” page as well. Why it’s called “archive”, I don’t know, but it works, so I’m not complaining. The title key provides the page’s title, and the author_profile key puts the blog author’s bio information on the same page to keep the same look and feel as the rest of the site. Note that the styling configuration change we made to the defaults: setting above doesn’t apply to this page. This is because it’s not a file within the _subscribers/ directory, hence we have to mention author_profile in its frontmatter.

The assign statement sets the entries_layout variable either to the value of page.entries_layout or to list if page.entries_layout isn’t set or doesn’t exist. We pass this variable as the type argument to the archive-single template used later within the include statement. It also helps with the styling of the entries appearing within the <div> block, via the <div>’s class attribute. Within the <div>, we loop over all posts in the list of files that Jekyll found in the _subscribers/ directory. In other words, we iterate over all elements in the site.subscribers list. Note that it’s necessary to use the variable name post as the loop variable here because the article-single template uses this variable name internally. I probably could have written something myself which used a more appropriate variable name, but it would have duplicated a lot of code for little reason.

The loop creates an overview of the available articles, showing their titles as well as the “teaser” text for that article.⁴

Let’s add another article to see what the list can look like with more than one article.

Open a new file called _subscribers/books-in-modern-society.md and fill it with this content:

---
title: Books in modern society
categories:
  - Books
---

Are parchment scrolls a thing of the past? Are books the future or just a
fad? We asked Pliny the Elder his opinion: "I think there is a world market
for maybe five books". Others have chimed in on this new trend with "I love
the smell of books in the morning". It seems that expert opinion is divided
on this contentious issue.

Rebuilding the development site and navigating to http://localhost:4000/subscribers/, you should see something like this:

We’ve thus created an area for a certain group of privileged people to read our articles. Great! The only problem is: anyone can read it. I.e. it’s public and not private. How do we fix that? By getting Apache to control access for us.

Make the “subscribers” area private

Now that we have a directory to protect, let’s protect it.

The Hetzner directory protection docs describe how to do this if you’re a Hetzner user. However, once you’ve gone through that process, you realise that this is just an automated way to create .htaccess and .htpasswd files in the directory you want to protect. It’s easier to create these files ourselves, rather than clicking through a GUI to achieve the same goal.

The way to do this is to create two files, a .htaccess file and a .htpasswd file, and upload them to the subscribers/ directory where our blog is hosted. The .htaccess file controls the access configuration, and the .htpasswd file contains the usernames and passwords of our “subscribers” so that only they can access this directory.

The .htaccess file looks like this:

AuthUserFile "/usr/www/users/<username>/subscribers/.htpasswd"
AuthName "Subscribers area"
AuthType Basic
require valid-user

which is a standard Apache .htaccess configuration allowing basic HTTP authentication.

The AuthUserFile option specifies the password file to use for authentication. In this case, the path is specific to Hetzner; thus, you will likely need to change the /usr/www/users component to some other value for your setup. The main point is that we specify the path to the .htpasswd file within the subscribers/ directory on our hosting provider’s infrastructure. We’ll copy the file there when deploying our Jekyll site to our hosting provider.

The AuthName value

sets the name of the authorization realm for a directory.

This value is just a name, so it’s easiest here to use the same name as the title we gave our subscribers’ area landing page.

As mentioned above, we use basic HTTP authentication; hence, the config option AuthType is set to Basic.

And of course, we require a valid user for login, hence we set require valid-user. We don’t want any old riff-raff looking at our nice preview articles!

With the configuration all set up, we now need to create a .htpasswd file. This file contains colon-separated usernames and hashed passwords for those who are allowed access to our “subscribers” area. The password hashing uses the bcrypt encryption option of the Apache htpasswd command. But first, what password should we use?

In my use case, I thought it would be best if I chose a long, random password and informed each user individually what their username and password are. It’s clear that this doesn’t scale to hundreds of users. But then, I only want to show some articles to friends for review, so this solution is ok for me. Also, if someone finds out the password for a user, it’s not that bad. I mean, whoever happens to nab this information only gets read-only access to a couple of articles. Not a major security incident. Of course, if you have hundreds of subscribers, you’ll need to come up with a much better solution!

To create a good random password, my tool of preference is pwgen. If you’re on Debian, you can install it in the usual manner:

$ sudo apt install pwgen

To create a single 20-character password, call the program like so:

$ pwgen 20 1

We then feed this output into the htpasswd program (which is part of the apache2-utils Debian package):

$ htpasswd -nbB alice password-generated-from-pwgen
alice:$2y$05$iFwBDdjNbE3sef7siS2/dOu.DqAkVhiw8zVcMLiPfE.y9tw.i8ypO

where the -n option displays the output on the terminal, -b uses the password specified on the command line, and -B forces bcrypt encryption. The two arguments to htpasswd are the username (in this case, alice) and the password generated by pwgen.

You then paste the output from htpasswd into the .htpasswd file which is located in the _subscribers/ directory. If you don’t want to do that by hand, then you can get htpasswd to update the file directly by removing the -n option and specifying the .htpasswd file explicitly.

First, create the file with touch (htpasswd requires the password file to exist beforehand):

$ touch _subscribers/.htpasswd

and set the password:

$ htpasswd -bB _subscribers/.htpasswd alice password-generated-from-pwgen
Adding password for user alice

You can add further users in the same manner, e.g. for the user bob:

$ htpasswd -bB _subscribers/.htpasswd bob password-generated-from-pwgen
Adding password for user bob

You don’t want everyone in the world to be able to read this file, so set its file permissions to remove group and world read permissions:

$ chmod 0600 _subscribers/.htpasswd

The last step in this process is to upload the .htaccess and .htpasswd files to the subscribers/ subdirectory where your blog is hosted. This process depends upon your hosting provider, so I’m not going to describe it here. I’m guessing that there’s either an FTP or rsync connection that you can use to copy your blog’s _site contents to the appropriate location on your hosting provider’s servers. You’ll have to work that bit out on your own, sorry. :-/

One thing I will note: if you want to get Jekyll to copy your .htaccess and .htpasswd files into the _site/subscribers directory when building your site, you’ll need to tell Jekyll to include them. You do this by adding these filenames to the include: directive in your _config.yml file:

include:

  <snip>

  - .htaccess
  - .htpasswd

Now, when you copy your site to your hosting provider, the .htaccess and .htpasswd files will be copied to the correct location automatically.

Wrapping up

And that’s it! Now you have most of the pieces to the puzzle of how to password-protect a subdirectory of your Jekyll-built site. Although I can’t flesh out a complete solution,⁵ I hope you’ve been able to see enough of the shape of the solution to be able to fill in the gaps on your own.

I hope that helps, and I wish you luck!

I deduced this detail from the Hetzner docs, where configuration of web server extensions uses mod_http2 and redirecting a domain to a subdomain requires using mod_rewrite and setting up a .htaccess file. I.e. classic Apache config. ↩
Look, I know it’s not hosted there, but we’ll make the assumption for now and use that as a placeholder for the actual domain. ↩
That is, of course if they’re not already reading a book, which is always difficult to avoid. ↩
The “teaser text” is the article’s first paragraph. ↩
This is because I don’t know what theme you might be using and how that impacts things nor what hosting provider you have. If you want help with this, give me a yell! ↩

Action of a simple harmonic oscillator

Paul Cochrane 🇪🇺 — Wed, 09 Jul 2025 22:00:00 +0000

What’s the action of a simple harmonic oscillator? And how does this change depending upon the path? Some mates of mine and I worked through this recently. This is my write-up of that calculation.

While discussing special relativity recently, some friends of mine and I realised that we’d not understood the concept of action. It seems that this isn’t a well-understood concept, at least not at an intuitive level: it “just works” and most people seem happy with that. As mentioned in a Veritasium video a while ago, Euler and Lagrange gave the concept of action a firm mathematical foundation. The idea that action is minimised is known as the principle of least action.

To understand this topic better (or at least, try to), we decided to consider a simple harmonic oscillator as a toy example. Since we know the solution, we could then calculate the action for the path matching the known solution and, as a comparison, the path of a linear ramp. The idea here being that we’d find the action to be minimised over an oscillatory path and that a path differing from that would give a non-minimal action. Thus, we could develop a better intuition for the principle of least action and hence for why the Lagrangian formalism works the way it does.

Let’s look at how to calculate the action for a simple harmonic oscillator.

Setting the stage

Consider a simple harmonic oscillator (SHO) such as a mass on a spring. We wish to calculate the action for a quarter oscillation of this system to try and convince ourselves that the standard solution for the equations of motion of a SHO (in a simplified form) gives the minimal (and hence optimal) action. We will later compare this to an alternative motion with the same boundary conditions¹–but following a different path between them–to see that the standard solution minimises the action. We will also show that the general solution for the action of a SHO reduces to the same value as our simplified form when considering these boundary conditions.

We know that the general definition of the action is defined as

\begin{equation} S = \int_{t_1}^{t_2} L(\dot{q}, q, t)\ dt, \end{equation}

with $q$ and $q˙\dot{q}$ being the generalised positions and velocities of the objects in motion, and where $t$ is time. The values $t_1$ and $t_2$ denote the start and end points of the interval over which we are measuring the action. The quantity $L$ is the Lagrangian, which is defined by

\begin{equation} L = T - V, \end{equation}

with $T$ being the kinetic energy and $V$ being the potential energy.

We simplify the situation by considering only motion in the $x$ -direction. Hence, $q$ and $q˙\dot{q}$ reduce to $x$ and $x˙\dot{x}$ , respectively.

The kinetic energy is written in its familiar form:

\begin{equation} T = \frac{1}{2} m v^2 \end{equation}

which, given we only consider motion in the $x$ -direction, we write as:

\begin{equation} T = \frac{1}{2} m \dot{x}^2 \end{equation}

For motion under the influence of a conservative force,² the potential energy is the negative of the integral of all forces in the system, i.e.

\begin{equation} V = - \int F\ dx, \end{equation}

where in this case we only consider forces acting in the $x$ -direction. To find the potential energy of a mass on a spring, we use Hooke’s Law for the force:

\begin{equation} F = -kx \end{equation}

and thus integrate over $x$ . Hence we have

\begin{equation} V = - \int F\ dx \end{equation}

\begin{equation} = \int kx\ dx \end{equation}

\begin{equation} = \frac{1}{2} kx^2. \end{equation}

Combining (4) and (9) we write the Lagrangian like so:

\begin{equation} L = \frac{1}{2} m \dot{x}^2 - \frac{1}{2} k x^2. \end{equation}

The action is therefore

\begin{equation} S = \int_{t_1}^{t_2} \biggl[\frac{1}{2} m \dot{x}^2 - \frac{1}{2} k x^2 \biggr]\ dt. \end{equation}

With the general form for the action of a simple harmonic oscillator in hand, we can consider the action over different paths of motion.

Action for sinusoidal motion

For the mass-on-a-spring example we’re considering here, we choose the situation where the mass has its maximum speed (and hence maximum kinetic energy) at $t_1 = 0$ corresponding to the position $x(t_1) = 0$ . The endpoint, at $t_2$ , we choose to be where the mass has maximum amplitude and hence maximum potential energy. In other words, the final position is $x(t_2) = A$ , where $A$ is the amplitude of oscillation.

We can visualise this situation like so:

The interval over which we wish to calculate the action is thus a quarter oscillation of the SHO.

The equation for the motion of a simple harmonic oscillatorcan be written as

\begin{equation} x(t) = A \sin (\omega t + \phi), \end{equation}

where $x (t)$ is the mass’ position along the $x$ -axis at time $t$ , $A$ is the amplitude, $ω\omega$ is the angular velocity of the oscillation and $ϕ\phi$ is the phase offset. In our situation, we’ve chosen the boundary conditions such that the phase offset can be set to zero; hence, we now have this reduced form of the equation describing the motion:

\begin{equation} x(t) = A \sin (\omega t). \end{equation}

Differentiating this with respect to time, we obtain a relation for the velocity

\begin{equation} \dot{x}(t) = \omega A \cos (\omega t). \end{equation}

With this information in hand, we’re now able to write the action (Equation 11) explicitly in terms of $t$

\begin{equation} S = \int_{t_1}^{t_2} \biggl[\frac{1}{2} m \omega^2 A^2 \cos^2 (\omega t) - \frac{1}{2} k A^2 \sin^2 (\omega t) \biggr]\ dt. \end{equation}

This now makes it possible to perform the integral. Before we can do that, however, we need to work out what the limits of integration are.

The boundary conditions

\begin{equation} x(t_1) = 0 \quad \text{and} \end{equation}

\begin{equation} x(t_2) = A, \end{equation}

imply that the limits of integration become

\begin{equation} t_1 = 0 \quad \text{and} \end{equation}

\begin{equation} t_2 = \frac{\pi}{2 \omega}. \end{equation}

The squared sine and cosine terms in (15) make performing the integral difficult. We can simplify things by using the trigonometric double-angle identities:

\begin{equation} \cos^2 \theta = \frac{1 + \cos (2 \theta)}{2}\quad \text{and} \end{equation}

\begin{equation} \sin^2 \theta = \frac{1 - \cos (2 \theta)}{2}. \end{equation}

Therefore, we can replace the squared sine and cosine expressions with simpler expressions only in terms of cosine. The integral calculating the action thus becomes:

\begin{equation} S = \int_{t_1}^{t_2} \Biggl[\frac{1}{2} m \omega^2 A^2 \frac{1 + \cos (2 \omega t)}{2} - \frac{1}{2} k A^2 \frac{1 - \cos (2 \omega t)}{2} \Biggr]\ dt. \end{equation}

Let’s now focus on only the first term in (22) and perform the integration. We begin with this expression:

\begin{equation} S_1 = \int_{t_1}^{t_2} \Biggl[\frac{1}{2} m \omega^2 A^2 \frac{1 + \cos (2 \omega t)}{2} \Biggr]\ dt, \end{equation}

where $S_1$ is the integral of the first term in (22).

We collect constants outside the integral to get:

\begin{equation} S_1 = \frac{1}{4} m \omega^2 A^2 \int_{t_1}^{t_2} 1 + \cos (2 \omega t)\ dt. \end{equation}

Performing the integration, we get:

\begin{equation} S_1 = \frac{1}{4} m \omega^2 A^2 \biggl[t + \frac{1}{2 \omega} \sin (2 \omega t) \biggr]_{t_1}^{t_2} \end{equation}

\begin{equation} = \frac{1}{4} m \omega^2 A^2 \biggl[t + \frac{1}{2 \omega} \sin (2 \omega t) \biggr]_{0}^{\frac{\pi}{2 \omega}}. \end{equation}

Evaluating this at the integration limits, we have:

\begin{equation} S_1 = \frac{1}{4} m \omega^2 A^2 \biggl[t + \frac{1}{2 \omega} \sin (2 \omega t) \biggr]_{0}^{\frac{\pi}{2 \omega}} \end{equation}

\begin{equation} = \frac{1}{4} m \omega^2 A^2 \biggl[\frac{\pi}{2 \omega} + \frac{1}{2 \omega} \sin \biggl( \frac{2 \omega \pi}{2 \omega} \biggr) - 0 - 0 \biggr] \end{equation}

\begin{equation} = \frac{1}{4} m \omega^2 A^2 \biggl[\frac{\pi}{2 \omega} + \frac{1}{2 \omega} \sin (\pi) \biggr] \end{equation}

\begin{equation} = \frac{1}{8} \pi m \omega A^2. \end{equation}

Focusing now on only the second term in (22), we have this expression:

\begin{equation} S_2 = \int_{t_1}^{t_2} \Biggl[\frac{1}{2} k A^2 \frac{1 - \cos (2 \omega t)}{2} \Biggr]\ dt, \end{equation}

where $S_2$ is the integral of the second term in (22).

Again, collecting constants in front of the integral, we have:

\begin{equation} S_2 = \frac{1}{4} k A^2 \int_{t_1}^{t_2} 1 - \cos (2 \omega t)\ dt. \end{equation}

Integrating over $t$ , we obtain this expression

\begin{equation} S_2 = \frac{1}{4} k A^2 \biggl[t - \frac{1}{2 \omega} \sin (2 \omega t) \biggr]_{t_1}^{t_2} \end{equation}

\begin{equation} = \frac{1}{4} k A^2 \biggl[t - \frac{1}{2 \omega} \sin (2 \omega t) \biggr]_{0}^{\frac{\pi}{2 \omega}}. \end{equation}

Evaluating the integration limits, we get:

\begin{equation} S_2 = \frac{1}{4} k A^2 \biggl[t - \frac{1}{2 \omega} \sin (2 \omega t) \biggr]_{0}^{\frac{\pi}{2 \omega}} \end{equation}

\begin{equation} = \frac{1}{4} k A^2 \biggl[\frac{\pi}{2 \omega} - \frac{1}{2 \omega} \sin \biggl( \frac{2 \omega \pi}{2 \omega} \biggr) - 0 - 0 \biggr] \end{equation}

\begin{equation} = \frac{1}{4} k A^2 \biggl[\frac{\pi}{2 \omega} - \frac{1}{2 \omega} \sin (\pi) \biggr] \end{equation}

\begin{equation} = \frac{1}{8} \frac{\pi k A^2}{\omega}. \end{equation}

Combining Equations (30) and (38), we can write the equation for the action specified in (22) as:

\begin{equation} S = \frac{1}{8} \pi m \omega A^2 - \frac{1}{8} \frac{\pi k A^2}{\omega}, \end{equation}

which simplifies to

\begin{equation} S = \frac{1}{8} \pi A^2 \biggl[m \omega - \frac{k}{\omega} \biggr]. \end{equation}

From the solution of the differential equation describing simple harmonic oscillation, we know that $ω\omega$ can be written in terms of the spring constant $k$ and the object’s mass $m$ , i.e.:

\begin{equation} \omega = \sqrt{\frac{k}{m}}. \end{equation}

Substituting this value into (40), we get:

\begin{equation} S = \frac{1}{8} \pi A^2 \biggl[m \omega - \frac{k}{\omega} \biggr] \end{equation}

\begin{equation} = \frac{1}{8} \pi A^2 \biggl[m \sqrt{\frac{k}{m}} - k \sqrt{\frac{m}{k}} \biggr] \end{equation}

\begin{equation} = \frac{1}{8} \pi A^2 \biggl[\sqrt{\frac{m^2 k}{m}} - \sqrt{\frac{m k^2}{k}} \biggr] \end{equation}

\begin{equation} = \frac{1}{8} \pi A^2 \biggl[\sqrt{m k} - \sqrt{m k} \biggr] \end{equation}

\begin{equation} = 0. \end{equation}

Thus, we find that the action for a simple harmonic oscillator for the boundary conditions

\begin{equation} x(0) = 0 \quad \text{and} \end{equation}

\begin{equation} x(\pi / 2 \omega) = A \end{equation}

is equal to zero. This result implies that the action is minimised over the path described by the equations of motion of a simple harmonic oscillator.

Comparison with the general result

The general solution for the (classical) action of a simple harmonic oscillator is given by (Quantum Mechanics and Path Integrals; Richard Feynman and Albert R. Hibbs):

\begin{equation} S = \frac{m \omega}{2 \sin(\omega T)} \biggl[(x_b^2 + x_a^2) \cos(\omega T) - 2 x_b x_a \biggr], \end{equation}

where $x_a$ and $x_b$ are the start and end points of the path, with time at $x_a$ being at $t = 0$ and the time at $x_b$ being $t = T$ .³

Inserting the boundary conditions in our calculation from above, we have the correspondence:

\begin{equation} x(0) = x_a = 0 \quad \text{and} \end{equation}

\begin{equation} x(T) = x_b = x(\pi / 2 \omega) = A. \end{equation}

Therefore, we can write the general solution (49) as

\begin{equation} S = \frac{m \omega}{2 \sin\bigl( \omega \frac{\pi}{2 \omega} \bigr)} \biggl[(A^2 + 0) \cos \biggl( \omega \frac{\pi}{2 \omega} \biggr) - 2 A \cdot 0 \biggr] \end{equation}

\begin{equation} = \frac{m \omega}{2 \sin\bigl( \frac{\pi}{2} \bigr)} \biggl[A^2 \cos \biggl( \frac{\pi}{2} \biggr) \biggr] \end{equation}

\begin{equation} = \frac{m \omega}{2 \cdot 1} \biggl[A^2 \cdot 0 \biggr] \end{equation}

\begin{equation} = \frac{m \omega}{2} \cdot 0 \end{equation}

\begin{equation} = 0. \end{equation}

We can see that–for the boundary conditions considered here–the general solution for the action reduces to the same result as that given by our simplified SHO solution in (13).

We’ve now convinced ourselves that the action for the simple harmonic oscillator seems to be minimised over the path given by the standard solution. What happens if we choose another path? What effect does that have on the action? Let’s look into this now.

A different path

Consider a situation other than simple sinusoidal motion. For instance, a linear ramp from zero amplitude up to maximum amplitude, progressing over the same time interval. The mass now moves with constant velocity from the beginning ( $x (t = 0) = 0$ ) to the end ( $\pi/2 \omega) = A$ ). What is the action in this situation? If the action for the simple harmonic oscillator solution is $0$ (and hence minimal), then we expect this alternate path to give a larger action.

Let’s calculate everything and see. The main point about considering such a situation is that the system now takes a different path between the two boundary conditions. Whether or not the motion itself is 100% physical is irrelevant for the calculation. The goal in performing this calculation is to support our intuition about the action being minimised in the optimum situation.

As before, we write the Lagrangian like so:

\begin{equation} L = T - V \end{equation}

with

\begin{equation} T = \frac{1}{2} m v^2 = \frac{1}{2} m \dot{x}^2 \quad \text{and} \end{equation}

\begin{equation} V = \frac{1}{2} k x^2. \end{equation}

Because the mass moves between the start and end points with constant velocity–and we want the equations of motion to match these boundary conditions–we write the position and velocity of the mass over time as:

\begin{equation} x(t) = \frac{A}{t_2} t = \frac{2 A \omega}{\pi} t \quad \text{and} \end{equation}

\begin{equation} \dot{x}(t) = \frac{A}{t_2} = \frac{2 A \omega}{\pi} \quad \text{(i.e. constant velocity)}. \end{equation}

The path that the mass takes can be visualised like so:

With this information, we can write the equation for the action

\begin{equation} S = \int_0^{\frac{\pi}{2 \omega}} \frac{1}{2} m \frac{4 A^2 \omega^2}{\pi^2}\ dt - \int_0^{\frac{\pi}{2 \omega}} \frac{1}{2} k \frac{4 A^2 \omega^2}{\pi^2} t^2\ dt. \end{equation}

Collecting constants in front of the integral sign simplifies the situation somewhat:

\begin{equation} S = \frac{2 A^2 \omega^2}{\pi^2} \int_0^{\frac{\pi}{2 \omega}} m - k t^2\ dt. \end{equation}

Performing the integration, we have:

\begin{equation} S = \frac{2 A^2 \omega^2}{\pi^2} \biggl[m t - \frac{1}{3} k t^3 \biggr]_0^{\frac{\pi}{2 \omega}} \end{equation}

\begin{equation} = \frac{2 A^2 \omega^2}{\pi^2} \biggl[m \frac{\pi}{2 \omega} - \frac{1}{3} k \frac{\pi^3}{8 \omega^3} - 0 - 0 \biggr] \end{equation}

\begin{equation} = \frac{A^2 \omega}{\pi} \biggl[m - \frac{1}{3} k \frac{\pi^2}{4 \omega^2} \biggr] \end{equation}

\begin{equation} = \frac{A^2 \omega}{\pi} \biggl[m - \frac{1}{12} k \frac{\pi^2}{\omega^2} \biggr]. \end{equation}

We remember that

\begin{equation} \omega = \sqrt{\frac{k}{m}}. \end{equation}

Although this equation follows from the sinusoidal result we derived above, we still need to use it in the linear ramp case due to our boundary conditions. This is in particular due to the expression for $t_2$ (Equation (19)), which is defined in terms of $ω\omega$ .

Substituting the expression for $ω\omega$ from (68) into (67), we get:

\begin{equation} S = \frac{A^2 \omega}{\pi} \biggl[m - \frac{1}{12} k \frac{\pi^2}{\omega^2} \biggr] \end{equation}

\begin{equation} = \frac{A^2}{\pi} \sqrt{\frac{k}{m}} \biggl[m - \frac{1}{12} k \pi^2 \frac{m}{k} \biggr] \end{equation}

\begin{equation} = \frac{A^2}{\pi} \sqrt{\frac{k}{m}} m \biggl[1 - \frac{\pi^2}{12} \biggr] \end{equation}

\begin{equation} = \frac{A^2}{\pi} \sqrt{k m} \biggl[1 - \frac{\pi^2}{12} \biggr]. \end{equation}

Since

\begin{equation} \biggl[1 - \frac{\pi^2}{12} \biggr] \end{equation}

is greater than zero (and hence greater than the action calculated in the sinusoidal case), we can conclude that this motion is not optimal. Thus, the simple harmonic oscillation path gives the best result because it minimises the action.

In other words, it worked! We confirmed our premise from the beginning. 🎉

If you want a deeper understanding, you have to put in the work

Going through this process really helped my understanding. For instance, when I first watched the Veritasium video about action, I thought “That’s interesting”, but nothing much more. After having done the calculation above myself, I re-watched the video and was like “Oh, that’s what they were talking about!”.

Going over the material after someone has presented it to you consolidates your knowledge of the topic. Who would’ve thought? 😉

The boundary conditions must be the same when comparing the actions of two separate paths. I.e. the start and end points in space need to be the same, and the start and end times need to be the same. ↩
In other words, the work done in taking a particle between two points in space is independent of the path taken. ↩
Note that $T$ here does not represent the kinetic energy; it represents the time at the end of the path between $x_a$ and $x_b$ . ↩

Towards an intuition for the simple harmonic oscillator solution

Paul Cochrane 🇪🇺 — Wed, 02 Jul 2025 22:00:00 +0000

The solution of the differential equation describing simple harmonic motion is often presented as-is, or “handed down from above”. Usually, there is no attempt at justifying where the mathematical solution comes from. To me, this wasn’t good enough. Here, I try to provide some intuition for the solution’s form.

Once a physicist, always a physicist

Some friends of mine and I have been doing some undergraduate-level physics again, just for fun.¹ As you do. It’s nice that we’ve now got enough distance from the pressures of studying and learning physics that we can take some time and really try to understand it. This is opposed to undergrad which was a case of trying to cope and not drown in a flood of information. In this vein, we’ve made lots of errors and hence progress has been slow. But who cares? We’ve gained more depth of understanding in what we’ve been looking at than what any of us managed to achieve at university.

Our current topic of focus has been the concept of action. To understand this better, we’ve been using the simple harmonic oscillator as an illustrative example.

Now, one way of writing the general solution of the simple harmonic oscillator is like so:

\begin{equation} x(t) = A \sin (\omega t + \phi) \end{equation}

where $x$ is the displacement from the origin, $A$ is the amplitude, $ω\omega$ is the angular frequency of the oscillation, $t$ is time and $ϕ\phi$ a phase shift.

Warning: rant ahead!

This is where I got annoyed. Everyone knows that this is the solution (or at least one form thereof). When I was at university, we were provided this answer as-is without anyone explaining why. It’s like an adult answering a child’s question of “Why?” with “It just is!”. Of course, usually, the solution was presented in a much nicer way at university. I.e. something along the lines of “Let’s assume the answer and then show that it works”. It’s like beginning a journey at the end. I think this is the bit that frustrates me: how do you know that you can assume that solution? How did you get the idea to try out such a solution in the first place (assuming that one starts from only the maths and ignores the physical situation)? What intuition do you have which leads to such a conclusion? If the answer to that question is: “You’ll see with experience”, then that’s silly. After all, that’s one reason to go to university: to get the benefit of the experience of those who have gone before us. That’s how we get to stand on the shoulders of giants.

Also, almost all books on differential equations² simply state the answer as a given. Sort of like knowledge that has been handed down by an appropriate celestial being that is just known and doesn’t need to be questioned.

As it turns out, this is knowledge handed down by a kind of celestial being. It was Euler’s intuition. But more on that later.

A popular comic on lecturer’s office doors, when I was at university, showed two mathematicians staring at a blackboard, with one pointing at part of the calculation, which reads “Then a miracle occurs” and making the comment “I think you need to be more explicit here in step two”. Here is said cartoon:³

That’s what it felt like when presented with “let’s assume we know the solution” demonstrations. It seemed miraculous and I came away lacking any understanding as to why, and that irked me.

My issue here is not with the physical intuition–that’s obvious to anyone who’s tugged on a spring and seen it bounce back and forth. My issue is rather with the mathematics. Where is the mathematical intuition leading to this solution when only considering the equations of motion? That’s what was unclear to me.

I guess that’s enough of me bemoaning this particular situation. It’s time I move on and find my own perspective. Let’s see what I came up with.

Setting up the scene

I’m stubborn.⁴ Along with that and the frustration of not having had much of any intuition imparted to me, I set out to find out more myself. What follows is my attempt to form an intuition for why we get a sinusoid for the solution of the differential equation describing a simple harmonic oscillator.

Let’s understand the physical problem we’re trying to solve. Imagine an object of mass $m$ attached to a spring. One end of the spring is connected to an immovable wall and the mass slides along a frictionless surface. The motion of the mass is confined to the $x$ -direction and the stiffness of the spring is characterised by its spring constant, $k$ .⁵

If we move the mass away from the equilibrium position, the restoring force due to the spring is given by Hooke’s law:

\begin{equation} F_H = -k x. \end{equation}

This force accelerates the mass towards its equilibrium position, which we are defining to be the position of zero amplitude. Thus, this force is equal to the force described by Newton’s second law, i.e.:

\begin{equation} F_N = m a. \end{equation}

Setting these two quantities equal to one another, we have

\begin{equation} F_N = F_H \end{equation}

\begin{equation} \Rightarrow ma = -kx \end{equation}

\begin{equation} \Rightarrow m \ddot{x} = -k x \end{equation}

where we only consider motion in the $x$ -direction.

Ensuring the perfect conditions

We require a solution which satisfies this equation as well as the boundary conditions:

\begin{equation} x(t_1) = 0 \end{equation}

\begin{equation} x(t_2) = A. \end{equation}

Considering (6), we can see that the second derivative of the solution with respect to time needs to be equal to the negative of the solution itself (up to a factor). We know that a sine function has this property:⁶

\begin{equation} x(t) = \sin(t) \end{equation}

\begin{equation} \ddot{x}(t) = -\sin(t). \end{equation}

This is our first hint at an intuition for why the solution is a sinusoid: we need the solution and its second derivative to balance each other out. It’s also why the second derivative must be the negative (up to a factor) of the proposed solution. It just so happens that a sine function has this desired property.

With this idea in mind, we ~~pull the following solution out of the hat~~ propose the following solution:⁷

\begin{equation} x(t) = A \sin(\omega t + \phi) \end{equation}

where the parameters are as described earlier.

We can see that this equation satisfies the boundary conditions if we set $ϕ=0\phi = 0$ :

\begin{equation} x(t_1 = 0) = A \sin(\omega \cdot 0) = 0 \end{equation}

\begin{equation} x \biggl( t_2 = \frac{\pi}{2 \omega} \biggr) = A \sin \biggl( \omega \cdot \frac{\pi}{2 \omega} \biggr) = A \sin \biggl( \frac{\pi}{2} \biggr) = A \end{equation}

where we’ve defined $t_2$ in terms of $ω\omega$ to be the end of a quarter oscillation of the sine function.

With this solution, we have the following equations for the position, velocity, and acceleration:

\begin{equation} x(t) = A \sin (\omega t) \end{equation}

\begin{equation} \dot{x}(t) = \omega A \cos (\omega t) \end{equation}

\begin{equation} \ddot{x}(t) = -\omega^2 A \sin (\omega t) \end{equation}

Inserting (14) and (16) into the force relation (6), we have:

\begin{equation} -m \omega^2 A \sin (\omega t) = -k A \sin (\omega t) \end{equation}

\begin{equation} \Rightarrow m \omega^2 = k \end{equation}

\begin{equation} \Rightarrow \omega^2 = \frac{k}{m} \end{equation}

\begin{equation} \Rightarrow \omega = \sqrt{\frac{k}{m}} \end{equation}

which is the familiar result for the angular velocity of a simple harmonic oscillator in terms of the spring constant, $k$ , and the object’s mass, $m$ .

That’s all well and good. We’ve trodden a very well-worn path: we showed that the answer we assumed to be true worked. But where did we get that from? What led us to choosing that answer?

Digging deeper

As mentioned above, I wasn’t satisfied with the usual “let’s assume we know the answer and then show that it works” method of solving the differential equation in (6). Thus, I went on a bit of a hunt for intuitive ways of understanding how this becomes a solution. I found that almost no-one provides an intuition for why

\begin{equation} x(t) = A \sin (\omega t + \phi) \end{equation}

is a general solution to (6): it is simply stated. Perhaps this has something to do with there being no general process for finding solutions to differential equations. There are some known ways to tackle particular forms but, in general, solving things seems to be a mix of experience and squinting at the problem from the right angle.

The closest I’ve gotten to a good explanation is from A Modern Introduction to Differential Equations (3rd. edition) by Henry J. Richard. In Chapter 4, he discusses second- and higher-order equations. Page 144 contains the comment that

If we consider a homogeneous first-order linear equation with constant coefficients, $a y^{'} + b y = 0$ , where $\neq 0$ , we know that the general solution is $e^{-\frac{b}{a} t}$ . In 1739, aware of this solution, Euler proposed solving an $n$ th-order homogeneous linear equation with constant coefficients by looking for solutions of the form $e^{\lambda t}$ , where $λ\lambda$ is a constant to be determined.

In other words, Euler had some insight back in the 18th century and we’ve taken that for granted ever since. Given that the insight comes from Euler, it’s not a bad idea to use it.⁸ Still, it’d be nice if this way of looking at things were better known.

Searching for understanding

As far as I can tell, the main insight is that the exponential function differentiates to itself multiplied by a constant. This reflects the observation we made earlier that the second derivative of a sine is a negative sine. In other words, if we begin with the exponential function as a guess for the solution:

\begin{equation} x = e^{\lambda t}, \end{equation}

then we get the first and second derivatives like so:

\begin{equation} \dot{x} = \lambda e^{\lambda t} \end{equation}

\begin{equation} \ddot{x} = \lambda^2 e^{\lambda t}. \end{equation}

I think this is what we’re after. The search for a solution involves looking for particular properties of functions and seeing if they fit the problem. It’s like doing a jigsaw puzzle where you’re looking for that single piece out of thousands that fits the current situation just so.

With this trial solution and its derivatives, we can write the homogeneous linear second-order differential equation

\begin{equation} a \ddot{x} + b \dot{x} + c x = 0 \end{equation}

in terms of these derivatives:

\begin{equation} \Rightarrow a \lambda^2 e^{\lambda t} + b \lambda e^{\lambda t} + c e^{\lambda t} = 0. \end{equation}

Given that $eλte^{\lambda t}$ doesn’t equal zero, we can cancel out the terms in the exponential to give

\begin{equation} a \lambda^2 + b \lambda + c = 0. \end{equation}

This is known as the characteristic equation of the differential equation. Since this is a quadratic equation, we know that it has two solutions, namely

\begin{equation} \lambda = \frac{-b \pm \sqrt{b^2 - 4 a c}}{2a}. \end{equation}

There are three possibilities for what these roots could be:

Either they are both different real numbers,
Or they are equal real numbers (i.e. the roots are degenerate; there’s also a tricky way to work out the general solution in this case, which (again) Euler worked out yonks ago),
Or they are complex numbers.

It turns out that for the simple harmonic oscillator case, they’re complex numbers. Let’s see that now.

A complex situation

Inserting the trial solution $e^{\lambda t}$ and its second derivative into (6) we get

\begin{equation} m \lambda^2 e^{\lambda t} + k e^{\lambda t} = 0 \end{equation}

\begin{equation} \Rightarrow \quad \lambda^2 e^{\lambda t} + \frac{k}{m} e^{\lambda t} = 0 \end{equation}

\begin{equation} \Rightarrow \quad \lambda^2 + \frac{k}{m} = 0 \end{equation}

\begin{equation} \Rightarrow \quad \lambda = \sqrt{-\frac{k}{m}} \end{equation}

\begin{equation} \Rightarrow \quad \lambda = i \sqrt{\frac{k}{m}}. \end{equation}

This might look a bit dodgy because of the $i$ , but if we plug $λ\lambda$ into our trial solution, we get

\begin{equation} x = e^{i \sqrt{\frac{k}{m}} t} \end{equation}

Now, if we define the angular frequency, $ω\omega$ , like so:

\begin{equation} \omega = \sqrt{\frac{k}{m}} \end{equation}

then we get

\begin{equation} x = e^{i \omega t} \end{equation}

which (again, thanks to Euler, crikey was he a busy boy!) one can write as

\begin{equation} x = \cos (\omega t) + i \sin (\omega t). \end{equation}

This, because of the cosine and sine, describes sinusoidal behaviour. And I guess that’s it. Sinusoidal motion has sort of popped out of the insight that the exponential function keeps differentiating to effectively itself and that was a handy property to have when looking for a solution. It’s almost an anti-climax.

Alternatively, we could assume that the proposed solution uses $\lambda$ in the exponent instead of only $λ\lambda$ , then the result is a bit clearer.⁹ The solution and its derivatives are:

\begin{equation} x = e^{i \lambda t} \end{equation}

\begin{equation} \dot{x} = i \lambda e^{i \lambda t} \end{equation}

\begin{equation} \ddot{x} = - \lambda^2 e^{i \lambda t}. \end{equation}

Where we see more clearly this nice property that the second derivative is the negative (up to a factor) of the solution itself.

Using the solution and its second derivative, we get

\begin{equation} m \lambda^2 e^{i \lambda t} + k e^{i \lambda t} = 0 \end{equation}

\begin{equation} \Rightarrow \quad -\lambda^2 e^{i \lambda t} + \frac{k}{m} e^{i \lambda t} = 0 \end{equation}

\begin{equation} \Rightarrow \quad -\lambda^2 + \frac{k}{m} = 0 \end{equation}

\begin{equation} \Rightarrow \quad \lambda = \sqrt{\frac{k}{m}}. \end{equation}

This seems a bit more direct, but yeah, I’m using knowledge of the solution to make it seem more direct, and this was my criticism from earlier. Oh well.

Chasing my tail?

So what can I conclude? Well, it looks like the oscillatory nature simply comes out in the wash. This seems to be a consequence of the insight that differentiating an exponential function gives you back the exponential function (modulo some constant), which makes your life easier when looking for a solution. Also, because we have this form for the differential equation

\begin{equation} \ddot{x} + K x = 0 \end{equation}

where $\geq 0$ ,¹⁰ then the solution of the characteristic equation has complex roots which in turn gives a differential equation solution with sinusoidal oscillations. Quod erat demonstrandum. Erm, hopefully.

I feel like I’m going around in circles, but maybe that’s just because everything is internally consistent, so if you start at one point you naturally end up at the other.

I think I’m at the end of my journey. Perhaps the answer is so banal that no-one can be bothered explaining it. Maybe Euler’s insight got lost in the hundredth or thousandth retelling? Dunno. Either way, I’m more satisfied with my understanding of where the solution comes from now, even if I do feel like I’m chasing my tail!

As one of my high-school teachers would have put it: “phun”. Ironically, it wasn’t one of my high-school physics teachers who said that. ↩
And there are lots of books about differential equations! Wow! I mean, how many perspectives on that topic do you need? So yeah, I searched through quite a few while looking into this issue. ↩
The comic was by cartoonist Sidney Harris. ↩
In a good way, honest! Once I get my teeth into a problem, I tend not to give up until I’ve worked out the root cause and have a solution. Maybe you’ve got some bug you can’t crack? If so, give me a yell! I’m available for Perl and Python development and maintenance work. Send an email to paul@peateasea.de. ↩
Given the number of simplifying assumptions we’re making here (frictionless surface, immovable wall, the spring stiffness described by a constant) it’s amazing that this setup approximates so many physical situations so well. ↩
This property we have to know from experience, unfortunately. Such experience comes from having done many exercises at high school involving derivatives of sines and cosines. ↩
I had a funny maths lecturer at university who would often use the phrase “And it is easy to show that…” followed by the comment (said in a conspiratorial fashion) “We know it’s not easy to show, but we’ll continue anyway”. Whereupon he’d do a little chuckle to himself. Given how dry the material was, this was amusing. He was a nice bloke, too; one of my favourite lecturers. ↩
Yes, that was an understatement. ↩
Although I realise I’m myself running into the problem of “given I know the solution, let’s just use it and show that it works”. I guess it’s an easy trap to fall into. ↩
Note that if $K < 0$ then the solution is no longer obviously sinusoidal: it turns out to be a linear combination of hyperbolic functions. TIL. ↩

Of sines, cosines, and phase shifts

Paul Cochrane 🇪🇺 — Wed, 25 Jun 2025 22:00:00 +0000

An alternative form for the solution of a simple harmonic oscillator caused some short-lived confusion and consternation. Resolving the conflict turned out to be fairly straightforward.

While looking at some calculations recently, I stumbled across this as the general solution of a simple harmonic oscillator:

\begin{equation} x(t) = A \sin (\omega t) + B \cos (\omega t). \end{equation}

I was a bit confused because I’d only ever seen the solution presented as¹

\begin{equation} x(t) = A \sin (\omega t + \phi), \end{equation}

where $x$ is the displacement from the origin, $A$ is the amplitude, $ω\omega$ is the angular velocity, $t$ is time, and $ϕ\phi$ a phase shift.

I remember thinking “Hä?² Is that right?” It turns out it is, and that the two forms are equivalent, but it wasn’t directly apparent. I certainly wasn’t familiar with the first form.

I’m guessing some people will probably look at those equations and my comment and think “duuhhhh!”, or “no shit, Sherlock!”. Well, it wasn’t obvious to me, nor was it obvious to the two other physicists with PhDs who were in the room at the time. Anyway, I thought I’d write this up so that it might help someone else. I’m hoping it will also help solidify my own knowledge.

Let’s see how the two forms are equivalent.

Fix up the notation

Usually, we’re used to seeing the parameter $A$ as “amplitude”. This certainly makes sense in the form containing the explicit phase shift (Equation (2)). However, that’s not the case in the first form (Equation (1)), which doesn’t include an explicit phase shift.

To be able to equate these two forms, we need to change the notation, so that we’re not using one symbol for two concepts. Hence, let’s replace $A$ in Equation (2) with $C$ to get

\begin{equation} x(t) = C \sin (\omega t + \phi). \end{equation}

We’re now in a position to see if we can write $C$ and $ϕ\phi$ in terms of $A$ and $B$ from Equation (1).

Trig identities to the rescue

The first thing to identify is that one can write the sine of a sum of angles (as we have in Equation (3)) as the sum of products of sine and cosine of the angles. In other words, we use the identity

\begin{equation} \sin (\theta + \psi) = \sin \theta \cos \psi + \cos \theta \sin \psi \end{equation}

to reformulate Equation (3) as

\begin{equation} x(t) = C \sin (\omega t) \cos \phi + C \cos (\omega t) \sin \phi. \end{equation}

Collecting terms in $ωt\omega t$ to the right, we have

\begin{equation} x(t) = C \cos \phi \sin (\omega t) + C \sin \phi \cos (\omega t). {eq:collected-terms} \end{equation}

Since $C$ and $ϕ\phi$ are not dependent on $t$ , the factors

\begin{equation} C \cos \phi \quad \text{and} \quad C \sin \phi \end{equation}

are constants, hence we can see that Equation (\ref{eq:collected-terms}) has the same form as Equation (1), where

\begin{equation} A = C \cos \phi \quad \text{and} \quad B = C \sin \phi. \end{equation}

In other words, the two forms of the simple harmonic oscillator solution are equivalent.

Writing one in terms of the other

The question now becomes: how can we write $C$ and $ϕ\phi$ in terms of $A$ and $B$ ? If we sum the squares of $A$ and $B$ , we have

\begin{equation} A^2 + B^2 = C^2 \cos^2 \phi + C^2 \sin^2 \phi \end{equation}

\begin{equation} = C^2 (\cos^2 \phi + \sin^2 \phi) \end{equation}

\begin{equation} = C^2 \end{equation}

\begin{equation} \Rightarrow C = \sqrt{A^2 + B^2}. \end{equation}

How, then, is $ϕ\phi$ related to $A$ and $B$ ? Again, we use the relations

\begin{equation} A = C \cos \phi \quad \text{and} \quad B = C \sin \phi \end{equation}

but this time we divide $B$ by $A$ :

\begin{equation} \frac{B}{A} = \frac{C \sin \phi}{C \cos \phi} \end{equation}

\begin{equation} = \frac{\sin \phi}{\cos \phi} \end{equation}

\begin{equation} = \tan \phi \end{equation}

\begin{equation} \Rightarrow \phi = \tan^{-1} \biggl(\frac{B}{A}\biggr). \end{equation}

Rewriting Equation (3) in terms of $A$ and $B$ , we have the equivalence of the two forms of the general solution of the simple harmonic oscillator:

\begin{equation} A \sin(\omega t) + B \cos(\omega t) = \sqrt{A^2 + B^2} \sin \Biggl( \omega t + \tan^{-1} \biggl(\frac{B}{A}\biggr) \Biggr). \end{equation}

Summing up

Thus, in terms of the $A$ and $B$ parameters from Equation (1), we find that the amplitude in Equation (3) is

\begin{equation} C = \sqrt{A^2 + B^2} \end{equation}

and the phase shift is given by

\begin{equation} \phi = \tan^{-1} \biggl(\frac{B}{A}\biggr). \end{equation}

So that’s it! The two forms are equivalent. Hopefully, the next time I see the solution written as a sum of sines and cosines, I won’t be as surprised.³

Note that the $A$ appearing in each of the above equations are not the same. ↩
“Hä” is a very compact expression in German evoking a sense of deep bewilderment. It can be roughly translated into English as “I am exceedingly confused by that which you have just told me”. So much for German always using really long words… ↩
I later found a similar formulation on the Wikipedia page for simple harmonic motion. ↩

Analysing FIT data with Perl: interactive data analysis

Paul Cochrane 🇪🇺 — Tue, 17 Jun 2025 22:00:00 +0000

Printing statistics to the terminal or plotting data extracted from FIT files is all well and good. The problem is that the feedback loops are too long. Sometimes questions are better answered by playing with the data directly. Enter the Perl Data Language.

Interactive data analysis

For more fine-grained analysis of our FIT file data, it’d be great to be able to investigate it interactively. Other languages such as Ruby, Raku and Python have a built-in REPL.¹ Yet Perl doesn’t.² But help is at hand! PDL (the Perl Data Language) is designed to be used interactively and thus has a REPL we can use to manipulate and investigate our activity data.³

Getting set up

Before we can use PDL, we’ll have to install it:

$ cpanm PDL

After it has finished installing (this can take a while), you’ll be able to start the perlDL shell with the pdl command:

perlDL shell v1.357
 PDL comes with ABSOLUTELY NO WARRANTY. For details, see the file
 'COPYING' in the PDL distribution. This is free software and you
 are welcome to redistribute it under certain conditions, see
 the same file for details.
ReadLines, NiceSlice, MultiLines enabled
Reading PDL/default.perldlrc...
Found docs database /home/vagrant/perl5/perlbrew/perls/perl-5.38.3/lib/site_perl/5.38.3/x86_64-linux/PDL/pdldoc.db
Type 'help' for online help
Type 'demo' for online demos
Loaded PDL v2.100 (supports bad values)

Note: AutoLoader not enabled ('use PDL::AutoLoader' recommended)

pdl>

To exit the pdl shell, enter Ctrl-D at the prompt and you’ll be returned to your terminal.

Cleaning up to continue

To manipulate the data in the pdl shell, we want to be able to call individual routines from the geo-fit-plot-data.pl script. This way we can use the arrays that some of the routines return to initialise PDL data objects.

It’s easier to manipulate the data if we get ourselves a bit more organised first.⁴ In other words, we need to extract the routines into a module, which will make calling the code we created earlier from within pdl much easier.

Before we create a module, we need to do some refactoring. One thing that’s been bothering me is the way the plot_activity_data() subroutine also parses and manipulates date/time data. This routine should be focused on plotting data, not on massaging its requirements into the correct form. Munging the date/time data is something that should happen in its own routine. This way we encapsulate the concepts and abstract away the details. Another way of saying this is that the plotting routine shouldn’t “know” how to manipulate date/time information to do its job.

To this end, I’ve moved the time extraction code into a routine called get_time_data():

sub get_time_data {
    my @activity_data = @_;

    # get the epoch time for the first point in the time data
    my @timestamps = map { $_->{'timestamp'} } @activity_data;
    my $first_epoch_time = $date_parser->parse_datetime($timestamps[0])->epoch;

    # convert timestamp data to elapsed minutes from start of activity
    my @times = map {
        my $dt = $date_parser->parse_datetime($_);
        my $epoch_time = $dt->epoch;
        my $elapsed_time = ($epoch_time - $first_epoch_time)/60;
        $elapsed_time;
    } @timestamps;

    return @times;
}

The main change here in comparison to the previous version of the code is that we pass the activity data as an argument to get_time_data(), returning the @times array to the calling code.

The code creating the date string used in the plot title now also resides in its own function:

sub get_date {
    my @activity_data = @_;

    # determine date from timestamp data
    my @timestamps = map { $_->{'timestamp'} } @activity_data;
    my $dt = $date_parser->parse_datetime($timestamps[0]);
    my $date = $dt->strftime("%Y-%m-%d");

    return $date;
}

Where again, we’re passing the @activity_data array to the function. It then returns the $date string that we use in the plot title.

Both of these routines use the $date_parser object, which I’ve extracted into a constant in the main script scope:

our $date_parser = DateTime::Format::Strptime->new(
    pattern => "%Y-%m-%dT%H:%M:%SZ",
    time_zone => 'UTC',
);

I’ve also made it our so that both subroutines needing this information have access to it.

Making a mini-module

It’s time to make our module. I’m not going to create the full Perl module infrastructure here, as it’s not necessary for our current goal. I want to import a module called Geo::FIT::Utils and then access the functions that it provides.⁵ Thus–in an appropriate project directory–we need to create a file called lib/Geo/FIT/Utils.pm as well as its associated path:

$ mkdir -p lib/Geo/FIT
$ touch lib/Geo/FIT/Utils.pm

Opening the file in an editor and entering this stub module code:

package Geo::FIT::Utils;

use Exporter 5.57 'import';

our @EXPORT_OK = qw(
    extract_activity_data
    show_activity_statistics
    plot_activity_data
    get_time_data
    num_parts
);

1;

we now have the scaffolding of a module that (at least, theoretically) exports the functionality we need.

Line 1 specifies the name of the module. Note that the module’s name must match its path on the filesystem, hence why we created the file Geo/FIT/Utils.pm.

We import the Exporter module (line 3) so that we can specify the functions to export. This is the @EXPORT_OK array’s purpose (lines 6-12).

Finally, we end the file on line 14 with the code 1;. This line is necessary so that importing the package (which in this case is also a module) returns a true value. The value 1 is synonymous with Boolean true in Perl, hence why it’s best practice to end module files with 1;.

Copying all the code except the main() routine from geo-fit-plot-data.pl into Utils.pm, we end up with this:

package Geo::FIT::Utils;

use strict;
use warnings;

use Exporter 5.57 'import';
use Geo::FIT;
use Scalar::Util qw(reftype);
use List::Util qw(max sum);
use Chart::Gnuplot;
use DateTime::Format::Strptime;

our $date_parser = DateTime::Format::Strptime->new(
    pattern => "%Y-%m-%dT%H:%M:%SZ",
    time_zone => 'UTC',
);

sub extract_activity_data {
    my $fit = Geo::FIT->new();
    $fit->file( "2025-05-08-07-58-33.fit" );
    $fit->open or die $fit->error;

    my $record_callback = sub {
        my ($self, $descriptor, $values) = @_;
        my @all_field_names = $self->fields_list($descriptor);

        my %event_data;
        for my $field_name (@all_field_names) {
            my $field_value = $self->field_value($field_name, $descriptor, $values);
            if ($field_value =~ /[a-zA-Z]/) {
                $event_data{$field_name} = $field_value;
            }
        }

        return \%event_data;
    };

    $fit->data_message_callback_by_name('record', $record_callback ) or die $fit->error;

    my @header_things = $fit->fetch_header;

    my $event_data;
    my @activity_data;
    do {
        $event_data = $fit->fetch;
        my $reftype = reftype $event_data;
        if (defined $reftype && $reftype eq 'HASH' && defined %$event_data{'timestamp'}) {
            push @activity_data, $event_data;
        }
    } while ( $event_data );

    $fit->close;

    return @activity_data;
}

# extract and return the numerical parts of an array of FIT data values
sub num_parts {
    my $field_name = shift;
    my @activity_data = @_;

    return map { (split ' ', $_->{$field_name})[0] } @activity_data;
}

# return the average of an array of numbers
sub avg {
    my @array = @_;

    return (sum @array) / (scalar @array);
}

sub show_activity_statistics {
    my @activity_data = @_;

    print "Found ", scalar @activity_data, " entries in FIT file\n";
    my $available_fields = join ", ", sort keys %{$activity_data[0]};
    print "Available fields: $available_fields\n";

    my $total_distance_m = (split ' ', ${$activity_data[-1]}{'distance'})[0];
    my $total_distance = $total_distance_m/1000;
    print "Total distance: $total_distance km\n";

    my @speeds = num_parts('speed', @activity_data);
    my $maximum_speed = max @speeds;
    my $maximum_speed_km = $maximum_speed*3.6;
    print "Maximum speed: $maximum_speed m/s = $maximum_speed_km km/h\n";

    my $average_speed = avg(@speeds);
    my $average_speed_km = sprintf("%0.2f", $average_speed*3.6);
    $average_speed = sprintf("%0.2f", $average_speed);
    print "Average speed: $average_speed m/s = $average_speed_km km/h\n";

    my @powers = num_parts('power', @activity_data);
    my $maximum_power = max @powers;
    print "Maximum power: $maximum_power W\n";

    my $average_power = avg(@powers);
    $average_power = sprintf("%0.2f", $average_power);
    print "Average power: $average_power W\n";

    my @heart_rates = num_parts('heart_rate', @activity_data);
    my $maximum_heart_rate = max @heart_rates;
    print "Maximum heart rate: $maximum_heart_rate bpm\n";

    my $average_heart_rate = avg(@heart_rates);
    $average_heart_rate = sprintf("%0.2f", $average_heart_rate);
    print "Average heart rate: $average_heart_rate bpm\n";
}

sub plot_activity_data {
    my @activity_data = @_;

    # extract data to plot from full activity data
    my @times = get_time_data(@activity_data);
    my @heart_rates = num_parts('heart_rate', @activity_data);
    my @powers = num_parts('power', @activity_data);

    # plot data
    my $date = get_date(@activity_data);
    my $chart = Chart::Gnuplot->new(
        output => "watopia-figure-8-heart-rate-and-power.png",
        title => "Figure 8 in Watopia on $date: heart rate and power over time",
        xlabel => "Elapsed time (min)",
        ylabel => "Heart rate (bpm)",
        terminal => "png size 1024, 768",
        xtics => {
            incr => 5,
        },
        ytics => {
            mirror => "off",
        },
        y2label => 'Power (W)',
        y2range => [0, 1100],
        y2tics => {
            incr => 100,
        },
    );

    my $heart_rate_ds = Chart::Gnuplot::DataSet->new(
        xdata => \@times,
        ydata => \@heart_rates,
        style => "lines",
    );

    my $power_ds = Chart::Gnuplot::DataSet->new(
        xdata => \@times,
        ydata => \@powers,
        style => "lines",
        axes => "x1y2",
    );

    $chart->plot2d($power_ds, $heart_rate_ds);
}

sub get_time_data {
    my @activity_data = @_;

    # get the epoch time for the first point in the time data
    my @timestamps = map { $_->{'timestamp'} } @activity_data;
    my $first_epoch_time = $date_parser->parse_datetime($timestamps[0])->epoch;

    # convert timestamp data to elapsed minutes from start of activity
    my @times = map {
        my $dt = $date_parser->parse_datetime($_);
        my $epoch_time = $dt->epoch;
        my $elapsed_time = ($epoch_time - $first_epoch_time)/60;
        $elapsed_time;
    } @timestamps;

    return @times;
}

sub get_date {
    my @activity_data = @_;

    # determine date from timestamp data
    my @timestamps = map { $_->{'timestamp'} } @activity_data;
    my $dt = $date_parser->parse_datetime($timestamps[0]);
    my $date = $dt->strftime("%Y-%m-%d");

    return $date;
}

our @EXPORT_OK = qw(
    extract_activity_data
    show_activity_statistics
    plot_activity_data
    get_time_data
    num_parts
);

1;

… which is what we had before, but put into a nice package for easier use.

One upside to having put all this code into a module is that the geo-fit-plot-data.pl script is now much simpler:

use strict;
use warnings;

use Geo::FIT::Utils qw(
    extract_activity_data
    show_activity_statistics
    plot_activity_data
);

sub main {
    my @activity_data = extract_activity_data();

    show_activity_statistics(@activity_data);
    plot_activity_data(@activity_data);
}

main();

Poking and prodding

We’re now ready to investigate our power and heart rate data interactively!

Start pdl and enter use lib 'lib' at the pdl> prompt so that it can find our new module:⁶

$ pdl
perlDL shell v1.357
 PDL comes with ABSOLUTELY NO WARRANTY. For details, see the file
 'COPYING' in the PDL distribution. This is free software and you
 are welcome to redistribute it under certain conditions, see
 the same file for details.
ReadLines, NiceSlice, MultiLines enabled
Reading PDL/default.perldlrc...
Found docs database /home/vagrant/perl5/perlbrew/perls/perl-5.38.3/lib/site_perl/5.38.3/x86_64-linux/PDL/pdldoc.db
Type 'help' for online help
Type 'demo' for online demos
Loaded PDL v2.100 (supports bad values)

Note: AutoLoader not enabled ('use PDL::AutoLoader' recommended)

pdl> use lib 'lib'

Now import the functions we wish to use:

pdl> use Geo::FIT::Utils qw(extract_activity_data get_time_data num_parts)

Since we need the activity data from the FIT file to pass to the other routines, we grab it and put it into a variable:

pdl> @activity_data = extract_activity_data

We also need to load the time data:

pdl> @times = get_time_data(@activity_data)

which we can then read into a PDL array:

pdl> $time = pdl \@times

With the time data in a PDL array, we can manipulate it more easily. For instance, we can display elements of the array with the PDL print statement in combination with the splice() method. The following code shows the last five elements of the $time array:

pdl> print $time->slice("-1:-5")
[54.5333333333333 54.5166666666667 54.5 54.4833333333333 54.4666666666667]

Loading power output and heart rate data into PDL arrays works similarly:

pdl> @powers = num_parts('power', @activity_data)

pdl> $power = pdl \@powers

pdl> @heart_rates = num_parts('heart_rate', @activity_data)

pdl> $heart_rate = pdl \@heart_rates

In the previous article, we wanted to know what the maximum power was for the second sprint. Here’s the graph again for context:

Eyeballing the graph from above, we can see that the second sprint occurred between approximately 47 and 48 minutes elapsed time. We know that the arrays of time and power data all have the same length. Thus, if we find out the indices of the $time array between these times, we can use them to select the corresponding power data. To get array indices for known data values, we use the PDL which command:

pdl> $indices = which(47 < $time & $time < 48)

pdl> print $indices
[2821 2822 2823 2824 2825 2826 2827 2828 2829 2830 2831 2832 2833 2834 2835
 2836 2837 2838 2839 2840 2841 2842 2843 2844 2845 2846 2847 2848 2849 2850
 2851 2852 2853 2854 2855 2856 2857 2858 2859 2860 2861 2862 2863 2864 2865
 2866 2867 2868 2869 2870 2871 2872 2873 2874 2875 2876 2877 2878 2879]

We can check that we’ve got the correct range of time values by passing the $indices array as a slice of the $time array:

pdl> print $time($indices)
[47.0166666666667 47.0333333333333 47.05 47.0666666666667 47.0833333333333
 47.1 47.1166666666667 47.1333333333333 47.15 47.1666666666667
 47.1833333333333 47.2 47.2166666666667 47.2333333333333 47.25
 47.2666666666667 47.2833333333333 47.3 47.3166666666667 47.3333333333333
 47.35 47.3666666666667 47.3833333333333 47.4 47.4166666666667
 47.4333333333333 47.45 47.4666666666667 47.4833333333333 47.5
 47.5166666666667 47.5333333333333 47.55 47.5666666666667 47.5833333333333
 47.6 47.6166666666667 47.6333333333333 47.65 47.6666666666667
 47.6833333333333 47.7 47.7166666666667 47.7333333333333 47.75
 47.7666666666667 47.7833333333333 47.8 47.8166666666667 47.8333333333333
 47.85 47.8666666666667 47.8833333333333 47.9 47.9166666666667
 47.9333333333333 47.95 47.9666666666667 47.9833333333333]

The time values lie between 47 and 48, so we can conclude that we’ve selected the correct indices.

Note that we have to use the bitwise logical AND operator here because it operates on an element-by-element basis across the array.

Selecting $power array values at these indices is as simple as passing the $indices array as a slice:

pdl> print $power($indices)
[229 231 232 218 210 204 255 252 286 241 231 237 260 256 287 299 318 337 305
 276 320 289 280 301 320 303 395 266 302 341 299 287 309 279 294 284 266 281
 367 497 578 512 762 932 907 809 821 847 789 740 657 649 722 715 669 657 705
 643 647]

Using the max() method on this output gives us the maximum power:

pdl> print $power($indices)->max
932

In other words, the maximum power for the second sprint was 932 W. Not as good as the first sprint (which achieved 1023 W), but I was getting tired by this stage.

The same procedure allows us to find the maximum power for the first sprint with PDL. Again, eyeballing the graph above, we can see that the peak for the first sprint occurred between 24 and 26 minutes. Constructing the query in PDL, we have

pdl> print $power(which(24 < $time & $time < 26))->max
1023

which gives the maximum power value we expect.

We can also find out the maximum heart rate values around these times. E.g. for the first sprint:

pdl> print $heart_rate(which(24 < $time & $time < 26))->max
157

in other words, 157 bpm. For the second sprint, we have:

pdl> print $heart_rate(which(47 < $time & $time < 49))->max
165

i.e. 165 bpm, which matches the value that we found earlier. Note that I broadened the range of times to search over heart rate data here because its peak occurred a bit after the power peak for the second sprint.

Looking forward

Where to from here? Well, we could extend this code to handle processing multiple FIT files. This would allow us to find trends over many activities and longer periods. Perhaps there are other data sources that one could combine with longer trends. For instance, if one has access to weight data over time, then it’d be possible to work out things like power-to-weight ratios. Maybe looking at power and heart rate trends over a longer time can identify things such as overtraining. I’m not a sport scientist, so I don’t know how to go about that, yet it’s a possibility. Since we’ve got fine-grained, per-ride data, if we can combine this with longer-term analysis, there are probably many more interesting tidbits hiding in there that we can look at and think about.

Open question

One thing I haven’t been able to work out is where the calorie information is. As far as I can tell, Zwift calculates how many calories were burned during a given ride. Also, if one uploads the FIT file to a service such as Strava, then it too shows calories burned and the value is the same. This would imply that Strava is only displaying a value stored in the FIT file. So where is the calorie value in the FIT data? I’ve not been able to find it in the data messages that Geo::FIT reads, so I’ve no idea what’s going on there.

Conclusion

What have we learned? We’ve found out how to read, analyse and plot data from Garmin FIT files all by using Perl modules. Also, we’ve learned how to investigate the data interactively by using the PDL shell. Cool!

One main takeaway that might not be obvious is that you don’t really need online services such as Strava. You should now have the tools to process, analyse and visualise data from your own FIT files. With Geo::FIT ,Chart::Gnuplot and a bit of programming, you can glue together the components to provide much of the same (and in some cases, more) functionality yourself.

I wish you lots of fun when playing around with FIT data!

REPL stands for read-eval-print loop and is an environment where one can interactively enter programming language commands and manipulate data. ↩
It is, however, possible to (ab)use the Perl debugger and use it as a kind of REPL. Enter perl -de0 and you’re in a Perl environment much like REPLs in other languages. ↩
Many thanks to Harald Jörg for pointing this out to me at the recent German Perl and Raku Workshop. ↩
This is an application of “first make the change easy, then make the easy change”(paraphrasing Kent Beck). An important point often overlooked in this quote is that making the change easy can be hard. ↩
Not a particularly imaginative name, I know. ↩
The documented way to add a path to @INC in pdl is via the -Ilib command line option. Unfortunately, this didn’t work in my test environment: the local lib/ path wasn’t added to @INC and hence using the Geo::FIT::Utils module failed with the error that it couldn’t be located. ↩

Analysing FIT data with Perl: producing PNG plots

Paul Cochrane 🇪🇺 — Wed, 11 Jun 2025 22:00:00 +0000

Last time, we worked out how to extract, collate, and print statistics about the data contained in a FIT file. Now we’re going to take the next logical step and plot the time series data.

Start plotting with Gnu

Now that we’ve extracted data from the FIT file, what else can we do with it? Since this is time series data, the most natural next step is to visualise the data values over time. Since I know that Gnuplot handles time series data well,¹ I chose to use Chart::Gnuplot to plot the data.

An additional point in Gnuplot’s favour is that it can plot two datasets on the same graph, each with its own y-axis. Such functionality is handy when searching for correlations between datasets of different y-axis scales and ranges that share the same baseline data series.

Clearly Chart::Gnuplot relies on Gnuplot, so we need to install it first:

$ sudo apt install -y gnuplot

Now we can install Chart::Gnuplot with cpanm:

$ cpanm Chart::Gnuplot

Putting our finger on the pulse

Something I like looking at is how my heart rate evolved throughout a ride; it gives me an idea of how much effort I was putting in. So, we’ll start off by looking at how the heart rate data varied over time. In other words, we want time on the x-axis and heart rate on the y-axis.

One great thing about Gnuplot is that if you give it a format string for the time data, then plotting “just works”. In other words, explicit conversion to datetime data for the x-axis is unnecessary.

Here’s a script to extract the FIT data from our example data file. It displays some statistics about the activity and plots heart rate versus time. I’ve given the script the filename geo-fit-plot-data.pl:

use strict;
use warnings;

use Geo::FIT;
use Scalar::Util qw(reftype);
use List::Util qw(max sum);
use Chart::Gnuplot;

sub main {
    my @activity_data = extract_activity_data();

    show_activity_statistics(@activity_data);
    plot_activity_data(@activity_data);
}

sub extract_activity_data {
    my $fit = Geo::FIT->new();
    $fit->file( "2025-05-08-07-58-33.fit" );
    $fit->open or die $fit->error;

    my $record_callback = sub {
        my ($self, $descriptor, $values) = @_;
        my @all_field_names = $self->fields_list($descriptor);

        my %event_data;
        for my $field_name (@all_field_names) {
            my $field_value = $self->field_value($field_name, $descriptor, $values);
            if ($field_value =~ /[a-zA-Z]/) {
                $event_data{$field_name} = $field_value;
            }
        }

        return \%event_data;
    };

    $fit->data_message_callback_by_name('record', $record_callback ) or die $fit->error;

    my @header_things = $fit->fetch_header;

    my $event_data;
    my @activity_data;
    do {
        $event_data = $fit->fetch;
        my $reftype = reftype $event_data;
        if (defined $reftype && $reftype eq 'HASH' && defined %$event_data{'timestamp'}) {
            push @activity_data, $event_data;
        }
    } while ( $event_data );

    $fit->close;

    return @activity_data;
}

# extract and return the numerical parts of an array of FIT data values
sub num_parts {
    my $field_name = shift;
    my @activity_data = @_;

    return map { (split ' ', $_->{$field_name})[0] } @activity_data;
}

# return the average of an array of numbers
sub avg {
    my @array = @_;

    return (sum @array) / (scalar @array);
}

sub show_activity_statistics {
    my @activity_data = @_;

    print "Found ", scalar @activity_data, " entries in FIT file\n";
    my $available_fields = join ", ", sort keys %{$activity_data[0]};
    print "Available fields: $available_fields\n";

    my $total_distance_m = (split ' ', ${$activity_data[-1]}{'distance'})[0];
    my $total_distance = $total_distance_m/1000;
    print "Total distance: $total_distance km\n";

    my @speeds = num_parts('speed', @activity_data);
    my $maximum_speed = max @speeds;
    my $maximum_speed_km = $maximum_speed*3.6;
    print "Maximum speed: $maximum_speed m/s = $maximum_speed_km km/h\n";

    my $average_speed = avg(@speeds);
    my $average_speed_km = sprintf("%0.2f", $average_speed*3.6);
    $average_speed = sprintf("%0.2f", $average_speed);
    print "Average speed: $average_speed m/s = $average_speed_km km/h\n";

    my @powers = num_parts('power', @activity_data);
    my $maximum_power = max @powers;
    print "Maximum power: $maximum_power W\n";

    my $average_power = avg(@powers);
    $average_power = sprintf("%0.2f", $average_power);
    print "Average power: $average_power W\n";

    my @heart_rates = num_parts('heart_rate', @activity_data);
    my $maximum_heart_rate = max @heart_rates;
    print "Maximum heart rate: $maximum_heart_rate bpm\n";

    my $average_heart_rate = avg(@heart_rates);
    $average_heart_rate = sprintf("%0.2f", $average_heart_rate);
    print "Average heart rate: $average_heart_rate bpm\n";
}

sub plot_activity_data {
    my @activity_data = @_;

    my @heart_rates = num_parts('heart_rate', @activity_data);
    my @times = map { $_->{'timestamp'} } @activity_data;

    my $date = "2025-05-08";

    my $chart = Chart::Gnuplot->new(
        output => "watopia-figure-8-heart-rate.png",
        title => "Figure 8 in Watopia on $date: heart rate over time",
        xlabel => "Time",
        ylabel => "Heart rate (bpm)",
        terminal => "png size 1024, 768",
        timeaxis => "x",
        xtics => {
            labelfmt => '%H:%M',
        },
    );

    my $data_set = Chart::Gnuplot::DataSet->new(
        xdata => \@times,
        ydata => \@heart_rates,
        timefmt => "%Y-%m-%dT%H:%M:%SZ",
        style => "lines",
    );

    $chart->plot2d($data_set);
}

main();

A lot has happened between this code and the previous scripts. Let’s review it to see what’s changed.

The biggest changes were structural. I’ve moved the code into separate routines, improving encapsulation and making each more focused on one task.

The FIT file data extraction code I’ve put into its own routine (extract_activity_data(); lines 17-54). This sub returns the array of event data that we’ve been using.

I’ve also created two utility routines num_parts() (lines 57-62) and avg() (lines 65-69). These return the numerical parts of the activity data and average data series value, respectively.

The ride statistics calculation and display code is now located in the show_activity_statistics() routine. Now it’s out of the way, allowing us to concentrate on other things.

The plotting code is new and sits in a sub called plot_activity_data() (lines 109-137). We’ll focus much more on that later.

These routines are called from a main() routine (lines 10-15) giving us a nice bird’s eye view of what the script is trying to achieve. Running all the code is now as simple as calling main() (line 139).

Particulars of pulse plotting

Let’s zoom in on the plotting code in plot_activity_data(). After having imported Chart::Gnuplot at the top of the file (line 7), we need to do a bit of organising before we can set up the chart. We extract the activity data with extract_activity_data() (line 11) and pass this as an argument to plot_activity_data() (line 14). At the top of plot_activity_data() we fetch an array of the numerical heart rate data (line 112) and an array of all the timestamps (line 113).

The activity’s date (line 115) is assigned as a string variable because I want this to appear in the chart’s title. Although the date is present in the activity data, I’ve chosen not to calculate its value until later. This way we get the plotting code up and running sooner, as there’s still a lot to discuss.

Now we’re ready to set up the chart, which takes place on lines 117-127. We create a new Chart::Gnuplot object on line 117 and configure the plot with various keyword arguments to the object’s constructor.

The parameters are as follows:

output specifies the name of the output file as a string. The name I’ve chosen reflects the activity as well as the data being plotted.
title is a string to use as the plot’s title. To provide context, I mention the name of the route (Figure 8) within Zwift’s main virtual world (Watopia) as well as the date of the activity. To highlight that we’re plotting heart rate over time, I’ve mentioned this in the title also.
xlabel is a string describing the x-axis data.
ylabel is a string describing the y-axis data.
the terminal option tells Gnuplot to use the PNG²“terminal”³ and to set its dimensions to 1024x768.
timeaxis tells Gnuplot which axis contains time-based data (in this case the x-axis). This enables Gnuplot to space out the data along the axis evenly. Often, the spacing between points in time-based data isn’t regular; for instance, data points can be missing. Hence, naively plotting unevenly-spaced time data can produce a distorted graph. Telling Gnuplot that the x-axis contains time-based data allows it to add appropriate space where necessary.
xtics is a hash of options to configure the behaviour of the ticks on the x-axis. The setting here displays hour and minute information at each tick mark for our time data. We omit the year, month and day information as this is the same for all data points.

Now that the main chart parameters have been set, we can focus on the data we want to plot. In Chart::Gnuplot parlance, a Chart::Gnuplot::DataSet object represents a set of data to plot. Lines 129-134 instantiate such an object which we later pass to the Chart::Gnuplot object when plotting the data (line 136). One configures Chart::Gnuplot::DataSet objects similarly to how Chart::Gnuplot objects are constructed: by passing various options to its constructor. These options include the data to plot and how this data should be styled on the graph.

The options used here have the following meanings:

xdata is an array reference to the data to use for the x-axis. If this option is omitted, then Gnuplot uses the array indices of the y-data as the x-axis values.
ydata is an array reference to the data to use for the y-axis.
timefmt specifies the format string Gnuplot should use when reading the time data in the xdata array. Timestamps are strings and we need to inform Gnuplot how to parse them into a form useful for x-axis data. Were the x-axis data a numerical data type, this option wouldn’t be necessary.
style is a string specifying the style to use for plotting the data. In this example, we’re plotting the data points as a set of connected lines. Check out the Chart::Gnuplot documentation for a full list of the available style options.

We finally get to plot the data on line 136. The data set gets passed to the Chart::Gnuplot object as the argument to its plot2d() method. As its name suggests, this plots 2D data, i.e. y versus x. Gnuplot can also display 3D data, in which case we’d call plot3d(). When plotting 3D data we’d have to include a z dimension when setting up the data set.

Running this code

$ perl geo-fit-plot-data.pl

generates this plot:

A couple of things are apparent when looking at this graph. It took me a while to get going (my pulse rose steadily over the first ~15 minutes of the ride) and the time is weird (6 am? Me? Lol, no). We’ll try to explain the heart rate behaviour later.

But first, what’s up with the time data? Did I really start riding at 6 o’clock in the morning? I’m not a morning person, so that’s not right. Also, I’m pretty sure my neighbours wouldn’t appreciate me coughing and wheezing at 6 am while trying to punish myself on Zwift. So what’s going on?

For those following carefully, you might have noticed the trailing Z on the timestamp data. This means that the time zone is UTC. Given that this data is from May and I live in Germany, this implies that the local time would have been 8 am. Still rather early for me, but not too early to disturb the neighbours too much.⁴ In other words, we need to fix the time zone to get the time data right.

Getting into the zone

How do we fix the time zone? I’m glad you asked! We need to parse the timestamp into a DateTime object, set the time zone, and then pass the fixed time data to Gnuplot. It turns out that the standard DateTime library doesn’t parse date/time strings. Instead, we need to use DateTime::Format::Strptime. This module parses date/time strings much like the strptime(3) POSIX function does and returns DateTime objects.

Since the module isn’t part of the core Perl distribution, we need to install it:

$ cpanm DateTime::Format::Strptime

Most of the code changes that follow take place only within the plotting routine (plot_activity_data()). So, I’m going to focus on that from now on and won’t create a new script for the new version of the code.

The first thing to do is to import the DateTime::Format::Strptime module:

 use Scalar::Util qw(reftype);
 use List::Util qw(max sum);
 use Chart::Gnuplot;
+use DateTime::Format::Strptime;

Extending plot_activity_data() to set the correct time zone, we get this code:

sub plot_activity_data {
    my @activity_data = @_;

    # extract data to plot from full activity data
    my @heart_rates = num_parts('heart_rate', @activity_data);
    my @timestamps = map { $_->{'timestamp'} } @activity_data;

    # fix time zone in time data
    my $date_parser = DateTime::Format::Strptime->new(
        pattern => "%Y-%m-%dT%H:%M:%SZ",
        time_zone => 'UTC',
    );

    my @times = map {
        my $dt = $date_parser->parse_datetime($_);
        $dt->set_time_zone('Europe/Berlin');
        my $time_string = $dt->strftime("%H:%M:%S");
        $time_string;
    } @timestamps;

    # determine date from timestamp data
    my $dt = $date_parser->parse_datetime($timestamps[0]);
    my $date = $dt->strftime("%Y-%m-%d");

    # plot data
    my $chart = Chart::Gnuplot->new(
        output => "watopia-figure-8-heart-rate.png",
        title => "Figure 8 in Watopia on $date: heart rate over time",
        xlabel => "Time",
        ylabel => "Heart rate (bpm)",
        terminal => "png size 1024, 768",
        timeaxis => "x",
        xtics => {
            labelfmt => '%H:%M',
        },
    );

    my $data_set = Chart::Gnuplot::DataSet->new(
        xdata => \@times,
        ydata => \@heart_rates,
        timefmt => "%H:%M:%S",
        style => "lines",
    );

    $chart->plot2d($data_set);
}

The timestamp data is no longer read straight into the @times array; it’s stored in the @timestamps temporary array (line 6). This change also makes the variable naming a bit more consistent, which is nice.

To parse a timestamp string into a DateTime object, we need to tell DateTime::Format::Strptime how to format the timestamp (lines 8-12). This is the purpose of the pattern argument in the DateTime::Format::Strptime constructor (line 10). You might have noticed that we used the same pattern when telling Gnuplot what format the time data was in. We also specify the time zone (line 11) to ensure that the date/time data is parsed as UTC.

Next, we fix the time zone in all elements of the @timestamps array (lines 14-19). I’ve chosen to do this within a map here. I could extract this code into a well-named routine, but it does the job for now. The map parses the date/time string into a Date::Time object (line 15) and sets the time zone to Europe/Berlin⁵ (line 16). We only need to plot the time data,⁶ hence we format the DateTime object as a string including only hour, minute and second information (line 17). Even though we only use hours and minutes for the x-axis tick labels later, the time data is resolved down to the second, hence we retain the seconds information in the @times array.

One could write a more compact version of the time zone correction code like this:

my @times = map {
    $date_parser->parse_datetime($_)
        ->set_time_zone('Europe/Berlin')
        ->strftime("%H:%M:%S");
} @timestamps;

yet, in this case, I find giving each step a name (via a variable) helps the code explain itself. YMMV.

The next chunk of code (lines 22-23) isn’t related to the time zone fix. It generalises working out the current date from the activity data. This way I can use a FIT file from a different activity without having to update the $date variable by hand. The process is simple: all elements of the @timestamps array have the same date, so we choose to parse only the first one (line 22)⁷. This gives us a DateTime object which we convert into a formatted date string (via the strftime() method) composed of the year, month and day (line 23). We don’t need to fix the time zone because UTC is sufficient in this case to extract the date information. Of course, if you’re in a time zone close to the international date line you might need to set the time zone to get the correct date.

The last thing to change is the timefmt option to the Chart::Gnuplot::DataSet object on line 41. Because we now only have hour, minute and second information, we need to update the time format string to reflect this.

Now we’re ready to run the script again! Doing so

$ perl geo-fit-plot-data.pl

creates this graph

where we see that the time information is correct. Yay! 🎉

How long can this go on?

Now that I look at the graph again, I realise something: it doesn’t matter when the data was taken (at least, not for this use case). What matters more is the elapsed time from the start of the activity until the end. It looks like we need to munge the time data again. The job now is to convert the timestamp information into seconds elapsed since the ride began. Since we’ve parsed the timestamp data into DateTime objects (in line 15 above), we can convert that value into the number of seconds since the epoch (via the epoch() method). As soon as we know the epoch value for each element in the @timestamps array, we can subtract the first element’s epoch value from each element in the array. This will give us an array containing elapsed seconds since the beginning of the activity. Elapsed seconds are a bit too fine-grained for an activity extending over an hour, so we’ll also convert seconds to minutes.

Making these changes to the plot_activity_data() code, we get:

sub plot_activity_data {
    my @activity_data = @_;

    # extract data to plot from full activity data
    my @heart_rates = num_parts('heart_rate', @activity_data);
    my @timestamps = map { $_->{'timestamp'} } @activity_data;

    # parse timestamp data
    my $date_parser = DateTime::Format::Strptime->new(
        pattern => "%Y-%m-%dT%H:%M:%SZ",
        time_zone => 'UTC',
    );

    # get the epoch time for the first point in the time data
    my $first_epoch_time = $date_parser->parse_datetime($timestamps[0])->epoch;

    # convert timestamp data to elapsed minutes from start of activity
    my @times = map {
        my $dt = $date_parser->parse_datetime($_);
        my $epoch_time = $dt->epoch;
        my $elapsed_time = ($epoch_time - $first_epoch_time)/60;
        $elapsed_time;
    } @timestamps;

    # determine date from timestamp data
    my $dt = $date_parser->parse_datetime($timestamps[0]);
    my $date = $dt->strftime("%Y-%m-%d");

    # plot data
    my $chart = Chart::Gnuplot->new(
        output => "watopia-figure-8-heart-rate.png",
        title => "Figure 8 in Watopia on $date: heart rate over time",
        xlabel => "Elapsed time (min)",
        ylabel => "Heart rate (bpm)",
        terminal => "png size 1024, 768",
    );

    my $data_set = Chart::Gnuplot::DataSet->new(
        xdata => \@times,
        ydata => \@heart_rates,
        style => "lines",
    );

    $chart->plot2d($data_set);
}

The main changes occur in lines 14-23. We parse the date/time information from the first timestamp (line 15), chaining the epoch() method call to find the number of seconds since the epoch. We store this result in a variable for later use; it holds the epoch time at the beginning of the data series. After parsing the timestamps into DateTime objects (line 19), we find the epoch time for each time point (line 20). Line 21 calculates the elapsed time from the time stored in $first_epoch_time and converts seconds to minutes by dividing by 60. The map returns this value (line 22) and hence @times now contains a series of elapsed time values in minutes.

It’s important to note here that we’re no longer plotting a date/time value on the x-axis; the elapsed time is a purely numerical value. Thus, we update the x-axis label string (line 33) to highlight this fact and remove the timeaxis and xtics/labelfmt options from the Chart::Gnuplot constructor. The timefmt option to the Chart::Gnuplot::DataSet constructor is also no longer necessary and it too has been removed.

The script is now ready to go!

Running it

$ perl geo-fit-plot-data.pl

gives

That’s better!

Reaching for the sky

Our statistics output from earlier told us that the maximum heart rate was 165 bpm with an average of 142 bpm. Looking at the graph, an average of 142 bpm seems about right. It also looks like the maximum pulse value occurred at an elapsed time of just short of 50 minutes. We can check that guess more closely later.

What’s intriguing me now is what caused this pattern in the heart rate data. What could have caused the values to go up and down like that? Is there a correlation with other data fields? We know from earlier that there’s an altitude field, so we can try plotting that along with the heart rate data and see how (or if) they’re related.

Careful readers might have noticed something: how can you have a variation in altitude when you’re sitting on an indoor trainer? Well, Zwift simulates going up and downhill by changing the resistance in the smart trainer. The resistance is then correlated to a gradient and, given time and speed data, one can work out a virtual altitude gain or loss. Thus, for the data set we’re analysing here, altitude is a sensible parameter to consider. Even if you had no vertical motion whatsoever!

As I mentioned earlier, one of the things I like about Gnuplot is that one can plot two data sets with different y-axes on the same plot. Plotting heart rate and altitude on the same graph is one such use case.

To plot an extra data set on our graph, we need to set up another Chart::Gnuplot::DataSet object, this time for the altitude data. Before we can do that, we’ll have to extract the altitude data from the full activity data set. Gnuplot also needs to know which data to plot on the primary and secondary y-axes (i.e. on the left- and right-hand sides of the graph). And we must remember to label our axes properly. That’s a fair bit of work, so I’ve done the hard yakka for ya. 😉

Here’s the updated plot_activity_data() code:

sub plot_activity_data {
    my @activity_data = @_;

    # extract data to plot from full activity data
    my @heart_rates = num_parts('heart_rate', @activity_data);
    my @timestamps = map { $_->{'timestamp'} } @activity_data;
    my @altitudes = num_parts('altitude', @activity_data);

    # parse timestamp data
    my $date_parser = DateTime::Format::Strptime->new(
        pattern => "%Y-%m-%dT%H:%M:%SZ",
        time_zone => 'UTC',
    );

    # get the epoch time for the first point in the time data
    my $first_epoch_time = $date_parser->parse_datetime($timestamps[0])->epoch;

    # convert timestamp data to elapsed minutes from start of activity
    my @times = map {
        my $dt = $date_parser->parse_datetime($_);
        my $epoch_time = $dt->epoch;
        my $elapsed_time = ($epoch_time - $first_epoch_time)/60;
        $elapsed_time;
    } @timestamps;

    # determine date from timestamp data
    my $dt = $date_parser->parse_datetime($timestamps[0]);
    my $date = $dt->strftime("%Y-%m-%d");

    # plot data
    my $chart = Chart::Gnuplot->new(
        output => "watopia-figure-8-heart-rate-and-altitude.png",
        title => "Figure 8 in Watopia on $date: heart rate and altitude over time",
        xlabel => "Elapsed time (min)",
        ylabel => "Heart rate (bpm)",
        terminal => "png size 1024, 768",
        xtics => {
            incr => 5,
        },
        y2label => 'Altitude (m)',
        y2range => [-10, 70],
        y2tics => {
            incr => 10,
        },
    );

    my $heart_rate_ds = Chart::Gnuplot::DataSet->new(
        xdata => \@times,
        ydata => \@heart_rates,
        style => "lines",
    );

    my $altitude_ds = Chart::Gnuplot::DataSet->new(
        xdata => \@times,
        ydata => \@altitudes,
        style => "boxes",
        axes => "x1y2",
    );

    $chart->plot2d($altitude_ds, $heart_rate_ds);
}

Line 7 extracts the altitude data from the full activity data. This code also strips the unit information from the altitude data so that we only have the numerical part, which is what Gnuplot needs. We store the altitude data in the @altitudes array. This we use later to create a Chart::Gnuplot::DataSet object on lines 53-58. An important line to note here is the axes setting on line 57; it tells Gnuplot to use the secondary y-axis on the right-hand side for this data set. I’ve chosen to use the boxes style for the altitude data (line 56) so that the output looks a bit like the hills and valleys that it represents.

To make the time data a bit easier to read and analyse, I’ve set the increment for the ticks on the x-axis to 5 (lines 37-39). This way it’ll be easier to refer to specific changes in altitude and heart rate data.

The settings for the secondary y-axis use the same names as for the primary y-axis, with the exception that the string y2 replaces y. For instance, to set the axis label for the secondary y-axis, we specify the y2label value, as in line 40 above.

I’ve set the range on the secondary y-axis explicitly (line 41) because the output looks better than what the automatic range was able to make in this case. Similarly, I’ve set the increment on the secondary y-axis ticks (lines 42-44) because the automatic output wasn’t as good as what I wanted.

I’ve also renamed the variable for the heart rate data set on line 47 to be more descriptive; the name $data_set was much too generic.

We specify the altitude data set first in the call to plot2d() (line 60) because we want the heart rate data plotted “on top” of the altitude data. Had we used $heart_rate_ds first in this call, the altitude data would have obscured part of the heart rate data.

Running our script in the now familiar way

$ perl geo-fit-plot-data.pl

gives this plot

Cool! Now it’s a bit clearer why the heart rate evolved the way it did.

At the beginning of the graph (in the first ~10 minutes) it looks like I was getting warmed up and my pulse was finding a kind of base level (~130 bpm). Then things started going uphill at about the 10-minute mark and my pulse also kept going upwards. This makes sense because I was working harder. Between about 13 minutes and 19 minutes came the first hill climb on the route and here I was riding even harder. The effort is reflected in the heart rate data which rose to around 160 bpm at the top of the hill. That explains why the heart rate went up from the beginning to roughly the 18-minute mark.

Looking back over the Zwift data for that particular ride, it seems that I took the KOM⁸ for that climb at that time, so no wonder my pulse was high!⁹ Note that this wasn’t a special record or anything like that; it was a short-term live result¹⁰ and someone else took the jersey with a faster time not long after I’d done my best time up that climb.

It was all downhill shortly after the hill climb, which also explains why the heart rate went down straight afterwards. We also see similar behaviour on the second hill climb (from about 37 minutes to 42 minutes). Although my pulse rose throughout the hill climb, it didn’t rise as high this time. This indicates that I was getting tired and wasn’t able to put as much effort in.

Just in case you’re wondering how the altitude can go negative,¹¹ part of the route goes through “underwater tunnels”. This highlights the flexibility of the virtual worlds within Zwift: the designers have enormous room to let their imaginations run wild. There are all kinds of fun things to discover along the various routes and many that don’t exist in the Real World™. Along with the underwater tunnels (where it’s like riding through a giant aquarium, with sunken shipwrecks, fish, and whales), there is a wild west style town complete with a steam train from that era chugging past. There are also Mayan ruins with llamas (or maybe alpacas?) wandering around and even a section with dinosaurs grazing at the side of the road.

Here’s what it looks like riding through an underwater tunnel:

I think that’s pretty cool.

At the end of the ride (at ~53 minutes) my pulse dropped sharply. Since this was the warm-down phase of the ride, this also makes sense.

Power to the people!

There are two peaks in the heart rate data that don’t correlate with altitude (one at ~25 minutes and another at ~48 minutes). The altitude change at these locations would suggest that things are fairly flat. What’s going on there?

One other parameter that we could consider for correlations is power output. Going uphill requires more power than riding on the flat, so we’d expect to see higher power values (and therefore higher heart rates) when climbing. If flat roads require less power, what’s causing the peaks in the pulse? Maybe there’s another puzzle hiding in the data.

Let’s combine the heart rate data with power output and see what other relationships we can discover. To do this we need to extract power output data instead of altitude data. Then we need to change the secondary y-axis data set and configuration to produce a nice plot of power output. Making these changes gives this code:

sub plot_activity_data {
    my @activity_data = @_;

    # extract data to plot from full activity data
    my @heart_rates = num_parts('heart_rate', @activity_data);
    my @timestamps = map { $_->{'timestamp'} } @activity_data;
    my @powers = num_parts('power', @activity_data);

    # parse timestamp data
    my $date_parser = DateTime::Format::Strptime->new(
        pattern => "%Y-%m-%dT%H:%M:%SZ",
        time_zone => 'UTC',
    );

    # get the epoch time for the first point in the time data
    my $first_epoch_time = $date_parser->parse_datetime($timestamps[0])->epoch;

    # convert timestamp data to elapsed minutes from start of activity
    my @times = map {
        my $dt = $date_parser->parse_datetime($_);
        my $epoch_time = $dt->epoch;
        my $elapsed_time = ($epoch_time - $first_epoch_time)/60;
        $elapsed_time;
    } @timestamps;

    # determine date from timestamp data
    my $dt = $date_parser->parse_datetime($timestamps[0]);
    my $date = $dt->strftime("%Y-%m-%d");

    # plot data
    my $chart = Chart::Gnuplot->new(
        output => "watopia-figure-8-heart-rate-and-power.png",
        title => "Figure 8 in Watopia on $date: heart rate and power over time",
        xlabel => "Elapsed time (min)",
        ylabel => "Heart rate (bpm)",
        terminal => "png size 1024, 768",
        xtics => {
            incr => 5,
        },
        ytics => {
            mirror => "off",
        },
        y2label => 'Power (W)',
        y2range => [0, 1100],
        y2tics => {
            incr => 100,
        },
    );

    my $heart_rate_ds = Chart::Gnuplot::DataSet->new(
        xdata => \@times,
        ydata => \@heart_rates,
        style => "lines",
    );

    my $power_ds = Chart::Gnuplot::DataSet->new(
        xdata => \@times,
        ydata => \@powers,
        style => "lines",
        axes => "x1y2",
    );

    $chart->plot2d($power_ds, $heart_rate_ds);
}

On line 7, I swapped out the altitude data extraction code with power output. Then, I updated the output filename (line 32) and plot title (line 33) to highlight that we’re now plotting heart rate and power data.

The mirror option to the ytics setting (lines 40-42) isn’t an obvious change. Its purpose is to stop the ticks from the primary y-axis from being mirrored to the secondary y-axis (on the right-hand side). We want to stop these mirrored ticks from appearing because they’ll clash with the secondary y-axis tick marks. The reason we didn’t need this before is that all the y-axis ticks happened to line up and the issue wasn’t obvious until now.

I’ve updated the secondary axis label setting to mention power (line 43). Also, I’ve set the range to match the data we’re plotting (line 44) and to space out the data nicely via the incr option to the y2tics setting (lines 45-47). It seemed more appropriate to use lines to plot power output as opposed to the bars we used for the altitude data, hence the change to the style option on line 59.

As we did when plotting altitude, we pass the power data set ($power_ds) to the plot2d() call before $heart_rate_ds (line 63).

Running the script again

$ perl geo-fit-plot-data.pl

produces this plot:

This plot shows the correlation between heart rate and power output that we expected for the first hill climb. The power output increases steadily from the 3-minute mark up to about the 18-minute mark. After that, it dropped suddenly once I’d reached the top of the climb. This makes sense: I’d just done a personal best up that climb and needed a bit of respite!

However, now we can see clearly what caused the spikes in heart rate at 25 minutes and 48 minutes: there are two large spikes in power output. The first spike maxes out at 1023 W;¹² what value the other peak has, it’s hard to tell. We’ll try to work out what that value is later. These spikes in power result from sprints. In Zwift, not only can one try to go up hills as fast as possible, but flatter sections have sprints where one also tries to go as fast as possible, albeit for shorter distances (say 200m or 500m).

Great! We’ve worked out another puzzle in the data!

A quick comparison

Zwift produces what they call timelines of a given ride, which is much the same as what we’ve been plotting here. For instance, for the FIT file we’ve been looking at, this is the timeline graph:

Zwift plots several datasets on this graph that have very different value ranges. The plot above shows power output, cadence, heart rate, and altitude data all on one graph! A lot is going on here and because of the different data values and ranges, Zwift doesn’t display values on the y-axes. Their solution is to show all four values at a given time point when the user hovers their mouse over the graph. This solution only works within a web browser and needs lots of JavaScript to work, hence this is something I like to avoid. That (and familiarity) is largely the reason why I prefer PNG output for my graphs.

If you take a close look at the timeline graph, you’ll notice that the maximum power is given as 937 W and not 1023 W, which we worked out from the FIT file data. I don’t know what’s going on here, as the same graph in the Zwift Companion App shows the 1023 W that we got. The graph above is a screenshot from the web application in a browser on my laptop and, at least theoretically, it’s supposed to display the same data. I’ve noticed a few inconsistencies between the web browser view and that from the Zwift Companion App, so maybe this discrepancy is one bug that still needs shaking out.

A dialogue with data

Y’know what’d also be cool beyond plotting this data? Playing around with it interactively.

That’s also possible with Perl, but it’s another story.

I’ve been using Gnuplot since the late 90’s. Back then, it was the only freely available plotting software which handled time data well. ↩
By default, Gnuplot will generate Postscript output. ↩
One can interpret the word “terminal” as a kind of “screen” or “canvas” that the plotting library draws its output on. ↩
I’ve later found out that they haven’t heard anything, so that’s good! ↩
I live in Germany, so this is the relevant time zone for me. ↩
All dates are the same and displaying them would be redundant, hence we omit the date information. ↩
All elements in the array have the same date, so using the first one does the job. ↩
KOM stands for “king of the mountains”. ↩
Yes, I am stoked that I managed to take that jersey! Even if it was only for a short time. ↩
A live result that makes it onto a leaderboard is valid only for one hour. ↩
Around the 5-minute mark and again shortly before the 35-minute mark. ↩
One thing that this value implies is that I could power a small bar heater for one second. But not for very much longer! ↩