Forem: Adam Leskis

From Active Learning to Deliberate Practice: an iximiuz Labs case study

Adam Leskis — Tue, 24 Mar 2026 08:59:53 +0000

Much of the educational tech content on the internet is in the form of blogs/videos/tutorials/walkthroughs/etc, and a lot of it is good, some great, some not so great. One thing that the great stuff has in common is that it encourages a "Learning by Doing" approach. After all, you get better at building a REST API by actually building a REST API...not just reading about it, watching somebody else do it, or drawing out a diagram on a whiteboard.

To be sure, a lot of those things I just mentioned (eg, reading about REST APIs) are absolutely necessary input to build your conceptual framework of technical concepts, foundational components, and how the entire system fits together to accomplish a goal. There's very little chance you could just guess at what it should be and get it right on the first, or even 100th time, without SOME context around what you should do.

And with this context built through content, active learning is an excellent learning strategy to extend and consolidate the knowledge that you gain through reading/watching/hearing/etc about different technical concepts. By actually performing the behavior required to create the thing you've learned about (eg, our example REST API), you get all sorts of added benefits:

feedback on whether the thing works
having to deal with errors when it doesn't work
seeing where the gaps in your knowledge still are
exposing hidden assumptions about whether certain steps in the tutorial are obvious (they usually aren't)
the satisfaction of completing something

How This Content Gets Created

One of the greatest shortcomings, however, in this wealth of content on active learning is that each of the examples/scenarios/challenges needs to be created by a human.

In before: I hope you're not suggesting we just create content entirely with AI...no, I'm not going to argue that, since I think it would be unnecessarily wastefully, and we can get pretty far anyway without it.

Take, for example, a learning exercise where a learner is dropped into a linux environment, and has to find out why a systemd service has stopped. It used to be running, but now it's not, and eventually, with some helpful hints from the learning platform, the learner finds that permissions have been updated on some files that the service needs, and fixing those allows it to start again.

The learner has actually applied some knowledge to fix a system issue and rejoices in the glow of active learning success!

And now the learner would like to do it again, so they press "start" again on the activity, and they're dropped back into the same linux system, with the exact same instructions, the exact same error, and the exact same fix.

They remember probably 80% of what they just did (it was only 5 minutes ago), and so they don't have much trouble fixing it again. After repeating this about 5 times, the learner can go on autopilot, just repeating the commands from memory, and maybe even jumping directly to the fix without needing to go through the whole debugging flow.

This active learning can become passive when it's the same thing over and over, and can begin to be more of a test of memory than actual knowledge application.

Deliberate practice

A much stronger paradigm supported by the work of researchers like Ericsson (1993), is the framework of deliberate practice (the seminal article on deliberate practice is here).

The main points are that the learner needs to be engaged in narrowly focused intentional repetition, centered on a particular skill that they're trying to improve. So we could reimagine our previous example within this framing to have the following structure:

The learner still enters the familiar linux environment, but it's unclear which systemd service is having issues (since this is randomized at the start of the activity). Through investigation, the learner isolates which service it is and proceeds to update the file permissions to fix the service. On the next attempt, it's a different service that is having issues, and so the learner is prompted to use a specific process to isolate which service is misbehaving. This can continue as long as the learner wants to continue practicing.

Note that this new framing exposes a hidden complexity in the original activity...it's actually combining two realistic debugging activities: identifying a failing service and using file system permissions to fix the issue.

So you could also imagine a slightly different path based on the original scenario as the learner in the linux system, and where the learner already knows (or is told) which system is behaving problematically, but the underlying permissions issue is randomized and requires investigation and remediation:

the user the service is running under is randomized, which has implications for the file permissions necessary
the binary of the service isn't executable
the log directory isn't writable due to permissions mismatch
the ownership of a file directory is correct, but new files don't inherit the ownershipt because setgid is missing
various permission bit issues

Compared to the single repeated scenario, this has some structural differences and advantages:

the learning objective can be scoped down and isolated, which also allows more targeted feedback and assessment
the practice is necessarily more deliberate and intentional, since the fix can't be guessed ahead of time, so the role of short-term memory changes to be about the process rather than the solution
there's actually a reason for the learner to repeat the activity multiple times
the variation now forces the learner to exhibit the behavior we're trying to gain improvement in (related to the specific learning objective)
this approach is supported by research (as mentioned above)

Case Study #1 - Kubernetes OWASP Top Ten

https://labs.iximiuz.com/playgrounds/my-my-k3s-de88e13a

This is a playground set up to give the learner practice with running a vulnerability scanning tool (in this case, kubescape) to identify and fix a randomized security vulnerability from the OWASP Kubernetes Top 10 list in a running cluster.

The very specific and measurable learning objective is editing a kubernetes manifest/resource to fix the vulnerability. While this does look a bit like our original example of both finding/fixing an issue, the scanner bit is very much just repeated commands, and there's not much of a learning objective to target there.

Additionally, this first part of the activity is arguably not very authentic. Not many people are actually manually running scanners to identify vulnerable misconfigurations in their K8s clusters, since this is usually delegated to the CI pipelines (and rightly so!).

The second part is more defensible, as a means to give learners a mapping between vulnerability category (K01 - insecure workload configurations, like running as root) and what that actually looks like in a manifest.

While it does have gaps, it's a working MVP of a system that automatically presents a situation and gives feedback about whether the learner achieved a given outcome. So in terms of our discussion of deliberate practice, it's a bit closer to something that's supported by the research.

Case Study #2 - Log Parsing in Linux

https://labs.iximiuz.com/playgrounds/log-parser-lab-k3s-dd84febf

This playground is configured to present the learner with an opportunity to use common Linux tools (eg, grep, sed, awk, wc) to parse various log types (nginx, apache, syslog) for various investigative purposes:

finding the top unique IP address triggering failures
summing of bytes for an IP with a specific HTTP method over a given time period
finding the user with the most failed SSH logins over the last hour

Similar to the previous case, a strength here is that the learner needs to convert the requirements into a chain of commands. So it's also narrowly focused, is completely self-contained, and can continue as long as the learner wants to practice.

Since both the log type and investigation focus are varied at random for each new exercise, it's possible for the learner to use their short-term memory for applying knowledge of the various linux commands, rather than the keystrokes to type.

Disclaimer - We Need More Tools in General

Having said all of the above, why might it still be the wrong thing to do?

While I'm obviously biased towards the deliberate practice approach, it's not a format that a lot of learners are familiar with, specifically because it isn't very common.

In addition, other strands of cognitive research which discuss desirable difficulty (another good thing that deliberate practice incorporates), has a downside in that learners can feel like they're learning less. This is also a proposed reason for why, given the choice, learners still gravitate towards youtube walkthroughs and blog tutorials:

They feel like they're learning more because it's easy and they enjoy it

So it could be that this format isn't something that learners are even interested in, because they themselves feel like they're learning less from it.

Another challenge is how to implement these types of activities from a technical perspective. There are a few different challenges like:

where/how is this activity presented to learners (web browser, terminal cli, etc)?
what mechanism is used to randomize the activities?
how do we assess whether the learning objective has been achieved?
what format is the feedback on unsuccessful attempts, and when is it given?

Nevertheless, more choice in how users like to learn new things is never a bad thing, and with the internet, we can create and share these things at basically zero marginal cost. More is better, and I'm here for it!

An Iximiuz Cluster of Clusters with Tailscale and Cilium

Adam Leskis — Thu, 19 Mar 2026 21:57:18 +0000

Editors note: This was originally much more ambitious, but for various reasons detailed below it had to be scaled back a bit. It's still pretty cool.

The final k8s cluster project is here and the thing I build to visualize it is here, but read on for the story of how I got there!

So I got hooked on the kubernetes-the-hard-way walkthroughs a couple years back and ran through them on various different public cloud providers. Since iximiuz Labs is my current favorite place to play, I thought I would give it a go there as well.

It turns out that somebody else already did this, and it's way better than I would have done anyway, so I guess I might as well just forget about it.

...or maybe not!!!!

What if, I thought to myself...just what IF instead of the traditional 1-3 controller nodes and 2-3 worker nodes (these numbers, I assume, are used in the typical k8s the hard way due to cost), I instead networked together not only more worker nodes, but different networks altogether?!?!

The canonical walkthrough uses only 4 machines (1 controller and 3 workers) all in the same subnet (this, as we shall see, MASSIVELY simplifies the networking). I decided that since I had so much fun playing with Tailscale in a previous post, I would use it a bit more to connect the various subnets across multiple iximiuz Labs playgrounds.

So the original idea was to do all the inter-node routing via Tailscale, so that each controller and worker node would be able to address each other by hostname, which would map to the Tailscale overlay IP address (eg, 100.1.20.144). So even though we were wiring together a bunch of different playgrounds/subnets, all of the VMs (and hence also the eventual pods) would think they were on a flat network.

My Kingdom for Some Cool Viz

Then I decided that just a big ole kubernetes cluster is pretty cool all by itself, but it's a bit difficult to get a good picture of exactly HOW that differs from just a regular one (ie, all the other k8s the hard way walkthroughs).

So I thought, maybe we could have some sort of visual, and even better to have a live traffic graph UI that would make it very obvious when we scaled a deployment up/down, or even just had a traffic generator hit one of the services with a ton of traffic and actually see the difference between a 3 worker cluster and a 9 worker cluster.

And so after a bit of research, it seemed like the easiest way to do this would be with some sort of a service mesh (the pods don't care about it, and there's usually baked in telemetry, so a ready data source that I could consume).

The ones I was looking at were Istio/Envoy/Cilium, and since the iximiuz Labs VMs have a maximum of 4CPU and 8GiB RAM, I was trying to minimize the resource usage of the service mesh. Of those, not only is Cilium the clear winner, but it uses eBPF, which I though was really cool, and I wanted to play with it more.

Okay, so I decided on Cilium, now what?

I started looking around for traffic visualization projects using Cilium (more specifically, hubble-observatory, which is Cilium's observability platform). The default view for hubble-ui, which is the included UI for hubble-observatory, just shows static traffic routes.

Basically nothing I found was live traffic. There was a super old Netflix project, but it wasn't maintained anymore.

Hmmmm...there's nothing that does what I need, and I've been spending the last year or so getting acquainted with agentic workflows...

Let's Just Make The Thing We Need

Okay, so I need something running in the cluster that will grab data from hubble-observatory and I need a frontend that will open up and display something in a browser tab. Let's mix some Go and React and do the damned thing!

I used codex for this, and I was actually surprised at how far it got on the first attempt. Most of the ~~work~~ prompting was getting a local dev example up and running with sample data so I could see what it looked like and iterate on stuff...

no, those graph edges need to be thinner
...thinner!
alright, and stop resetting everything every 2 seconds, I said DECOUPLE the frontend and backend!
WHY DID YOU SUGGEST WEBSOCKETS AGAIN?!?!! THIS COULD EASILY BE SERVER SENT EVENTS!!!! (to be fair, it probably doesn't matter that much in this case, since it's just a throwaway demo, but the agent always starts out with websockets even when it's definitely a worse choice than SSE)
that's okay, but we need to auto-zoom when the topology changes
forget everything I just said, we need to also cater for a mobile UI (I'm obsessed with running iximiuz Labs stuff on my phone, so this is a must for any of my projects)

...eventually, I had something that looked like I wanted, and I was ready to get it showing some actual live data.

My MVP meets Live Cluster Reality

Okay, so this is where things went a bit pear-shaped, and I spent probably 75% of the total project time.

To start with, actually running a service inside kubernetes was going to require all kinds of extra stuff like RBAC, network policies, and pod specs for the deployment. No biggie, I've got an Intern to do that.

I'll also admit to heavy ulitization of the scripts in the project, since there's no way I was gonna "hard way" build it from scratch every day (the playgrounds have a max lifetime of 8 hours, and since it's a group of them, I guess we could have saved VM state and restarted them, but I also wanted to be sure the project would always come up from zero, so I went with the scripts), and I started to see some really weird failures and intermittent behavior:

Everything would come up, but there would be no data coming through hubble-gazer
I could see initial data in hubble-gazer, but then every 2 seconds (the same interval as the SSE pushes to the frontend) it got less and less until it completely disappeared
sometimes the scripts would just choke and things wouldn't start up

The short answer to "why?" is that not only was I using an overlay network (Cilium VXLANs) wrapped in another overlay network (Tailscale), but the topology was such that we had 5 different subnets and 12 kubernetes nodes all communicating with each other for reachability.

We ended up with a situation where some of the playgrounds would end up with the same public IP address but be on different private subnets, and so Tailscale would have to fall back to a relayed connection, which still works, but introduces latency.

So things I had initially wanted to try, like HPA scaling based on CPU from the metrics server to show a cool demo of dynamic pod growth, just wouldn't work. I couldn't even get just KEDA with queues in redis to work, since the jitter and general network conditions just wouldn't support it.

This was all very frustrating and a bit confusing to me, and I'm sure somebody with better networking knowledge could have sorted it out, but I had already learned a bunch and a deadline (I wanted this done by Kubecon EU 2026) was fast approaching.

Updating the topology so all of the worker nodes were in the same subnet basically solved everything, since now internode traffic didn't have to transit two layers of tunnelling, and we only used Tailscale for communication between the workers and the control plane.

So we ended up with a much simplified topology, but it also finally worked and I could finish aligning hubble-gazer to show the live traffic.

I had been hoping that the KEDA-based scaling I added would be more obvious, but oh well.

The final topology (3 playgrounds, 9 VMs) looks like this:

And here are some tasty gifs of the live traffic being portforwarded from the jumpbox back to localhost to be accessible from the laptop I used to set up the cluster (that's why the browser URL bar says localhost:8888).

All services and pods

Application L4 traffic, also grouped by worker node

Application L7 traffic, as well as DNS queries

Custom rootFS for playgrounds in iximiuz Labs

Adam Leskis — Sat, 21 Feb 2026 13:48:10 +0000

The playgrounds on https://labs.iximiuz.com have always been amazing, but they recently got an upgrade to enable creation with a custom image for their root filesystem. Not only is this super cool, but it unlocks the ability for authors to significantly speed up the boot times of their playgrounds and labs by pre-installing and configuring everything to achieve massive startup speed improvements.

I recently went on a journey to try this out for myself, since I'd never done it before, and this is a writeup of where I made silly mistakes, so you don't have to.

The New Project

I'm heading to Kubecon EU next month, and I thought it would be cool to have a new project to demo. The committee didn't accept my proposal to present, but the joke's on them, since obviously, with an iximiuz Labs playground, I can demo it to anybody with a web browser.

Only problem is that when I run this locally, it downloads and compiles the kubernetes operator from source, plus a bunch of other stuff that slows down the initialization and basically means the user has to wait for about 5 minutes.

The Challenge

I knew that recent work had been done to allow you to bring your own rootFS to an iximiuz Labs playground, and that sounded like exactly what I needed, but how to actually do it?

The things that make this easy:

you can just set a link to an image in an OCI compliant container registry as the root file system for a VM

there are a ton of examples that have been opensourced on what these look like for the current playgrounds
when trying it out, you can just run a playground and see what happens (if/when it fails, there's some helpful error messages to show you what went wrong)

The things that make this hard:

conceptually, I didn't initially get how a container image and a VM could be the same (wasn't the whole Docker/VM thing like "the great divide" in how to run software?!?!)
the examples can be used as base images, but how do I add my stuff on top of it?
when you try out a new custom image as a rootFS mount...and it doesn't work...how do you debug it? (* - this actually has been improved since I started drafting this article...that's how awesome it is to be in the iximiuz discord and ask for a feature!)

The Stuff I Did

So I started out with one of the rootFS images, the one for Ubuntu 24.04. Then I took some of the stuff from my custom playground and added it to the Dockerfile...basically the apt install -y stuff, and also the curl -s https://raw.githubusercontent.com/kubescape/kubescape/master/install.sh | /bin/bash stuff, cause I gots to have my kubescape scanning in this playground.

That built okay, and so I pushed to GitHub's container registry (editor's note: this is preferable to just pushing to dockerhub because the rate limiting is less of a pain).

The VM immediately blew up though, and wouldn't even boot.

Lesson: Github packages, which container images are, are private by default, and you have to go through the whole rigmarole of switching their visibility to public before they can be pulled by anyone but you (this is why just pulling it locally isn't a good test!).

Now having switched my image to public, it finally pulled and built fine, but nothing worked, because there was no kubectl installed, nor any of the stuff I actually needed to run k3s. I needed to use the same underlying image as the actual k3s playground I had based on this Dockerfile.

Lesson: base the custom rootFS on the base image that actually has the stuff you need.

There was another kerfuffle when I tried to copy in the kubernetes operator (y'know, the WHOLE POINT of the playground) into the image, but it kept disappearing.

Lesson: don't copy stuff into /tmp, cause just like in AWS ECS (which you think I would have remembered by now cause it's happened to me about a bajillion times) it gets wiped when the system mounts it. It works in the playground configuration because all that stuff happens after mounting.

I was also trying out different things, pushing images to ghcr, but not seeing any differences in the lab VMs. Turns out there is some pretty aggressive image caching going on, and if you use the same tag, it might not get picked up.

Lesson: while prototyping, you might just need to cycle through different tags, and that's how I ended up with v4 as my (current) working tag.

First Result

So with all my efforts the startup time went from about 6 minutes to...about 5 minutes. It turns out that the package installation was already fairly quick, and the real slowdown was happening when I compiled all the kubernetes operator stuff (there's a LOT of stuff that goes on there!).

So the next obvious move was to move all that compilation to build time, swap in a multi-stage build so that I could get rid of the entire golang toolchain in the final image, and just copy in the binary and CRD yaml.

Second Result

Now with the binary all ready to go on boot, the VM startup went down to around 1-2 minutes. The remaining time is starting up all the pods in the k3s cluster and making sure things are running smoothly before we're all ready for the user to start doing thing.

Lesson: If you want to actually make sure things are ready to interact with, don't get rid of all your init scripts, cause those are what make the fancy "Warming up playground" screen, and then users are immediately dumped into the terminal and might think things are ready, but they might not be.

So I added a

kubectl wait --for=condition=Ready --timeout=60s pods --all -n test-lab

to my lab just to make sure that screen kept the user waiting while things get ready.

I also was wondering if it would be necessary to create two different versions of the playground with different specs (obviously, more cpu/RAM would mean that it can start up and be ready even faster), but thankfully that's not even an issue, since you can just set it to the max of the premium tier, and when a free tier user starts it, it defaults to free tier limits (important - test out your playground as a free user if you want to make sure this actually works)

The End Result

So I've got a playground where you can practice running scans with kubescape to find security vulnerabilities in a running k3s cluster, then test out your fix (it's usually just updating the deployment), and the cluster will reset to another random vulnerability.

The end result is a combo of Dockerfile, playground manifest, and some operator CRD stuff, which you can see here on GitHub.

the reason I got confused initially between docker/VM is that in this case, we're only creating an image to use as a root volume, which is kind of what docker also does, but docker also layers on all of the fancy namespace (network/file/process/etc) stuff, which we don't use here, so a volume is a volume is a volume.

I'll definitely be using this custom rootFS method going forward, and I encourage you to give it a try if you find yourself in need of some speed!

What to DevOps...??? An Iximiuz Labs Speedrun to the Rescue!

Adam Leskis — Fri, 22 Aug 2025 12:54:06 +0000

The video version of this blog is here. I used box-01/02/03 instead of server-01/02/03 in the video, but the rest is exactly the same.

A lot of people who are interested in breaking into DevOps (or whatever title seems most appropriate for "learning about managing the things that actually run the software") often have the same question:

"What should I do?"

And they often get the same answer:

"Just build something"

But then they want to know:

"What should I build?"

And this is where it gets tricky...

While building something that you actually want to use yourself is often one of the most motivating ways to learn while doing, for a lot of beginners, they're not even sure what the basic building blocks of "stuff that runs the software" are.

What to build or how to build

In very basic terms, those building blocks (especially as related to SaaS products) are easy enough to read about:

a database
a backend server
a webserver sending a javascript frontend bundle to a browser

A lot of the web apps we use are basically web frontend wrappers around databases running CRUD, and that's okay!

Though even knowing this, a lot of people can get stuck on the stage of "what kind of web frontend wrapper around a database should I build?"

A lot of the tutorials walk learners through running everything locally, which is indeed a great way to get started on the software side of things.

But what about the operating of that software on VMs somewhere? And yes, even containers run inside VMs, although a lot of platforms like ECS Fargate or Google Cloud Run abstract that away...

Forget the apps, lets deal with some VMs!

It used to be that if you wanted to deploy an application on a VM and get experience doing that, you would have needed to sign up to a cloud provider like AWS, Digital Ocean, or Hetzner. This took time and oftentimes involved a financial commitment (even just entering your credit card details, which kept some people from going this route).

I've written before about Iximiuz Labs, which is the perfect place to play around with VMs for free*!

* - requires a GitHub user

In the rest of this post, I'll walk through deploying and configuring a "3-Tier Web App" ™️ in one of the playgrounds, completely from scratch.

...and here's what it will look like.

We will:

Set up a postgres database
Set up a backend application to interact with the database
Set up a frontend application served via nginx to load in the browser

We won't have to deal with the tougher aspects of DevOps, like DNS, firewalls, subnets, monitoring, or CI/CD pipelines...we want to keep this as simple as possible (though maybe we'll look at some of those in later blog posts).

But don't worry...just in case you're worried it'll be too easy, we will need to care about IP addresses and ports, so there's still a little bit of networking going on here.

I cheated a little bit...the code for both the backend and frontend is already written (it's VERY simple), but that makes it easier for other people, especially those who have no idea what to build, to follow along themselves!

My Kingdom for a Database!

Let's fire up the Flexbox playground and start installing stuff!

You'll just need to add another machine and make sure the Rootfs (the root file system) for each of them is Ubuntu 24.04.

Ubuntu 22.04 would also still work, but we might as well go for the latest OS version.

I've called my machines server-01, server-02, and server-03, but you can call them whatever you want. Once you hit the "Start Playground" button, the playground will boot up, and you should see the following screen:

Let's go ahead and install the database on server-03:

sudo apt update && sudo apt install -y postgresql postgresql-client-16

and now let's start the database

sudo systemctl start postgresql
sudo -u postgres psql -x

This is a neat trick to get into the database without needing the password, since it automatically lets a user named "postgres" in if you're on the same server.

Now you're in the database and can start configuring the application user

CREATE DATABASE drizzle;

\c drizzle;

CREATE USER laborant WITH PASSWORD 'laborant';

(in the above, because these are isolated ephemeral environments, it doesn't really matter what you set as your password, so you might as well make it easy to remember...don't do this in production, obviously)

and now we can grant the privileges for the new user to interact with the database:

GRANT ALL PRIVILEGES ON DATABASE drizzle TO laborant;

GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO laborant;

GRANT USAGE, CREATE ON SCHEMA public TO laborant;

now the user is ready to be used by the backend service.

Backend user service...get that data!

Now that we've got a database, we can use that for our backend application. On server-02, let's clone the code:

git clone https://github.com/lpmi-13/speed-run-backend

Because this backend service is written in typescript, we have to install node and npm to build and run it. I like fnm for this, but you could use nvm if you want to.

Once that's installed and ready (don't forget to run source ~/.bashrc, since you can't restart the shell inside the VM), go ahead and run the following:

npm i

Inspecting the code in src/index.ts, you can see that it looks for an environment variable called DATABASE_URL, and if that's not set, it tries a default connection.

import express from 'express';                                                                                                                                                                  
import { drizzle } from 'drizzle-orm/node-postgres';                             
import { migrate } from 'drizzle-orm/node-postgres/migrator';                    
import { seed } from 'drizzle-seed';                                             
import { Pool } from 'pg';                                                       
import dotenv from 'dotenv';                                                     
import path from 'path';                                                         

import { usersTable } from './db/schema';                                        

dotenv.config();                                                                 

const app = express();                                                           
app.use(express.json());                                                         

const connectionString =                                                         
    process.env.DATABASE_URL ||                                                  
    'postgres://postgres:mypassword@localhost:5432/drizzle';                     
const pool = new Pool({ connectionString });

We have a user and password (the one we created in the last section in our database), so let's create a .env file and set up our DATABASE_URL.

...there's a problem though...what's the hostname going to be? You can't use localhost, because the database isn't running on the same VM.

That's easy enough to fix, let's jump back on server-03 and find out what the IP address is (hint: they will all be 172.X.X.X, because that's the virtual subnet they get run in).

You can run either ip addr or good ole ifconfig and find the address for the eth0 interface that starts with 172..., that's the IP address of server-03.

Now with that information, you can fill in the connection string:

echo 'DATABASE_URL=postgres://laborant:laborant@172.16.0.4:5432/drizzle' > .env

We're specifying the drizzle database, since that's the one the application is configured to try and connect to, but if you're feeling adventurous, go ahead and try with your own custom-defined database.

Once that's all configured, let's build and run the application!

npm run build
node dist/index.js

OH NO! Our first big problem!

laborant@server-02:speed-run-backend$ node dist/index.js 
Connecting to PostgreSQL database...
Running migrations if needed...
Failed to start server: Error: connect ECONNREFUSED 172.16.0.4:5432
    at /home/laborant/speed-run-backend/node_modules/pg-pool/index.js:45:11
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async PgDialect.migrate (/home/laborant/speed-run-backend/node_modules/drizzle-orm/pg-core/dialect.cjs:56:5)
    at async migrate (/home/laborant/speed-run-backend/node_modules/drizzle-orm/node-postgres/migrator.cjs:27:3)
    at async startServer (/home/laborant/speed-run-backend/dist/index.js:38:9) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '172.16.0.4',
  port: 5432
}

We can't connect to the database. Let's debug that 🕵️...

With a quick netcat command, we can see that the port is not accepting connections

laborant@server-02:speed-run-backend$ nc -vz 172.16.0.4 5432
nc: connect to 172.16.0.4 port 5432 (tcp) failed: Connection refused

And the reason is simple enough. We didn't update the postgres config to accept incoming connections from outside its own VM 💡.

Let's update that now.

Back on server-03, we want to edit /etc/postgresql/16/main/pg_hba.conf to change the following line:

host    all             all             127.0.0.1/32            scram-sha-256

into this

host    all             all             0.0.0.0/0            scram-sha-256

That basically means "instead of just listening for connections from inside the VM, listen to connections trying from outside the VM". See? Easy networking fix!

...but we're not done yet. We still need to tweak one more bit of config. Let's also edit /etc/postgresql/16/main/postgresql.conf to change the following line:

#listen_addresses = 'localhost'   # what IP address(es) to listen on;

into this

listen_addresses = '*'   # what IP address(es) to listen on;

make sure to remove the initial #, which turns the above line into a comment, as well as update the listen_addresses 💡

The above is a similar update to tell the database that instead of just listening for connections from inside the VM on localhost, we want to listen for connections from outside.

To apply these configuration changes, we need a full restart of the database, so we can run:

sudo systemctl restart postgresql

And now let's try our netcat test from server-02 again, and we should see:

laborant@server-02:speed-run-backend$ nc -vz 172.16.0.4 5432
Connection to 172.16.0.4 5432 port [tcp/postgresql] succeeded!

Great, now we're ready to try to run the application again.

we could run this as a background process, but we're not going to keep this playground around very long, so we're not doing anything fancy like setting up a systemd service or anything.

laborant@server-02:speed-run-backend$ node dist/index.js
Connecting to PostgreSQL database...
Running migrations if needed...
Migrations completed successfully
Database seeded with 10 users
Server running on port 4000

🎉 our server is running and connected to the database!

If you'd like to confirm it's running fine, let's send a request to its endpoint...FROM ANOTHER VM!!!! (so exciting!)

Frontend user service...show that data!

Just like we planned in the previous step, let's head over to server-01 and connect to the backend server and query the list of users:

curl 172.16.0.3:4000/users | jq

we're just using jq because it comes pre-installed on these VMs, and it makes the output easier to read.

Awesome, we can see the users data coming through from the backend server

laborant@server-01:~$ curl 172.16.0.3:4000/users | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   815  100   815    0     0  85411      0 --:--:-- --:--:-- --:--:-- 90555
[
  {
    "id": 1,
    "name": "Mouhamed",
    "age": -118655047,
    "email": "motivational_shakeia@outlook.com"
  },
  {
    "id": 2,
    "name": "Gisele",
    "age": 1415217029,
    "email": "bronchial_myrtice@orange.fr"
  },
  {
    "id": 3,
    "name": "Garvin",
    "age": -1302366560,
    "email": "logical_mariacristina@live.com"
  },
  {
    "id": 4,
    "name": "Klay",
    "age": 1243263009,
    "email": "blameless_annalyn@web.de"
  },

...

ignore the ludicrous ages, this post isn't gonna cover a data-cleaning pipeline...

So we have a working backend server getting data from a database...but how do we get that to load in the browser for a user?

You might think that you could just clone the frontend code and run the dev server with npm start, but you're gonna have all kinds of problems with binding to 0.0.0.0/0 (remember that means "listens for outside connections") as well as host headers and general frontend-y types of issues.

...but don't worry! We're gonna use Nginx to do what Nginx does (serve bundled frontend code to users and redirect requests to the backend server).

So let's first clone the frontend code:

git clone https://github.com/lpmi-13/speed-run-frontend

Now we can do the same trick of installing fnm/nvm then building the frontend assets.

...previous step of installing fnm or nvm...
npm i
npm run build

The assets (here, we mean html/js/css) are now located in a build/ directory, and we just need to put those in a place where Nginx knows how to find them.

Let's also install Nginx, since we haven't done that yet:

sudo apt update && sudo apt install -y nginx
sudo service nginx start

You'll know that Nginx is up and running since it listens on localhost port 80 by default, so you should be able to run

laborant@server-01:speed-run-frontend$ curl localhost
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

And see the delightful Nginx html coming back.

Now we need to copy over the files in the build/ directory to an Nginx location.

sudo rm -rf /var/www/html/*
sudo cp -r build/* /var/www/html

You'll know it's copied correctly because you can now re-run the curl command and you'll see the react html coming back

laborant@server-01:speed-run-frontend$ curl localhost
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="/favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="Web site created using create-react-app"/><link rel="apple-touch-icon" href="/logo192.png"/><link rel="manifest" href="/manifest.json"/><title>React App</title><script defer="defer" src="/static/js/main.d1e5e332.js"></script><link href="/static/css/main.e6c13ad2.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>

We also need to update the Nginx config a bit so that it sends the requests made by the browser to the correct backend (the request will come into Nginx first running on server-01, which will pass it on to the backend running on server-02)

Now we can cheat again (since I have a working nginx config in the frontend code repo we cloned earlier), and copy it into the right place:

sudo cp nginx/nginx.conf /etc/nginx/nginx.conf

You can take a look at it yourself if you want, it's very basic:

user www-data;
worker_processes auto;
pid /run/nginx.pid;
error_log /var/log/nginx/error.log;
include /etc/nginx/modules-enabled/*.conf;

events {
    worker_connections 1024;
}

http {

    server {
        listen 80; 

        root /var/www/html;

        # Serves the frontend html/js on page load
        location / { 
            try_files $uri $uri/ /index.html;
        }

        # proxy the requests for the backend to the other server
        # the domain for server-02 is resolvable by the hostname
        location /users {
            proxy_pass http://server-02:4000;
            proxy_http_version 1.1;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
}

It basically says "listen for connections on port 80, and send the html/javascript for the root domain, and also forward any requests with /users at the end to the other server on port 4000."

Once that's all copied, go ahead and reload the config with sudo service nginx reload (I forgot this step in the demo and had to quickly debug it).

So we've got the index.html in the correct place, we just need to expose it to the world.

This is one of the best parts of Iximiuz Labs...you can expose a port directly and get a nice public URL.

In the upper right of the UI, there's a little box icon with an arrow coming out of it...click that, and type in 80 for the port (the default nginx port).

and BOOM! We've got users showing up in the browser

Wrapping up

Actually deploying and connecting real things (even if they're temporary and disappear when you close the playground) is the heart of DevOps.

Ideally, there would also be a bit of automation in there, but in this speedrun, the whole idea was to do something super super simple and connect it across independent VMs.

It would be a great addition to add in some database replication or caching or load balancing or...

...but that's probably gonna be a different speed run.

Remote Homelab Admin with Tailscale

Adam Leskis — Thu, 26 Jun 2025 21:02:45 +0000

For a video version of me walking through this is you can watch here.

Tailscale is a super cool networking tool, and Iximiuz Labs Playgrounds are a great place to run a Remote Homelab, so I thought I'd smash em together and make a super cool tech sandwich!

Author's note: This particular playground will only run on the Iximiuz Labs premium tier, but you can sign up for a tailscale account and play with your own VMs for free.

Playground direct link

Source config for the playground.

The setup

To make things at least a little authentic, I replicated a very basic homelab running two main services:

home assistant
jellyfin

and two monitoring components:

grafana
prometheus

along with tailscale on each of the nodes to allow me to connect securely via my phone (which I do via termux)

all inside of a multi-node k3s cluster. The graphic below shows these components and how they're deployed across the various 4 VMs

The cool networking part

The TL;DR of tailscale is that once you connect running nodes to your tailnet, they can find each other by exchanging public keys with the centralized tailscale key servers, and using UDP holepunching to find out their public IP addresses.

The traffic heading out of a node first goes to the virtual network interface, where the wireguard process is running and signs the messages with the public keys of the other nodes they send traffic to. Then it gets forwarded onto the physical network interface with the IP address of the destination node (found via UDP holepunching earlier).

It's all very slick and I've found it to be a great UX. If you want to read more in depth about how it all works, this is a good article.

And so if your phone has also joined the tailnet, it can communicate directly with the VMs inside the homelab (or in this case, remotelab) with end-to-end encrypted tunnels straight through to the private subnets behind NAT!

Your servers are never exposed to the public internet.

The scenario

I wanted to demo an authentic scenario to show off the capabilities, and so I did a bunch of work up front so I could crash one of the services (jellyfin) and then bring it back to life, all connecting via my phone over the secure tailnet.

...okay, so I cheat a little bit and crash the server from the Iximiuz Labs web interface, but you probably wouldn't run that command from your phone anyway, so I'm gonna claim it's still totally authentic ™️.

First, we set up our slideshow of http status cats via jellyfin...

Then we start filling up the server with fake logs

...and then all of a sudden, the slideshow turns to sadness!

OH NO! The cats have crashed and we're far away from home...time to debug remotely!

Luckily, we have a (very basic) monitoring stack set up with Prometheus and Grafana, so we can see if that tells us anything interesting...

Ah ha! It looks like one of the nodes is down, and we can also see that the file usage spiked to 100%...that's probably what crashed the server :lightbulb:

Let's dive into the node and see if we can work out where all the logs are filling up the server (we've already stopped the log generation script via the Iximiuz Labs web UI, so we don't need to worry about that anymore).

Lets first connect to the node with termux. Grab either the magic DNS or just the private IP of the node and use that when you run

ssh phoneuser@IP_ADDRESS_FROM_TAILSCALE_UI

Note, this requires having a valid authorized_keys file in your ssh config for the phoneuser user on the node, containing the public key from a keypair generated on your phone.

Then we can run this command to see where all the disk bloat is

sudo du -h --max-depth=1 /

Looks like it's the /var directory. Looking further down the directory tree, we can see the culprit is in /var/log/fake_logs.

We found all the (fake) logs...let's clean them up. Now run sudo rm -rf /var/log/fake_logs/*.

Now that there's free space on the server, let's restart the k3s-agent (the thing that crashed when we ran out of disk space).

sudo systemctl restart k3s-agent

and we probably also want to clean up all the left over crud in the pod namespace...

k delete po -l app=jellyfin
k delete po -l app=node-exporter

since these are stateless services that mount in their config, they're completely fine to be deleted and recreated

And now we can see that the node-exporter is back up and the pod has returned to normal functioning. Our jellyfin server is back up, and we can continue admiring how cats' facial expressions never change, and that's why they're so funny.

Wrapping up

Both Tailscale and Iximiuz Labs are very fun and approachable tools to help us learn about different homelab components, and they let us do it (basically) for free! What an internet!

Pre-shaved Y-Axis: the case for preconfigured Rstudio workshop environments

Adam Leskis — Sun, 16 Feb 2025 23:08:12 +0000

You might think that not being an avid user of either Rstudio or the R programming language in general makes me unqualified to write about them, and you would be 100% correct; but it's my blog, so I'm gonna do it anyway!

Now that we're past my extremely obscure joke in the title related to data viz in R and yak shaving, I will tell you about how I had a lot of fun creating a prepackaged playground on labs.iximiuz.com to follow along with a recent paper presenting a tutorial for visualizing linguistics data using the R package ggplot2.

I'd like to briefly discuss why I did it, and also how, in case you are interested in doing something similar.

Why did I do this?

While I may not be accustomed to using R for...well...anything, really...what I do have loads of experience in is listening to friends in the academic realm talk about how difficult it is to work with.

The dependency management is very difficult, it's nontrivial to upgrade Rstudio to a version that works with the packages that are needed for a given task, and even just setting up working environments on students' personal devices to take part in class work can be a multi-day job.

Imagine it didn't have to be this way...

Imagine that students in a workshop could open up any modern web browser, load the URL, and click one button to have a 100% deterministic working version of Rstudio loaded with all the packages needed for the activity...

Well, imagine no more (and you already knew this was possible if you read my last post), this is possible right now!

The very cool platform on labs.iximiuz.com has playgrounds that can be customized for this exact usecase and shared publicly. I know because I made one. Given the ability to run basically whatever you want in the ephemeral environments there, this is the perfect platform to create on-demand environments pre-configured with almost any software your heart desires.

To borrow a term from the testing and security domains, we can "shift left" and get the user to actually doing the thing they wanna do faster and more effectively. This is devops for education, and I'm here for it!

How did I do this?

Because the Iximiuz platform is basically a free-form sandbox, the most difficult part was figuring out how to run the appropriate code in there. I knew it could be done, I just needed to connect all the fun lego blocks.

I started with the rocker/tidyverse container image, pinned to base R version 4.4.1 (this was originally 4.4.2 since that's the latest, but I updated it after noticing the author specifying in GitHub that the base version of R they used was 4.4.1).

After firing that up locally with:

docker run -it --rm -p 127.0.0.1:8787:8787 -e PASSWORD=rstudio rocker/tidyverse

I had a running version of Rstudio at localhost:8787. Then it was a matter of just running the commands from the tutorial, seeing what broke and then fixing it by installing stuff via an iteratively growing Dockerfile.

Once that was all set, all I had to do was build the container locally and push to a public GitHub container registry, then add one line to customize a linux playground with Docker pre-installed to run my container.

But that's not all!

There's a super slick feature of the platform where you can basically create a new tab that accesses a port from the running container, and this can be configured to be available on startup.

So the entire flow for the user is

Click "Start" for the playground
Wait for playground to be ready
Click "Rstudio" tab
Log in with the supplied username/password
Start running commands from the tutorial

If you're interested in seeing the complete Dockerfile, that's here.

This is a completely disposable learning environment where the users can have a running start without worrying about any annoying configuration challenges, and when they're done, they can just close the browser tab. They don't even need to know that Docker is a thing!

The yak is fully pre-shaved! The future is wow!

Turning Markdown into Learning: publishing a challenge on labs.iximiuz.com

Adam Leskis — Tue, 14 Jan 2025 20:26:46 +0000

This post will detail my experience (which was a very good one) of authoring my first challenge on the very cool new(ish) labs.iximiuz.com platform.

...but first, a short digression into why it's so cool, and why I'm excited about it!

It used to be that to create an entire learning environment, you had to create the whole thing yourself. If you wanted, like I did, to make a mongo dojo, you had to set up the various VMs with something like vagrant, and then manage the machine states and the networking, and the storage, etc etc...

But now, for particular learning cases, this platform makes it ridiculously straightforward. It's a very simple/easy way to get console access directly in the browser to running linux VMs in the cloud. Since it uses firecracker microVM's under the hood, it boots very fast as is ready to learn with minimal wait time. The accompanying content is also very comprehensive and easy to read.

(For the full description of the tech stack that powers the platform, check out this post.)

So instead of needing to know about how to orchestrate all the VMs behind the scenes and wire that up to a very slick looking frontend with networking all smoothed out and support various users starting and stopping playgrounds and challenges constantly...you can just write yaml and markdown, and the platform can convert it into a working learning environment.

Anatomy of a Challenge

If you'd like to see for yourself what all the challenges look like, you can just go there now. There are loads of free ones, as well as a premium tier with all the content and higher usage limits for things like CPU and bandwidth. Also, the platform even works (fairly well) in a mobile UI (though turn off predictive text and spelling fixes for your keyboard or it's much harder).

...for a bit more of a descriptive introduction to the challenges, read on...

Each challenge is based on one of the existing playgrounds (or can even use a custom playground you design yourself), and so can be situated in environments like a single node k3s cluster or a multi-node VM configuration with Docker preinstalled.

(for a full list of playgrounds, check out this page)

The challenge starts with some description (and ideally also a nice graphic) to set the stage for what the user needs to do. The platform handles all the formatting and styling, so all you need to do is bring the text, and it'll automatically show up looking very sharp.

Each step can have associated hints that can be exposed if you're a bit stuck. Often these are links to the docs, but can also be examples of commands to try running.

The UI has verification elements that go green when you've satisfied the requirements for that particular step. These are backed by simple bash scripts in the markdown file to check, for example, whether a certain number of pods are running with given labels, or whether two pods are able to reach each other over the network.

One last aspect that's available if you need it, is init tasks that can run before the user is presented with the terminal to play around in. These would be for installing some specific software that isn't available in the default playground base, or potentially generating some sample data that needs to be available when the user starts the challenge.

My CKA Challenge Creation

Even though I don't have that much experience with kubernetes, it's still one of the most popular technologies in the DevOps space, and a lot of people are interested in it. So I figured I would enhance my own knowledge of how networking policies are configured and used in those systems.

And if you've ever read this blog before, you'll know that my attitude is that the only thing better than building to learn is building to teach...so this platform is the perfect opportunity to do that.

A very nice companion piece of tech for the platform is the official Iximiuz Labs CLI. You can use it to easily create new challenges, update existing challenges, and generally do cool stuff from the comfort of your terminal. The obvious first step in this case was creating new content via the CLI.

The default file autogenerated via the CLI has lots of helpful comments, and can guide you towards how to use the different fields available for configuration:

(If you want to see what the markdown for my finished challenge looks like, that's here)

There are a few things to be aware of, though, that tripped me up the first time:

the categories are intended as higher level concepts like "kubernetes" that can be used to filter content in the platform UI, whereas the tagz are more for specific topics (like CKA) that should not overlap with the categories

the machine and tab names need to be values from the existing system, and if you try to configure something custom, it (currently) has the risk of breaking things.

if you expect to have some long-running tasks (or some very short-running ones), you might need to tweak the default timeouts for those in your configuration. I've seen a couple times now where I had tasks that didn't play nice with the default timeouts and I had to either explicitly make them shorter or longer to get things working.

Iterating on the content

The CLI isn't only able to generate/push content...it also has the ability to hot reload changes on save so you can get very fast feedback loops while authoring content. It's a total gamechanger to not need to wait for a daemon running somewhere on a server to pick up changes to see what you've updated.

I found it very smooth to sketch out my ideas, see them running immediately in the environment, and then fix things that weren't yet quite right.

And you can join the Discord server if you want to report a bug or request a feature. Ivan is very helpful and very interested in making sure learners (and authors!) have a positive experience on the platform.

Some things I learned along the way to pass on:

it's better to avoid using Dockerhub images in your challenges, since the rate limiting can be a bit aggressive. Much preferred is ghcr.io or another more robust container registry that allows public access.

Instead of batching together related verification tasks into one check, it's a much nicer learner experience to separate them into discrete tasks that give iterative feedback more quickly as the user is completing the challenge. For example, in my challenge I originally had all the checks for correct labels in one big scripted check, though it was much nicer to split that into 3 checks and 3 pieces of feedback.

I definitely recommend https://excalidraw.com/ for getting simple diagrams together (in general, and for the purposes of putting in challenges). It (somewhat) hides my absolutely awful design skills.

And here's the finished product, my very first challenge on this very cool platform!

https://labs.iximiuz.com/challenges/cka-network-policies-between-deployments

Scary Git Commands Made Easy with Terminal-based Micromaterials

Adam Leskis — Thu, 10 Sep 2020 09:57:21 +0000

rebase, reflog, submodule...these git commands can seem very scary and unapproachable when you first hear about them. They definitely seemed that way to me.

Since I had been avoiding rebase for that very reason, the first time I actually need to use it in my professional life as a developer was very stressful, and thankfully I had a much more experienced git user to pair with while doing it.

It definitely DOES NOT have to be that way.

Existing resources

There are an overwhelming number of free resources online to help learn about every conceivable git command, so why add more?

While there are many informational resources (such as the excellent documentation on rebase), these can tell you a lot "about" git, but they don't necessary tell you about the process of actually using it.

In addition, there are also numerous tutorials that do actually guide users through the process of using these commands. Though by necessity, they tend to be very broad, speaking about rebasing in general rather than presenting one very specific objective detailing the context around why you would want to do something like rebase.

I'd add that I'm sure a lot of git users have learned perfectly well using these kinds of resources, and if they're adequate for your purposes, by all means, go ahead and use what works.

Personally, I always found them to be a bit lacking, so I decided to just make the stuff that I wish existed.

Micromaterials for git

My aim was to create some very simple learning materials, which were very specific, open-source, and fast. I also wanted to focus on learning by doing, involving actually typing out the git commands in your terminal to see what happens.

Unlike a situation involving a time-sensitive project at work, using these commands in your own local environment is very low-pressure, you can go at your own pace, and making a mistake is no big deal.

for rebase, I made https://github.com/lpmi-13/rebasic, which asks you to just remove two commit messages while keeping the commits themselves. I do this all the time after finishing work on a feature branch and before opening a pull request.
for reflog, I made https://github.com/lpmi-13/reflog-power, which simulates deleting a local branch and then bringing it back to life. This is also something that I didn't know was possible until I accidentally deleted the local branch with all my work, and then I had to get it back.
for submodule I made both https://github.com/lpmi-13/submodz and https://github.com/lpmi-13/snubmodule. The first one involves a very basic scenario of updating a submodule, and the second one deals with moving a submodule into your main repository. I don't do much with submodules, but I did have to update one in my job recently, and it took me a long time to figure out what to do.

Whatever works

In such a wide field of software development, involving so many different types of people, there can never be one learning resource that works for everyone. Whatever works best for you is the correct resource to use. Successful learners use learning strategies, and the most successful learners select different learning strategies based on the situation.

I'm hoping that these micromaterials can add to the wealth of resources online to help people learn about using git, and also start the wider conversation on what other micromaterials could help people develop their tech skills.