Forem: Morten Olsen

Hyperconnect: A Theory of Seamless Device Mesh

Morten Olsen — Tue, 13 Jan 2026 16:48:26 +0000

Apple's "Continuity" features feel like magic. You copy text on your phone and paste it on your Mac. Your watch unlocks your laptop. It just works. But it only works because Apple owns the entire vertical stack.

For the rest of us living outside the walled garden, device communication is stuck in the 90s. We are still manually pairing Bluetooth or debugging local IP addresses. Why is it harder to send 10 bytes of data to a device three feet away than it is to stream 4K video from a server on the other side of the planet?

I have "ecosystem envy," and I think it’s time we fixed it. I want to build a service mesh that treats Bluetooth, WiFi, and LTE as mere implementation details, not hard constraints.

"But doesn't Tailscale solve this?" you might ask. Tailscale (and WireGuard) are brilliant technologies that solve the connectivity problem by creating a secure overlay network at Layer 3 (IP). However, they don't solve the continuity problem. They assume the physical link exists. They can't ask the radio firmware to scan for BLE beacons because the WiFi signal is getting weak.

Similarly, projects like libp2p (used by IPFS) do an excellent job of abstracting transport layers for developers, but they function more as a library for building P2P apps rather than a system-wide mesh that handles your text messages and file transfers transparently. I want something that sits deeper—between the OS and the network.

Furthermore, I have a strong distaste for the "walled garden" approach. I don't believe you should have to buy every device from a single manufacturer just to get them to talk to each other reliably. An open-source, vendor-neutral framework would unlock this kind of "hyperconnectivity" for the maker community, allowing us to mix and match hardware without sacrificing that magical user experience.

So, I’ve been toying with a concept I call Hyperconnect.

If a person is hyperconnected, it generally means they are available on multiple different channels simultaneously. I want to build a framework that allows my devices to do the same.

The Big Idea

The core idea is to build a framework where all your personal devices create a device mesh (distinct from the backend "service mesh" concept often associated with Kubernetes) that can span different protocols. This mesh maintains a live service graph and figures out how to relay messages from one device to another, using different strategies to do so effectively.

This isn't just about failover; it's about context-aware routing.

The Architecture

To make this work without turning into a security nightmare, we need a few foundational blocks:

1. Passports (Identity)

We can't just let any device talk to the mesh. The user starts by creating an authority private key. This key is used to sign "Passports" for devices. A passport is a cryptographic way for a device to prove, "I belong to Morten, and I am allowed in the mesh."

Crucially, this passport also includes a signed public key for the device. This allows for end-to-end encryption between any two nodes. Even if traffic is relayed through a third device (like the phone), the intermediary cannot read the payload.

2. Lighthouses (Discovery)

How do isolated devices find each other? We need a Lighthouse. This is likely a cloud server or a stable home server with a public IP. When a device connects for the first time, it gets introduced through the Lighthouse to find other nodes and build up its local service graph. While an always-available service helps established devices reconnect, the goal is to be as peer-to-peer (P2P) as possible.

3. The Service Graph

Every device advertises the different ways to communicate with it. It might say: "I am available via mDNS on local LAN, I have an LTE modem accessible via this IP, and I accept Bluetooth LE connections"

4. Topology & Gossip

Once introduced, the Lighthouse steps back. The goal is a resilient peer-to-peer network. However, a naive "spaghetti mesh" where everyone gossips with everyone is a battery killer.

Instead, the network forms a tiered topology:

Anchor Nodes: Mains-powered devices (NAS, Desktop) maintain the full Service Graph and gossip updates frequently. They act as the stable backbone.
Leaf Nodes: Battery-constrained devices (Watch, Sensor) connect primarily to Anchor Nodes. They typically do not route traffic for others unless acting as a specific bridge (like a Phone acting as an LTE relay).

When a device rejoins the network (e.g., coming home), it doesn't need to check in with the Lighthouse. It simply pings the first known peer it sees (e.g., the Watch sees the Phone). If that peer is authorized, they sync the graph directly. The Lighthouse is merely a fallback for "cold" starts or when no known local peers are visible.

The Scenario: A Smartwatch in the Wild

To explain how this works in practice, let's look at a specific scenario. Imagine I have a custom smartwatch that connects to a service on my desktop computer at home to track my steps.

Stage 1: At Home

Initially, the watch is connected at home. It publishes its network IP using mDNS. My desktop sees it on the local network. Since the framework prioritizes bandwidth and low latency, the two devices communicate directly over IP.

The watch also knows it has an LTE modem, and it advertises to the Lighthouse that it is reachable there. It also advertises to my Phone that it's available via Bluetooth. The Service Graph is fully populated.

Stage 2: Leaving the House

Now, it's time to head out. I leave the house, and the local WiFi connection drops.

This is where the framework needs to be smart. It must have a built-in mechanism to handle backpressure. For the few seconds I am in the driveway between networks, packets aren't lost; they are captured in a ring buffer (up to a safe memory limit), waiting for the mesh to heal.

The Connection Owner (in this case, my desktop, chosen because it has the most compute power and no battery constraints) looks at the graph. It sees the WiFi path is dead. It checks for alternatives. It sees the Watch advertised P2P capabilities over LTE.

The desktop re-establishes the connection over LTE. The buffer flushes. No packets dropped, just slightly delayed.

Stage 3: The Metro (The Relay)

I head down into the metro. The LTE coverage is spotty, and the smartwatch's tiny antenna can't hold a stable connection to the cell tower. The connection drops again. The buffer starts to fill.

The desktop looks at the Service Graph. Direct IP is gone. LTE is gone. But, it sees that the Phone is currently online via 5G (better antenna) and that the Phone has previously reported a Bluetooth relationship with the Watch.

The desktop contacts the Phone: "Hey, I need a tunnel to the Watch."

The Phone acts as a relay. It establishes a Bluetooth Low Energy link to the Watch. The data path is now Desktop ↔ Internet ↔ Phone ↔ Bluetooth ↔ Watch.

The step counter updates. The mesh survives.

Beyond the Basics: Strategy and Characteristics

So far, I've mostly talked about the "big three": WiFi, Bluetooth, and LTE. But the real power of a personal mesh comes when we start integrating niche protocols that are usually siloed.

Expanding the Protocol Stack

Imagine adding Zigbee or Thread (via Matter) to the mix. These low-power mesh protocols are perfect for stationary home devices. Suddenly, your lightbulbs could act as relay nodes for your smartwatch when you are in the garden, extending the mesh's reach without needing a full WiFi signal.

Or consider LoRa (Long Range). I could have a LoRa node on my roof and one in my car. Even if I park three blocks away and the car has no LTE signal, it could potentially ping my home node to report its battery status or location. The bandwidth is tiny, but the range is incredible.

Connection Characteristics

However, just knowing that a link exists isn't enough. The mesh needs to know the quality and cost of that link. We need to attach metadata to every edge in our service graph.

I believe we need to track at least four dimensions:

Bandwidth: Can this pipe handle a 1080p stream, or will it choke on a JSON payload?
Latency: Is this a snappy local WiFi hop (5ms), or a satellite uplink (600ms)?
Energy Cost: This is critical for battery-powered devices. Waking up the WiFi radio on an ESP32 is expensive. Sending a packet via BLE or Zigbee is much cheaper.
Monetary Cost: Am I on unlimited home fiber, or am I roaming on a metered LTE connection in Switzerland?

Smart Routing Strategies

Once the mesh understands these characteristics, the routing logic becomes fascinating. It stops being about "shortest path" and starts being about "optimal strategy."

The "Netflix" Strategy: If I am trying to stream a video file from my NAS to my tablet, the mesh should optimize for Bandwidth. It should aggressively prefer WiFi Direct or wired Ethernet, even if it takes a few seconds to negotiate the handshake.
The "Whisper" Strategy: If a temperature sensor needs to report a reading every minute, the mesh should optimize for Energy. It should route through the nearest Zigbee node, avoiding the power-hungry WiFi radio entirely.
The "Emergency" Strategy: If a smoke detector goes off, we don't care about energy or money. The mesh should blast the alert out over every available channel—WiFi, LTE, LoRa, Bluetooth—to ensure the message gets through to me.

The Developer Experience

As a developer, I don't want to manage sockets or handle Bluetooth pairing in my application code. I want a high-level intent-based API.

It might look something like this for a pub/sub pattern:

import { mesh } from '@hyperconnect/sdk';

// Subscribe to temperature updates from any node
mesh.subscribe('sensors/temp', (msg) => {
  console.log(`Received from ${msg.source}: ${msg.payload}`);
});

// Publish a command with constraints
await mesh.publish('controls/lights', { state: 'ON' }, {
  strategy: 'energy_efficient', // Prefer Zigbee/BLE
  scope: 'local_network'        // Don't route over LTE
});

For use cases that require continuous data flow (like video streaming) or legacy application support, the mesh could offer a standard stream interface that handles the underlying transport switching transparently:

// Stream-based API (Socket-compatible)
// 'my-nas-server' resolves to a Public Key from the Passport
const stream = await mesh.connect('my-nas-server', 8080, {
  strategy: 'high_bandwidth'
});

// Looks just like a standard Node.js socket
stream.write(new Uint8Array([0x01, 0x02]));
stream.on('data', (chunk) => console.log(chunk));

There are other massive topics to cover here—like handling delegated guest access (a concept I call 'Visas') or how this becomes the perfect transport layer for Local-First (CRDT) apps—but those deserve their own articles. For now, let's look at the downsides.

But first, the downsides

I am painting a rosy picture here, but I want to be honest about the challenges.

Battery Life: Maintaining multiple radio states and constantly updating a service graph is expensive. A protocol like this needs to be aggressive about sleeping. The "advertising" phase needs to be incredibly lightweight.

Complexity: Implementing backpressure handling across different transport layers is hard. TCP handles some of this, but when you are switching from a UDP stream on WiFi to a BLE characteristic, you are effectively rewriting the transport layer logic.

Security: While end-to-end encryption (enabled by the keys in the Passport) solves the privacy issue of relaying, implementing a secure cryptographic protocol is notoriously difficult. Ideally, we would need to implement forward secrecy to ensure that if a device key is compromised, past traffic remains secure. That is a heavy lift for a weekend project.

Platform Restrictions: Finally, there is the reality of the hardware we carry. Efficiently managing radio handovers requires low-level system access. On open hardware like a Raspberry Pi, this is accessible. However, on consumer devices like iPhones or Android phones, the OS creates a sandbox that restricts direct control over the radios. An app trying to manually toggle network interfaces or scan aggressively in the background will likely be killed by the OS to save battery or prevent background surveillance (like tracking your location via WiFi SSIDs).

A Call to Build

This is a project I have long wanted to build, but never found the time to.

I am posting this idea hoping it might inspire someone else to take a crack at it. Or, perhaps, this will just serve as documentation for my future self if I ever clear my backlog enough to tackle it.

The dream of a truly hyperconnected personal mesh is vivid. We have the radios, we have the bandwidth, and we have the hardware. We just need the software glue to make it stick.

The Clubhouse Protocol: A Thought Experiment in Distributed Governance

Morten Olsen — Mon, 12 Jan 2026 13:24:33 +0000

I am a huge admirer of the open-source ethos. There is something magical about how thousands of strangers can self-organize to build world-changing software like Linux or Kubernetes. These communities thrive on rough consensus, shared goals, and the freedom to fork if visions diverge.

But there is a disconnect. While we have mastered distributed collaboration for our code (Git), the tools we use to talk to each other are still stuck in a rigid, hierarchical past.

Even in the healthiest, most democratic Discord server or Slack workspace, the software forces a power imbalance. Technically, one person owns the database, and one person holds the keys. The community remains together because of trust, yes—but the architecture treats it like a dictatorship.

The Problem: Benevolent Dictatorships

Most online communities I am part of are benevolent. The admins are friends, the rules are fair, and everyone gets along. But this peace exists despite the software, not because of it.

Under the hood, our current platforms rely on a "superuser" model. One account has the DELETE privilege. One account pays the bill. One account owns the data.

This works fine until it doesn't. We have seen it happen with Reddit API changes, Discord server deletions, or just a simple falling out between founders. When the social contract breaks, the one with the technical keys wins. Always.

I call this experiment The Clubhouse Protocol. It is an attempt to fix this alignment—to create a "Constitution-as-Code" where the social rules are enforced by cryptography, making the community itself the true owner of the platform.

This post is part of a series of ideas from my backlog—projects I have wanted to build but simply haven't found the time for. I am sharing them now in the hope that someone else becomes inspired, or at the very least, as a mental note to myself if I ever find the time (and skills) to pursue them.

Disclaimer: I am not a cryptographer. The architecture below is a napkin sketch designed to explore the social dynamics of such a system. The security mechanisms described (especially the encryption ratcheting) are illustrative and would need a serious audit by someone who actually knows what they are doing before writing a single line of production code.

The Core Concept

In the Clubhouse Protocol, a "Channel" isn't a row in a database table. It is a shared state defined by a JSON document containing the Rules.

These rules define everything:

Who is allowed to post?
Who is allowed to invite others?
What is the voting threshold to change the rules?

Because there is no central server validating your actions, the enforcement happens at the client level. Every participant's client maintains a copy of the rules. If someone tries to post a message that violates the rules (e.g., posting without permission), the other clients simply reject the message as invalid. It effectively doesn't exist.

The Evolution of a Community

To understand why this is powerful, let's look at the lifecycle of a theoretical community.

Stage 1: The Benevolent Dictator

I start a new channel. In the initial rule set, I assign myself as the "Supreme Owner." I am the only one allowed to post, and I am the only one allowed to change the rules.

I invite a few friends. They can read my posts (because they have the keys), but if they try to post, their clients know it's against the rules, so they don't even try.

Stage 2: The Republic

I decide I want a conversation, not a blog. So, I construct a start-vote message.

Proposal: Allow all members to post.
Voting Power: I have 100% of the votes.

I vote "Yes." The motion passes. The rules update. Now, everyone's client accepts messages from any member.

Stage 3: The Peaceful Coup

As the community grows, I want to step back. I propose a new rule change:

Proposal: New rule changes require a 51% majority vote from the community.
Proposal: Reduce my personal voting power from 100% to 1 (one person, one vote).

The community votes. It passes.

Suddenly, I am no longer the owner. I am just a member. If I try to ban someone or revert the rules, the community's clients will reject my command because I no longer have the cryptographic authority to do so. The community has effectively seized the means of production (of rules).

The Architecture

How do we build this without a central server?

1. The Message Chain

We need a way to ensure order and prevent tampering.

A channel starts with three random strings: an ID_SEED, a SECRET_SEED, and a "Genesis ID" (a fictional previous message ID).
Each message ID is generated by HMAC'ing the previous message ID with the ID_SEED. This creates a predictable, verifiable chain of IDs.
The encryption key for the message envelope (metadata) is derived by HMAC'ing the specific Message ID with the SECRET_SEED.

This means if you know the seeds, you can calculate the ID of the next message that should appear. You can essentially "subscribe" to the future.

2. The Envelope & Message Types

The protocol uses two layers of encryption to separate governance from content.

The Outer Layer (Channel State):
This layer is encrypted with the key derived from the SECRET_SEED. It contains the message metadata, but crucially, it also contains checksums of the current "political reality":

Hash of the current Rules
Hash of the Member List
Hash of active Votes

This forces consensus. If my client thinks "Alice" is banned, but your client thinks she is a member, our hashes won't match, and the chain will reject the message.

The Inner Layer (The Payload):
Inside the envelope, the message has a specific type:

start-vote / cast-vote: These are visible to everyone in the channel. Governance must be transparent.
mutany: A public declaration of a fork (more on this later).
data: This is the actual chat content. To be efficient, the message payload is encrypted once with a random symmetric key. That key is then encrypted individually for each recipient's public key and attached to the header. This allows the group to remove a member simply by stopping encryption for their key in future messages.

3. Storage Agnosticism

Because the security and ordering are baked into the message chain itself, the transport layer becomes irrelevant.

You could post these encrypted blobs to a dumb PHP forum, an S3 bucket, IPFS, or even a blockchain. The server doesn't need to know what the message is or who sent it; it just needs to store a blob of text at a specific ID.

The Killer Feature: The Mutiny

The most radical idea in this protocol is the Mutiny.

In a standard centralized platform, if 45% of the community disagrees with the direction the mods are taking, they have to leave and start a new empty server.

In the Clubhouse Protocol, they can Fork.

A mutiny message is a special transaction that proposes a new set of rules or a new member list. It cannot be blocked by existing rules.

When a mutiny is declared, it splits the reality of the channel.

Group A (The Loyalists) ignores the mutiny message and continues on the original chain.
Group B (The Mutineers) accepts the mutiny message. Their clients apply the new rules (e.g., removing the tyrannical admin) and continue on a new fork of the chain.

Crucially, history is preserved. Both groups share the entire history of the community up until the fork point. It’s like git branch for social groups. You don't lose your culture; you just take it in a different direction.

Implementation Challenges

As much as I love this concept, there are significant reasons why it doesn't exist yet.

The Sybil Problem: In a system where "one person = one vote," what stops me from generating 1,000 key pairs and voting for myself? The solution lies in the protocol's membership rules. You cannot simply "sign up." An existing member must propose a vote to add your public key to the authorized member list. Until the community votes to accept you, no one will encrypt messages for you, and your votes will be rejected as invalid.

Scalability & The "Header Explosion": The encryption method described above (encrypting the content key for every single recipient) hits a wall fast. If you have 1,000 members and use standard RSA encryption, the header alone would be around 250KB per message. This protocol is designed for "Dunbar Number" sized groups (under 150 people). To support massive communities, you would need to implement something like Sender Keys (used by Signal), where participants share rotating group keys to avoid listing every recipient in every message.

The "Right to be Forgotten": In an immutable, crypto-signed message chain, how do you delete a message? You can't. You can only post a new message saying "Please ignore message #123," but the data remains. This is a privacy nightmare and potentially illegal under GDPR.

Key Management is Hard: If a user loses their private key, they lose their identity and reputation forever. If they get hacked, there is no "Forgot Password" link to reset it.

The Crypto Implementation: As noted in the disclaimer, rolling your own crypto protocol is dangerous. A production version would need to implement proper forward secrecy (like the Signal Protocol) so that if a key is compromised later, all past messages aren't retroactively readable. My simple HMAC chain doesn't provide that.

Why it matters

Even if the Clubhouse Protocol remains a napkin sketch, I think the question it poses is vital: Who owns the rules of our digital spaces?

Right now, the answer is "corporations." But as we move toward more local-first and peer-to-peer software, we have a chance to change that answer to "communities."

We need more experiments in distributed social trust. We need tools that allow groups to govern themselves, to fork when they disagree, and to evolve their rules as they grow.

If you are a cryptographer looking for a side project, feel free to steal this idea. I just want an invite when it launches.

A Small LLM Trick: Giving AI Assistants Long-Term Memory

Morten Olsen — Sat, 10 Jan 2026 16:13:07 +0000

I have a confession to make: I often forget how my own projects work.

It usually happens like this: I spend a weekend building a Proof of Concept, life gets in the way for three weeks, and when I finally come back to it, I’m staring at a folder structure that makes sense to "Past Morten" but is a complete mystery to "Current Morten."

Nowadays, I usually have an AI assistant helping me out. But even they struggle with what I call the "session void." They see the code, but they don't see the intent behind changes made three weeks ago. They learn something during a long chat session, but as soon as you start a new one, that "aha!" moment is gone.

I’ve been experimenting with a new technique to solve this. It’s early days, but the initial results have been promising enough that I wanted to share it. I call it the AGENTS.md trick.

Beyond the System Prompt

Many developers using AI for coding are already familiar with the idea of a project-level prompt or an instructions file. But usually, these are static—you write them once, and they tell the agent how to format code or which library to prefer.

The experiment I'm running is to stop treating AGENTS.md as a static instruction manual and start treating it as long-term memory.

The Strategy: Resemblance of Recall

The goal is to give the agent a set of instructions that forces it to continuously build and maintain an understanding of the project, across sessions and across different agents.

I’ve started adding these core "memory" instructions to an AGENTS.md file in the root of my projects:

Maintain the Source of Truth: Every time the agent makes a significant architectural change or learns something new about the project's "hidden rules," it must update AGENTS.md.
Externalize Discoveries: Any time the agent spends time "exploring" a complex logic flow to understand it, it should write a short summary of that discovery into a new file in ./docs/.
Maintain the Map: It must keep ./docs/index.md updated with a list of these discoveries.
Reference Personal Standards: I point it to a global directory (like ~/prompts/) where I keep my general preferences for things like JavaScript style or testing patterns. The power here is that these standards are cross-project. (Note: This works best if your AI tool—like Cursor or GitHub Copilot—allows you to reference or index files outside the current project root.) If you're in a team, you could host these in a shared git repository and instruct the agent to clone them if the folder is missing. This ensures the agent learns "how we build things" globally, not just "how this one project works."

Why I'm Liking This (Especially for Existing Projects)

LLMs are great at reading what's right in front of them, but they have zero "recall" for things that happened in a different chat thread or a file they haven't opened yet.

By forcing the agent to document its own "aha!" moments, I'm essentially building a bridge between sessions.

Continuity: When I start a new session, the agent reads AGENTS.md, sees the map in index.md, and suddenly has the context of someone who has been working on the project for weeks.
Organic Growth: I don't have to sit down and write "The Big Manual." The documentation grows exactly where the complexity is, because that's where the agent had to spend effort understanding things.
Legacy Code: This has been a lifesaver for older projects. I don't need to document the whole thing upfront. I just tell the agent: "As you figure things out, write it down."

Evolutionary Patterns

One of the coolest side effects of this setup is how it handles evolving standards.

If I decide I want to switch from arrow functions back to standard function declarations, I don't just change my code; I tell the agent. Because the agent has instructions to maintain the memory, it can actually suggest updating my global standards.

I've instructed it that if it notices me consistently deviating from my javascript-writing-style.md, it should ask: "Hey, it looks like you're moving away from arrow functions. Should I update your global pattern file to reflect this?" This keeps my preferences as alive as the code itself.

Early Results

Is it perfect? Not yet. Sometimes agents need a nudge to remember their documentation duties, and I'm still figuring out the best balance to keep the ./docs/ folder from getting cluttered.

But so far, it has drastically reduced the "What was I thinking?" factor for both me and the AI.

The Hidden Benefits: Documentation Gardening

Beyond just "remembering" things for the next chat session, this pattern creates a virtuous cycle for the project's health.

Automated Gardening: Periodically, you can ask an agent to go over the scattered notes in ./docs/ and reformat them into actual, structured project documentation. Since the agent has already captured the technical nuances it needed to work effectively, these docs are often more accurate and detailed than anything a human would write from scratch.
Context for Reviewers: When you open a Pull Request, the documentation changes serve as excellent context for human reviewers. If you’ve introduced a change large enough to document, seeing the agent’s "memory" update alongside the code makes the why behind your changes much more transparent.
Advanced Tip: The Auto-Documenting Hook: For the truly lazy (like me), you can set up a git hook that runs an agent after a commit. It reviews your manual changes and ensures the ./docs/ folder is updated accordingly. This means that even if you bypass the AI for a quick fix, your project's "memory" stays in sync.

The "Memory" Template

If you want to try this out, here is the behavioral template I've been using. Please note: this is a work in progress. I expect to refine these instructions significantly over the coming months.

# Agent Guidelines for this Repository

> **Important**: You are responsible for maintaining the long-term memory of this project.

## 1. Your Behavioral Rules
- **Incremental Documentation**: When you figure out a complex part of the system or a non-obvious relationship between components, create a file in `./docs/` explaining it.
- **Self-Correction**: If you find that the existing documentation in `./docs/` or this file is out of date based on the current code, fix it immediately.
- **Index Maintenance**: Ensure `./docs/index.md` always points to all relevant documentation so it's easy to get an overview.
- **Personal Standards**: Before starting significant work, refer to `~/prompts/index.md` to see my preferences for `javascript-writing-style.md`, `good-testing-patterns.md`, etc.
- **Evolutionary Feedback**: If you notice me consistently requesting or writing code that contradicts these standards, ask if you should update the global files in `~/prompts/` to match the new pattern.

## 2. Project Context
[User or Agent: Briefly describe the current state and tech stack here to give the agent an immediate starting point]

Making AI a Partner, Not a Guest

The difference between a tool that sees your code for the first time every morning and one that "remembers" your previous architectural decisions is massive.

It’s a simple experiment, but it's one that anyone can try today. It turns the agent from a temporary guest in your codebase into a partner that helps you maintain a rolling understanding of what you are actually building.

Would this work for you? I don't know yet, but I'm excited to keep refining it.

A meta talk about Git strategies

Morten Olsen — Mon, 05 Dec 2022 13:27:54 +0000

Let me start with a (semi) fictional story; It is Friday, and you and your team have spent the last five weeks working on this excellent new feature. You have written a bunch of unit tests to ensure that you maintain your project's impressive 100% test coverage, and both you, your product owner and the QA testers have all verified that everything is tip-top and ready to go for the launch!
You hit the big "Deploy" button.
3-2-1
Success! it is released to production, and everyone gets their glass of Champagne!

You go home for the weekend satisfied with the great job you did.

On Monday, you open your email to find it flooded with customers screaming that nothing is working! Oh no, you must have made a mistake!!! So you set about debugging and quickly locate the error message in your monitoring, so you checkout the code from Git and start investigating. But the error that happens isn't even possible. So you spend the entire day debugging, again and again, coming to the same conclusion; This is not possible.

So finally, you decide to go and read the deployment log line-by-painstakingly-line, and there, on line 13.318, you see it! One of your 12 micr-oservices failed deployment! The deployment used a script with a pipe in it. Unfortunately, the script did not have pipe fail configured. The script, therefore, did not generate a non-zero exit code, so the deployment just kept humming along, deploying the remaining 11 with success. This chain of events resulted in a broken infrastructure state and unhappy customers, and you spend the entire Monday debugging and potentially the ENTIRE EXISTENCE coming to an end!

I think most developers would have a story similar to the one above, so why is getting release management right so damn hard? Modern software architecture and the tools that help us are complex machinery, which goes for our deployment tools as well. Therefore ensuring that every little thing is as planned means that we would have to check hundreds, if not thousands of items, each more to decipher than the last (anyone who has ever tried to solve a broken Xcode builds from an output log will know this).

So is there a better way? Unfortunately, when things break, any of those thousands of items could be the reason, so when stuff does break, the answer is most likely no, but what about just answering the simple question: "Is something broken?". Well, I am glad you asked because I do believe that there is a better way, and it is a way that revolves around Git.

Declaring your expected state

So I am going to talk about Kubernetes, yet again - A technology I use less and less but, for some reason, ends up being part of my examples more and more often.

At its core Kubernetes has two conceptually simple tasks; it stores an expected state of the resources that it is supposed to keep track of two; if any of those resources are, in fact, not in the expected state, it tries to right the wrong.

This approach means that when we interact with Kubernetes, we don't ask it to perform a specific task - We never tell it, "create three additional instances of service X," but rather ", There should be five instances of service X".

This approach also means that instead of actions and events, we can use reconciliation - no tracking of what was and what is, just what we expect; the rest is the tool's responsibility.

It also makes it very easy for Kubernetes to track the health of the infrastructure - it knows the expected state. If the actual state differs, it is in some unhealthy state, and if it is unhealthy, it should either fix it or, failing that, raise the alarm for manual intervention.

Git as the expected state

So how does this relate to Git? Well, Git is a version control system. As such, it should keep track of the state of the code. That, to me, doesn't just include when and why but also where - to elaborate: Git is already great at telling when something happened and also why (provided that you write good commit messages), but it should also be able to answer what is the code state in a given context.

So let's say you have a production environment; a good Git strategy, in my opinion, should be able to answer the question, "What is the expected code state on production right now?"
And note the word "expected" here; it is crucial because Git is, of course, not able to do deployments or sync environments (in most cases) but what it can do is serve as our expected state that I talked about with Kubernetes.

The target is to be able to compare what we expect, with what is actually there completely independent of all the tooling that sits in between, as we want to remove those as a source of error or complexity.

We want to have something with the simplicity of the Kubernetes approach - we declare an expected state, and the tooling enforces this or alerts us if it can not.

We also need to ensure that we can compare our expected state to the actual state.

To achieve this we are going to focus on Git SHAs, so we will be tracking if a deployed resource is a deployment of our expected SHA.

For a web resource, an excellent way to do this could be through a /.well-known/deployment-meta.json while if you are running something like Terraform and AWS, you could tag your resources with this SHA - Try to have as few different methods of exposing this information as possible to keep monitoring simple.

With this piece of information, we are ready to create our monitor. Let's say we have a Git ref called environments/production, and its HEAD points to what we expect to be in production, now comparing is simply getting the SHA of the HEAD commit of that ref and comparing it to our ./well-known/deployment-meta.json. If they match, the environment is in the expected state. If not, it is unhealthy.

Let's extend on this a bit; we can add a scheduled task that checks the monitor. If it is unhealthy, it retriggers a deployment and, if that fails, raises the alarm - So even if a deployment failed and no one noticed it yet, it will get auto-corrected the next time our simple reconciler runs. This can be done simply using something like a GitHub workflow.

You could also go all in and write a crossplane controller and use the actual Kubernetes reconciler to ensure your environments are in a healthy state - Go as crazy as you like, just remember to make the tool work for you, not the other way around.

So, now we have a setup where Git tracks the expected state, and we can easily compare the expected state and the actual state. Lastly, we have a reconciliation loop that tries to rectify any discrepancy.

Conclusion

So as a developer, the only thing I need to keep track of is that my Git refs are pointing to the right stuff. Everything else is reconciliation that I don't have to worry about - unless it is unreconcilable - and in which case, I will get alerted.

As someone responsible for the infrastructure, the only thing I need to keep track of is that the expected state matches the actual state.

No more multi-tool lookup, complex log dives or timeline reconstruction (until something fails, of course)

I believe that the switch from Git being just the code to being the code state makes a lot of daily tasks more straightforward and more transparent, builds a more resilient infrastructure and is worth considering when deciding how you want to do Git.

My day is being planned by an algorithm

Morten Olsen — Sat, 07 May 2022 05:30:54 +0000

Allow me to introduce Bob. Bob is an algorithm, and he has just accepted a role as my assistant.

I am not very good when it comes to planning my day, and the many apps out there that promise to help haven't solved the problem for me, usually due to three significant shortcomings:

Most day planner apps do what their paper counterparts would do: record the plan you create. I don't want to make the plan; someone should do that for me.
They help you create a plan at the start of the day that you have to follow throughout the day. My days aren't that static, so my schedule needs to change throughout the day.
They can't handle transits between locations very well.

So to solve those issues, I decided that the piece of silicon in my pocket, capable of doing a million calculations a second, should be able to help me do something other than waste time doom scrolling. It should let me get more done throughout the day and help me get more time for stuff I want to do. That is why I created Bob.

Also, I wanted a planning algorithm that was not only for productivity. I did not want to get into the same situation as poor Kiki in the book "The circle", who gets driven insane by a planning algorithm that tries to hyper-optimize her day. Bob also needs to plan downtime.

Bob is still pretty young and still learning new things, but he has gotten to the point where I believe he is good enough to start to use on a day to day basis.

How does Bob work? Bob gets a list of tasks, some from my calendar (both my work and my personal calendar), some from "routines" (which are daily tasks that I want to do most days, such as eating breakfast or picking up the kid), and some tasks come from "goals" which are a list of completable items. These tasks go into Bob, and he tries to create a plan for the next couple of days where I get everything done that I set out to do.

Tasks have a bit more data than your standard calendar events to allow for good scheduling

An "earliest start time" and a "latest start time". These define when the task can add it to the schedule.
A list of locations where the task can be completed.
A duration.
If the task is required.
A priority

Bob uses a graph walk to create the optimal plan, where each node contains a few different things

A list of remaining tasks
A list of tasks that are impossible to complete in the current plan
A score
The current location
The present time

Bob starts by figuring out which locations I can go to complete the remaining tasks and then create new leaf notes for all of those transits. Next, he figures out if some of the remaining tasks become impossible to complete and when I will arrive at the location and calculate a score for that node.

He then gets a list of all the remaining tasks for the current node which can be completed at the current location, again figuring out when I would be done with the task, updating the list of impossible tasks and scoring the node.
If any node adds a required task to the impossible list, that node is considered dead, and Bob will not analyze it further.

Now we have a list of active leaves, and from that list, we find the node with the highest score and redo the process from above.

Bob has four different strategies for finding a plan.

First valid: this finds the first plan that satisfies all restrains but may lead to non-required tasks getting removed, even though it would be possible to find a plan that included all tasks. This strategy is the fastest and least precise strategy.
First complete: this does the same as "First valid" but only exits early if it finds a plan that includes all tasks. This strategy will generally create pretty good plans but can contain excess transits. If it does not find any plans that contain all tasks, it will switch to the "All valid" strategy.
All valid: this explores all paths until the path is either dead or completed. Then it finds the plan with the highest score. If there are no valid plans, it will switch to the "All" strategy.
All: This explores all paths, even dead ones, and at the end returns the one with the highest score. This strategy allows a plan to be created even if it needs to remove some required tasks.

Scoring is quite simple at the moment, but something I plan to expand on a lot. Currently, the score gets increased when a task gets completed, and it gets decreased when a task becomes impossible. How much it is increased or decreased is influenced by the task's priority and if the task is required. It also decreases based on minutes spent transiting.

The leaf picked for analysis is the one with the highest score. This approach allows the two first strategies to create decent results, though they aren't guaranteed to be the best. It all comes down to how well tuned the scoring variables are tweaked. Currently, they aren't, but at some point, I plan to create a training algorithm for Bob, which will create plans, score them through "All", and then try to tweak the variables to arrive at the correct one with as few nodes analyzed as possible when running the same plan through "First valid"/"First complete".

This approach also allows me to calculate a plan with any start time, so I can re-plan it later in the day if I can't follow the original plan or if stuff gets added or removed. So this becomes a tool that helps me get the most out of my day without dictating it.

Bob can also do multi-day planning. Here, he gets a list of tasks for the different days as he usually would and a "shared" list of goals. So he runs the same calculation, adding in the tasks for that day, along with the shared goal list, and everything remaining from the shared list then gets carried over to the next day. This process repeats for all the remaining days.

I have created a proof of concept app that houses Bob. I can manage tasks, generate plans, and update my calendar with those plans in this app.

There are also a few features that I want to add later. The most important one is an "asset" system. For instance, when calculating transits, it needs to know if I have brought the bike along because if I took public transit to work, it doesn't make sense to calculate a bike transit later in the day. This system would work by "assets" being tied to a task and location, and then when Bob creates plans, he knows to consider if the asset is there or not. Assets could also be tied to tasks, so one task may be to pick up something, another to drop it off. In those cases, assets would act as dependencies, so I have to have picked up the asset before being able to drop it off. The system is pretty simple to implement but causes the graph to grow a lot, so I need to do some optimizations before it makes sense to put it in.

Wrapping up; I have only been using Bob for a few days, but so far, he seems to create good plans and has helped me achieve more both productive tasks, also scheduling downtime such as reading, meditation, playing console etc. and ensuring that I had time for that in the plan.

How to hire engineers, by an engineer

Morten Olsen — Wed, 16 Mar 2022 21:13:25 +0000

It has been a few years since I have been part of the recruitment process. Still, I recently went through the hiring process when looking for a new job. Therefore, I will mix a bit from both sides for this article. You get some experience from hires, what worked, and some experiences from the other side of the table, which caused me not to consider a company because of the spoiler alert: Engineers are contacted a lot!

So first, I need to introduce a hard truth as this will be underpinning a lot of my points and is most likely the most important takeaway from this: Your company is not unique!

Unless your tech brand is among the X highest regarded tech companies in the world, your company alone isn't a selling point. I have been contacted by many companies that thought that because they were the leader in their field or had a "great product", that would make candidates come banging at their door. If I could disclose all those messages, it would be easy to see that except for the order of information, all say almost the same thing, and chances are your job listing is the same. Sorry.

If everything is equal, any misstep in your hiring process can cost you that candidate, so if you are not amongst the most exciting tech brands, you need to be highly aware, or you will not fill the position, at least not with the best candidate.

Okay, after that slap in the face, we can take a second to look at something else.

Many people focus on skills when hiring, and of course, the candidate should have the skills for the position, but I will make a case to put less focus on the hard skills and more focus on passion.

Usually, screening skills through an interview is hard, and techniques like code challenges have their issues, but more on that later.

Screening for passion is more accessible. Usually, you can get a good feeling if a candidate is passionate about a specific topic and passionate people want to learn! So even if the candidate has limited skills, if they have passion, they will usually learn and outgrow a candidate with experience but no passion.

Filling a team with technical skills can solve an immediate requirement, but companies, teams, and products change and your needs will change. A passionate team will adjust and evolve with your product. A skilled team without passion will stay where you hired them.

Another issue I see in many job postings is requiring a long list of skills. It would be awesome to find someone skilled in everything to solve all tasks. However, in the real world, whenever you add another skill to that list, you are limiting the list of candidates that would fit. Hence, chances are you will not find anyone, or the skills of any candidate in that very narrow list will be way lower than in the broader pool.

A better way is to add the essential skills and let them learn any less critical ones at the job. If you hired passionate people, this should be possible (remember to screen for passion about learning new things)

While we are on the expected skill list, many companies have this list of "it would be nice if you had these skills". Well, those could be framed as learning experiences instead. If you have recruited passionate people, they will see new skills as a plus, and any candidate who already has the skill will see it and think, "awesome, I am already uniquely suited for this job!"

I promised to talk a bit about code challenges: They can be helpful to screen a candidate's ability to go in and start to work from day one, and if done correctly, can help a manager organize the process to best suit the teams' unique skills but;

Hiring at the moment is hard! And as stated, pretty much any job listing I have seen are identical, so like in a competitive job market where a small outlier on your resume lands you in a pile that is never read through, as likely is it in a competitive hiring market that your listing never gets acted upon if the process is more requiring than the others.

Recruiters often contact engineers, and speaking to all would require a lot of work, so if a company has a prolonged process, it quickly gets sorted out, especially by the best candidates. The latter most likely get contacted the most and most likely have a full-time job, so time is a scarce resource.

So be aware that if you use time-consuming processes such as the code challenge, you might miss out on the best candidates.

Please disclose the salary range. From being connected to a few hundred recruiters here on LinkedIn, I can see that this isn't just me but a general issue. As mentioned before, it takes very little to have your listings ignored, and most likely, most of your strongest potential candidates already have full-time jobs and would not want to move to a position paying less (unless the position were highly unique which, again, yours most likely isn't). Therefore if you choose not to disclose the salary range, be aware that you miss out on the best candidates. A company will get an immediate NO from me if they do not disclose the salary range.

Let's close on a more upbeat note. I have used many words telling you that your company or position isn't unique, and well, we both know that is not accurate; your company most likely has something special to offer! Be that soft values or hard benefits. Be sure to put them in your job listing to add this uniqueness. It is what is going to set you apart from the other listing. There are a lot of different companies with the same tech stack, using an agile approach, with a high degree of autonomy, with a great team, etc.... But what can you offer that no one else can? Get it front and center... Recruiting is marketing and good copy-writing.

Cover image resources: Hiring stickers created by Stickers - Flaticon

My home runs Redux

Morten Olsen — Tue, 15 Mar 2022 12:43:18 +0000

I have been playing around with smart homes for a long time; I have used most of the platforms out there, I have developed quite a few myself, and one thing I keep coming back to is Redux.

Those who know what Redux is may find this a weird choice, but for those who don't know Redux, I'll give a brief introduction to get up to speed.

Redux is a state management framework, initially built for a React talk by Dan Abramov and is still primarily associated with managing React applications. Redux has a declarative state derived through a "reducer"-function. This reducer function takes in the current state and an event, and, based on that event, it gives back an updated state. So you have an initial state inside Redux, and then you dispatch events into it, each getting the current state and updating it. That means that the resulting state will always be the same given the same set of events.

So why is a framework primarily used to keep track of application state for React-based frontends a good fit for a smart home? Well, your smart home platform most likely closely mimics this architecture already!

First, an event goes in, such as a motion sensor triggering, or you set the bathroom light to 75% brightness in the interface. This event then goes into the platform and hits some automation or routine, resulting in an update request getting sent to the correct devices, which then change the state to correspond to the new state.

...But that is not quite what happens on most platforms. Deterministic events may go into the system, but this usually doesn't cause a change to a deterministic state. Instead, it gets dispatched to the device, the devices updates, the platform sees this change, and then it updates its state to represent that new state.

This distinction is essential because it comes with a few drawbacks:

Because the event does not change the state but sends a request to the device that does it, everything becomes asynchronous and can happen out of order. This behaviour can be seen either as an issue or a feature, but it does make integrating with it a lot harder from a technical point of view.
The request is sent to the device as a "fire-and-forget" event. It then relies on the success of that request and the subsequent state change to be reported back from the device before the state gets updated. This behaviour means that if this request fails (something you often see with ZigBee-based devices), the device and the state don't get updated.
Since the device is responsible for reporting the state change, you are dependent on having that actual device there to make the change. Without sending the changes to the actual device, you cannot test the setup.

So can we create a setup that gets away from these issues?

Another thing to add here is more terminology/philosophy, but most smart home setups are, in my opinion, not really smart, just connected and, to some extent, automated. I want a design that has some actual smartness to it. In this article, I will outline a setup closer to that of the connected, automated home, and at the end, I will give some thoughts on how to take this to the next level and make it smart.

We know what we want to achieve, and Redux can help us solve this. Remember that Redux takes actions and applies them in a deterministic way to produce a deterministic state.

Time to go a bit further down the React rabbit hole because another thing from React-land comes in handy here: the concept of reconciliation.

Instead of dispatching events to the devices waiting for them to update and report their state back, we can rely on reconciliation to update our device. For example, let's say we have a device state for our living room light that says it is at 80% brightness in our Redux store. So now we dispatch an event that sets it to 20% brightness.

Instead of sending this event to the device, we update the Redux state.

We have a state listener that detects when the state changes and compares it to the state of the actual device. In our case, it seems that the state indicates that the living room light should be at 20% but are, in fact, at 80%, so it sends a request to the actual device to update it to the correct value.

We can also do schedule reconciliation to compare our Redux state to that of the actual devices. If a device fails to update its state after a change, it will automatically get updated on our next scheduled run, ensuring that our smart home devices always reflect our state.

Sidenote: Yes, of course, I have done a proof of concept using React with a home build reconciliation that reflected the virtual dom unto physical devices, just to have had a house that ran React-Redux

Let's go through our list of issues with how most platforms handle this. We can see that we have eliminated all of them by switching to this Redux-reconciliation approach: we update the state directly to run it synchronously. We can re-run the reconciliation so failed or dropped device updates get re-run. We don't require any physical devices as our state is directly updated.

We now have a robust, reliable, state management mechanism for our smart home, time to add some smarts to it. It is a little outside the article's main focus as this is just my way of doing it; there may be way better ways, so use it at your discretion.

Redux has the concept of middlewares which are stateful functions that live between the event going into Redux and the reducer updating the state. These middlewares allow Redux to deal with side effects and do event transformations.

Time for another piece of my smart home philosophy: Most smart homes act on events, and I have used the word throughout this article, but to me, events are not the most valuable thing when creating a smart home, instead I would argue that the goal is to deal with intents rather than events. For instance, an event could be that I started to play a video on the TV. But, that state a fact, what we want to do is instead capture what I am trying to achieve, the "intent", so lets split this event into two intents; if the video is less than one hour, I want to watch a TV show, if it is more I want to watch a movie.

These intents allow us to not deal with weak-meaning events to do complex operations but instead split our concern into two separate concepts: intent classification and intent execution.

So last thing we need is a direct way of updating devices, as we can not capture everything through our intent classifier. For instance, if I sit down to read a book that does not generate any sensor data for our system to react to, I will still need a way to adjust device states manually. (I could add a button that would dispatch a reading intent)

I have separated the events going into Redux into two types:

control events, which directly controls a device
environment events represent sensor data coming in (push on a button, motion sensor triggering, TV playing, etc.)

Now comes the part I have feared, where I need to draw a diagram.

...sorry

So this shows our final setup.

Events go into our Redux setup, either environment or control.

Control events go straight to the reducer, and the state is updated.

Environment events first go to the intent classifier, which uses previous events, the current state, and the incoming event to derive the correct intent. The intent then goes into our intent executor, which converts the intent into a set of actual device changes, which gets sent to our reducer, and the state is then updated.

Lastly, we invoke the reconciliation to update our real devices to reflect our new state.

There we go! Now we have ended up with a self-contained setup. We can run it without the reconciliation or mock it to create tests for our setup and work without changing any real devices, and we can re-run the reconciliation on our state to ensure our state gets updated correctly, even if a device should miss an update.

Success!!!

But I promised to give an idea of how to take this smart home and make it actually "smart."

Let's imagine that we did not want to "program" our smart home. Instead, we wanted to use it; turning the lights on and off using the switches when we entered and exited a room, dimming the lights for movie time, and so on, and over time we want our smart home to pick up on those routines and start to do them for us.

We have a setup where we both have control events and environments coming in. Control events represent how we want the state of our home to be in a given situation. Environment events represent what happened in our home. So we could store those historically with some machine learning and look for patterns.

Let's say you always dim the light when playing a movie that is more than one hour long; your smart home would be able to recognize this pattern and automatically start to do this routine for you.

Would this work? I don't know. I am trying to get more skilled at machine learning to find out.