Forem: Jeremy Kahn

The Agony and Ecstasy of PWAs

Jeremy Kahn — Sun, 05 Nov 2023 16:33:52 +0000

This was originally posted on my principal blog.

I love PWA technology. I think it's the most powerful and empowering technology that has come along since the creation of the internet. And it's never going to succeed.

Why is this? How could a great technology utterly fail to make meaningful market share inroads? Let's explore why PWAs are awesome, and why nobody cares.

User-centric design

Progressive Web Apps (PWAs) are literally just web pages with some additional functionality to make them behave like a native application. The key components of this illusion are the ability to install the web page such that it appears alongside actually-native apps like Microsoft Paint or Apple Messages, and some level of offline functionality. When opened, PWAs render in a dedicated window as their own app, distinct from any browser. Under the hood they're just another browser tab, but they present as a fully-formed, installed application.

Because PWAs are web pages at their core, they natively offer all of the benefits of a standard web page. That includes accessibility, security sandboxing, the potential to be customized via browser extensions, and observability via browser devtools. Every mainstream web browser has devtools that allow the user to take a look at the code that's running on their device. Most people won't do this, but the important thing is that they can. Observability is a de facto right of web users that cannot be taken away by web developers¹. And this is wonderful. I can't think of any other mainstream technology platform that puts the user's interests ahead of the developer's in this way. Because of this, PWAs are inherently user-centric in a manner that native apps uniformly choose not to be.

In general the tech industry has sort of lost the plot of what we're trying to accomplish. Computers are magical and amazing things, and they could do so much great stuff for people. Unfortunately "tech" has been largely co-opted by VCs into being a wealth extraction mechanism, so user-centric design has taken a back seat to investor-centric design. This has led to widespread enshittification and a general degradation of product quality. I don't see this changing in the near future, but the user-centric design built in at the platform level of PWAs at least gives us a hope that it someday could.

Simple deployment

Not only do PWAs offer the most user-centric design, but they have the most developer-centric deployment mechanism available as well: The web! Depending on how a project is set up, deployment could be as simple as modifying a file on a server. By contrast: If you want to update a native app, you need to submit your change to the app store operator (Apple, Google, etc.), wait, and pray that they accept it. Native app developers have no real control over the software that is delivered to their users because app store operators could capriciously deny updates for any reason. They own the deployment platform.

With web technology, the server is the deployment platform. You could run one for free from home, if you'd like. The web also has no gatekeepers or censors. There is no inherent payment processor that skims 30% off of every transaction. The web lets you own and control the code, the infrastructure, and the content. And the best part? PWAs run on every device! There's only one build target. It is the most successful implementation of "write once, run everywhere" in history. I struggle to conceive of a better way to ship software.

So far I've extolled the myriad virtues of PWAs and you're hopefully as excited about them as I am. If you are, prepare to be disappointed!

PWAs have struggled to gain meaningful app market share, and I don't expect that to ever really change. There are two main reasons for this: Gatekeeping incumbents, and user expectations.

Corporate incentives stand in the way

Apple effectively invented the concept of a modern app store, and it makes them billions of dollars a year. Many people seem to consider their practices unfair. I don't quite see it that way (nobody is requiring developers to publish with Apple), but I do see what such a market position means for us. iOS and its derivatives offer only one web browser², Safari. The problem is that Safari has always lagged behind the other major browsers in its feature set. This holds iOS back, and it therefore holds the potential for PWAs back as well because it's such a major platform. After all, why would a developer invest resources into a project if it won't work properly on over a quarter of mobile devices?

Apple has an obvious incentive to limit its browser's capabilities: Apple can't monetize the web. Being the cunning profit-driven corporation that it is, Apple is going to seek to widen its various moats as much as possible. Why invest in a technology that doesn't produce revenue when you could invest in one that does? It's hard to fault their logic, but the end result hurts users because it disincentivizes developers from creating PWAs and therefore diminishes user freedom.

User expectations

PWAs, as a concept, don't make sense to the average person. PWA installation is effectively a power user feature because typical users don't even consider the concept. Computer users have been implicitly trained for decades to think of web browsers as a means of basic content access and to rely on installed native apps for more advanced work and engagement. Can you honestly think of anyone in your life (who isn't a web developer or technology enthusiast) that would default to installing an app via the web rather than their app store?

This isn't actually a technology or design problem. No, it's a marketing problem. Marketing problems can only be solved with money, and there isn't a clear ROI in solving this particular one. So things will likely remain this way.

"We just need an app store for PWAs!" I hear you say. Bad news: It's been tried and it hasn't succeeded. Appscope has been abandoned and nobody is talking about store.app. I want these stores to win. I just don't see the business model to support them, though.

I really want to be wrong. I want to see PWAs flourish and render native apps irrelevant. But there needs to be a killer business model behind them for it to happen. Without that, the primary audience for PWAs is other web developers.

What we can do

The good news is that PWAs aren't going anywhere. It's a web standard that's in wide deployment. It's an effective technology choice for hobby projects and side hustles. I don't see Apple ceding its market share any time soon, but that doesn't mean that we can't enjoy PWAs today and into the future. I think we just have to accept that it will always be a less successful platform, especially for Apple users.

I plan to keep making open source PWAs because it's such an effective solution for the hobby projects I like to build. I encourage you to do the same. If you value developer control and user freedom, PWAs are the obvious choice.

It's worth mentioning that developers can (and generally should, for performance reasons) minify their code before deploying it. This makes it less than trivial to figure out what the code is doing, but the point is that it's possible to do so, rather than impossible (or harder, anyways) like it is with native apps. ↩
Yes, I know there's other browsers on the iOS App Store. But due to Apple's restrictions, they're implemented as reskins of Safari and therefore cannot implement the web platform features that Safari is missing. ↩

Adding a chat room to any app with one line of code

Jeremy Kahn — Sun, 08 Oct 2023 19:19:47 +0000

I just added a new feature to Chitchatter that I'm really excited about!

If you haven't seen it before, Chitchatter is an open source web app that instantly enables private, anonymous communication between people. Once connected you can easily chat, share audio and video, and transfer files. All communication is encrypted and peer-to-peer.

The feature I just added is chat room embedding. Much like a YouTube video, you can now embed a Chitchatter room into any web app. Here's a demo of an embedded Chitchatter room being loaded via CodePen:

To create your own embedded chat room: Go to https://chitchatter.im/, enter a room name (or use the unique default), and click "Get embed code." Copy and paste the presented code snippet into your HTML code and you're done.

That's it! Happy chatting. 🙂

Taking the Power Back with Web Meshes

Jeremy Kahn — Sat, 26 Nov 2022 20:38:02 +0000

We live in a strange time. As we trade independence for convenience, control of the web is steadily being concentrated in the hands of a few major tech companies. Platforms like AWS have become the defacto standard for architecting apps. When we cede control to for-profit megacorporations, we lose the ability to control our own destiny and do things in the collective interest.

Web apps have historically been designed around a central service. The service is the sun, and clients orbit around it. This architecture is generally good. It enables highly-available systems and a good user experience. One tradeoff with this is that users have to hope that the service will be operated with the user's best interests in mind. This has become an increasingly bad bet, as you don't have to look far to find examples of it unfairly creating catastrophic problems for people. We don't have to settle for this state of affairs. We have to rethink the traditional relationship between web apps and servers, but there is a solution to this problem of service dependence.

Don't worry, this is not a Web3 manifesto.

Web browsers are unbelievably powerful these days. They are featureful and performant enough to meet many users' needs entirely client-side. Central services will always be necessary for applications that require reliable access to massive data sources (think Google Search or Spotify), but there are a lot of applications that don't strictly require such data access. Many web apps' purpose is to connect users, and it is generally assumed that a central service is needed to enable the connection. But what if it wasn't? What if people could connect with each other without relying on a central service?

Introducing: Web meshes

P2P is nothing new. It is a long-established means of connecting two or more people directly over a network. Web browsers are very capable of a wide range of P2P connections. Many apps use WebRTC to enhance realtime apps, but it is still an underutilized technology. Even with WebRTC, many apps are designed around the dependence on a central app server with WebRTC performing a user experience enhancement. Web meshes turn this idea on its head: Instead of using P2P connections to enhance the user experience, what if P2P connections were the foundation of the user experience? In other words, what if there was no central server?

Decentralized architecture has been explored time and again to varying degrees of success. Typically it is discussed in terms of distributed server nodes. Web meshes take this a step further and treat each user's browser as a node by which the network is established. In other words, users' browsers connect with each other rather than going through a server.

The primary benefit to cutting out dependence on a server is that truly private data transmission becomes possible. Privacy is inherently valuable and it cannot be guaranteed when data flows through a third-party server. Many services make bold claims about how responsibly they handle user data, but as a user all you can really do is take the service operator at their word and hope for the best. This isn't good enough in a world where service operators are known to act against users' best interests. When sensitive data never touches a service provider to begin with, there can be no risk of its misuse or mishandling.

How Web meshes work

First, I need to give credit to Dan Motzenbecker and his incredible work with the Trystero JavaScript library. Trystero serves as the direct inspiration for the web mesh concept. Trystero is one of the most powerful and fascinating libraries I've ever found and I don't understand how it doesn't have many thousands of stars on GitHub. Please give Trystero a look and consider how you might use it in your own projects.

A web mesh starts by a browser connecting to one or more WebTorrent servers and offering its SDP data. This SDP data contains information about how to connect to the offering browser via WebRTC. It also requests SDP data for any other browser that has already connected to the WebTorrent server similarly. These SDP data offers and requests are scoped to a specific, client-determined namespace (or "room") so that users only connect with others that have the same namespace/room name. Once SDP data is exchanged, browsers can then connect directly to each other. Each browser maintains a connection to the WebTorrent trackers to automatically connect to other browsers that try to connect to the room similarly later, but that is the only type of dynamic client/server interaction.

This effectively turns the WebTorrent servers into STUN servers. There is significant value in additionally setting up a proper TURN server to ensure a reliable connection between browser peers, but it isn't strictly necessary unless any peers are on a local network that prevents a P2P connection (which is all but guaranteed to happen in practice).

After the initial peer connections are established, the WebTorrent server's job is done. From this point, the web mesh has been created and connected browsers communicate directly with each other. Users can join and leave the mesh on an adhoc basis provided that they each know the room name beforehand. When all users leave the room, the mesh no longer exists until it is recreated via WebTorrent SDP exchange like it was initially.

Web meshes in practice

I discovered web mesh architecture by building Chitchatter, a privacy-focused anonymous chat app. Chitchatter connects peers to a chat room where they can communicate via text, video, and audio, as well as do screen and file sharing. My goal for this project was to enable people to communicate simply and safely, and I was able to achieve that by building it as a web mesh.

When you first get to the Chitchatter home page, a client-side generated UUID is used as the default room name. When users enter the room, they join the web mesh as described above. If there were already people in the room, the new user will be connected to that room's web mesh. A Chitchatter "room" doesn't exist in terms of records in a database. As a web mesh, a Chitchatter room is nothing more than a named conceptual construct for the connection between peers.

There are many more potential use cases for this sort of design such as collaboration apps (like a decentralized Google Docs or Figma-like app) and multiplayer games.

Why web meshes?

P2P apps aren't novel, but there is a significant user benefit in restricting the connections to just web browsers. Modern browsers are sandboxed from the rest of the OS, so users are guarded against system compromises due to malicious code and bugs.

Another benefit to designing around web browsers is the backbone of any web app: The URL. Users shouldn't need a service account to run code that's downloaded on their machine, but they will need to know how to find others to connect to. The room name can be embedded into the URL so that users just need to share their unique room link in order to connect.

Web meshes change the relationship between browsers and servers. Aside from the generic WebTorrent and TURN servers that establish the peer connections, the app's server is reduced to being a static asset delivery mechanism that is cheap and easy to manage.

Web mesh advantages

Web meshes are designed for maximum user benefit. Rather than creating reliance on service operators like more traditional web architectures, web meshes minimize the relationship between the client and server as much as possible. Sensitive information is never provided to service operators to misuse or mishandle. It is only ever passed back and forth between peers in a secure manner.

Web meshes are resistant to censorship and surveillance. Any sort of web monitoring would require intervention at the server level, but web meshes have no dynamic server component to intervene with.

Web mesh disadvantages

Web meshes are not a silver bullet, as they have a number of limitations that must be considered before adopting the design. They characteristically suffer from a lack of data availability and durability, as there is no cloud component to store data in. Users have come to expect reliable access to their data, so that would need to be considered and managed appropriately for any app designed around a web mesh. Because of this, web meshes would not be a good fit for many business needs. IndexedDB offers an excellent solution for persisting data to disk locally, but it's not geo-redundant.

Web meshes only work so long as a P2P connection can be established. Well-configured TURN servers are the solution to this problem, but they can be costly to operate.

Power to the people

Web meshes are a great fit for projects that prioritize benefit for users over developers or service operators. With Trystero, they are an effective way to connect people in a safe and private way. As more and more software concentrates power in the hands of a few service operators, it's important to build ways for people to connect, share ideas, and organize. Web meshes put the user first, and that's the way that software should work.

To channel the late Steve Jobs:

It is now 2022. It appears Big Tech wants it all. The open web is perceived to be the only hope to offer Big Tech a run for their money. Users, initially welcoming Big Tech with open arms, now fear a Big Tech dominated and controlled future. They are increasingly and desperately turning back to the web as the only force that can ensure their future freedom. Big Tech wants it all. Will Big Tech dominate the entire computer industry? The entire information age? Was George Orwell right?

Making LeetCode interviews more inclusive

Jeremy Kahn — Sat, 22 Jan 2022 23:49:21 +0000

Something that has plagued me throughout my career is LeetCode-style interview challenges. For whatever reason, most of the tech industry has converged on this format as the canonical litmus test for whether an individual can code, or whether they are an Imposter Programmer. This is a shame, because LeetCode-style interviews are deeply flawed:

An astonishing amount of false negatives. See: All the jobs you were qualified for but missed out on simply because you choked in the coding challenge.
Even more false positives. See: All the engineer coworkers you've ever had that were difficult to work with.
They are exclusionary to the point of being discriminatory against neurodivergent people. (This is the one that I'm personally most bothered by, because I am Autistic and therefore considered neurodivergent.)

Zooming out a bit, we all deserve better than this ineffective form of interviewing. This post will attempt to illustrate the many flaws with LeetCode interviews and offer some simple and more effective alternatives to use instead.

WhatCode?

When I talk about LeetCode-style challenges, I'm talking about live-coding challenges where you're given a prompt like this, on the spot:

// Given a string s, return the longest palindromic substring in s.

/**
 * @param {string} s
 * @return {string}
 */
var longestPalindrome = function(s) {

};

(This is a verbatim example from LeetCode.)

It's not an unreasonably difficult challenge, and I estimate that nearly all professional programmers could solve it given the necessary resources and time. The issue is that interview candidates are given a specific time limit (typically 45-60 minutes), and are actively scrutinized while they work. When they complete the challenge, they are given another one repeatedly until time is up. The experience is every bit as nerve-wracking as it sounds, and the sheer anxiety causes many people to freeze up and appear as though they genuinely can't code. However, that is typically not the case, as candidates probably wouldn't have gotten far enough in their career to be having the interview to begin with.

The case for having these sorts of trials is typically "to see how the candidate thinks." This focus on the "how" obscures the "what" of the end result. Most companies are not in the business of selling how their employees think. They are in the business of selling what they produce. So wouldn't that be the better thing to focus on in an interview?

Interview exclusion

LeetCode-style interviews, by design, filter for a very specific type of mind: One that can performatively produce complete and working code on the spot. This would be completely acceptable if that's what an engineering job actually entailed, but this is not the case. I've met many, many tremendously talented engineers from many walks of life in my career, and none of them work this way. The way actual, professional code is generally produced is by some mixture of quietly mulling over a problem, maybe going for a walk, talking it out with a coworker, sketching out some ideas, but most importantly: Iterating over the course of hours, days, or even weeks depending on the scale of the problem. No engineer regularly produces production-grade work in under an hour in their day job.

Some among us can summon the energy and the strength to conjure working code into existence within a time limit and under active scrutiny, but that is a subset of the engineers that are worth hiring. The whole includes introverts, people with social anxiety, people who didn't sleep well the night before the interview because they were nervous about it, people with ADHD, Autistic people, and many, many more. The LeetCode filter prevents these qualified people from passing the interview and everybody loses out because of it.

To expand a bit on the Autism point, since I can speak to that personally: The complexities of living with Autism and working with those who do are beyond the scope of this post. I also want to avoid speaking for all Autistic people, since it is an extremely varied and diverse group. However, the abbreviated version of what I want to say is that being an Autistic interviewee being actively judged by a neurotypical interviewer is inequitable to the point of being exclusionary. When I'm trying to navigate a LeetCode-style challenge, am also spending many mental cycles on distracting thoughts like the following:

Am I talking enough?
Am I talking too much?
Am I passing as normal?
Am I doing this the way the interviewer expects it to be done, or the way that makes sense to me? Which one should I be doing?
Should I be applying academic concepts for the sake of showing that I know them, or should I keep things simple?

Only the remainder of available mental cycles are actually going towards solving the problem, so you're only seeing maybe 30% of my actual coding ability. Many people might read this and say "just relax and focus on the challenge at hand." That is like telling a paraplegic to just believe in themself and stand up. This difficulty is the very essence of Autism and it cannot be willed away.

A better alternative

Given the many flaws of LeetCode-style interviews, how can we do better? I've got good news: It's pretty easy. The first thing you need to do is take a step back and consider the sorts of characteristics you're actually hiring for. I'll make the bold assumption that the engineers on your team aren't actually solving contrived code puzzles in their day-to-day (and if they are, feel free to stick with LeetCode interviews and please never contact me for a job). Instead, list out the technical duties they are actually performing to make the company successful. It hopefully looks something like this:

Understanding, improving, and extending business logic
Participating constructively in code reviews
Collaboratively problem-solving practical technical issues ("The API is experiencing a performance degradation, what do we do?")
Finding and fixing bugs

These are critical qualities that can be teased out of a much wider pool of candidates within an hour, simply by recreating relevant scenarios in the interview. Rather that handing interviewees an unimplemented function, show them your team's code and see how they might improve it. You'll learn what you need to know about them, and it will help them make a more informed decision about whether they and your company are a match. Let the candidate make a modest code change and see how they go about leading a Pull Request to get it merged. It's okay if you're working out of an alternate repo that's meant specifically for interviewing. In fact, that's the best way to create a consistent and fair challenge for each potential candidate.

"This sounds hard to do."

It isn't, really. It requires a bit of upfront investment and experimentation to get a balanced and streamlined process, but hiring good people is inherently challenging and you need to put some work into it. Having a more thoughtful and effective interviewing process will save you money in the long run by setting up the candidates you actually want to hire for success.

If you simply must stick with LeetCode challenges, provide candidates the specific problems they will be given a week in advance instead of on the spot. Allow them to come to the interview with the solution already implemented. This will set candidates up for success by allowing them to solve the problem their way, on their schedule, and in whatever way works best for their unique situation. You can then collaboratively analyze the solution and extend it via a pairing session that they lead. This is not an unreasonable advantage, it is an accessibility accommodation. The great thing about accessibility accommodations is that they make things better for everyone. Having ample time to think about a solution ahead of time would be helpful in setting all candidates up for success. And in this case, it changes the nature of the interview to be more of an exercise in collaboration rather than a gate designed to keep people out of the company.

"But what if the candidate cheats by getting help?"

So what if they do? Getting outside assistance ahead of the interview won't help a candidate during an interactive review and pairing session on the code they provide. Also, what's the harm in getting some help? Is there a rule at your company that forbids engineers from getting assistance to develop a solution? As an interviewer, you should be looking for signals that indicate a candidate can produce results by any means necessary, be receptive to feedback, and iterate based on evolving requirements.

"It sounds like you're just bad at coding interviews and are taking it out on the world via a blog post."

You're right, I am bad at coding interviews. I will never be good at them, because I have a disability that makes them disproportionately difficult for me. Funny thing about that, though: I am actually a very capable programmer. I'm not egotistical enough to believe I am exceptional, but speaking objectively: My coding ability has never been called into question as lacking on any job I've ever had. I've never been too insufficient of a programmer to get my job done well. Minimally, I am a "good enough" programmer.

If you're asking how I ever got a job given how bad I am at coding interviews, it's because I tend to get job offers from the exceptional companies that don't have LeetCode-style interviews as part of their hiring process.

We all deserve better

There are many, many exceptional programmers who struggle tremendously with LeetCode-style interviews. Max Howell, the creator of the mission-critical package manager Homebrew that millions of developers depend on, famously got rejected by Google (who uses Homebrew) because of such an interview:

// Detect dark theme var iframe = document.getElementById('tweet-608682016205344768-610'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=608682016205344768&theme=dark" }

There is something deeply wrong with this way of interviewing. It is depriving companies of excellent and diverse engineers, and it is making people miss out on good jobs that they actually deserve. It's time to go back to the drawing board (but not the whiteboard) on these ineffective hiring practices.

Automating Godot game releases to itch.io

Jeremy Kahn — Tue, 18 Jan 2022 00:21:13 +0000

If you're a game developer, you most likely want to spend your time actually designing and developing your game. Unfortunately, a lot of time goes into not-so-interesting manual administrative tasks. One of those tasks is publishing new builds. What if this work could be completely automated away and happen as you work on your game, in the background? With workflow automation tools like GitHub Actions, this is not only possible, but free to use and easy to achieve!

The goal

Me and my friend Luke are building Farmhand Go!, a Godot game that's based around making in-game money by growing and harvesting crops with timed actions. What we wanted was for every merge to the main Git branch to initiate a release to https://rainbowcow-studio.itch.io/farmhand-go, where the game lives and can be played. We also wanted our Discord channel to be automatically notified of the new release as soon as it's available.

With a little elbow grease, we came up with this GitHub Action that does all of this for us:

The rest of this article will discuss how we got this set up, and how you can have something similar for your Godot games!

Prerequisites

In order to set up this automation, you will need your GitHub repo and itch.io page already published. If you want the optional Discord notification functionality, you will need to have your server and relevant channel set up as well.

Setting it up

In the Git repo, your Godot project needs to have an export_presets.cfg file at its root. In Farmhand Go!, the actual Godot project is at ./project rather than the repo root, but your setup may vary. Be sure that export_presets.cfg is not in your .gitignore (it is by default), or that it is at least present for when the actual Godot export occurs. This is what our committed export_presets.cfg looks like:

Once that's set up, you'll need your modified version of the YAML file from above. Here's a more generic version you can use: https://gist.github.com/jeremyckahn/ff4f0e409f089ec36bdecb5a5adb6819. You'll just need to replace the following stubbed values:

NAME-OF-YOUR-GAME
NAME-OF-YOUR-USER

These occur in a few places in this file, so I recommend using your editor's Find and Replace feature and carefully reviewing the changes. Move this file to your repo's .github/workflows/ directory, and name it whatever you'd like. For Farmhand Go!, it's .github/workflows/deploy.yml.

As written, this automation will run whenever a commit is pushed, either directly or via Pull Request merge, to the main branch. If you want to use a different branch for this automation, change the branches value towards the top of the file.

Once that's in place, you will need an API key for itch.io and optionally a webhook URL for Discord.

The itch.io API key

To create your itch.io API key, go to your settings in itch.io:

Then, in the left-hand nav, go to "API keys"

Input your password, and then click "Generate new API key." You should now have a screen that looks something like this:

And you're done here! Copy the new key and head over to the Settings page for your game's GitHub repo and click on "Secrets" in the left-hand nav:

From here, click "New Repository Secret." For "Name," put BUTLER_API_KEY. For the value, paste in the key that you copied from itch.io. Click "Add secret," and you should now have a screen that looks something like this:

The Discord webhook URL (optional)

In your Discord server, go to "Server Settings":

In the left-hand nav, click "Integrations":

Click "Webhooks:"

Click "New Webhook" and fill out the form appropriately. Here's how ours looks:

Click "Copy Webhook" to copy the URL to your clipboard, and then "Save Changes" at the bottom. We're done here!

Flip back to your GitHub repo's Secrets page, and create a new secret named DISCORD_WEBHOOK with the webhook URL from Discord as the value. Your Secrets page should now look like this:

Now that all the necessary GitHub secrets are in place, all that's left to do is commit .github/workflows/deploy.yml to your main branch via PR. Starting with this commit and going forward on the main branch, all commits will trigger a GitHub Action that deploys your web game to itch.io and notifies your community of the new release:

And that's it! You can customize this workflow to your liking, perhaps by changing the Discord messaging or adding additional build targets. For the latter, I recommend looking at the Action that I learned from in order to see how to support Linux, Mac, and Windows builds: https://github.com/RudyMis/Bubbles/blob/master/.github/workflows/godot-ci.yml

What do you think? How do you think this automation could be improved? Please share your thoughts in the comments. Thanks for reading, and have fun with developing your Godot games!

How I designed an abuse-resistant, fault-tolerant, zero cost, multiplayer online game

Jeremy Kahn — Tue, 28 Dec 2021 23:47:58 +0000

Nearly a year ago I deployed a multiplayer feature for Farmhand, an open source and web-based farming game that I created. Since that initial deployment, the multiplayer system has experienced no downtime or service degradation. And best of all, I've paid nothing to host the service and therefore I am able to allow others to play for free. This article is an overview of how I designed this system from the ground up.

The game

In short, Farmhand is a game that mixes farming and market mechanics. The goal is to buy seeds for a low price, plant and harvest them, and then sell the crops at a high price. Prices fluctuate from day-to-day, so you'll have to be smart about your buy/sell decisions.

Farmhand was initially designed as a single-player game and seed/crop values were randomly generated at the start of each game day. One day I thought it would be cool to create a shared, online market that players around the world could participate in together. My vision was for one player's buy/sell decisions to affect the global market that determines the seed/crop values for all connected players.

In order for this market system to be fun, it needed to be simple and reliable. I gave myself the following constraints:

Zero hosting costs. I'm not making money from Farmhand, so I don't want to spend money to host it.
Minimal devops involvement. Farmhand is just a hobby for me, and I have a day job. I don't want to be dealing with managing service outages during the work day (or the middle of the night).
Fault-tolerance and abuse-resistant. If you're putting a service online, expect people to abuse it. I wanted this system to not only be highly-available, but resistant to griefing as well.

In the end I was able to ship a fun and functional multiplayer system that adhered to all of these constraints.

The tech

There are a few pieces to this system:

The client

Farmhand is implemented as a PWA that runs in a web browser. The client's overall architecture is outside the scope of this article, but for the purposes of online multiplayer it uses Trystero with the WebTorrent matchmaking strategy to connect peers to each other. It interacts with the central market server via a REST API.

The server

Farmhand's API is hosted on Vercel's Hobby tier. Vercel provides an excellent Serverless platform that offers scalable runtime performance, as well as static file hosting, automatic preview builds (great for testing out PRs), and more.

The Vercel-based API is backed by a Redis instance for data "persistence." "Persistence" is in quotes because the data only ever lives in memory, so a system failure would result in complete data loss. However, the application logic is designed such that this kind of failure would be a feature and not a bug. The Redis instance is hosted on Redis Labs' free tier.

For Farmhand, both Vercel and Redis Labs are configured to run in AWS.

System architecture

At any given time, a player can go to https://www.farmhand.life/, switch the "Play online" toggle and join a room of their choosing (global by default). When this happens, two things occur:

A request is made to GET https://farmhand.vercel.app/api/get-market-data?room=global, which is a Serverless function. This retrieves the latest market data and also informs the server of the player's presence. This request is repeated to serve as a heartbeat until the player leaves the room to maintain an "active" session with the API.
A WebSocket connection to a WebTorrent tracker is made. The tracker then connects the client to any other clients in the requested room. The peer-to-peer connection is persistent until the player leaves the room. This complexity is abstracted away via Trystero.

The API manages room data that is stored in Redis. When a GET https://farmhand.vercel.app/api/get-market-data?room=global request (source) is made, the API checks to see if a value associated with the key room-global exists. If not, it is initialized. Here's an example of a room object:

{
  "activePlayers": {
    "4a793fe2-9eb1-4041-935b-5caf55177dde": 1640727668293,
    "58f90cc1-1089-4394-a7e7-2f079f87ed4d": 1640727669934,
    "b26b2d59-79f5-40f3-bc91-cfc0554bb994": 1640727674791,
    "d1e34686-925e-4344-b7cb-e15ce6d7dad3": 1640727667860
  },
  "valueAdjustments": {
    "asparagus": 0.6798235686529905,
    "asparagus-seed": 0.9797840434970977,
    "carrot": 0.5382522777963925,
    "carrot-seed": 1.1233740954422615,
    "corn": 1.1524067154896047,
    "corn-seed": 1.2309158460921086,
    ...
  }
}

activePlayers is a map of unique player IDs (determined by clients via uuid to timestamps of when they last made a GET https://farmhand.vercel.app/api/get-market-data?room=global request. Each time the function is invoked, it examines the map to see which timestamps are older than the HEARTBEAT_INTERVAL_PERIOD (currently 10 seconds) and deletes any that are expired. This data is returned to the client and also written back to Redis to be persisted across function invocations. This is how the active room participants are tracked.

valueAdjustments is the current state of the room's market. The map's keys refer to an item ID in the game and the values represent their respective in-game market value. Market values are bound between 0.5 and 1.5 and go up or down based on individual player activity. When a player ends their in-game day, an API request to POST https://farmhand.vercel.app/api/post-day-results is made with a payload that looks something like:

{
  "positions": {
    "carrot-seed": 1
  },
  "room": "global"
}

positions represents all the items that the player either increased or decreased inventory of in their most recent in-game day. 1 means that they increased their inventory of the associated item ID (either by buying seeds or harvesting crops), which increases the item's market value. -1 means they decreased their inventory (typically by selling the item), which decreases the item's market value. Here's the source for that logic:

const applyPositionsToMarket = (valueAdjustments, positions) => {
  return Object.keys(valueAdjustments).reduce(
    (acc, itemName) => {
      const itemPositionChange = positions[itemName]

      const variance = Math.random() * 0.2

      const MAX = 1.5
      const MIN = 0.5

      if (itemPositionChange > 0) {
        acc[itemName] = Math.min(MAX, acc[itemName] + variance)
      } else if (itemPositionChange < 0) {
        acc[itemName] = Math.max(MIN, acc[itemName] - variance)
      } /* itemPositionChange == 0 */ else {
        // If item value is at a range boundary but was not changed in this
        // operation, randomize it to introduce some variability to the market.
        if (acc[itemName] === MAX || acc[itemName] === MIN) {
          acc[itemName] = Math.random() + MIN
        }
      }

      return acc
    },
    {
      ...valueAdjustments,
    }
  )
}

The updated market data is again persisted back to Redis.

Abuse mitigation

One of the nice things about this server-side logic is that it naturally mitigates abuse. There is nothing stopping people from crafting custom POST https://farmhand.vercel.app/api/post-day-results requests to manipulate the market however they want. However, once the adjusted value for any given item reaches the upper or lower bound (1.5 or 0.5, respectively), the item's value is randomized within those bounds. So while nefarious people can manipulate the market, it will reset itself and balance out before long. Even in those cases, it only presents as normal (if somewhat volatile) market dynamics to players.

Fault tolerance

Farmhand room data is only stored in memory via Redis and never written to a disk. Because of this, it is inherently ephemeral. The worst case scenario with this design is that room data gets lost either due to the Redis server shutting down or something like a FLUSHALL command being issued. However, since the API initializes requested room data that doesn't already exist, this would only present to the user as a bit of market volatility that would likely go unnoticed.

Peer-to-peer interaction

The Vercel-based API effectively manages the shared market data, but I wanted players to have a sense of who else is playing with them and how they are affecting the experience for everyone else. This is where the peer-to-peer communication comes into play.

Instead of a server and logic to act as a broker between clients, they connect to each other directly via Trystero and WebTorrent as explained previously. As players perform various actions such as buying or selling items, messages are broadcast to all connected peers to display in the "Active Players" modal:

In an effort to mitigate abuse, player names are the result of a simple hashing algorithm based on player IDs (which are stable).

export const getPlayerName = memoize(playerId => {
  const playerIdNumber = playerId
    .split('')
    .reduce((acc, char, i) => acc + char.charCodeAt() * i, 0)

  const adjective = adjectives[playerIdNumber % adjectives.length]
  const adjectiveNumberValue = adjective
    .split('')
    .reduce((acc, char, i) => acc + char.charCodeAt() * i, 0)

  const animal =
    animalNames[(playerIdNumber + adjectiveNumberValue) % animalNames.length]

  return `${adjective} ${animal}`
})

Retrospective analysis

This system has been effective so far. Vercel and Redis Labs have provided excellent performance and availability since this system launched nearly a year ago, which is impressive given that I'm using the free tier of both services. The fault-tolerance and abuse mitigation measures have resulted in the minimal maintenance burden I was hoping to achieve. The only manual intervention that's been required from me so far to keep things running is having to log into Redis Labs every couple of months to indicate that my account is still active.

I'm quite pleased with how this multiplayer system has turned out so far. I'd like to expand on Farmhand's multiplayer features and further develop its online market mechanics. I'd like to know what others think as well, as I've never designed a full stack system before this and I would like to learn how it can be improved. Let me know what you think via the comments below! And if you're up for some easygoing farming fun, give Farmhand a try sometime. 🙂

TIL: Reading stdin to a BASH variable

Jeremy Kahn — Mon, 14 Dec 2020 02:44:39 +0000

Today I wanted to read stdin to a BASH script variable for additional processing. It's not completely straightforward, but it's pretty easy once you know the syntax:

YOUR_VARIABLE=$(</dev/stdin)

A full example:

With this, you can easily make pipe-able BASH script files. For instance, with these two files using this technique:

You can do:

echo some-text | ./to-uppercase.sh | ./to-snake-case.sh

For:

SOME_TEXT

Shoutout to StackOverflow user Ingo Karkat for explaining this. It's a very handy little trick for making BASH scripts more flexible and reusable!

The case for async/await-based JavaScript animations

Jeremy Kahn — Tue, 18 Aug 2020 22:19:37 +0000

async/await is one of my favorite features of modern JavaScript. While it's just syntactic sugar around Promises, I've found that it enables much more readable and declarative asynchronous code. Recently I've started to experiment with async/await-based animations, and I've found it to be an effective and standards-based pattern.

The problem

There is no shortage of great JavaScript animation libraries available. For most use cases, GreenSock is the gold standard and the library you should default to (and I am saying this as an author of a "competing" library). GreenSock, like most animation libraries such as Tween.js, anime.js, or mo.js, has a robust and comprehensive animation-oriented API. This API works well, but like any domain-specific solution, it's an additional layer of programming semantics on top of the language itself. It raises the barrier of entry for newer programmers, and you can't assume that one bespoke API will integrate gracefully with another. What if we could simplify our animation scripting to be more standards-based to avoid these issues?

The solution: Enter `async`/`await`

async/await enables us to write asynchronous code as though it was synchronous, thereby letting us avoid unnecessarily nested callbacks and letting code execute more linearly.

Bias warning: For the examples in this post I am going to use Shifty, an animation library I am the developer of. It is by no means the only library you could use to build Promise-based animations, but it does provide it as a first-class feature whereas it's a bit more of an opt-in feature for GreenSock and other animation libraries. Use the tool that's right for you!

Here's an animation that uses Promises directly:

import { tween } from 'shifty'

const element = document.querySelector('#tweenable')

tween({
  render: ({ x }) => {
    element.style.transform = `translateX(${x}px)`
  },
  easing: 'easeInOutQuad',
  duration: 500,
  from: { x: 0 },
  to: { x: 200 },
}).then(({ tweenable }) =>
  tweenable.tween({
    to: { x: 0 },
  })
)

This is straightforward enough, but it could be simpler. Here's the same animation, but with async/await:

import { tween } from 'shifty'

const element = document.querySelector('#tweenable')

;(async () => {
  const { tweenable } = await tween({
    render: ({ x }) => {
      element.style.transform = `translateX(${x}px)`
    },
    easing: 'easeInOutQuad',
    duration: 500,
    from: { x: 0 },
    to: { x: 200 },
  })

  tweenable.tween({
    to: { x: 0 },
  })
})()

For an example this basic, the difference isn't significant. However, we can see that the async/await version is free from the .then() chaining, which keeps things a bit terser but also allows for a flatter overall code structure (at least once it's inside the async IIFE wrapper).

Because the code is visually synchronous, it becomes easier to mix side effects into the "beats" of the animation:

It gets more interesting when we look at using standard JavaScript loops with our animations. It's still weird to me that you can use a for or a while loop with asynchronous code and not have it block the thread, but async/await allows us to do it! Here's a metronome that uses a standard while loop that repeats infinitely, but doesn't block the thread:

Did you notice the while (true) in there? In a non-async function, this would result in an infinite loop and crash the page. But here, it does exactly what we want!

This pattern enables straightforward animation scripting with minimal semantic overhead from third-party library code. await is a fundamentally declarative programming construct, and it helps to wrangle the complexity of necessarily asynchronous and time-based animation programming. I hope that more animation libraries provide first-class Promise support to enable more developers to easily write async/await animations!

Addendum: Handling interruptions with `try`/`catch`

After initially publishing this post, I iterated towards another powerful pattern that I wanted to share: Graceful animation interruption handling with try/catch blocks.

Imagine you have an animation running that is tied to a particular state of your app, but then that state changes and the animation either needs to respond to the change or cancel completely. With async/await-based animations, this becomes easy to do in a way that leverages the fundamentals of language.

In this example, the ball pulsates indefinitely. In the async IIFE, notice that the tweens are wrapped in a try which is wrapped in a while (true) to cause the animation to repeat. As soon as you click anywhere in the demo, the animation is rejected, thus causing the awaited animation's Promise to be treated as a caught exception which diverts the control flow into the catch block. Here the catch block awaits reposition, another async function that leverages a similar pattern to move the ball to where you clicked. Once reposition breaks and exits its while loop, the async IIFE proceeds to repeat.

This demo isn't terribly sophisticated, but it shows how async/await-based animations can enable rich interactivity with just a bit of plain vanilla JavaScript!