Forem: Em Lazer-Walker

Adding a license to your open source art project

Em Lazer-Walker — Tue, 14 Jun 2022 17:44:21 +0000

As someone who spends a lot of time straddling the world of “professional” corporate open-source software and digital artists making weird new tech, I encounter something regularly that makes me sad.

Someone will release a cool art project they’ve made, or a creative tool, or some other awesome bit of creativity! They're excited to put it out in the open so that others can use what they've made and learn from their process! That’s great! However, they didn’t include an open-source license. That’s not so great!

Why do you need a license?

By default, if you put source code (or art assets) up on the Internet, that does not mean that anyone is allowed to use it. You still own the copyright to that code, and anyone using your code is committing copyright infringement. In order to for someone to legally be able to use your IP, you need to explicitly grant them the right to do so. In practice, if you publish your project's source code and assets online without a license, you’re probably unlikely to sue someone who uses those assets, but on paper that’s illegal infringement.

Regardless of whether your goal is restricting the use of your source code to specific uses or just telling people “hey, do whatever you want with this”, being straight-forward about your intentions is valuable. Telling someone explicitly what you do or don’t want them to do with what you’ve made, instead of needing them to assume what rights they don’t have, is going to lead to people feeling more comfortable looking at and potentially using your work!

Additionally, if your project accepts contributions from the community, if you get any contributions that don’t include a license, you technically don’t have the right to use those changes! A model of “this is the way my code is licensed, and anyone contributing is agreeing to the same terms” is a valuable way to protect yourself.

The solution to all of these ambiguities is to add a license: a bit of legalese to this git repo that explains the legal IP rights that anyone visiting this repo has to the code and assets stored in it.

What license should I choose?

There's sadly no one "correct" answer for what license you should use. Even though this is relatively cut-and-dry for e.g. open-source infrastructure projects built by tech startups, it gets fuzzier for art projects like this, and there is a lot of wiggle room depending on your goals.

I've got a few different recommendations below, but first it's worth separating out that we're talking about two different things here. On the one hand, a license explicitly grants legal IP use rights to anyone who comes across this repo, and makes it legal for them to do certain things with your IP. On the other hand, you are presumably trying to communicate intent about how you as a creator want people to be able to use your work.

In an ideal world, these two things are one and the same! In practice, OSS licenses are rather blunt tools, and it's useful to separate out "what am I communicating about my intent?" from "what rights am I legally granting?".

That said, here are three(ish) good options to consider, with the caveat that I am not a lawyer and this is not legal advice.

An OSS license and a Creative Commons license

A tricky thing is that "open source licenses" are written with code in mind, and it's a bit ambiguous how they apply to non-code things like art assets. A good rule of thumb is that if a piece of IP can be versioned the same way one would use source code, an open source license might be a good fit for it -- this means that things like hardware design files from software like KiCad can probably be safely protected under an OSS license, but 2D images or music might benefit from other licensing options.

Outside of a few special cases (e.g. the SIL Open Font License for fonts), the gold standard for licensing binary assets is Creative Commons, who maintain a large number of licenses allowing for different types of use.

The inverse applies as well: because licenses like Creative Commons focused on artistic works don't care about the difference between binary distribution and source distribution, this makes them relatively unsuitable to use for covering code (even Creative Commons themselves don't recommend it).

A common technique is to use multiple licenses within the same open source repository: all code is under one OSS license, and all non-code assets are under a Creative Commons license. This is a great option to remove ambiguity for projects that include both code and not-code.

Choosing a Code License: MIT License vs GPL

There are a large number of open-source licenses you can use to license your source code. I want to focus on two today, the MIT License and the GPL License. Both are extremely popular, arguably moreso than any other open-source licenses (they're the two licenses GitHub points people towards with their Chose a License site). There are many popular licenses similar to the MIT license, but comparing MIT and GPL is useful since they're examples of two very different philosophies.

The MIT License basically says "you can do whatever the heck you want with my code, as long as you give me credit".

The GPL basically says "you can do whatever you want with my code, as long as any changes you make to my open-source code are themselves made open-source and licensed under the GPL".

The MIT license grants "freedom" in the sense that anyone can use your code to build a proprietary commercial product. The GPL grants "freedom" in the sense that, even though it restricts what people can do with your code, it does so in a way that leads to more open-source software and thus net more "freedom" for end users. This concept of freedom the GPL represents is often referred to as "copyleft" (as opposed to "copyright")

Most open-source software maintained by large tech companies is MIT or a similarly-philosophically-aligned license (Apache and BSD are two others you see a lot), because the economic model behind that sort of OSS is generally "a bunch of tech companies all contribute to this thing in order to benefit from it in their closed-source commercial products". These corporations would not consider being forced to open source all of their commercial products that happen to depend on GPL'd libraries to be a reasonable outcome, so this sort of corporate OSS tends to default to more "liberal" licenses that allow closed-source use.

There's no clear correct answer. For-profit corporations find financial benefit in making licenses like MIT more common, but that doesn't mean that's the right choice for you.

It's also awkward and worth noting that the GPL's author and main evangelist for decades is an alleged sex pest who (among many other things) resigned from his position at MIT after public comments defending Jeffrey Epstein. Regardless of how you feel about the ideas and ideals underlying the GPL and the Free Software Foundation, there's a lot of cultural and political baggage there. I don't think that's a reason to avoid the GPL, but it's worth being aware of.

It's also worth noting here that there are weird politics around the term "open source". In order for a license to be considered "open source" by the Open Source Initiative, there are a number of criteria the license must meet. One of those criteria is that it must allow equal use to everyone. You won't see an "open source" license that restricts commercial use, because that by definition makes it not capital-O Open Source. I personally think this is bad!

Various Creative Commons Licenses

As mentioned, Creative Commons is a non-profit that maintains a half-dozen different licenses intended to be used for creative works, rather than source code.

There are a half-dozen different Creative Commons licenses you can choose. They span more or less the same ideological spectrum of MIT vs GPL, but with a bit more fine-grained choice around what restrictions are placed on what people can do with your work.

The Creative Commons License Chooser is a good way to look at your options and how these differ. All CC licenses require attribution (except for CC0, but I'll talk about that later). You have a choice of whether or not you allow commercial use of your work, as well as a choice of whether adaptations of your work are allowed to be shared, not allowed to be shared, or require any derivative works to also be licensed under the same terms as your work (similar to the GPL).

While restricting commercial use in derivative works may sound extremely appealing, Creative Commons themselves are philosophically opposed to it, in favor instead of what they call "free cultural works". Their viewpoint is basically "eh, we don't want to allow this, but it's better that works be CC Non-Commercial than to not have any CC license". In particular, a concern I would have is how loosely "commercial use" is defined, and making sure that you’re actually disallowing the use cases you think you are.

To me, in a lot of situations where I'd be tempted to add a non-commercial clause, I'd instead consider adding a share-alike clause. If what you want is for people to be able to freely remix your work, letting people release commercial products based on it but requiring them to also license those works under Creative Commons feels like a good compromise -- it's philosophically similar to the GPL, but without a lot of the legal or cultural baggage.

Do you lean on the side of leniency or strictness?

Deciding whether to use a MIT-like license or a copyleft license (or whether or not your CC license should have a sharealike clause) is a tough decision.

In my own work, I personally usually explicitly want to encourage others to make their improvements to my work public, but often find that requiring that unintentionally limits some uses you'd like to enable. As a common example, if you would like people to be able to read your code as a reference, the GPL will allow that, but it technically forbids someone from copying a three-line snippet of code and using that in their own project, without making their entire project open-source under the terms of the GPL as well. That may be what you personally want, but in many cases I personally find that to be a bit excessive.

This is perhaps less likely to happen with a CC share-alike license (just because it's less likely someone would pull out and use a tiny fraction of an art asset), but your license virality being too overbearing is still a concern in either case.

Of course, people may still informally use your code in that way anyway, and you may be okay with that (read: not pursue legal action for violation of the terms of the GPL or CC). But choosing a more stringent legal position than you intend to perhaps enforce is a conscious choice you're making in that situation.

Putting it all together

It's worth emphasizing again the distinction between legal IP restrictions and communicating your intent.

I personally lean towards being more lenient in what I allow legally (read: default to using an MIT-style license and a CC license without non-commercial or sharealike clauses), and informally expressing in the project documentation what I do or don't want people to do.

I find explicitly writing "hey, I know you're allowed to do whatever you want with this, but PLEASE don't do X, Y, or Z" is a nice middle-ground of something that isn’t strictly legally enforceable, but communicates to reasonable human beings what I want.

It's also possible that your personal ideological leanings would rather err on the side of unintentionally restricting valid use cases rather than unintentionally enabling use cases you don't want to allow. That's okay too!

So, uh, what does this all mean? If I was choosing a license for this repo, and wanted to go the "OSS + CC" route, I'd probably pick MIT, either CC BY or CC BY-SA ("only attribution required" or "attribution and share-alike"), and put a paragraph in the README explaining my intent. But any of the other things mentioned here are reasonable options!

2. The Anti-Capitalist Software License

This is a unique license in that it adds restrictions on who can use your software. You can do whatever you want with the code, as long as you're an individual, a non-profit, an education institution, or a worker-owned co-op, with a carve-out banning military and law enforcement use.

I'm inclined to view this license through the lens of performance art. While this license probably most directly aligns with how I personally would want a lot of my projects to be used, I've read a number of arguments claiming it almost certainly would not hold up to litigation, and would basically equate to including no license at all (with the repeated caveat that I am not a lawyer, and I additionally suspect many/most of the people posing those arguments are also not lawyers). It's possible this is unjustified fear, uncertainty and doubt; the current tech industry culture extremely stigmatizes any software licenses like this that do not conform to the "proper" definition of "open source".

Along those lines, this WILL incidentally act as a deterrent to many larger tech companies using your work, as management will look at a non-standard license like this and say "the time and effort saved by using this code is worth less than the billable hours for the legal team to vet this license". This is possibly a positive for you.

This license is on the extreme end of tradeoffs: it is an incredibly strong socio-political statement about how you wish your code to be used, but realistically is likely to leave you in the same legal IP situation as if you had not included a license at all. Using this license does have the positive benefit of normalizing licenses like this, which could eventually lead to future iterations of the idea that are more proven to be legally-enforceable.

3. Creative Commons 0

A final option is to place your work in the public domain: you relinquish all rights to them, and anybody can use your IP for whatever purposes they want. This is "chaos mode" -- it's definitely the simplest of all these options, and the most easily-understood by people who don't want to have to become armchair IP lawyers. But it's also the least restrictive, for better and for worse. It's specifically worth calling out that people are not required to provide credit or attribution.

If your intent is to put your project in the “public domain”, you should still specifically apply the CC0 license. Many countries do not have a legal mechanism (or only have limited ways) by which a living IP holder can dedicate their work to the public domain. CC0 is explicitly written to work around this, and allows authors to explicitly waive all possible copyright protection and IP rights they have, to the extent possible in their jurisdiction.

Okay, so how do I actually add a license to my project?

So you've fretted and fussed and finally picked a license or set of licenses that you think are right for your project. What do you do now?

If you just have a single license, this is simple.

For a code license: in your project's repository (I'm assuming you're distributing this project via a git repo, but this could also just be in the folder root if you're e.g. distributing a zip file), include a file called README (or README.md, or similar) containing a copy of your license.

A bonus improvement is to include the license as a header comment in every single code file. This makes it somewhat less likely that your code will get (unintentionally or intentionally) taken out-of-context and have the licensing info removed from it. This is best practices for larger corporate OSS, but if you don't have good technical infrastructure to do that automatically on your small art project I wouldn't worry too much about it.

For a creative commons license: In whatever public documentation your project has (a README, a marketing website, etc), note the CC license prominently. The Creative Commons has great image buttons you can use.

If you have a mixed license, it gets slightly hairier. Combining those two approaches — a LICENSE file and noting the CC license — is broadly right, although you want to be explicit about which files get which license applied to them. This is simpler if your assets are divided up such that you can cleanly say "this specific set of folders are CC licensed, and the rest are using this other code license". There really isn’t a legal standard here for “you MUST indicate things this way or nothing matters”, your goal is primarily making your licensing intent clear and unambiguous.

...and that's it?

If you have clear documentation in your repository about how which files are licensed which ways, that's all there is to it! Your project now has a license, and people can feel more comfortable using your work safely, knowing that they're respecting your wishes and intentions!

This of course isn't a silver bullet. If someone willingly infringes on you, your options are still either to take them to court (potentially expensive and time-consuming!) or just let it be. But having an explicit license minimizes the chance you'll get into that sad situation, and also potentially makes it easier even if you do get into that sad degenerate state.

So you want to run a virtual event

Em Lazer-Walker — Tue, 10 May 2022 16:59:48 +0000

If you're an event organizer right now, you're in a tough spot. You're desperate to get back to running an in-person event, but you know it isn't quite safe yet.

Maybe you've seen in-person events in your community lead to large-scale COVID outbreaks or even deaths, and that terrifies you.

Maybe your planning team disagrees on what an acceptable level of COVID safety is, or you don't have the resources to provide what you believe to be adequate safety.

Even if you could run an event in-person, you're worried about losing the accessibility of a fully-remote event for attendees who are immunocompromised, or who live in far-off places and can't reasonably travel.

Whatever reason, you need an online event. But you're also justifiably worried, because online events tend to suck. And now that people in a lot of the world are seeing their friends in-person and spending time in places that aren't their homes, the prospect of spending a day or a weekend or a week plastered to a Twitch stream seems even more unpleasant than it did in 2020.

How the heck do you actually throw a virtual event worth attending?

I don't have the answers

Unfortunately, this isn't a "how-to" guide. If I could give you a concrete checklist for how to throw an online event that didn't suck, I would!

It's also hard to give concrete suggestions since it's so dependent on your conference's audience, and how "experimental" a space is appropriate for you. I've attended exciting, creative, community-driven in-person conferences hoested in found spaces that needed major modifications to work as event venues (I love to tell the story of the Berlin games festival where the two talk tracks where "in the karate dojo" and "on-stage at the combination nightclub and pool"). I've also attended plenty of corporate conferences in hotel ballrooms or city-owned convention centers.

The former has a lot of leeway to get experimental with online events, while the latter is going to have a harder time convincing attendees to try something different. Knowing your audience is key. A lot of the advice in this article is more focused on people who do have room to explore, but even if you're restricted to using a relatively-buttoned-up turnkey enterprise event platform, there are still usually ways you can nudge the social design of your space one way or another.

If it's not obvious, this article is also focused more on events that look like conferences or festivals — the sort of event where an in-person event would typically be centered around one or more tracks of talks. Designing other types of events, like parties, face all of the same conceptual design challenges, but a lot of the specific tools at your disposal may be different.

The point of this piece is really to convey that designing a virtual event is, in fact, a design problem, and a deeply complex and largely unsolved one at that. I'm hoping I can point you in the right direction for some of the questions you should be asking, and some of the design considerations that should be front-of-mind as you start to create an event for your community.

The first question: do you actually want to throw a live event?

A question I ask to a lot of event organizers is "why are you producing a livestream instead of a YouTube playlist?"

GitHub's most recent GitHub Universe conference completely bypassed the idea of live talks: it being a "two-day event" simply meant that each day featured a new YouTube playlist of new talks to watch asynchronously, plus a fifteen minute "keynote" previewing that day's new talks.

There are benefits to running a live event. A lot of people won't go out of their way to watch a half-hour talk on YouTube, even if it's directly applicable to their interests or professional development. A live conference feeling like a "happening" or a "moment" can give people the push they need to make time in their busy schedule for something they would enjoy and benefit from.

On the other hand, it's a big ask to get people to attend a synchronous virtual conference, especially as more in-person activities become allowed. People feel guilty taking time off work to attend a weekday virtual conference, and don't really disconnect from work when they do. For weekend events, they're hesitant to spend their limited free time staring at a computer screen instead of doing, well, literally anything else. This is especially true if your event has little to no meaningful networking or social interaction driving a sense that people will miss out on something valuable if they just catch the talks on YouTube after the fact.

There's no single correct answer here, but whether you throw a live virtual event or something more asynchronous, be sure to have a coherent reason why that's the correct choice for your community and attendees.

Your event needs to be an event

Why do people attend conferences or other events? They might say it's the talks, or the socializing and networking opportunities in the "hallway track", or for a certain class of professional events they might be honest and say "it's a free work-sponsored vacation".

Any reason is valid. But the real underlying reason people attend your event, instead of finding another way to achieve those goals, is precisely because it's an "event". It's the act of traveling to an in-person conference that allows your brain to shift into a new headspace where you're receptive to new ideas or meeting new people. It's why companies run "off-sites" for big-picture brainstorming: that context shift is essential.

It's hard to get this for an event that people attend from the comfort of their own home. It's even more difficult when you're asking people to use software — Discord, Slack, Zoom, Twitch — that they already use in their day jobs and social lives. There's no sense of moving to a new space. It's no wonder people find it hard to disconnect from their work when they're sitting at their work desk and using their work software!

How do you create that sense of place?

This is the million-dollar question.

With Roguelike Celebration, a game design conference I run, our answer was a custom-built social space blending design elements of chat apps like Discord and Slack with MUDs, the text-based precursors to MMORPGs. Consistent feedback we've gotten across two years of events is that, despite being a text-based chat space, attendees feel a sense of physical presence. They describe coming back year after year as having the same feeling as going back to an in-person venue, and they tend to use a lot of the same language that VR enthusiasts use around "presence" despite it being a text-only space. That's really cool!

I wouldn't necessarily start with trying to build your own custom event platform like we did, even if you have the resources. Instead, get creative: what existing tech platforms can you repurpose for your event that will help it feel special?

Online spatial chat platforms like Gather Town or Skittish can be nice, but require a lot of customization (I'll talk more about them later!). A lot of event organizers I've spoken with lately are interested in repurposing existing online spaces — like, say, finding a free MMO or online game that can serve as a social space — which I think is a really interesting path to explore, even if I haven't seen concrete success there.

Finding the right "found space" can also be tricky. I've seen events flop in spaces like Second Life, as anything that's 3D rather than 2D will have accessibility barriers to basic things like navigation unless somebody is already familiar with 3D games. I've seen events held in Roblox fall over from a technical standpoint, as its networking is generally extremely robust but has scaling issues if you want more than a dozen or so people in your space at once. I mention these not to say "X tool is bad!", but more to emphasize how important it is to test a potential space to the extent you can before committing to it.¹

If you have access to some development resources, a nice middle-ground might be building on top of existing open-source projects. Roguelike Celebration's social space is on GitHub, as is LIKELIKE Online, a 2D pixel art gallery space. If you go down this route, it's worth emphasizing how much custom-built event spaces are, in fact, built for their specific contexts, and how well they could work for your event depends on how much context overlap there is. I don't know how well Roguelike Celebration's setup would work for an audience that doesn't get immediately excited at the idea of a text-only world. Similarly, LIKELIKE does a phenomenal job of replicating the vibes of an art gallery opening party, where attendees can vibe with the energy of the space and briefly say hi to people they know, but the way it "feels" like a loud noisy party means it'd be a poor fit for a networking-focused "hallway track" where you want people to be able to have in-depth conversations with each other.

Case Study: Deserted Island DevOps

It's worth noting that you can create this sense of place without necessarily providing interactivity. I want to talk about a really clever and resourceful example: Deserted Island DevOps. A conference that first took place in April 2020, at the start of the pandemic and the height of the popularity of Animal Crossing: New Horizons for the Nintendo Switch, it's a conference that "took place in Animal Crossing".

Now, if you've played Animal Crossing, you know that "a conference in Animal Crossing" would be unwieldy and impractical. Animal Crossing has great multiplayer features, but it allows a maximum of 8 people in the same island, and chat options are limited.

In practical terms, Deserted Island DevOps was a "boring" virtual conference: it was little more than a Twitch stream and an associated Discord. The "Animal Crossing" piece came from the Twitch stream itself: a video feed of an Animal Crossing instance, showing the speaker's Animal Crossing avatar (they had full control over their avatar's appearance and could trigger custom emote animations) and their slides composited into the scene.

The event was a huge success, with nearly 12,000 live views over the course of the weekend, and a lot of buzz on social media for how innovative their approach was. I think a huge take-away from Deserted Island DevOps is that creating a sense of place doesn't necessarily mean creating a novel interactive space people can navigate. Sometimes sufficiently evocative theming can be enough!

To videochat or not to videochat?

A thing I have learned is that people have drastically different opinions about videochat. Some people desperately crave the ability to get literal "face time" with other event attendees. Others can't stand the idea of having to turn on their webcam to talk to strangers. Similarly, some people easily prefer audio-only chat to videochat, while others can't stand the loss of social cues that comes from having audio but not video. I'd urge you to make sure the design of your virtual event caters equally to all of these groups.

Videochat is a high-trust activity. People in videochat are more vulnerable to harassment than text chat, and moderation is also trickier when you don't have a perfect record of everything said. Different communities and sets of conference attendees have different levels of trust; while there are plenty of trust and safety tools you can use to mitigate issues once your event has started, I'd urge you to both think about how to cultivate as healthy a community as you can before your event begins, and also approach designing videochat at your event with an honest eye towards where your event's community falls on the trust spectrum.

In the past, I assumed the ideal was an event where attendees could freely choose to consensually escalate from text chat to audio or video at any point during the conference. For Roguelike Celebration 2021, we built out our own custom videochat, and spent a great deal of time designing an exceptionally thoughtful videochat system that did just that, seamlessly integrating videochat and text chat in our social space. To be blunt, our attempt was broadly a failure, with the focused exception of using our videochat for post-talk speaker Q&A breakout rooms.

Humans instinctively prioritize paying attention to people with higher-fidelity modes of communication. I don't know how you design videochat that doesn't give preferential treatment to video users over audio or text users. This is definitely true if using an off-the-shelf event platform or videochat tool, but even with a custom-built solution it's an unsolved problem. And that's even before tackling issues of Zoom fatigue; even people who love videochat will get tired if they stay on video for too long.

Videochat as a focused tool

Instead, I'd encourage you to think of videochat as a focused tool you can apply at specific times during your event. A lot of people who might not want to spend 8 hours on a call would probably love to spend a focused hour or two with a higher-fidelity chance to connect with other attendees. If these sessions are time-limited and opt-in, it's also easier to feel comfortable prioritizing videochat users over others in these sessions.

One specific hard problem to design for is allowing people to "preview" conversations. In real life, it's easy for someone to hover at the outside of a conversation circle before choosing to engage or find a different conversation. Most online tools do a poor job of recreating that dynamic — joining a Zoom call and then leaving ten seconds later feels socially awkward, as does walking up to a chat circle in a tool like Gather Town and then leaving. There are ways to minimize that awkwardness can with either technical or social solutions, but it's a hard problem you need to think about and actively solve for.

There are many ways to design focused video time. At Roguelike Celebration, we do "unconferencing" sessions, where people can self-select into Zoom rooms based on conversation topics proposed by attendees. I've also seen a lot of success with "speed dating"-style breakout rooms, where people are placed in random conversation groups for a preset period of time before being automatically split up and reformed into new groups.

I don't have a lot of concrete advice around unconferencing other than having decent tools to allow proposing and upvoting topics (and having an answer to the "how to poke your head into a room" problem), but it is a model that works really well for us.

Videochat breakout rooms / "speed dating" / "coffee chats"

If you want to try a "speed dating" or "coffee chat" model, that's typically easy to implement using the "breakout room" functionality of most commercial videochat software. The rigid, predictable structure makes it convenient to make new people, as it hand-waves over a lot of the awkwardness of how to start a conversation with someone in the first place.

The tricky thing is this pattern really requires you have a high-trust environment, even more than other videochat techniques. Because you can't simply leave a conversation while it's happening, it's easy for underrepresented groups to get frustrated if they feel like they aren't being listened to. You probably won't receive formal code of conduct reports, people will just quietly stop participating. While there are social tools you can use to try to shape these conversations to be healthy, the success of a session like this really hinges on your attendees being kind, respectful, and generous. If your community is on the lower end of the trust spectrum, this specific technique might not be right for your event.

If you do think that sort of "coffee chat" is appropriate for your community, I'd recommend groups of 3-4 people, 5 minute chats, and some sort of formal or informal structure for how to break the ice in each chat. 1-on-1 chats and longer durations can both work in higher-trust environments, but most of the time you want to aggressively optimize for mitigating just how miserable it is to be stuck in a bad conversation.

Ice-breakers depend a lot on the context. I've seen professional networking sessions use a convention where each person starts by giving a 30-second intro to themselves, and I've seen people doing this in context of a creative retreat naturally lead with "so, what are you working on?".

Facilitators: you probably need them

Regardless of what your videochat structure looks like, if you plan to have more than 5 or so people in a single videochat, you probably want facilitators to help keep the conversation moving. In work contexts, someone usually runs the meeting and plays this role; in social Zoom chats with friends, someone probably informally picks up this mantle. At an event where you are hoping that strangers or pseudo-strangers will talk to each other, making sure that you have a volunteer or staff member to play this role (instead of hoping an attendee might) is a good way to stack the deck and make sure things go well.

This obviously depends a lot on your format. If you're using a spatial chat tool, it might be awkward to have people whose explicit job is to go around and start conversations. But if you're going to invest in making videochat work for your event, it's worth thinking about how having a few human facilitators can help ensure the experience is a positive one.

How to create social interaction

At in-person events, there's a wide range of social interaction people might take part in, with varying levels of interactivity or room for freeform conversation. Sitting down to discuss Serious Topics with someone one-on-one is different from making small talk over the cheese table is different from playing a game of cards where the majority of the space in the conversation is taken up by the logistics of the game itself.

People are very good at subconsciously navigating these varied spaces and choosing the form of interaction for them that's right for them. But this requires an event that provides that wide variety of interactions. If your event just provides a single way to "network", it'll feel flat, and a large number of your attendees won't feel comfortable chatting.

Like so much else I've been talking about, I sadly can't give you a checklist or playbook to just implement. Figuring out how to provide that range of activities is so highly dependent on both your audience and on your event platform. It's also really hard to talk about! I don't think Roguelike Celebration is perfect at this yet, but it may be instructive for me to outline some of the conversation opportunities we intentionally create or support. Going from most involved to least involved:

Each day, we host "unconferencing" sessions, where people can propose topics (generally design or technical subjects) to discuss in focused Zoom calls
After each block of talks, speakers can optionally break out into a dedicated room for Q&A with attendees. Generally, the speaker is on a videocall while other attendees use text chat
During the mainstage talks, we encourage a vibrant Twitch-style chat. With our audience, this generally leads to a good blend of insightful discussion/commentary and silly memes
During the talks, people can also submit and upvote questions to be asked during any formal moderated Q&A time left at the end of a session (separate from the speaker breakout rooms). This gives people a chance to contribute to the discussion who are overwhelmed by the pace of the real-time chat
During unstructured breaks, we've seen a pattern of people running their own playful or game-like activities in the space, ranging from running short tabletop RPG sessions to someone hosting a fortune-telling table
In 2021, we introduced a short puzzle hunt / chain of riddles, which gave people an activity they could either do solo or team up with others and ask for help on
Our space has consistently included a number of "fidget toys": objects you can pick up and carry with you, magical potions that append an emoji to your name, ways to interact with the space and see a random text generator do something silly. These give you an easy way to strike up a conversation with someone else by, say, commenting on their emoji, or asking where they got their item, and also give you a way to passively do something in the space alongside someone else without talking.

It's worth calling out that our attendees spontaneously playing formal games and doing fortune tellings is not something to plan for. In most cases, that sort of attendee self-organizing doesn't happen, or happens relatively infrequently. If you want to try to encourage people to play online multiplayer party games together (which is a great form of social interaction!) I'd put it upon yourself as an organizer to make it happen. It's a lot easier to get attendees to jump into a game of Jackbox, say, if a facilitator actively schedules it, sets up the game, and tells people where to show up, rather than hoping an attendee will spontaneously choose to do all that work.

If this set of problems is something you're interested in solving or diving deeper on, I'd also recommend you read my earlier article about Roguelike Celebration, as I go into a few other topics not covered here. Kate Compton's talk about technology design for social creativity has also been formative in my thinking.

Purpose-built spatial event platforms

I've seen a lot of interest in online event platforms specifically built to be fun spaces that lean on playfulness to provide an alternative to traditional video calls. There are a ton of these "spatial" event platforms: Gather Town is perhaps the best-known, but I have familiarity with Skittish, Wonder, and Bramble, and you could easily find a dozen more.

They broadly share similar featuresets: each attendee controls an individual avatar in a custom space (almost always 2D, or 2D movement within a 3D space, often with a retro pixel-art aesthetic), and when you walk close to another attendee, you can see their webcam feed and/or hear them speak via microphone. The core idea behind these sorts of platforms is that spatial audio/videochat recreates an aspect of in-person events, where you can walk up to somebody and have a conversation with them without it being a giant unwieldy video call.

Different platforms differentiate themselves through things like their default aesthetic, their editing tools, whether you can host a live presentation in the space, the specifics of their audio/videochat, and things like that. But the core value proposition is often broadly similar. I often see these either adopted as the core platform for an event, or just used as the venue for a specific scheduled 1-2 hour "social hour" or networking session.

If you're considering one of these tools, the main thing I would urge you is to recognize them as just that: tools. Just because you may be using an event platform that allows people to walk up to each other and strike up a conversation, that doesn't mean they will unless encouraged to do so through thoughtful social design. You absolutely can build a space that accomplishes the sort of goals I set out above using tools like these, but in most cases I'd recommend thinking of them as jumping-off points to build on rather than ready-made solutions that will solve the hard social problems for you. This holds true whether you're looking to hold a "social hour" in tools like this or are hoping to use them as your primary event platform.

The linguistics conference LingComm has a great series of articles about running their 2021 online event, most notably one about the design of their Gather space (also to read: LingComm organizer Gretchen McCulloch's Wired article about these sorts of tools). If you're looking to use one of these readymade spatial event platforms, I think LingComm is a great example of the sort of care and effort you need to put into designing your space (disclaimer: both of those articles liberally cite my work with Roguelike Celebration :P)

Counter-intuitively, you may also have more success using one of these tools that is less popular. As I've discussed, a big part of the value of using a "non-traditional" event platform is the novelty of a space affording a context shift. If other events in your community regularly use one of these event platforms, it may already feel "same-y" to attendees. In other professional technical contexts, I often advise people to not make technology choices based on what's new and shiny, but that may be exactly what you want here.

Sidenote: VR spaces

One specific sub-flavor of this sort of chat tool you might encounter are VR spaces like AltSpace, VRChat, or Mozilla Hubs that are primarily focused on VR but also usually support flat displays or smartphones.

My advice here is simple: this is a great option if you're working with an audience that is already intimately familiar with VR or 3D virtual worlds. Otherwise, the technical barriers of getting people onboarded in a first-person 3D space are too large. You'll waste your time and your attendees will be frustrated.

If you do happen to be running an event for a VR audience, I urge you to explore your options for yourself, but my informal opinion is that AltSpace is best for large events where thousands of people will watch a single presenter and you need to shard audience instances, while Mozilla Hubs is the most flexible (and the most accessible across a wide range of non-VR devices) for smaller and more informal events.

How do you schedule it?

One of the biggest strengths of virtual events is that you can welcome attendees from all over the world without requiring expensive travel. Supporting that through a schedule that's equally inclusive is surprisingly difficult. It's easy to schedule an in-person conference: your conference day aligns with the working day. That doesn't necessarily make sense when all of your attendees are in different time zones.

I've seen two common approaches here, depending on your resources.

24-hour conference

One approach is to schedule a "24-hour conference". No matter what time zone an attendee is in, there will be something happening during the day their time. This is great for the way it doesn't center a specific region: it feels conceptually more inclusive, and concretely makes it easier for you to attract audiences outside of the geographic regions you usually operate in.

It's also difficult to pull off, requiring having strong geographic diversity for both speakers and conference staff to make sure that all times feel equal. If you go down this route, I'd also consider how you can intentionally design your schedule during common overlap times to encourage groups to mingle, so it doesn't just feel like your conference has different "shifts" where people only interact with people from the same region as them. Similarly, if you run a 24-hour event but don't actually have a globally diverse pool of attendees, you'll end up with "dead" periods of time, which also isn't great. A global-first event schedule needs to be accompanied by a global-first marketing plan.

A core time zone

The alternative is to pick a "core" time zone and schedule around that. Roguelike Celebration, for example, is scheduled around Pacific time, since we were historically hosted in-person in San Francisco. This is much easier for us to staff, although requires careful consideration to still be as accessible as possible. We need to be as accommodating as possible to speaker schedule requests, so someone isn't stuck speaking in the middle of the night their time.

We also consciously schedule our event so that people in non-Americas time zones can experience as diverse a sample of content as possible during their daylight-hours overlap. For example, we host "unconferencing" sessions each day, where attendees can post topics they'd like to discuss and then hop into a Zoom call with other attendees. We intentionally schedule these at different times each day — typically one in the morning and one in the afternoon — to try to maximize the chance that someone in Europe or the APAC region can at least attend one of them.

There's no one "correct" answer here. Maximizing the global accessibility of your schedule is a tricky problem that cuts across almost all of your other planning concerns.

Hybrid Events

Maybe you're really itching to run an in-person event, and you think it's possible to do something small, alongside a virtual event.

I have a hard truth for you. It's absolutely possible to run a successful "hybrid event", but it requires having a successful virtual event as a baseline. If you start from the standpoint of running a good in-person event, and attach a virtual component to it, virtual attendees will correctly recognize that they're an after-thought.

You need to start by designing a top-tier virtual event experience, and then from there figure out how to augment that experience with in-person moments. If you're not taking that approach, you're not running a "hybrid event", you're running an in-person event with a secondary livestream.

It hasn't happened yet, and I'm deeply critical of their almost total lack of COVID safeguards, but Apple's upcoming WWDC developer conference tentatively seems like a great example of how to do this right. WWDC is a weeklong virtual event; separately, there's a one-day in-person event you can submit a request to attend, that primarily consists of a viewing party for the WWDC keynote and Platform State of the Union speech as well as some tours of additional on-site Apple spaces. Presumably the keynote and SOTU are still being produced as livestream-first events, but allowing some people to capture the magic of an in-person live keynote experience (which was previously a tentpole of in-person WWDC) seems like tentatively a good way to allow some in-person camaraderie while not creating too much FOMO for attendees of an otherwise virtual-first event.

Odds and ends

There are a lot of other important aspects to running a virtual event, but talking about them starts to become more logisical than conceptual. I could write a whole blog point about each of these (and have in many cases!) but a few quick things to keep in mind:

Your event should ideally have live captions, provided by a human captioner. I've written about the logistics of this. The short version is that it requires budgeting, but relatively little work, and is infinitely better than using AI-generated captions. If you can't afford a human captioner, make sure you have automated captions.
Make sure your schedule is easy to understand regardless of whether someone is in the same time zone as the organizers. I recommend investing in a little bit of front-end JavaScript code so your website automatically shows times in each attendee's local time zone.
Your virtual event is likely more accessible due to people being able to attend worldwide, but that doesn't mean it's financially accessible. What you consider a "cheap" ticket might not be that for people elsewhere in the world. I've seen a handful of conferences adopt pay-what-you-want pricing — Roguelike Celebration has suggested tiers including "pay for yourself", "pay for yourself and someone else", and "I can't afford to pay". Our revenue has been roughly the same as it was before introducing that PWYW structure, and I've heard the same thing from other organizers using the same pricing plan.
How to technically host remote talks is an extremely deep subject (that I've written about!), but big picture, have a dedicated AV person on duty instead of just your MC, and use a hosted all-in-one platform (like Streamyard) rather than fiddling with your own OBS setup. Run AV tests beforehand with every speaker, allow them to pre-record if they'd prefer, and be respectful that speakers may have valid reasons to not show their face on-camera. Ideally have a budget to pay for cameras/mics for those who need them.
If the thousands of words you've just read weren't enough from me: aside from the various posts of mine I've linked throughout this piece, I also wrote about building a better hallway track (this was the earliest design thinking that eventually led to the Roguelike Celebration space) as well as a list of questions to ask yourself when designing a synchronous online social space.

I hope this massive brain-dump was helpful! If anything in here sparks inspiration for you to do something different in your own virtual event planning, I'd love to hear about it — my Twitter DMs are open.

A quick hot tip to share the sort of brainspace I'm currently in: I'm convinced, for the right experimental event, BYOND would be an incredible tool to build a custom space. It's an early-2000s low-code/no-code tool for building online multiplayer games, most notably Space Station 13. Getting the editor to run on a modern Windows 11 PC can be a challenge, but I'd love to see an event run in a custom-built BYOND space. ↩

Why Video Chat is a Hard Technical Problem

Em Lazer-Walker — Fri, 12 Mar 2021 17:06:56 +0000

Back over the summer, I began a series of experiments to play around with new forms of synchronous online social interaction while we're all stuck at home. These ranged from a virtual conference hosted in a custom text-based MMORPG to using real-time mocap in the browser to make 2D animated avatars:

// Detect dark theme var iframe = document.getElementById('tweet-1272894598214492160-818'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1272894598214492160&theme=dark" }

For these early experiments, I used WebRTC, a browser-based peer-to-peer videochat technology. Since I was churning out small experiments quickly, I cared about being able to build something as quickly as possible, and ideally without having to spin up complicated and/or expensive servers.

WebRTC sounds like it's perfect for this! Being peer-to-peer means you don't need complicated or expensive server infrastructure, and being a well-supported piece of browser tech means there are a lot of educational resources out there.

To jump straight to the punchline: after we built a WebRTC-based videochat service for Roguelike Celebration's event platform, we ripped it out and replaced it with a series of Zoom links for the actual event. Our WebRTC setup simply wasn't viable for production use.

I've since talked to many other folks who built out WebRTC setups, ranging from simple to complex, and similarly ran into unacceptable performance pitfalls. This doesn't mean that WebRTC as a technology isn't viable for things like this — all of the solutions I recommend later in this article ultimately still use WebRTC under the hood — but reality is significantly more complicated than just reading the WebRTC API spec and building against it.

The rest of this article will walk you through our learning process, and what we learned is necessary to make a WebRTC videochat setup work in a production environment. Our path to functioning videochat was long and winding; I want to outline what we learned to save other people from spending the same time and effort we did to come to that understanding.

Problem 1: Accessing AV Hardware

Before we even get to sending audio and video streams over a network, we need audio and video streams. This means using the browser MediaDevices API, not yet WebRTC. But this has a catch!

The API is simple. You call navigator.mediaDevices.getUserMedia() and get access to audio and video streams. The catch: the user doesn't get to specify which specific input devices they want to use, so someone with multiple microphones or webcams is going to have a hard time. You'd assume web browsers would provide their own UIs to let users select devices, but the reality is complicated.

If someone is using Firefox, they will in fact get a nice friendly popup asking which audio and video input they want to use. If they're using Chrome, that option is hidden deep in a settings menu, and it's extraordinarily bad at remembering your preferences. That UI doesn't exist at all anywhere in Safari.

Solution: building a production-ready app means you'll need to build your own in-app device selector for available audio and video inputs.

This is doable, but a pain. You also have to deal with inconsistencies in the ways different browsers surface the MediaDevices APIs for accessing that data. Ideally, you're using some sort of persistent local storage (e.g. the localStorage API) so you can remember the user's preference and not make them navigate a dropdown every single time they enter a chat.

Problem 2: Making a connection

Okay, so you've got proper audio and video streams, coming from the correct local input devices. Now we need a way to send that to other users!

The most straight-forward way to do a group videochat in WebRTC is using what's called a full-mesh network topology. This sounds complicated, but it just means "every client is connected to every other client". If there are 3 of us in a chat, each of our web browsers has a direct connection to each of the other two people's web browsers, and a new person joining would immediately initiate three new connections to each of us.

To open a WebRTC connection between two clients, one client generates an offer. The other client accepts that offer and generates a response. The initiating client accepts that response, and you're off to the races.

To send these offers and responses back and forth between clients, you need some sort of data transport mechanism. And since you don't yet have a WebRTC data connection you can use, this means you'll need some sort of server infrastructure. Building and scaling a backend to exchange handshake strings between clients is a lot less work than building one to send video data, but it's not nothing.

Solution: You'll need to build your own server backend that can transport strings between clients until they successfully open a peer-to-peer connection.

WebSockets are a great choice for this, but WebSockets are also a pain to scale compared to regular HTTP servers. I personally use a combination of Azure Functions and Azure SignalR Service to do this handshake (in an architecture similar to what I outline in this article), but this still requires maintaining server-side services!

Problem 3: What if network settings mean clients can't connect?

Let's say you've built out a simple WebRTC flow, where 4 different people are all connected to each other. This means there'll be 6 different WebRTC connections across all participants. You'll quickly find something pretty weird: chances are, at least one of those 6 connections will fail and two people won't be able to videochat with each other.

The short explanation for this is router settings. After the WebRTC signaling handshake is complete, a remote service called ICE tries to directly connect the two clients by getting publicly-accessible IP addresses for both.

An ICE service will first try to use a STUN server, which is a server that basically exists to tell a client what its public IP address is. In the ideal case, this just works to give you working IP addresses for both clients, and you're done.

If one or both clients are behind a particularly protective NAT layer (e.g. due to a corporate firewall), that STUN public IP dance isn't going to work. In that case, both clients need to connect to a relay, called a TURN server, that forwards all messages between the two since they can't connect directly.

If you're interested in a more detailed technical explanation for this issue, this article is a great resource.

Conventional wisdom says that about 80% of WebRTC connections will succeed with only STUN. This means that, unless you have a TURN server to fall back to, about 20% of all connections will fail!

Solution: Run your own TURN relay server for when clients' NAT settings don't allow them to connect directly.

STUN services are cheap to run, and it's pretty easy to find free ones that can scale with your prototype. Since TURN servers are more resource-intensive (given they're active beyond just the handshake stage of a connection), you'll probably need to host your own rather than find free community options.

One option is to use Twilio's hosted TURN service. Another is to host your own Docker image on a cloud provider such as Azure.

Problem 4: What if too many people are connected?

At this point, you've got a working videochat app. You've built your own AV selector UI to let people pick their devices. You've built server infrastructure to let clients complete offer handshakes. You're running a TURN server to make sure that everyone can connect regardless of their network setup. This all sounds great.

And then, you try to have a videocall with more than 4 people and your computer comes to a grinding halt.

This "full-mesh" setup - where each person in a 4-person videochat is sending and receiving video data from each of the other three participants - is incredibly wasteful.

For each additional participant, your own bandwidth and CPU/GPU consumption increase linearly. Even on a pretty beefy computer with a solid fast network connection, performance usually anecdotally starts degrading somewhere above 4-ish video participants or 10-ish audio-only participants.

And that assumes a solid network connection. If one participant has slow Internet speeds, ideally other clients would start sending them a lower-bitrate video stream, but that sort of selective real-time transcoding really isn't feasible to do in the browser.

It's worth noting that this is not just a technical concern but an accessibility issue: by building a system that falls over unless you have a top-of-the-line computer and a blazing fast Internet connection, you're building a system that only serves the most privileged.

There's no clear fix here other than not having to send out your same audio/video stream N times and having to simultaneously decode and present N remote A/V streams.

Solution: Move away from a full-mesh peer-to-peer system in favor of a centralized system, most likely a Selective Forwarding Unit (SFU).

A SFU is a server that acts as a single WebRTC peer to send and receive video data. Instead of connecting to all of the other people using your chat app directly, your client just connects to the SFU and sends its A/V streams to that single source. The SFU selectively decides which other connected clients should receive a given audio or video stream, and can also intelligently do things such as dynamic video reencoding to serve lower-bitrate streams to clients with lower bandwidth caps.

There are many different ways to run a SFU, but one common way is integrating the mediasoup library into your own Node.js server so you can configure and scale it exactly how you would like.

...but that's A LOT for just doing basic video chat!

I agree! My goal was initially to build some fun little prototypes of novel social interaction patterns, and instead I found myself deep in the technical weeds of networking protocols and peer-to-peer network topologies.

I hope this mile-high overview of the tricky bits of implementing WebRTC can at least get you to understand why this is a hard problem, and give you the lay of the land for coming up with your own solution.

In particular, I have two concrete recommendations:

If you're just experimenting, start out by using a fully-hosted video solution such as Azure Communication Service or Twilio Programmable Video. You'll get an easy-to-integrate API that doesn't require running your own server backend, audio and video chat that automatically scales to any number of simultaneous users, and relatively minimal costs for prototype-scale use.
If you're building a production piece of software where video or audio chat will be a core component, a hosted solution is still the most effort-free option, but you may want to build your own solution to save costs and have more control over your infrastructure. If that's the case, jump straight to running your own SFU. Trying to just get by with a full-mesh topology and maybe a TURN server is ultimately not going to be good enough. Learn from the experiences of myself and countless others and save yourself the time and effort.

Has this helped? Come up with your own solution to recommend? Let me know on Twitter, I'm always happy to hear from more folks tackling these hard problems :)

An (Incomplete) List of Questions To Ask When Designing a Synchronous Online Social Space

Em Lazer-Walker — Mon, 04 Jan 2021 16:52:51 +0000

We're in an exciting period of change where we're still figuring out what online events should be.

Most would agree that, whether you're talking about a birthday party or a professional conference, a large grid of faces in a group video chat likely isn't the ideal setup to foster meaningful online interaction. But we don't really know yet what is the ideal setup!

A lot of people are experimenting with new technical platforms to better support spontaneous interactions and small group conversations in online settings (myself included!), but it's still the wild west.

Here's a list of questions to ask yourself as you're trying to design a more thoughtful space for online communication. Many of these are unsubtle leading questions with a 'correct' answer, but others are more open-ended and more meant to encourage reflection.

This list emerged out of my work on the text-based social space that powered the Roguelike Celebration conference as well as some future events. My goal is to make sure that you're thinking about the right design elements and principles to create a well-considered environment for your event attendees.

To be clear, I'm talking about spaces meant for synchronous real-time communication, and generally (but not exclusively!) about temporary spaces for time-limited events rather than longer-persisting spaces. Think more meetups, parties, and conferences, than coworking spaces, persistent virtual worlds, or traditional social media.

Although a lot of the conversation is currently focused on cartoony 2D environments that include spatial video chat (Gather Town is currently the highest-profile, but I could easily name a dozen competitors), I'm trying to ask questions that are applicable regardless of whether you're a solo event host throwing a party on Zoom for your friends or are a VC-funded startup building a VR platform for 3D virtual worlds.

Input Methods

How do you mitigate or minimize exhaustion from the performative nature of extended group video chat (aka "Zoom fatigue")?
How do you accommodate people who are more comfortable communicating over text or audio chat instead of video?
If you allow multiple input modalities (text vs audio vs video, VR head and hand tracking vs mouse and keyboard or game controller, etc), how do you ensure that people with lower-fidelity communication methods don't feel like "lesser" attendees than those with a wider range of expression?
Are your software and communication methods fully accessible? Can people with vision impairment or low vision, people who are deaf or hard-of-hearing, and people with motor impairments all use your tool to communicate with each other?
What languages does your tool support? Can non-English speakers understand or read your UI? Do your user-editable text inputs (usernames, text chat, etc) support non-Roman alphabets and right-to-left languages?
If you provide a multitude of input methods, is it clear to attendees what their options are and what the tradeoffs are? This is especially true if you offer options that may be mutually exclusive in practice, such as turning on your webcam vs wearing a VR headset

Demographics

What is the average session time for an attendee to your space? A few hours for a meetup or party, a few days for a conference, a persistent long-term coworking space, etc...
What is the average number of attendees to an event in your space? 10 people, 100 people, and 1000 people have very different needs.
Do attendees to events in your space typically already know each other, are they all strangers, or somewhere in between?
If the answer to any of the previous questions is "it depends", what tools are you providing event hosts to make sure their event space is well-tailored to the needs of their specific event?
If there are specific types of events your space is better suited for, how do you communicate this to event hosts?
Do the hardware requirements and level of technical involvement required to access your space match the capabilities of an average attendee to your events?

Fostering Social Interaction

In what ways does your space provide activities or interactions that can serve as conversational hooks to encourage discussion?
What level of attention is required to do these activities? Do you offer activities with a range of involvement levels to allow attendees to self-select for how much room for freeform conversation they want versus focusing primarily on a structured activity?
Separate from level of involvement needed, do your activities provide varying levels of structure and rules, to accommodate people with different levels of creativity and willingness to jump in and try something new? Some people are excited to improvise and play make-believe with little to no prompting; others need more encouragement and structure to make it feel socially acceptable to engage in playful activities.
Do different types of activities or interactions appeal to different types of attendee personalities? The Bartle taxonomy of player types may be a helpful, if incomplete, lens
How "mandatory" are all of these activities?
To what extent are these activities or interactions explicit and broadcasted versus being secrets hidden throughout your space? How do you balance encouraging as much involvement as possible with creating a sense of exploration and mystery?
To the extent that your space has secrets or elements that are less obvious, how does knowledge-sharing about that tie into your other social scaffolding and conversational hooks?
If your event is centered around a singular activity (e.g. a talk or series of talks), how do you balance between pushing people to attend that activity versus allowing or encouraging people who would prefer to keep participating in the "hallway track" instead?
If your space is meant to host larger gatherings, how do you foster and encourage smaller group conversations?
Once attendees are having smaller conversations, how do they find new people to talk to or move to a different conversation?
If I don't know anyone at an event, how do I find people to talk to with similar interests as me or who want to talk about the same things as me? Is there a way for me to signal my interests, or a place I can go to indicate what I'm looking for?
If there is technical or design friction involved in moving to a new conversation (rather than social friction), is this an intentional choice designed to create specific conversational dynamics, or is this something you should aim to optimize out?
How do you balance actively encouraging fluidity of conversations versus letting people deeply engrossed in conversation stay there?
By default, online spaces won't have the equivalent of "I need to go to the bathroom" or "let me refill my cheese plate". If somebody wants to get out of a focused conversation, do you provide socially-acceptable excuses to leave?
How does your social scaffolding scale as users become familiar with your platform? To what extent is it focused on the novelty of the space itself (e.g. is most discussion focused on how cool and original the space is, how much it feels like you're talking in-person, etc?) or does it still provide useful conversational hooks for expert users?
How can a person or conversation group broadcast things such as "we'd love to talk to new people!" or "go away, we're having a private conversation" to others? In physical spaces, these would typically be communicated via subconscious body language cues that can be difficult to directly recreate digitally.

Aesthetics and World-Building

Does the aesthetic theming of your space match the tone of the event? A house party is not a professional conference is not a friendly coworking space
If you are a platform or a space where the answer to the previous question is "it depends", what creation tools do you offer event hosts (or attendees!) to customize the feeling of a space?
If you provide creation tools, how do you educate event hosts not just how to use them but how to build good things with them? Are you providing event hosts who aren't architects or videogame level designers the scaffolding they need to create spaces that succeed at an intentional design goal?
If your space is graphical, what sort of art assets or visual design creation tools do you provide? Can event hosts bring their own assets if they want to? Are they required to bring their own assets?
If event hosts or attendees are encouraged to provide their own art assets, what are the barriers to entry for creation? Creating 3D models is more work than 2D sprites is more work than writing prose text.
To what extent can attendees modify or shape the space? Do they have access to the same creation tools as event hosts? If not, are there alternative ways for them to express themselves in the space in a persistent or semi-persistent way?
How much control do attendees have over their own presentation? This could mean anything from usernames and user profiles to 2D or 3D avatars to something else entirely. How do these forms of self-expression themselves provide hooks for people to start conversations about?
Do your various forms of attendee self-expression provide space for in-jokes and spontaneous culture to emerge over the course of the event? How do your creative tools actively encourage this?

Trust and Safety

Do you provide event hosts the tools to effectively moderate their events and enforce a code of conduct? (e.g. the ability to ban attendees and remove individual messages, tools for users to report CoC violations and issues, perhaps some sort of secure auditable log to review in the case of CoC reports)
Do individual attendees have the trust and safety tools they need to minimize the damage of abuse or harassment without escalating to the event hosts? (e.g. robust muting and blocking tools)
Do you have sufficient live human moderators at your event to make attendees feel safe? Depending on your space, it may not be feasible (or even desirable!) to have an organizer present and listening in every possible space where people might congregate, but do attendees feel comfortable with the level of moderation when they need to e.g. report a CoC violation?
Many event organizers in VR social spaces feel the need to explicitly explain to new attendees that, as in the real world, standing too close to someone else in VR is viewed as an invasion of personal space. Does your space have cultural norms where unintentional violations may cause discomfort or harm, and if so how do you communicate and educate about them?
When designing various features and interactions between users, have you actively considered how those features might be vectors for abuse and harassment and designed defensively against that?
How do you balance a desire to allow pseudonymity with a desire to keep bad actors accountable for their actions? How does your user registration policy and user profile design reflect this?
If you as a larger platform place restrictions on allowed content, are your rules and enforcement policies explicit? Is it clear what will happen when the policies of a specific event conflict with the platform as a whole?
How do you control user access to specific events? Where do you strike the balance between making it as simple as possible to join an event versus preventing bad actors and trolls from entering events they were not invited to?

Technical limitations

What hardware is needed to access your space? Does it run on mobile devices? How about an underpowered 5-year-old computer?
If someone attempts to use your space with underpowered or unsupported hardware, are they warned about potential issues before joining? Are they prevented from joining entirely?
Does accessing your space require a downloadable executable, or can it run in a web browser?
If your space is focused on a certain technology (e.g. videochat, or VR head + hand tracking), does it meaningfully work without appropriate hardware?
Do event hosts feel like they need to spend a meaningful portion of their event providing instruction and technical assistance to attendees?

This is naturally an incomplete list of concerns, but hopefully is helpful as you work on your own novel online social spaces and events!

If you're working on something cool, I'd love to hear about it! Feel free to say hello, I'm always excited to check out exciting new experiments in this space.

Scaling an Online Virtual World with Serverless Tech

Em Lazer-Walker — Fri, 20 Nov 2020 15:59:15 +0000

I help run an annual game design conference called Roguelike Celebration. Naturally, this year we were a virtual event instead of in-person for the first time. However, instead of just broadcasting a Twitch stream and setting up a Discord or Slack instance, we built our own custom browser-based text-based social space, inspired by online games and MMOs!

I've written about the design underlying our social space, as well as our approach to AV infrastructure, but in this article I wanted to talk about the technical architecture and how we used serverless technology to design for scale.

From an engineering standpoint, we built an open-source real-time game and chat platform. We eventually ended up selling around 800 tickets, meaning we needed to support at least that many concurrent users in a single shared digital space.

Our timeline for the project was incredibly short — I built the platform in about three months of part-time work aided by a handful of incredibly talented volunteers — which meant we didn't really have time to solve hard scaling problems. So what did we do?

Overall Server Architecture

A "traditional" approach to building something like this would likely involve building a server that could communicate with game clients — likely a combination of HTTP and WebSockets, in the case of a browser-based experience — as well as read/write access to some sort of database.

If we ended up having more concurrent users than that one server could handle, I'd have two options: run the server process on a beefier computer ("vertical" scaling) or figure out how to span multiple servers and load-balance between them ("horizontal" scaling).

However, on such a tight time scale, I didn't want to get into a situation where I would need to design and run load tests to figure out what my needs and options were. It's possible none of these scaling issues would actually be relevant given the size of our conference, but it wasn't possible to be confident about that without investing time we didn't have into testing. Particularly, I knew from experience that scaling WebSockets is an especially frustrating challenge.

Instead, I reached for a "serverless" solution. Instead of provisioning a specific server (or set of servers), I would use a set of services that themselves know how to auto-scale without any input on my end, charging directly for usage.

This sort of architecture often has a reputation for being more expensive than just renting raw servers (more on costs later!), but in our case it was well worth the peace of mind of not having to think about scaling at all.

Here's a high-level look at the architecture we ended up building:

If I wanted to not have to think about scaling, we needed our server-side code to be running on a serverless platform such as Azure Functions. Instead of deploying a proper Node.js server (our code was all written in TypeScript), I wanted to be able to upload individual TypeScript functions to be mapped to specific HTTP endpoints, with our cloud provider automatically calling those functions as needed and automatically scaling up capacity as needed.

However, as a real-time game, we also needed real-time communication. The typical way to do this in a web browser is to use WebSockets, which require long-standing persistent connections. That model isn't compatible with the serverless function model, where by definition your computing resources are fleeting and each new HTTP request is processed by a different short-lived VM.

Azure SignalR Service

Enter Azure SignalR Service, a hosted SignalR implementation designed to solve this problem. If you're familiar with WebSockets but not SignalR, you can think of SignalR as a protocol layer on top of WebSockets that adds features like more robust authentication. But for our purposes, what matters isn't the use of SignalR instead of raw WebSockets, but the fact that Azure SignalR Service is a hosted service that can manage those long-standing connections and provide an API to communicate with them from short-lived Azure Functions code.

The only issue is that Azure SignalR Service only handles one-way communication: you can send messages to connected clients from the server (including our serverless functions), but clients can't send messages back to the server. This is a limitation of Azure SignalR Service, not SignalR as a protocol.

For our purposes, this was fine: we built a system where clients sent messages to the server (such as chat messages, or commands to perform actions) via HTTP requests, and received messages from the server (such as those chat messages sent by other clients) over SignalR. This approach also let us lean heavily on SignalR's group management tools, which simplified logic around things like sending chat messages to people in specific chat rooms.

HTTP requests and latency

Using HTTP requests for client-to-server messages did add extra latency to the system that wouldn't exist if we could do everything over WebSockets instead. Even using WebSockets by itself can be a performance issue for particularly twitch-heavy games, as being a TCP-based protocol often makes speedy packet delivery trickier than the UDP-based socket solutions most fast-paced multiplayer games use.

These are problems that any browser-based game needs to solve, but fortunately for us, we weren't dealing with 2D or 3D graphics and a networked physics model and other systems that typically complicate games netcode. As a text-based experience, the extra tens to hundreds of milliseconds of latency added by using HTTP requests for client-to-server messages were totally acceptable.

Redis as a persistence layer

From there, we also had a persistence layer to handle things such as remembering players' user profiles and who was in what room (we didn't store text messages, other than dumping them to a controlled audit log only accessed when addressing Code of Conduct violations).

We used Redis, a key-value store primarily intended to be used as a caching layer. It worked great for our purposes: its ease of use made it easy to integrate, and its emphasis on speed helped make sure that database access didn't add to latency, since we were already incurring extra latency from our reliance on HTTP requests. Redis sometimes isn't as suitable for long-term persistence compared to a proper database, but given we were running an ephemeral installation for two days that didn't matter.

That said, any sort of database or key-value store would likely have worked great for our admittedly simple needs.

So... did it work?

We were easily able to support hundreds of concurrent users in the same space during our live event, and had absolutely zero issues with server performance or load. Azure Functions can scale more or less infinitely, Azure SignalR Service gave us a clear path to upgrade to support more concurrent users if we needed to (up to 100,000 concurrents), and our Redis instance never went above a few hundred kilobytes of storage or above 1% of our available processing power, even using the cheapest instance Azure offers.

Most importantly, I didn't need to think about scale. The space cost about $2.50 per day to run (for up to 1,000 concurrent users), which might have been prohibitively expensive for a long-lasting space run as a non-profit community event, but was absolutely fine for a time-bounded two-day installation (and I'm already working on ways to bring that cost down).

I've used this general architecture before for a previous art installation, but seeing it work so flawlessly with a much larger conference gave me confidence it would have scaled up to even 10x as many attendees without any trouble.

Design your way around hard problems instead of solving them

In general, I'm really optimistic about this sort of serverless workflow as a way of building real-time games quickly. When working on experimental experiences such as Roguelike Celebration's space, I think it's essential to be able to spend your time focusing on hard questions surrounding what the most interesting experience to build is, rather than having to spend your limited engineering resources focused on hard scaling problems.

Scaling traditional real-time netcode is an incredibly difficult problem, even if it's a relatively solved one. Our approach let us functionally sidestep a whole bunch of those difficult problems and focus on building a truly unique and magical virtual event, which absolutely resulted in a better experience for attendees than if we'd invested our time manually scaling.

Whether you're literally trying to figure out how to scale your magical online experience, or you're working on some other interesting experiment, I'd recommend taking the same approach as us: sidestepping difficult problems with outside-the-box design can let you focus your attention on more mission-critical design questions.

If you're interested in learning more about the Roguelike Celebration space, you may want to check out the aforementioned design blog post or the code on GitHub.

Using Game Design to Make Virtual Events More Social

Em Lazer-Walker — Thu, 22 Oct 2020 15:31:26 +0000

This is part of a series of posts/etc about Roguelike Celebration 2020! If you like this, you may also like my post about the technical architecture of our social space, our streaming AV setup, or the open source codebase for the social space.

A few months ago, I had a conundrum: I couldn't stand virtual conferences.

I personally go to in-person conferences to talk to people: to catch up with friends, to make new friends, to have intellectually stimulating conversations. I'd personally rather watch talk videos at home on my own time than spend my limited time at a synchronous event watching them.

Current virtual events are almost entirely talks! We know how to record and broadcast talks over the Internet really well; we don't know how to replicate the social side of things, or the quote-unquote "hallway track".

At first, this wasn't a huge problem for me. I mostly just avoided virtual events except for the occasional speaking engagement, and started doing some experimentation on the side about designing new types of online social spaces to foster the sort of small-group conversation I was missing.

But then it became time to organize this year's Roguelike Celebration, a game design conference I've helped run for the past four years, and the conundrum revealed itself. If I was going to spend my time and effort bringing an online event into existence, I wanted it to be one I actually wanted to attend!

I pitched the team on something radical: instead of using Zoom and Discord, what if we built our own event platform and social space, built from the ground up to foster the sorts of intimate social interaction that made the in-person event special?

Roguelike Celebration ended up becoming a test-bed for a text-based social space and online game that served as the digital venue for our 2020 event, adopting design techniques taken from online games and virtual worlds to encourage meaningful interaction and conversation between attendees.

This article is going to walk through the underlying design decisions that led to what we built, as well as talk a bit about the space itself.

So what was the space?

To contextualize everything I'm about to say, let me explain the space itself.

As mentioned, Roguelike Celebration took place in a custom browser-based text-based social space. Most of the UI and UX design were based on modern chat apps like Discord and Slack, but structurally it much more resembled MUDs, the text-based precursors to modern online MMOs.

Each attendee starts by creating a profile that contained not only their name, pronouns, etc, but also a text description of what their avatar looks like, visible by all other attendees. They are then dropped into a chat room with a virtual "registration desk", from which they can navigate to other rooms in the virtual conference space we had built.

"Rooms" in this space are more like rooms in a MUD or online game than in Discord or Slack. Each room has a text description of what it contains, with hyperlinks to navigate to adjacent rooms, as well as fun novel things attendees could interact with.

We tried to strike a balance between being a "normal" event venue and being playful: there were locations like a quiet lounge and an exhibition hall where we were showing a curation of games, but there was also a dance floor with a DJ set your avatar could dance to, the bar was serving polymorph potions instead of alcohol, and the foyer happened to be haunted.

As a particularly important example, the "theater" contained our talk livestream embedded right in the page. So when talks were starting, attendees would all move to the theater just like they would at an in-person conference.

Each virtual room was also its own chat room. Like in other online virtual worlds, you can only take part in conversations in the room you're physically in. If you want to talk to other people, you have to move to another room.

On top of the theater broadcasting our live talk videos, most individual rooms had their own exciting special activities going on. The kitchen had a vending machine that would produce randomly-generated food items you could pick up and carry around with you, while the dance floor had embedded chiptune DJ sets you could make your character dance to. Fun easter eggs and things to explore or pick up or interact with filled every room of the space.

Entirely text-based?!

The fact that the space was text-based was largely done for logistical, rather than aesthetic, reasons.

The hardest problem we were solving was figuring out what sort of tone and what level of game-like interactions would help foster the social dynamics we were aiming for. Using text meant we could rapidly iterate on content and systems, rather than getting caught up in the additional complexity of building a 2D or 3D rendering system or having higher asset production costs.

Even using ASCII graphics (a la classic roguelikes) would have added in a lot of complex design problems to solve that we were able to sidestep with text descriptions.

That said, it wouldn't surprise me if being text-based would still still be the correct design choice even given more design resources. Text descriptions can be a lot more evocative than representational graphics, and having something look a bit less like a traditional videogame helps make the space feel appropriate for a professional conference.

Overall, the response was overwhelmingly positive. Many, many attendees remarked on how Roguelike Celebration felt the most like physically "attending" an event to them since quarantine started.

How did we get here?

My chief design goal was to create a social space where people could have small group conversations: say, a conversation with 2-10 people where maybe you know some of the people, but it's just as likely you won't know anyone.

Given that overarching goal, I quickly settled on a few key design tentpoles:

A novel space is inherently valuable

In games, we talk about the idea of the "magic circle", a boundary that clearly delineates the space where a game or play takes place as distinct from the normal world. Activities within the magic circle being distinct from normal reality gives people freedom to express themselves more freely and to, well, play (within reason and established safety limits, of course).

A similar thing happens with in-person conferences. The event venue (especially if in a destination location!) serves as a freeing liminal space that helps attendees be present and engaged. Whether you're at a conference to learn from the talks, to meet new people, or frankly to just enjoy a free employer-paid vacation, the act of being in a different physical place does a lot to get you in a mindset where you're ready to embrace new experiences.

This is difficult for online events in the time of quarantine! If you're like me, you're largely spending 40 hours a week sitting at home, using Slack and Zoom or some other similar text and videoconferencing software. Asking people to spend their weekends at their same computers attending a Zoom conference with a Discord or Slack doesn't accomplish that goal!

We realized that, even if our custom social space was otherwise a complete and utter failure, the mere act of having it be a new and novel space would still be valuable.

Allow small-group conversations at a technical level

Having a conference Discord or Slack means having a few hundred people in the same dozen text channels. This setup affords two different modes of interaction: people can talk in those large public channels with a few hundred participants, or they can slide into other people's DMs for 1:1 chats.

Neither of these are particularly great for enabling intimate group conversations with strangers! In particular, we talked to a large number of potential attendees who expressed extreme discomfort and anxiety about trying to have any sort of conversation in those large public hundred-person chat channels.

Conversely, VR social spaces such as AltspaceVR or Mozilla Hubs do a great job of enabling the sort of fluid small-group conversations you get naturally in-person. Physical presence, spatial audio, and body language cues from head-tracking and hand controllers mean that you can naturally split off from a group conversation to start a smaller conversation, and then effortlessly rejoin the larger conversation whenever you want, similar to how you would in a physical setting.

But I've regrettably found VR social spaces to be completely inaccessible to people who aren't VR enthusiasts, even when using software like AltspaceVR or Hubs that technically support non-VR desktop and mobile devices.

We knew we needed to find a technical model for chat that, while not in VR, was closer to what VR offers than to Slack.

Borrowing the spatial chat model from MUDs gave us the property we wanted where you could be in a room with a small group of people having an intimate conversation!

Playful design adds spontaneity

Even if we create a space where people can talk to each other in small groups, that doesn't mean they will. Striking up a cold conversation with a stranger is hard and scary!

At in-person events, there are a number of easy hooks that make it socially acceptable to initiate small-talk. You might strike up a conversation with the person sitting next to you in-between talks about the talk you've just seen. You might comment on a sticker on someone's laptop, or the logo on their t-shirt. You can walk up a sponsor booth and know that someone sitting there will be thrilled to chat about their company.

We don't have any of those affordances by default in online spaces!

My hypothesis for Roguelike Celebration was that we could fill that void by integrating game mechanics and playful elements borrowed from online games.

In particular, I was inspired by Dan Cook's work at Spry Fox around how to design MMOs to encourage formation of meaningful friendships. His design work and writing is primarily concerned with helping people form deep friendships over the course of months or years, rather than us trying to get people to be mildly friendly over two days, but a lot of the core concepts he talks about in his GDC talks were still directly applicable.

A particular piece of social science he explores is the idea that friendships are formed through repeated spontaneous interactions over time.

This model reinforces some design decisions I've already explained: if you want spontaneous interactions, that seemingly requires a more spatial chat model than a giant Discord server where everybody is always in the same chat rooms at the same time.

From there, adding game-like and playful activities to the space can encourage these moments of spontaneous interaction to happen more frequently.

To ground this in a concrete example, the space had a bar area where attendees could drink a polymorph potion (designed/implemented by Alexei and Kawa) that would add a random emoji to the front of their name. Let's look at all of the ways this simple playful element provided social value to attendees:

People who needed a break from conversation could wander off to the bar and drink a few more polymorph potions as a sort of fidget activity
Because this was an exciting special thing that was only available in the bar, this encouraged people to move in and out of the bar regularly, letting the bar serve as a sort of conversational nexus where you could bump into someone you knew (or didn't know but had seen in other rooms)
Seeing other attendees with cool emojis encouraged people to explore the space to find other secrets like that, helping attendees circulate to different rooms and increasing the chance of spontaneous interactions
When someone couldn't figure out how to add an emoji to their name, they'd ask people who already had emoji, giving people an excuse to be helpful and get to know each other in the process
Being able to have some control over your emoji — you could drink as many potions as you wanted with no ill effect, so you could keep chugging until you got an emoji you particularly liked — served as a form of player expression. This is both extremely satisfying as an attendee and serves as a great conversation opener for other attendees.
Running into someone with the same emoji as you was a particularly potent way to start a conversation and instantly feel affinity towards a stranger.

Most of our playful elements like this were fairly simple due to time constraints. I'd love to explore more complex and involved systems for future events, but you can see here how even the simplest game-like interactions can massively impact the sociability of the space.

Video chat is valuable in moderation

Video chat is far more effective than audio chat at helping convey emotional nuance, and audio chat is equally more effective than text chat. This is important if our goal is to foster new friendships and interesting connections!

But at this point in quarantine, we're all well aware of Zoom Fatigue. It's clear that running an entire conference on video or audio chat is a great way to burn everyone out.

As we spoke with potential attendees, we realized there are broadly two types of online communicators: those who are happier communicating online in text, and those who are happier using videochat to the extent that they have the emotional energy. Finding a way to make both groups of people feel comfortable and socially stimulated felt like a valuable goal.

This led us to aim for an event where communication over text chat was the default, but there were many opportunities for attendees to consensually opt-in to escalate into audio or video chat. Being primarily based in text chat lets attendees save up their emotional energy for focused higher-quality moments of video conversation, and grounding those moments in opt-in activities means that each individual attendee can self-moderate how much video chat they can handle.

Our plan for video chat in the space was twofold: we planned to schedule discrete blocks of time for video chat-based networking sessions, but also offer lower-key videoconferencing in every room that people in that room could join at any time.

For the former, we held unconferencing sessions where attendees could propose and upvote discussion topics, which were then assigned to specific rooms. We also intended to schedule breakout room-style networking sessions (where you would be randomly moved into a videochat with 3-5 random people for 10 minutes), but couldn't find room in our final conference schedule. These are the two structures I've seen work particularly well for structured video chat to avoid the anarchy and fatigue of unstructured 30-person Zoom calls.

Putting it all together

Combining all of these design ideas, we ended up with a space that, by all accounts, was fairly successful at achieving its goals.

As mentioned, I was blown away by how many of the remarks about the space talked about a sense of physical presence that mirrors the way people talk about VR — this felt to people like they were physically "attending" our conference in ways they hadn't felt before with virtual events.

One moment that sticks with me was when an attendee set up shop at a specific table in the kitchen and did tarot readings for anyone who brought her an offering (of any object taken from elsewhere in the space). To me, this level of an attendee buying into the fantasy and aesthetic of our space, and contributing their own playfulness to the mix, is a ringing endorsement for our game-like and playful approach.

One pain point we did run into was just not having enough ways to test what we were doing. We were able to run a "preview event" that served as a test of the space (both from a design and from a technical load-test standpoint), and that proved essential in shaping the design of the space for its final iteration. But for the most part, it's incredibly difficult to playtest or validate your designs ahead of time when doing so requires dozens to hundreds of people.

There are a ton of things I'm eager to change for our next iteration, as well as new hypotheses I have for how we can be more effective at encouraging conversations. But it's also still unbelievable to me that we accomplished as much as we did with about three months of development time (mostly me, plus contributions from a handful of amazing conference organizers and volunteers) and only that one public playtest session.

Call for Collaboration

This social space was essentially an experiment intended to test my hypothesis that borrowing elements from online games — physicality, playful interactions, etc — would create a space where people could have the sorts of small person-to-person interactions I've been missing from online events.

I think it was about as successful at that as I could reasonably hope, but it was also built for an audience perfectly suited to what I built. As a conference of mostly game designers, attendees were broadly familiar with the sorts of interface paradigms and game mechanics they were being presented with, and they're already used to the idea that a professional conference can be playful and silly and fun.

I think all the work I'm doing is interesting and broadly applicable to other communities, but figuring out how to make it accessible to wider audiences is a complicated problem!

All of the code for our social space is open-source, and anyone could technically build their own space based on my work without my input. But this design space is so nascent and so experimental that I think it would be hard to use this codebase without more context around the social design decisions we made. Which is to say, I suspect the most successful second deployment of this tech would be one that I continue to be involved in.

I'm excited to work with conference organizers to figure out what that would mean! If you have an event that you think could work well for something like this, shoot me an email or Twitter DM and let's chat about how the work I'm doing can benefit your community!

This isn't just about this space

Maybe you don't run synchronous online events. Maybe you do, but you think a text-based tool like this isn't right for your community. More than talking about great the specific thing I built is, I want to hammer home the idea that designers of online games and virtual worlds have been thinking about and solving these social design problems for literal decades.

The most effective way to make online events more engaging is going to be looking towards game design and virtual world design to learn what makes those spaces tick.

This doesn't (necessarily) mean making actual games. Our space more resembles Discord or Slack from a UI/UX perspective than a historical MUD.

It also doesn't mean building more of the same online event platforms we already have, but throwing in some 2D pixel art or traditional 'gamification' markers (leaderboards, badges, etc) or other surface-level signifiers.

What we actually need to take from game design is the understanding of how to use play and playful design to create environments whose architecture encourages and rewards positive social interactions through psychologically satisfying systems. This isn't by any means easy, but I hope the work I've done can show that it's doable!

Running A Virtual Conference: Roguelike Celebration’s AV Setup

Em Lazer-Walker — Tue, 13 Oct 2020 15:45:45 +0000

The Roguelike Celebration conference has been running for five years, but two weeks ago marks our first foray into an online-only event!

The most notable thing about this iteration of the event was the custom MMO-like social space that hosted the event. You can read more about that here, but today I wanted to talk about something a bit more universally applicable to any online event: the nuts-and-bolts of how we ran our AV setup.

We were a two-day single-track conference, with talks streamed to both Twitch and YouTube (more on that later), and the YouTube stream embedded directly within our custom event platform software.

This is a technical post for people who will directly be handling AV needs for their own virtual events. I walk through both the technologies we used and the structural/philosophical choices we made about how to work with our speakers.

To be clear, I was not the person actually operating the stream during the event itself: that honor goes to Kawa and Travis on the AV logistics side of things, and Alexei and Noah as emcees/hosts. That said, many of these high-level tooling/process decisions were mine, and the knowledge I'm sharing about our experience comes from both my personal observations and from speaking with the people who were actively running the stream.

StreamYard

We used StreamYard as our streaming studio. We liked the idea of having a solution that wasn’t reliant on the host’s home Internet connection, and that was easier for both the host and for speakers to deal with than messing around with videoconferencing software.

Having a browser-based tool meant that a conference organizer wasn't running OBS and a videochat client with support for NDI, and that it was easier for us to swap out which organizer was on-duty for technical setup during the event itself.

StreamYard was great. It was easy for us to use, easy for speakers to connect to, and it effortlessly let us stream to both Twitch and YouTube simultaneously.

Throughout our event, we had at least two organizers in StreamYard at all times. We intentionally split out the roles of "emcee" and "technical AV person" into two separate people, largely so that in case of technical issues the emcee could continue to stall and keep the audience occupied (tech permitting!) while the other person fixed the issues. I suspect StreamYard is easy enough to use you could likely get away with only one person, but this worked really well for us.

We did have a few minor issues with StreamYard:

While we were able to integrate our own custom overlays, we didn't have as much control over our display as we would have using something like OBS. It's possible that some of the things we thought we couldn't do (e.g. custom fonts or adding arbitrary text labels) are things that StreamYard is capable of doing, but the documentation wasn't great — we often found ourselves watching YouTube videos from the community when StreamYard's official documentation wasn't helpful.
Related to the previous issue: even though we had a human captioner providing live captions, we were unable to embed those directly as closed captions within the stream as we would have been able to with OBS. In our specific case, this was mostly okay: most attendees were watching via our custom social space, where we directly embedded the captions below the stream, and our Twitch and YouTube channels directed people to a website where they could view the captions.
Prerecorded video playback was still reliant on a home internet connection. StreamYard doesn’t let you upload video files and directly play them, so playing a pre-recorded video meant the emcee opening up a Chrome tab with the video file and screen-sharing that tab.
For times we played prerecorded videos, the volume was frequently lower than live speakers, and we didn't have a way to dynamically adjust this during the event. In the future, we'd likely take the time to normalize all prerecorded videos before streaming, but it's frustrating StreamYard doesn't appear to have any real-time audio mixing tools.
StreamYard's highest paid tier allows you to capture recordings of up to 8 hours long. Our conference had roughly 8 hours of video each day. To avoid unintentionally cutting off our recordings if our schedule went long, we split each day up into two separate StreamYard 'studio' recording instances. Switching over added some minor logistics hassle, and also caused some issues where the YouTube embed widget we were using wasn't capable of automatically switching from the morning to the afternoon video feeds, meaning attendees would occasionally need to refresh their browsers to switch over.
StreamYard is a bit finicky when it comes to playing audio over screen-sharing, particularly on MacOS. This wasn't an issue for our specific speaker pool, and is most likely a technical limitation for any browser-based technology, but is worth noting if you're running a particularly multimedia-heavy event.

There are other browser-based streaming services such as Restream.io and Stage Ten, but we didn't really spend time looking into them. I'd used StageYard before as a speaker, it was easy to use and affordable, we went with it.

It's possible one of these alternative services would have given us all of the features and ease-of-use did that StreamYard did, but without some of the other hassles we encountered. I'm not sure. I don't want to speak for my other organizers, but the next time I host an online event, I'll likely investigate to see if that's the case.

That said, I could also see myself just as easily using StreamYard again, and would heartily recommend it except for those minor caveats.

Streaming: Twitch vs YouTube

We’re a conference about games and game development. Streaming to Twitch makes sense to us, since that’s where our audience is.

However, Twitch doesn’t offer dynamic bitrate re-encoding of your streams unless you’re a Twitch Partner. This means that viewers on bad Internet connections can’t choose to load your stream at a low bitrate so they can actually watch it. This is exceedingly bad for accessibility!

YouTube does offer this! Our compromise was to stream to both Twitch and YouTube. Our custom social space embedded the YouTube stream, so everyone within the space was watching on YouTube. Even given that, our stream view counts were roughly equal across Twitch and YouTube.

One other thing to note is that we actively wanted to disable chat on both our YouTube and Twitch streams. Our approach was to have all text chat take place in our custom social space, where attendance was limited to people who had chosen to acquire a ticket (whether free or paid) and had agreed to our code of conduct, and where we had active moderation efforts.

YouTube easily lets you disable live chat on streams. Twitch does not.

On Twitch, we were able to set chat settings so that the only people who could what were people who had followed our account for more than 3 months. They could only post one message every five minutes, and it could only be emoji. This was functionally fine in practice, but it was frustrating Twitch wouldn’t let us just completely turn chat off, and meant we did have to keep an eye on it.

Which is to say: if you don’t have a solid reason to stream to Twitch (e.g. you’re a games conference), stream to YouTube instead of Twitch.

Speaker AV Tests

This may be obvious, but it's worth calling out: we scheduled 5-10 minutes for each speaker to pop into the StreamYard a week or so before the event to test out their AV setup and get used to the environment.

In our case, this was helpful to confirm that each speaker's audio + video situation was sufficient (we had a very small budget to buy speaker equipment when necessary). At a previous online event I spoke at, these AV tests going poorly are what led to the organizers switching from an OBS-based setup to StreamYard.

StreamYard made this particularly easy. We could give all speakers a link to join the StreamYard instance ahead of time. Since anyone who joins the stream session is put into a 'green room' by default (where they can chat but aren't on-stream) that also made it easy to manage back-to-back AV tests if we were running late.

Speaker AV tests are fairly easy to schedule and make happen, but are essential to making sure things go smoothly during the event itself!

Prerecording

We asked — but did not require — all speakers to send us recorded videos ahead of time, to use as a backup in case of technical failure. Most speakers did, which I’m extremely grateful for. I’m also grateful we didn’t need to unexpectedly fall back to a backup.

We also made it clear to speakers that, while our default assumption was that speakers would present live, choosing to air a prerecorded video was perfectly fine.

From what I’ve seen as a speaker and conference organizer, experienced public speakers tend to be split pretty evenly about whether they’d prefer to perform live or provide a recorded talk. Some thrive on the adrenaline of knowing there’s a live audience, while others appreciate being able to take the time to record a perfect take or edit after the fact. We wanted both of these groups to do what would make them most comfortable and result in the best possible talks.

At the same time, an aspect of Roguelike Celebration that I really appreciate is that many of our accepted talks tend to come from first-time public speakers. There are certainly exceptions, but in general I’ve found that many inexperienced speakers give better performances live than pre-recorded. This isn’t a knock against anyone in that situation; maintaining high energy levels when you know you’re not speaking to anyone is a skill that often needs to be consciously learned.

Framing live talks as the default of two equally valid options let us nudge our new speakers in the direction that best set them up for success, while still allowing everyone to make their own personal choice as to what would let them give their best performance.

I think this strategy worked out well for us. Worth noting that, even for speakers who opted to use a prerecorded video, we asked (but did not require) that they show up live after their talk for moderated live Q&A.

Don't Require Speakers to be On-Camera

We had a few speakers who did not want to show their faces. At least one speaker use a Snap Camera filter, and another speaker performed as a VTuber with a puppeteered 2D avatar.

These required more intensive AV setups, but the onus to get that working was generally on the speakers rather than us. Our job was primarily to be supportive: our goal as organizers is to enable speakers to give the best talks they can, and making sure they're comfortable is an important part of that.

If a speaker is uncomfortable showing their face, I would probably nudge them towards a solution that conveys some sense of body language over disabling their video feed entirely — even the Snap Camera solution did a great job of conveying emotion in a way that just audio wouldn't have — but again, speaker comfort needs to come first.

From a technical standpoint, it's worth noting that StreamYard didn't have any issue with streaming video from virtual camera sources (e.g. SnapChat Camera or an OBS scene exposed as a camera).

And that was the video portion of our conference!

All in all, the livestreaming portion of our event went remarkably smoothly. At times, we were even ahead of schedule, which I think speaks to how effortless our setup was.

Hopefully this might be useful to you if you're planning an online event and need to figure out how to handle the technical streaming aspect of your talks!

A rock-paper-scissors app with gesture detection and voice

Em Lazer-Walker — Thu, 09 Jul 2020 13:16:06 +0000

By David Smith and Em Lazer-Walker, Cloud Advocates at Microsoft

In this blog post, we’ll show you how to build a web application that will access your camera and say something whenever you make a specific gesture with your hand. This is a simplified version of the Rock, Paper, Scissors, Lizard, Spock application, and you can try out the app here or deploy it yourself with the instructions below. After you launch the app using a desktop browser, click Start and allow access to your camera, and then make one of the hand gestures from the game created by Sam Kass and Karen Bryla. Make sure your volume is turned up, and when the application sees a valid gesture, it will speak to you as it is recognized.

You can customize and run this application yourself by visiting this GitHub repository and following the directions shown. All you need is an Azure subscription, and it uses free services so it won’t cost you anything to try it out.

Let’s dive into the various components of the application:

Speech. The speech generated when the application detects a valid gesture is generated on demand with Cognitive Services Neural Text to Speech. Neural TTS can synthesize a humanlike voice in a variety of languages (with 15 more just added!) and speaking styles.

Vision. The hand gesture detection is driven by Custom Vision in Azure Cognitive Services. It’s based on the same vision model used by the Rock, Paper, Scissors, Lizard, Spock application, but running locally in the browser. No camera images are sent to the server.

Web Application. The application is built with Azure Static Web Apps, which means you can create your own website with a version of the application in just minutes – and for free!

Customizing the application

Because we’ve provided all of the code behind the application, it’s easy to customize and see the differences for yourself. As soon as you check in changes to your forked GitHub repository, Static Web Apps will automatically rebuild and deploy the application with your changes. Here are some things to try, and you can find detailed instructions in the repository.

Customize the speech. All of the speech generated by the application is defined using the SSML standard, which you can easily customize simply by modifying the text in a JavaScript object. Here are some things you can try:

Change the words spoken for each hand signal by modifying the text.
Try changing the default voice or language by configuring the default.
Try a different speaking style, like “newscast” or “empathetic” with SSML.

Customize what’s recognized by the camera. The GitHub repository includes only the exported rock-paper-scissors Custom Vision model, but not the source data used to train the model. You train your own vision model with Custom Vision, export it for TensorFlow.js, and replace the provided model.

Going Deeper

If you’d like to learn more about the technology used in this app, check out these Microsoft Learn modules on Static Web Apps, Custom Vision, and Text-to-Speech. If you have any feedback about the app itself, please leave an issue in the Github repository, or reach out to either of us (David and Em) directly. This was a fun app to make, and we hope you have fun playing with it too!

What is Spatial Audio, Why Does it Matter, and What's Apple's Plan?

Em Lazer-Walker — Tue, 30 Jun 2020 11:55:01 +0000

At WWDC 2020, Apple announced that iOS apps will soon be able to use motion data coming from your AirPods Pro to enable head-tracked spatial audio. They talked about this largely in context of playing movies with multi-channel surround sound, but that's probably the least interesting application of spatial audio.

As someone who's been working in the field for a long time — my research at the MIT Media Lab in 2015 and 2016 focused on location-based storytelling in public spaces using spatial audio — I wanted to try to give some context around why this is interesting and what it might enable.

What is spatial/positional audio?

Spatial or positional audio (these terms are typically used interchangeably) lets you position sounds anywhere in 3D space. Instead of just thinking about sound engineering at the level of "is this to the listener's left, right, or neither?" as you would with normal stereo sound, you can place specific sounds at specific 3D locations around the listener: say, a sound that's in front of you, a little bit to the left, and a meter or two above head/ear height.

If that sounds a lot like surround sound, you're not wrong, but the underlying technology is different. Surround sound systems play audio out of speakers that are placed in different physical locations. To play a sound that sounds like it's behind you and to the right, you use a speaker that is literally behind you and to the right.

The sort of spatial audio we're talking about is (usually, but not always) concerned with producing sound that is situated precisely in 3D space despite coming out of a pair of normal headphones.

How does spatial audio work?

Let's talk about how humans normally hear sounds in the real world.

If a loud noise happens directly to your left, those sound waves will reach both your left and right ears. But while they have a pretty direct route into your left ear, your right ear will receive them after they've passed through and been shaped by your skull, your brain, and pretty much everything else in there.

Humans — specifically, our brains, inner ears, and outer ears working together — are really good at processing the difference in these sound waves and transforming those raw signals into your conscious mind saying "ah yeah, that sound is coming from over there".

To produce spatial audio that lets you hear things in precise 3D locations through a set of headphones, the audio coming out of each headphone ear needs to essentially recreate the sound attenuation that happens naturally so you can to trick your low-level auditory systems into thinking the sound is coming from someplace else.

How does do you produce spatial audio using analog methods?

The traditional way of producing binaural audio recordings involves taking two microphones and sticking them in the 'ears' of a mannequin head that's been designed to roughly match the density of a human head. Done right, this gives you a stereo audio recording that truly does capture the 3D soundscape as it was recorded.

The audio walks by sound artist Janet Cardiff are a great example of these traditional analog methods. If you want to get a sense of how effective this technique can be, grab a pair of headphones and listen to a few minutes of her NYC audio walk Her Long-Black Hair.

Using digital techniques for spatial audio

Manually setting up two microphones and a test dummy is a lot of work. Modern audio production techniques typically involve math instead. Specifically, they use a set of models known as Head-Related Transfer Functions (HRTF) that describe the transformation that occurs when a "pure" sound is attenuated in such a way so as to mimic what a specific ear would hear.

Although specific HRTFs theoretically vary from person to person (all of our bodies are different!), in practice researchers have generated a few different algorithms that will work well for most situations. If you're using HRTF as a programmer or sound engineer, in most cases you'll just see "HRTF" as an option you can enable and that's really the extent to which you need to think about it.

How do you adapt to the listener?

Whether you're working with analog binaural recordings or digital tools that can apply HRTF, the results are usually stereo audio files that sound impressively like they're positioned in real space.

But they're still static recordings. Let's say that a sound is directly behind you. If you turn 90 degrees to your left, you'd expect that sound is now directy to your left. But because the audio recording was made so that sound is "behind" you, it will turn along with you, breaking the illusion.

This is where being able to create spatial audio in software is valuable. If you have some way of tracking the position of the listener's head, you can dynamically adjust your sound generation in real-time to keep a sound source fixed in the listener's concept of real-world space.

Right now, the main commercial application of something like this is VR and AR headsets. Since they already have high-quality head tracking data for graphics rendering, using that same data for positional audio is a no-brainer.

Where AirPods Pro come in

Given all of that, hopefully you can see how adding spatial audio features to AirPods Pro might work.

Apple already offers an API for producing spatial audio with HRTF, both integrated into ARKit for AR experiences and in the more general-purpose AVFoundation library. This has actually been part of iOS since 2014!

With iOS 14, Apple is adding a set of new APIs to use the motion sensors already in AirPods Pro to provide head-tracking, letting developers build positional audio experiences that are aware of how the listener is moving their head.

This isn't a completely new concept — until recently, Bose maintained a similar platform for head-tracked spatial audio using certain Bose headphone models and a third-party smartphone SDK — but being supported at a first-party system level, with Apple's incredibly popular headphones, will almost certainly help this see wider use than Bose's SDK.

The audio version of ARKit

You could technically build visual AR apps without using an AR framework like ARKit or ARCore. It's not a lot of work to show a live camera feed in a mobile app, and then overlay a 3D model on top of it. But unless you're doing a lot of manual computer vision work, you're not going to have the world awareness to keep that object's position fixed in real space as the user moves the phone around.

Head tracking for spatial audio is similar. Prior to this announcement, you could easily make experiences for iOS that play positional 3D audio soundscapes through users' headphones. But without head tracking, they lack an awareness of and connection to the physical world, and it's not possible to make the sounds feel like they're fixed in a concrete real-world position. This new API solves that.

What does this look like from a technical standpoint?

As of the writing of this piece, Apple's APIs aren't ready for public consumption. There's a new headphone motion API, and a new as-yet-unused configuration option for different spatialization algorithms that seems unconnected to the existing AVFoundation APIs.

The latter suggests to me that Apple may release a higher-level system that, say, automatically adds head-tracked spatial audio to any apps already playing audio through ARKit. I suspect they will heavily encourage developers to use ARKit when appropriate, as augmenting headphone motion data with camera-based world tracking will likely provide better tracking results.

That said, once new AirPods Pro firmware has been released that support sending motion data, the headphone motion manager will be enough for interested developers to dive in and start building spatial audio experiences.

Four years ago, I built some quick experiments using the iPhone's built-in accelerometer and gyroscope to control a spatial audio scene generated using Apple's existing AVFoundation spatial audio APIs. The code to wire up the two was straight-forward back then, and a similar approach should work just as well when it's using motion data coming from the headphones instead of from the phone itself.

Why does this matter?

This is all well and good. But what does head-tracked spatial audio actually enable? Providing a more immersive experience for films or 3D games, as Apple suggested, is a natural use case, but far from the most interesting one.

What's difficult about answering this question is that there isn't really yet a well-established field of design for building audio-only real-world experiences that take advantage of positional audio. Existing audio-only platforms like voice assistants don't really have a concept of grounding an audio experience in the physical world; even the people building rich gaming experiences for those platforms don't have the clearest answer of how spatiality might change things.

Based on my experience working with spatial audio, there are at least a few broad classes of potential applications that really excite me. This is far from an exhaustive list, but here's a taste of the sorts of experiences we might see as spatial audio becomes more of a thing:

Wayfinding

One of the first usecases people tend to think of for spatial audio are helping people navigate the world. Microsoft has already released an app called Soundscape that uses binaural audio to help people who are blind or have low vision to navigate the world.

It's easy to imagine turn-by-turn navigation apps adding in support for spatial audio cues, and interaction patterns such as "follow this sound that keeps moving in the direction you should walk" becoming commonplace.

As Apple improves their indoor location technology, this could also easily become a big part of making indoor wayfinding viable before they ship AR glasses, since the ARKit model of "hold your phone out in front of you while you walk through a space" is both socially and physically awkward.

Improving existing audio content

If you speak to anyone who's worked on a social platform for VR, they will be quick to point out how much of a difference spatial audio makes in fostering natural voice conversations. When human voices are mapped to distinct physical locations, it's like a switch is flipped in the brain that makes it easier to differentiate similar-sounding voices, even if you're on a platform that doesn't have great lip-syncing or other visual ways to indicate who's speaking.

It wouldn't surprise me to see applications like group voice chat apps or even podcast apps embrace spatial audio as a way to make conversation feel more natural and easier to make sense of at a subconscious level.

Real-world gaming and playful experiences

One of the projects that resulted from my MIT research into spatial audio was a site-specific generative poetry walk built for a park in San Francisco. Being built for consumer iPhones in 2016 meant it doesn't use head tracking for its positional audio, but key to the piece are the binaural audio soundscapes that subtly fade in depending on where in the park you are.

If you're in the main grassy field in the park, you may hear kids laughing and playing off in the distance, and you won't really be sure whether they exist in the real world or just in the audio; the cacophany of birds chirping as you enter the fenced-off community garden create a sense of magic and connection to nature in a visually stunning space.

The Janet Cardiff audio walk I mentioned earlier does similar magic tricks with (also non-head-tracked) positional audio. You'll hear a couple arguing behind you, or police sirens going off on the street outside the park, and not be sure whether it's reality or fiction.

Cardiff applies a ton of incredibly subtle psychological tricks to prevent you from turning your head and breaking the illusion of the static baked-in binaural audio. This means her work is generally in a league of its own, as this sort of work is so difficult to replicate without her sheer experience and talent.

Having readily-available consumer head-tracked audio means making these sorts of experiences will be so much more accessible to creators of all types, not just ones with extensive experience in traditional binaural audio production.

To be clear, I think the future of games and playful experiences focused on spatial AR audio isn't in extending games like Pokémon Go to be more "immersive", but in taking design cues from live-action role-playing and immersive theatre design communities. My design goal for my poetry walk was to create something that encourages players to appreciate the mundane beauty of a public space in their neighborhood, blurring reality with fiction and make-believe to elevate reality into something that feels magical.

Positional audio is so much more powerful — and so much cheaper to produce — than current-day 3D visual AR at causing that sort of emotional reaction in players. Apple helping to make head-tracked positional audio mainstream could bring on a waterfall of beautiful hyperlocal audio experiences.

So what's the takeaway?

I feel like I'm standing over here, wildly waving my arms at everyone to pay attention to my favorite pet technology that's finally on the verge of becoming mainstream. But it's true!

I think spatial audio in general is a much more powerful technology than a lot of people give it credit for, but good head-tracking available in consumer hardware is the piece that's been missing for it to find more widespread appeal. By piggy-backing off of existing popular headphones, Apple is well-positioned to make spatial audio tech explode in a way that it hasn't before.

I'm so excited to see what people make if this, and can't wait to dive in more as Apple updates the AirPods Pro firmware and makes beta API access available. Let me know if you're working on something cool or have cool ideas for ways to use spatial audio tech!

Using Data to Improve Your Narrative Games!

Em Lazer-Walker — Wed, 29 Jan 2020 16:53:43 +0000

Analytics in Interactive Fiction?

When I speak with indie game developers making narrative games, a lot of people perk up when I mention analytics. Gathering data about user behavior can feel like a weird dark art: it often sounds like something you should be doing, but you don't quite know what to track or how to make sense of that data, and you probably also wonder about whether or not it's actually ethical to capture that sort of data.

This article is about how I use analytics in my Twine games! I'll talk about how analytics are "traditionally" used in large free-to-play games, as well as the ways in which that approach does or does not apply to people making small experimental narratives.

I'll walk through a few examples of how I've concretely found gathering analytics data useful in my work, as well as give some links to the free tools I use to do that data-gathering in Twine (largely the PlayFab-Twine tool I maintain).

Although my experience here — and the tools I mention — are largely grounded in Twine, this should be applicable to those making choice-based interactive fiction games in most any environment.

How free-to-play games use analytics

Free-to-play games are by far the most regimented users of analytics data. To be clear, I don't want to endorse the way F2P uses data, and I think it's largely not relevant to people making free or premium interactive fiction games.

But it's still useful to get a sense for how they function before we talk about which specific elements to crib and which to leave be.

This is also a mile-high view; it goes without saying that I'm not trying to speak to how EVERY free-to-play game uses analytics. This is a sort of abstracted overview based on my personal experiences in free-to-play over the years.

Guiding Metrics

For the most part, free-to-play games are laser-focused on a funnel:

You get new users into your game. In 2020, you're probably paying money to acquire them through channels like video ads.
Some number of these players will keep playing your game.
Of the people who keep playing your game, some of them will become paying players.
Once people are paying you, you want them to pay you as much money as you can get from them.

The goal of metrics is mostly to quantify each of these steps, so you can then run measured experiments to make small improvements to each of those steps.

If you can make the average retention longer, or increase the amount of money your average player pays, that can have huge returns in your profits.

For the most part, there are a few metrics that are universally tracked:

Retention is measured in terms of what percentage of players return a certain number of days after installing.

Day 1 — or D1 — retention is how many players come back the day after they install, D7 is how many people return 7 days after installing, so on and so forth. You'll typically track D0, D1, D7, and D30 retention.

Retention numbers beyond day 30 are often ignored. I'd argue that's likely a mistake, but that's a discussion for another day!

Revenue is exactly what it sounds like. You might track things like the average revenue per daily active user (ARPDAU), average revenue per paying user (ARPPU), or what percentage of your players give you any money.

Although you'll focus on those metrics as the most important things to track, you'll typically also use them as jumping-off points for finding other things to keep an eye on. A product owner on a free-to-play game will typically add lots of tracking into their game, with an eye towards answering specific questions:

When launching new features: are people engaging with this the way it was anticipated? If you visualize a feature of a game as a funnel, where are people dropping off of that funnel?
Health checks: games will often have a specific number that's indicative of overall game health. When I worked on Words With Friends, we looked at the total number of turns played by all players each day. If that number was unexpectedly low, that was a sign that there was something wrong, either with our servers or with a new design
Deciding what new features to prioritize: new feature work will often be oriented around raising a specific metric, usually one of the key revenue or retention metrics

This sounds gross. Why does this apply to me?

As a conceptual framework, this sort of numbers-driven approach to game design is intended to maximize how much time and money a player is spending on your game, typically using psychological tricks and addiction patterns taken from gambling. That's not great.

The good news is, even a I've just spent time describing this whole approach to analytics, it probably doesn't apply to you!

F2P narrative games certainly exist, but I'm writing this article for people who don't work in that space. If you're working on a game that's a bounded narrative experience — whether free or "premium" — you're (hopefully!) not focused on the sort of questions around optimizing money and engagement time from players that F2P designers are, you're much more focused on conveying some aspect of the human experience or evoking some sort of emotion in players.

In that situation, a strict application of F2P analytics techniques is actively going to make you a worse designer! But I do think it's possible to use similar techniques and tools to get a better sense of how players are interacting with your game, in ways that are qualitatively different from what you'll get from e.g. in-person playtesting. Having that information can help inform design decisions in ways that can result in a better experience for your players, in particular in the moments when your expectations turn out to not match up with reality.

It's also absolutely possible to gather that data without capturing personally-identifiable data on any given individual player, which helps me personally feel a lot less squeamish about this sort of data collection.

Metrics aren't about individual players

An important point with metrics data is they're mainly useful in aggregate. If you're interested in charting out the journey a single player takes, just looking at numbers isn't going to tell you what that player is thinking or experiencing. Tools like in-person playtesting or quantitative surveys are going to be far more effective at that personal individual scale.

So what are metrics useful for?

I recently shipped a small Twine game for work that's essentially a short escape room game: you're in a spooky old house, and need to solve a bunch of puzzles.

Using my PlayFab-Twine tool (more on that later!), we were able to gather data on whenever a player visited a different Twine node in our game, with the ability to write complex queries around that dataset.

We broadly found analytics to be incredibly useful for answering a whole bunch of different design problems.

Who are our players?

Data about how many people played our game, how often repeat players came back, where players were coming from (geographic region / OS and browser version / etc), and how long people on average spent with the game was super useful to get a sense for who was playing our game.

Is our game broken?

Many interactive fiction tools have some sort of mechanism for validating that every passage is theoretically visible, basically by brute-forcing every possible path in the game.

For us, analytics was able to serve a similar goal: once the game was out in the hands of a large number of users, we were able to look at visit numbers for every Twine passage relative to the total user count. If any passage had 0 visits, or an exceedingly low number of visits, that was a strong sign there was likely some sort of engineering error that prevented players from reaching that content.

This isn't a replacement for traditional manual QA, but can still serve as helpful extra validation your game works as expected!

Where are players the most unhappy?

With our game's initial release, we were disappointed with how few players were playing the game all the way to the end.

Our game is a puzzle game whose structure is generally fairly open, but we have a few strict chokepoints where players must solve a specific puzzle to progress. This made finding the problem fairly easy: taking each of these chokepoints, we graphed out how many people viewed the Twine passage where they needed to essentially input the answer to the puzzle as opposed to how many people viewed the passage immediately following the puzzle.

This made it clear which exact puzzle in our game was too difficult. Adding some more aggressive hinting earlier in the game quickly fixed things, which we were able to verify both by looking at passage view counts for the "puzzle solved" passage as well as our metrics for overall game completion.

Using similar techniques, we were also able to look at the more nonlinear and multilinear parts of our game: when given freedom to wander wherever they wanted, what paths did players take? We were able to look both at overall view counts for individual passages (e.g. did more people go into the dining room than the study?) as well as ordering of the two (e.g. for people who went both upstairs and downstairs, which area did they explore first?).

Our game didn't have a particularly complex puzzle graph, so we didn't need to really make changes to the flow of our game in order to make it work, but having that sense of where players' eyes were naturally drawn would have given us a clearer sense of how to rearrange the game space's architecture if those had become problems.

But what if you're not making a puzzle game?

Your game might not as clearly break down into explicit discrete chokepoints, but this same general approach can tell you a lot about player data.

Often you'll design a choice with two options, expecting that roughly 50% of players will select each option. Tracking those passage views can help confirm whether player behavior meets your expectations.

If they're not, that then gives you the opportunity to either revisit the choice you're presenting players to try to get it closer to what you were imagining, or to lean into it and redesign other aspects of the game based on what that tells you about what your players are actually interested in.

If you're making an episodic game, you could also see how tracking this data can help influence future decisions. You don't want to make creative decisions completely at the mercy of your players, sure, but knowing that your players prefer a certain character or a certain type of gameplay might be useful data to have as you continue to write and design. And while asking players directly what they like is also a useful technique, it's equally useful to look at what players do rather than what they tell you.

How can we convey choices to players?

We didn't end up shipping anything like this in the final game itself, but we were really interested in being able to show Telltale style metrics to users within the game itself. After completing the game, we imagined players might be able to see "for this major choice, X% of players made the same choice as you".

Once you have aggregated data about how players interact with your game, you can imagine other interesting ways to surface this to players. It's not the right fit for every game, sure, but using your data to directly empower players can also help you allay your own concerns that you're using these metrics techniques for good rather than evil.

Be data-informed, not data-driven

All of these tools are incredibly powerful, but it's worth reiterating that you shouldn't solely rely on metrics. Even in live F2P games, people talk a lot about having the wisdom to know when to disregard what the data is telling you.

Whatever your design goals are, you shouldn't look at metrics as numbers to obsessively optimize, but a very specific tool to use to augment your decision-making process. When you have a question about how players are interacting with your game, looking at data and constructing a specific query can be one way to get one specific perspective of what's going on.

Which is to say: analytics are great! Use them! But looking at metrics can itself become an unhealthy Skinner Box-esque system that triggers compulsive behavior in you as a designer.

So how can I do this in my game?

Maybe you're reading all of this and saying "wow, that sounds great! I want to apply these techniques to my own game!"

How to get started depends a lot on what you're making, what tools you're using, and how technical you are.

I'm personally partial to PlayFab. It's a hosted web service that basically aims to power your game's backend. It's intended to be used in rather large F2P games, where it can do things like manage your game's entire economy and social systems, but I really like using it for small-scale projects just for things like user authentication, leaderboards, cloud saves, and analytics. Every personal project I've worked on has fit within its free tier (even my game Flappy Royale, which had nearly 200,000 DAU at its peak)

For analytics in particular, I appreciate that it's focused specifically on the needs of game designers, rather than people building business-focused web applications.

Being owned by Microsoft is also a plus from a privacy standpoint: most popular large analytics services are owned by ad providers, which gives them a perverse incentive to sell your users' data. There are a ton of smaller startup analytics tools that are great as well, but they're historically likely to be acquired by those same large ad companies. Using a tool owned by a large stable company like Microsoft that isn't in the advertising business seems like the best of all worlds.

If you're a coder, PlayFab is easy to get started with. Whether you're writing in Unity, Unreal, JavaScript on the web, or anything else, it's usually only a few lines of code to authenticate with PlayFab and start sending real-time analytics data. Check out their getting started guide.

If you're using Twine, I maintain a tool called PlayFab-Twine designed to make it as easy as possible to add analytics to your game via PlayFab without writing any custom code.

With only 5-6 lines of JavaScript copy/pasted into your Twine project, and a few minutes' worth of configuration, you'll be able to use PlayFab's web UI to answer the same sorts of questions I did for the Mystery Mansion project.

Go forth and measure things!

Hopefully this has been a useful intro to the world of how analytics can (and can't) help your narrative game! I'm excited for you to start gathering and making decisions influenced by (but not too heavily!) actual real-world data from your players!

Please do drop me a note on Twitter if you do something cool with what I've talked about!

A Modern Developer's Workflow For Twine

Em Lazer-Walker — Thu, 16 Jan 2020 16:56:23 +0000

I love Twine! Whether you're trying to prototype a larger work or make something on your own, it's such a powerful and easy-to-use tool to make hypertext-based narrative games.

That said, a common complaint I've heard from most people I've talked to who use it seriously is how readily its workflows fall apart at scale.

A visual graph editor is a fantastic approach for small projects, but gets unmanageable quickly on larger projects. Additionally, the way the Twine 2 editor handles files means using using tools like version control can be difficult, and merging changes from multiple collaborators can be nearly impossible.

But there's a solution! I'm going to spend the next few minutes walking you through my Twine development workflow. There are three important parts of it I want to talk about:

Plain text files. I use VS Code to write my games, rather than using the visual Twine editor.
Modern version control, storing my games in git on GitHub.
Automatic publishing. Every time I push a new version of my game to GitHub, it's instantly playable via GitHub Actions and GitHub Pages.

Let's step through the tools I use, and how you can get set up with a similar toolchain!

Writing in a Text Editor

Why is it valuable to be able to write Twine games as text files instead of as nodes in a visual graph?

It scales better. When your game grows to be tens of thousands of words, navigating Twine's node-based visual editor can be a pain. Having your entire game be in a single text file, that you can manipulate and browse however you'd like, is far easier for even medium-sized projects. And that's even before considering that being able to split your script up into multiple files, which can greatly reduce the cognitive load for larger projects.

It allows for reuse. Have some macros or other bits of scripting you'd like to reuse between passages, or across multiple game projects? Being able to copy/paste text in an IDE is a lot easier than managing it in the visual editor.

It gives you access to better writing tools. I'm more comfortable writing in the same text editor I use for other programming and writing tasks than I am in Twine's text boxes. It also means I can use the tools they provide to make my life easier!

VS Code has extensions to add syntax highlighting for both Harlowe and Sugarcube. More than that, access to its entire IDE ecosystem means I can pull in tools to help with creative prose writing. This means basic things like spell check and an omnipresent word counter, but it can also mean more powerful tools to do things like warn me if I'm using subtly sexist/racist/ableist language or even spark my creativity by collaborating with an AI!

It enables more robust versioning and collaboration. More on this later, but writing my game in a text file means it's stored in a human-readable text file, which is what enables all of the other great tools and techniques I'll be talking about next.

This all sounds great! To get all of these benefits, we can use a special programming language called Twee!

What is Twee?

In the olden days of Twine 1, there were two officially-supported ways to make games: using the Twine visual editor, or by writing code in a scripting language called twee that could be compiled by an official CLI tool, also called twee.

(A fun historical sidenote: even though the Twine's visual editor is the more popular tool, the twee CLI predates it by 3 years!)

Twee code is conceptually the same as a Twine graph, with different blocks of text in a file referring to different passages.

:: Start
This is the first passage in a Twine game!

[[This is a link|Next Passage]]


:: Next Passage
The player just clicked a link to get here!

When Twine 2 came out, support for the twee language was officially killed, and the only officially supported path was to use the Twine 2 visual editor and its greatly-expanded support for story formats.

How do you use Twee with Twine 2?

When Twine 2 wasn't accompanied by a "Twee 2", the community stepped up, and a number of third-party twee CLI tools emerged. The twee language needed to adapt, though, since Twine 2 handles story formats in a vastly different way from Twine 1.

What follows is a bit of a technical explanation of the development of modern Twee tools. I think it's interesting, but if you want to skip over it, the main practical takeaway is that I use the Tweego CLI tool to write a newer version of Twee that's called Twee 3.

Twine 2 Story Formats: A Technical Explanation

To understand why we can't just use the old twee tool with Twine 2, we need to understand how Twine 2 story formats work.

Internally, Twine 2 stores your work as an XML document. When you click the "publish" button in the Twine 2 editor, that XML document is passed to the selected "story format", which is essentially an HTML template. A story format will typically embed JS within that template to parse and modify the Twine story data as appropriate to display it as a playable game.

This is why/how different story formats present vastly different authoring syntax: as far as Twine the engine is concerned, a passage's text is just an arbitrary text blob (except insofar as it parses links to draw lines in the visual graph editor), and it's then up to the story format to decide how to parse a passage to provide narrative functionality.

If you're curious to see a "minimum viable story format", I maintain a story format called Twison that converts Twine story data XML into JSON, with a few bits of computation and data-munging meant to make the JSON easier to consume if you're integrating it into your own game engine.

This all means a story format is essential to actually going from a script to a playable game! It isn't enough for a hypothetical CLI tool to just take your twee code and bundle it up into the same XML format that Twine 2 uses internally, it also needs to then pass that XML to a story format and generate an HTML file from that interaction.

So... is there or isn't there a Twee 2?

The last few years have been a tumultuous time for people who would want to write Twee. After quite some time of different people building out different competing Twine 2-compatible twee compilers, there is now a formal language specification for Twee 3, maintained by the Interactive Fiction Technology Foundation (IFTF).

It's designed to be a superset of the original twee language (retroactively known as Twee 1), and to be fairly easy to convert between twee code and the internal format used by the Twine 2 visual editor.

If you're interested in the history and politics of how we got here, this oral history is a great overview.

There are multiple functioning Twee 3 compilers, but I personally use Tweego. I'm sure others are great as well, but Tweego works well, is actively maintained, and is easy to get support for in the official Twine Discord.

How to use Tweego

If you're comfortable using CLI tools, Tweego is quite easy to use. After downloading the correct binary from the website, you can call it directly to simply compile a .twee file into a compiled .html file you can play in a browser:

$ /path/to/tweego -o example.html example.twee

Here's the sample code from earlier updated to Twee 3 and with some metadata:

::StoryData
{
    "ifid": "F2277A49-95C9-4B14-AE66-62526089F861",
    "format": "Harlowe",
    "format-version": "3.1.0",
    "start": "Start"
}

::StoryTitle
My test story!

:: Start
This is the first passage in a Twine game!

[[This is a link|Next Passage]]


:: Next Passage
The player just clicked a link to get here!

That ifid is a random unique identifier for a game. If you try to compile a Twee file without including that, tweego will automatically generate one for you.

Similarly, tweego has a ton of other options and flags you can pass in, that you can see by running tweego --help. For the options that do things like specify a story format, I'd highly recommend just specifying that in a metadata block like I have above.

Also worth calling out is the --watch option. If you run tweego -o example.html example.twee --watch, it will start up a server that watches for file changes and then recompiles. If you have a text editor open in one window and a web browser open in another one pointed to your compiled output, this is a great way to quickly test changes!

But I want to use the visual editor!

If you have a reason to use the Twine 2 visual editor for something, you can also use it with Tweego. You can take the .html file output by Tweego and import it directly into Twine 2. When you're done, you can convert back from a .html file produced by Twine 2 into Twee by using the -d flag (e.g. tweego -o example.twee example.html -d).

As an aside: the Twee language includes import functionality that lets you spread your game across multiple files and then join them at compilation time. That can be a really powerful technique for managing larger games, or reusing macros across projects, but that sort of workflow can make jumping back and forth with the visual editor trickier. See the tweego docs for more info.

Version Control

As mentioned, one of the coolest parts about writing Twine games in plain text files is how much easier they are to version.

If you've ever tried to revisit previous versions of a Twine game you've made, or tried to collaborate with other writers, you know how difficult this can be when you're operating purely on .html files! Whether you're using git or just storing .html files on a server somewhere, having to import and export files that aren't particularly human readable is a major pain.

In the past, I've often given up on trying to fix merge conflicts with other writers, and just manually copy-pasted changes into the Twine editor by hand. That's frustrating, and avoidable by storing everything in Twee files instead!

I'm not going to walk through how I use git and GitHub, but I will say one important thing that I do is not store my compiled .html files in git at all. Rather, I'm going to set up a build process so that GitHub is responsible for automatically compiling my .twee files into .html files. This means we can keep the git repository clean and readable!

Automatically building on GitHub

The concepts of CI and CD (continuous integration and continuous delivery, respectively) are very popular in non-game software development. The high-level idea is that it shouldn't require a lot of manual work to deploy a new version of your software.

As soon as you push up new code to your version control server, it should be responsible for making sure things aren't broken and then compiling it, deploying it, or whatever else might need to be done to get your code into the hands of users.

This might seem foreign, or perhaps overkill, if you're just used to the flow of writing a game, getting an HTML file, and uploading that to something like https://itch.io.

However, GitHub Actions are a lightweight free service we can use to easily set up a deployment pipeline! In the previous section, I mentioned I don't store the compiled HTML files in my git repos for Twine/Twee games. Instead, GitHub Actions handles everything.

Every time I push a new version of a Twine game to GitHub, a GitHub Action runs that uses Tweego to compile my game, and then publishes it to GitHub Pages. The end result is I don't need to think about how to publish my game, or worry if I've forgotten to deploy the latest version or not: whatever version of my Twee code I can read on GitHub, that's the version players are playing!

Getting this set up with your own Twine/Twee project is easy. Let's walk through it!

Add the story format to git

When your Twee specifies that you're using a story format like Harlowe or Sugarcube, Tweego can find the correct story format because the version of Tweego you've downloaded from the Tweego website includes a half-dozen standard ones. The way we'll be installing Tweego on GitHub Actions won't have access to those.

Within your git directory, create a folder called storyformats. Go into wherever you've downloaded Tweego, and move the appropriate story format(s) from its storyformats directory into the one you've just created. Commit and push that to git.

This is also generally a good thing for maintaining your game in the future! If you come back to this in five years, it's possible this specific version of the story format you're using might not still be available, and tracking it down might be hard; including the exact story format bundle within your git repo will help ensure (although not guarantee) your ability to edit and compile your game.

Getting Started with GitHub Actions

To set up a GitHub Action, all you need to do is add a new file into your git repo.

GitHub Actions are based on "workflows", which are configuration files. If you add a file called .github/workflows/build.yml (or any .yml file inside that directory), it will read that config and try to use it.

That file should look like this:

name: Build

on:
  push:
    branches:
      - master

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1

      - name: Use Go 1.13
        uses: actions/setup-go@v1
        with:
          go-version: 1.13.x

      - name: build game
        run: |
          go get github.com/tmedwards/tweego
          export PATH=$PATH:$(go env GOPATH)/bin
          tweego YOUR_TWEE_FILE.twee -o dist/index.html

      - name: Deploy to Pages
        uses: peaceiris/actions-gh-pages@v3
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_branch: gh-pages
          publish_dir: ./dist

Be sure to swap out YOUR_TWEE_FILE.twee for the actual filename, and change any other tweego settings you might need to. If you're not sure what you're doing, you probably want to leave the output file as dist/index.html.

This script uses GitHub Pages to host your games. It's a free hosting service for static sites such as Twine games that's integrated right into GitHub. It's totally free, and can scale to support any amount of traffic. I think it's absolutely the best and easiest way to host small websites like Twine games that don't require any sort of backend server services.

If you don't want to use GH Pages to host your game, you'll want to replace the last "Deploy" step with whatever you're using instead.

Testing your GitHub Action

If you make a new commit and push it to your game's master branch on GitHub, after a few minutes it should be live on the web! By default, it should be available at https://[your-github-username].github.com/[repo-name], although it's also possible to configure GitHub Pages to work with a custom domain name.

The GitHub Action can take a few minutes to compile and deploy, so be patient! You can also click through to the "Actions" tab in your repository and see the build as it progresses.

For those who are interested, let's walk through what this config file is doing:

name: Build

This just names the workflow. It can be anything you want; it'll show up in the Actions UI.

on:
  push:
    branches:
      - master

This indicates the series of steps that follow will execute whenever someone pushes code to the master branch.

jobs:
  build:
    runs-on: ubuntu-latest

    steps:

Now we've started to define the task itself. Specifically, it runs on Linux, although that doesn't really matter to us.

Conceptually, a workflow is made up of a number of steps. A step can either be some code we manually write, or it can be a preset collection of actions provided by the community.

- uses: actions/checkout@v1

This checks out the latest version of our code

- name: Use Go 1.13
  uses: actions/setup-go@v1
  with:
    go-version: 1.13.x

Tweego is written in the programming language Go. We'll be compiling Tweego's code from scratch, which means we need a Go compiler. This gives us a working environment for Go code, and lets us specify which version of Go we want.

- name: build game
    run: |
      go get github.com/tmedwards/tweego
      export PATH=$PATH:$(go env GOPATH)/bin
      tweego YOUR_TWEE_FILE.twee -o dist/index.html

This is a custom script! The first go get line downloads and compiles the Tweego tool itself. The next line does some fiddly environment setup you don't particularly need to worry about (modifying our PATH so we can just call the tweego binary without specifying a full filepath). Finally, we run tweego itself.

- name: Deploy
  uses: peaceiris/actions-gh-pages@v3
  env:
    github_token: ${{ secrets.GITHUB_TOKEN }}
    publish_branch: gh-pages
    publish_dir: ./dist

At this point, we have an HTML file in a directory called dist. This is a third-party action created by another GitHub user that deploys code straight to GitHub Pages. This config uses an automatically-generated access token (so it has permissions to commit/deploy), and specifies that we want to take all of the files in the dist directory and publish them to the gh-pages branch.

...and that's it!

And with all of that, we should be good to go!

As someone used to working with more programer-focused tools, I've found this workflow to make it WAY easier and more pleasant to work on games with Twine. Hopefully it's helpful to you too!

If this is interesting to you, you might also be interested in PlayFab-Twine, my tool to easily and automatically add free analytics to your Twine games. The GitHub repo for that site is also a great example of a Twine project developed using this workflow!

Drop me a note if you're using any of this stuff, I'd love to hear from you!

Making a weird GIF wall using Azure Functions and SignalR

Em Lazer-Walker — Fri, 06 Dec 2019 19:14:42 +0000

At this year’s XOXO festival, one of the top-secret closing party happenings was a special live listening of Neil Cicerega's latest mashup album. If you're not familiar with Neil's work, his previous album Mouth Moods might give you an idea of what was played: a weird and surprising concept album that sort of amounts to cramming an excessive amount of Pure Internet™ into your ear through mashups, references, and very clever mixing.

One of the XOXO organizers approached Reed Kavner and I to make some sort of interactive installation to accompany the listening party: a sort of gif wall where listeners could post GIFs and other weird Internet ephemera as a way of annotating the piece.

I had just started my new job on the Microsoft Azure Advocates team, so I took this as a chance to try out a whole bunch of Azure tech for the first time!

A Wall of Pure Internet

The goal was to create a completely overwhelming wall of GIFs and text. We wanted people to be able to live-annotate the music by pulling up memes the music itself was referencing, while itself playing into a sort of Internet-y vaporwave visual aesthetic.

We decided to rely on Slack rather than build out our own UI. XOXO has an active year-round Slack community, and most attendees were already logged into the festival Slack on their phones. This handled a whole bunch of hard problems for us: authentication, mapping posts to real names (important to handle Code of Conduct violations) and fully handling GIF search (including explicit content filters).

The level of trust we put in our community (along with our real-name policy) meant we could also allow people to post plaintext messages instead of just GIFs. Along with that, it mattered to us that we supported all of the custom emoji that our Slack supports, since the community has built up a large collection of meaningful ones.

One other conscious design decision was to not rate-limit how often anybody could post. When you post a GIF or some text, it shows up on screen and slowly grows over time, but any newer GIFs that come after yours will cover yours up. We simply set the starting size of a post based on how recently the author last posted. If somebody wanted to sit there and spam GIFs as quickly as they could, we wanted to let them do that, but making their content start smaller meant their fun wouldn't come at the expense of annoying others.

Serverless? With a long-running client?!

While Reed built out the JS front-end (available on GitHub), I was responsible for the server infrastructure to send messages to a web browser.

I was interested in using Azure Cloud Functions to avoid needing to spin up my own server on something like EC2 or Heroku. With "serverless" tools like Azure Cloud Functions, you just upload a single free-floating function (JS in my case), and instead of you maintaining a server runtime, Azure is responsible for spinning up an instance and running your function any time somebody hits a specified HTTP endpoint. In our case, that endpoint is a webhook being triggered by a Slack API app.

On the browser side, we assumed we'd use a WebSocket connection to send messages to the client. However, WebSockets require a long-living connection. With serverless functions, we only have an execution environment at the moment our function is being called, which makes it rather difficult for the browser app to have a persistent WS connection!

Enter SignalR!

SignalR is a technology designed to make it easy for servers to broadcast real-time messages to various clients. It’s different from WebSockets in that it’s unidirectional — it can only be used to send messages from servers to clients, not the other way around.

It's mostly meant for larger, more enterprise-focused uses: it gracefully handles things that WebSockets doesn’t like more complex authentication and connection handshakes. It operates at a higher level of abstraction than WebSockets: by default, it even uses WebSockets in the browser as its transport mechanism, but can fall back to alternate methods automatically (e.g. polling) without you needing to worry about it as a developer.

We don't care about the security or reliability promises of SignalR, but we do care that Azure offers a hosted SignalR service that can interoperate with Azure Cloud Functions. This lets us overcome the issue of needing a long-running connection to a short-lived server!

The browser client connects to the Azure SignalR service, which maintains that connection for as long in the browser is open. In the meanwhile, any time an Azure Function instance spins up and executes, it can independently connect to the SignalR service and push messages to the queue. We get the flexibility of using serverless functions to build our node app, but can still maintain a long-running WebSocket connection to the client app. Neat!

Using SignalR with Cloud Functions: Declaring Inputs and Outputs

I'm not going to explain in here how to get set up with Azure Functions — check out this tutorial for getting started using the official VS Code extension, which is by far the easiest way to manage the fiddly bits — but I do want to talk a bit about how I integrated SignalR with my cloud Function.

Azure Functions have a really elegant way of handling external dependencies into your code. An Azure Function is just a single file with a single code function, but accompanying it is a function.json config file that specifies all inputs and outputs the function accepts. Add a bunch of dependencies to your function.json file, and they'll automatically be injected into your function as arguments!

Setting up SignalR requires two different functions. First, there's a short setup handshake required: a browser that wants to connect to our SignalR instance needs to hit an HTTP endpoint that returns the magic connection string it needs to complete the connection

{
  "disabled": false,
  "bindings": [
    {
      "authLevel": "anonymous",
      "type": "httpTrigger",
      "direction": "in",
      "name": "req"
    },
    {
      "type": "http",
      "direction": "out",
      "name": "res"
    },
    {
      "type": "signalRConnectionInfo",
      "name": "connectionInfo",
      "hubName": "chat",
      "direction": "in"
    }
  ]
}

module.exports = async function (context, req, connectionInfo) {
  context.res.json(connectionInfo);
};

You can see here we're setting up a function that has standard ExpressJS request/response inputs/outputs, as well as an extra connectionInfo argument that we specify in our function.json file should contain SignalR connection info to a message queue called "chat".

Our actual "post a message" Slack webhook function has a slightly different function.json file, as it uses the SignalR connection as an output (essentially a message queue it pushes messages onto) rather than an input:

{
  "disabled": false,
  "bindings": [{
      "authLevel": "anonymous",
      "type": "httpTrigger",
      "direction": "in",
      "name": "req",
      "methods": [
        "post"
      ]
    },
    {
      "type": "http",
      "direction": "out",
      "name": "res"
    },
    {
      "type": "signalR",
      "name": "$return",
      "hubName": "chat",
      "direction": "out"
    }
}

The "name": "$return" property means that whatever our function returns ends up getting pushed onto the "chat" SignalR queue as a message, which in turn gets pushed to all connected SignalR clients.

With these two functions in place, the actual client code to connect to the SignalR queue is fairly simple:

const connection = new signalR.HubConnectionBuilder()
    .withUrl(`https://xoxo-closing-party.azurewebsites.net/api`)
    .configureLogging(signalR.LogLevel.Information)
    .build();

connection.on('newMessage', function(m) {
    addPost(m); // m is a JSON blob containing whatever our function sends
});

connection.onclose(() => console.log('disconnected'));

connection.start()
    .then(() => console.log("Connected!"))
    .catch(console.error);

You'll notice the SignalR library itself is responsible for hitting the handshake endpoint and then subscribing to new messages.

Emojis are Hard!

With this code so far, my backend was sending messages to Reed's JS webapp containing message text and, if applicable, GIF data. But all emoji were coming through as Slack-style text shortnames. e.g. instead of the "🎉" emoji, the messages contained the string :tada:.

Fixing this actually meant handling two totally separate things: proper Unicode emoji, and our Slack instance's custom emoji set.

For “official” emoji, I was able to find someone else who already wrote a quick script to fetch Slack's mapping. This CLI one-liner I modified from the web gave me a JSON object mapping from short name to Unicode code point.

curl -s https://raw.githubusercontent.com/iamcal/emoji-data/master/emoji.json | \
  npx ramda-cli \
    'reject (.unified.includes("-"))' \
    'chain (emoji) -> emoji.short_names.map -> {...emoji, short_name: it}' \
    'sort-by (.short_name)' \
    'index-by (.short_name)' 'map -> "0x#{it.unified}"' > emoji.json

{
  ...,
  "abacus": "0x1F9EE",
  "abc": "0x1F524",
  "abcd": "0x1F521",
  "accept": "0x1F251",
  "adult": "0x1F9D1",
  "aerial_tramway": "0x1F6A1",
  "airplane_arriving": "0x1F6EC",
  "airplane_departure": "0x1F6EB",
  "alarm_clock": "0x23F0",
  "alien": "0x1F47D",
  "ambulance": "0x1F691",
  "amphora": "0x1F3FA",
  "anchor": "0x2693",
  "angel": "0x1F47C",
  "anger": "0x1F4A2",
  "angry": "0x1F620",
  "anguished": "0x1F627",
  "ant": "0x1F41C",
  "apple": "0x1F34E",
  "aquarius": "0x2652",
  ...
}

From there, I was able to use built-in JS string replacement functions to replace all valid Unicode emoji with the proper Unicode code points:

const replaceEmoji = message => {
  const standardEmojiMap = require("./emoji");
  return message.replace(/\:(.*?)\:/g, (original, name) => {
    if (standardEmojiMap[name]) {
      return String.fromCodePoint(standardEmojiMap[name]);
    } else {
      // This isn't in our list of Unicode emoji — either it's a custom emoji or nonsense
      return original;
    }
  });
};

Custom emoji were a bit trickier. Slack offers an API endpoint to grab the custom emoji for any given Slack instance.

Crucially, although it returns a map whose keys are emoji names, the values can be one of two things: a URL to a CDN-hosted image for that emoji, or the name of another emoji name that it's an alias for. So when doing my own find/replace, I needed to check if it was an alias, and if so make sure to resolve that. When I eventually landed on an actual URL, I replaced the :emoji: with an HTML <img> tag pointed at the CDN URL.

This made things slightly trickier for Reed: however he was rendering this text on-screen, he now needed to make sure that <img> tags were rendered properly as HTML, but also do that in a way where <script> tags wouldn't be executed as arbitrary JavaScript. It added some complexity, but we concluded that was easier than alternative methods of specifying "this image should be injected at this point within the text".

I cached this custom emoji data from Slack in an Azure CosmosDB database. While it's not like our custom emoji updated all that frequently, I needed to build out that caching infrastructure to handle fetching names as well.

Messages from Slack only contained unique user IDs, not human-readable names, so just like emoji I ended up needing to make some API calls to Slack's user list API endpoint so I could do my own lookup.

I'm not going to go into that process of using CosmosDB right now — our name cache (but not our emoji cache!) ended up falling over in production, and it was suggested to me after-the-fact that Azure Table Storage would have been a better fit for our needs.

The End-Result

...and that's (more or less) all there was to it! I glossed over a whole lot here, but you can check out the GitHub repo to see the code itself. I was impressed how well Azure Functions and SignalR worked — messages came through within a second or two of people sending them, it scaled effortlessly even when we were getting hundreds of messages per minute, and everybody loved the installation!

I'd love to see someone else take our code (or just inspiration from us) and make something similar! Shout at me on Twitter if you do anything cool like this.