Forem: Josh Wulf

Creating an Extraordinary Contributor Experience

Josh Wulf — Tue, 22 Sep 2020 08:38:01 +0000

Hacktoberfest is coming! And Camunda is participating, with the opportunity to get a sweet limited-edition Camunda Hacktoberfest t-shirt by making just two pull requests to any of our open source projects.

We've also put together a guide for open source repository maintainers, to empower you to create an extraordinary contributor experience.

Creating an Extraordinary Contributor Experience

This document is for open source project maintainers who want to increase community contributions to their project.

We'll look at the Contributor Experience and how you can create an Extraordinary Contributor Experience.

So You're a Maintainer

You're an open source project maintainer.

You're probably also a contributor to the project - maybe the only one.

Wouldn't it be great to have other people making contributions to the codebase?

Or maybe you're part of a team that works professionally on a code base. Maybe you have users asking for features and bug fixes. Wouldn't it be great if the contributions grew at the same rate as the requests?

Contributors not only provide value to the project by testing, opening issues, and contributing code, they also become advocates for the project in the wider community. And why not? It's their project too, now. They'll tweet out when their code goes into a release, and add it to their CV.

Maybe you've been an external contributor to an open source project prior to become a project maintainer - maybe you haven't.

Either way, you probably haven't thought about creating a contributor experience, systematically.

That is what we are going to look at here.

Why? Because repeat contributions, like repeat customers, come from an extraordinary experience. Not everyone will repeat - but if your contributor experience is extraordinary, you will increase the amount of contributions.

Who Contributes

Let's first consider who contributes.

We will create a few archetypes.

First-timer: The first-timer is someone who has never contributed to any open source project before. This is all new to them. They don't know how to "open a pull request", they are not sure if their contribution will be good enough to be accepted, and if they do take the plunge, how this pull request goes will determine how their entire open source contribution career goes. No pressure.

Enthusiastic Amateur: The Enthusiastic Amateur hacks code together and submits it fast and furiously. They have enthusiasm, but their code is not always up to scratch, and may come in without tests, or with spelling errors. They may lose their enthusiasm if required to make too many changes to get their pull request accepted.

Seasoned Pro: The Seasoned Pro is not only a solid developer who knows not only how to write good code, but also how open source works. They've done a bunch of PRs to various open-source projects, so they know what to expect. This person compares their experience with your project with their experience with other projects.

Characteristics of External Contributors

People are usually busy. They might be using your project for their day job. You are focused on the project itself - it's the main event. For a lot of potential contributors, however, your project probably is one piece in a complex stack of technology that they are wrestling with. It's an extra on the show.

Any contribution that people make to the project is a generous contribution of their valuable time.

How to create an Extraordinary Contributor Experience

We're going to cover a number of specific tactical actions that you can take, strategically.

However, there are two things that will make everything work. These are orientations, rather than actions. If you just get these two things, everything will work. If you do everything else and neglect these, your results will be variable.

Always Deal with The Person Behind the Pull Request

Behind the bug report, feature request, or pull request is a person.

Always deal with the person. Deal with the person about the pull request, rather than just dealing with the pull request.

It's obvious and easy to think that we are building software. Something that is just as easy to forget is that in open source we are building software together. So while we are building software, we are also building an experience of building software.

Before joining Camunda's DevRel team, Josh was an external contributor to the Zeebe project. While evaluating Zeebe against other open source workflow engines, he opened various issues and pull requests.

The pull requests to other projects languished, but the Zeebe ones got feedback, and got merged. He says that his experience was that his contribution was acknowledged and appreciated. That gave him confidence that if his company at that time went with Zeebe, they would be able to contribute fixes to the engine for their use case, and they would get merged.

Think about the person behind the pull request. Put yourself in their shoes. Think back to the best experience that you've had when contributing to a project. Maybe it was your first pull request, or another time. You probably experienced being acknowledged and appreciated, as a person, as well as respected professionally for your contribution.

On the flipside, a prominent open source company let one of its best engineers go because of their interactions with the community. This was an A-grade rockstar contributor - legendary in the open source world for their work on the Linux kernel. Unfortunately, they were also legendary for their interactions with contributors - and not in a good way. A typical PR response went along the lines of "your code is stupid, and so are you".

Don't be that guy.

If you treat your contributors well, they will come back.

Have the Contributor Win

People contribute for different reasons.

Some people are fixing a problem or implementing a feature for their use case, and they want it merged upstream so that they don't have to maintain a separate fork.

Some people are out to learn, and to build a reputation for themselves that is portable and visible.

Some people are just good open source citizens. Fixing things they come across is like picking up a piece of litter on the street.

Some people are evaluating the project for use, and want to get a sense of how amenable and responsive it is to Pull Requests.

For all of these cases, if you take the stand that you are the champion of this contributor and it is your role to "have them win", then you will find the way to empower them, wherever they are coming from.

A contributor may have time to work on further changes to a Pull Request, such as documentation or tests - and they may not.

Sometimes, it is a "drive-by" contribution. Someone fixing something in a dependency of their project and moving on with their day job. They may not have time to do additional changes to make it just right.

If you deal with the person behind the Pull Request, and see yourself as their champion, then you will be sensitive to their ability and appetite for extended collaboration.

Maybe they want to learn, and are eager to make further changes. Maybe not. But if your goal is to "have them win" - to get their contribution merged and their name in lights, then you will figure that out with them.

You have to balance something here: if a pull request comes in that requires a significant amount of work to get over the line, you may have a problem.

The contributor may not have had in mind that getting the PR over the line would turn into another full-time job for them. And you already have other things that you are working on, so fixing a half-done pull request is putting more work on your plate.

One of the things to factor in - as you work out whether it is worth the effort for you to get it over the line - is the longer-term investment in community capital.

Merged pull requests beget more pull requests. And one of the metrics used to evaluate project health - for people who are considering betting on your project by basing their project on it - is how many open pull requests there are, and how long they have been open.

If you can't see your way to get it done, and the contributor says that they can't get it done, then you have a couple of options.

You can close the pull request, and they can reopen it later, when they have time to work on it. Or you can pull it into a branch to work on later. If you do the latter, they will get props when it does get merged to the main code base.

Either way, acknowledge them and thank them.

If you do invest the time, either to work with them, or to get it over the line for them, know that you are making a long-term investment in the community capital of the project.

Prepare for guests

Here we get into some tactical steps that you can take to create an extraordinary contributor experience.

Before anyone comes to contribute, you can prepare the project for them.

Think of the project repository like a store, with a customer experience design. Users are one set of customers, and contributors are another.

A good example of design is the Apple Store layout. Everything is clearly marked, and discoverable. When people come in to an Apple Store, they are not wandering around a labyrinth, left to puzzle things out for themselves.

Add templates to the repo for Pull Requests and Issues. This gives users guidance when they open a Here is a great collection of templates: https://github.com/stevemao/github-issue-templates.

Put the sign out in front

Every business needs to let people know that they exist, and that they are open for business. Let people know that you accept contributions. Put it at the top of the README.md file, so that it's one of the first things that they read, with a link to the contribution instructions.

It's one thing to include a CONTRIBUTING.md file in your repo, it's another thing to "stand out on the sidewalk and hand out fliers". "Putting the sign out in front" doesn't mean the front of the project - it means in front of the contributor. People will land in different places in your project from a search engine, or by focusing in on a specific area of interest. If you catch them at the right moment, in the right place, with the right message, a brief moment of inspiration can convert into a contribution.

Everywhere that represents a touch point or an area that the community can contribute to, weave it into the narrative.

Put it in the Getting Started instructions: "If you find something wrong or missing in here, feel free to submit a pull request." (with a link to the documentation contribution section of your contribution instructions).

Put it in the API documentation. Put it in the issue template. Put it everywhere (but don't overdo it).

You want to "walk the floor". Start with a blank mind, and approach your project as a new user. Go to your repository, your documentation, and your package / image hosting page, and walk through them as a new user.

Is it obvious that I can contribute to this project? Is it obvious how I do that, or how I find out how to do that?

Make it Easy

The lower the barrier to entry, the more people will cross over it.

If you put links in your documentation so that someone can see something wrong or missing, click a link, correct it and submit a Pull Request, people are more likely to do it.

If you make it a single click from your README to report a bug, with a template that they can fill out, people are more likely to do it. This also makes your job easier, because you are not constantly going back to ask users for the details you need to triage and assess their issues / contributions.

If you want people to search existing bug reports first, then give them a link to the existing issues filter, with instructions to search, followed by a link to open a new issue.

If you want people to contribute code, you need to document how to develop the project. This means installing all prerequisites and building the project. The more of this that you can script, the easier you make it. Potential contributors can burn all of their inspiration trying to get the project to build, before they even think about writing code.

Again, "walk the floor". Starting with a blank mind, approach the project as a new user. How easy is it to check out and build? Can you make it easier?

This can be challenging, because your computer is configured to build the project, and may have dependencies or tools that are required, but not covered by your instructions. Again, this is a perfect opportunity to weave in opportunities for people to contribute. You can make it explicit - "if you find something missing in these instructions, feel free to submit a patch".

Acknowledgement

People contribute to open source for many reasons, but something that every person thrives on is acknowledgement. There is a saying "what gets recognised gets repeated".

Acknowledging people for their contribution goes a long way.

One thing you can do is label any Pull Request coming from a remote fork with a label saying "Thank you".

And you can add a comment on the pull request, leaving the person behind the pull request acknowledged and appreciated.

Timeliness

People are busy - including you!

Remember, though, that we are not just building software here - in open source, we are building a community that builds software.

Just like you attend to infrastructure or production outages, or customer issues, to create an extraordinary contributor experience, you'll need to attend to contributor requests.

Contributors may have a limited attention span or period of inspiration. It is important to acknowledge their contributions as soon as possible, and then to let them know what time frame they can expect to get a response within.

When evaluating the Zeebe project as an external contributor, Josh submitted a pull request to fix a bug on a Friday. When he got back to work on Tuesday after a long weekend holiday in Australia, the pull request had been merged and released.

That went a long way to building confidence in the project, and that company decided to base their project on Zeebe.

You can't always turn things around that fast, but you can acknowledge receipt and set an expectation. Automation is your friend in this case. You can create a "Contributor SLA" and create automation to trigger reminders.

Conclusion

You can get code contributions through Hacktoberfest, but if you take care of the person who makes the contribution, you can get a contributor. And that is way more valuable.

A Concierge workflow for Hacktoberfest using the Zeebe GitHub Action

Josh Wulf — Mon, 24 Aug 2020 10:08:58 +0000

My Workflow

This is a workflow that uses the Zeebe GitHub Action to enable a "community concierge" to support open source maintainers in providing an extraordinary contributor experience during Hacktoberfest.

You have one repository that runs a GitHub workflow on a schedule to send notification emails to the concierge email address, and a couple of yaml GitHub workflows that you add to each of the project repos participating in Hacktoberfest.

You can modify the workflow, but the example one I've created notifies the concierge:

When an external PR is opened.
24 hours later, to make sure that the contribution has been acknowledged by the maintainer.
Every two days until the PR is closed, to remind the concierge to check that it is moving forward.
When an external PR is closed.

Submission Category:

Maintainer Must-Haves

Yaml File or Link to Code

Project with instructions on GitHub.

Additional Resources / Info

At Camunda, the DevRel team are using this automation to support the maintainers of our various open source projects in providing an extraordinary contributor experience during Hacktoberfest.

Refactoring the Zeebe Node gRPC State Machine for Camunda Cloud: Part One

Josh Wulf — Sat, 29 Feb 2020 10:32:51 +0000

Originally published at https://joshwulf.com/blog/2020/02/camunda-cloud-connection

Preparing for the launch of Camunda Cloud, I turned on continuous integration testing for the Zeebe Node.js client against a running instance of a Camunda Cloud Zeebe cluster.

I was immediately forced to deal with something I’d been conveniently ignoring:

Camunda Cloud gRPC connections always report failure initially, before eventually succeeding.

This is because the connection to Camunda Cloud is via a TLS-enabled Nginx reverse proxy with OAuth authentication. This causes the current gRPC client state machine to emit intermediate connection failure events before emitting an eventual “READY” state.

This “connection characteristic” differs from the behaviour of the non-proxied connection over a Docker network (the current CI test environment in CircleCI) or against a non-proxied broker on a remote machine (my personal production setups not using Camunda Cloud).

How that looks for a user:

(This is using the new, in 0.23.0-alpha.1, default ZBSimpleLogger that outputs unstructured human-readable strings)

I had been ignoring this as “expected behaviour (for now)”, and my reticular activating system had conveniently made it invisible to me.

The failure of the integration tests made it abundantly clear that this is actually not expected behaviour in the formal specifications for the client (the tests).

DESIGNING FOR DEVELOPER UX

This is bad Developer Experience (DX) design.

When developers are using Zeebe Node for the first time against Camunda Cloud, they don’t know what they are doing and they don’t know if they are doing it right.

When it is not working as they expect, they don’t know whether it is that their expectation is erroneous, or they have done something incorrectly. Users are often unaware even that they have a model of the system composed of expectations they hold as a hypothesis. Instead, they think: “Something is wrong. This is not working.”

There are four things that can be at the cause of this:

The user has missed a step or made a mistake in their code or configuration.
There is an intermittent failure condition (network down, service interruption).
There is a bug in the library.
The user’s expectation is incorrect (actual behaviour is correct, expected behaviour is incorrect).

Surfacing that last one — the unexamined hypotheses that the user holds about the system as their working model of the system — is why bug reports request “Expected Behaviour” and “Actual Behaviour” in their templates.

Any message presented to the user while they are developing their model of the system must take into account that the user’s model is unformed, and also that the user is usually not consciously forming the model.

I mean any message. A DEBUG level informational message in the Zeebe broker log has been a source of confusion for new users. They frequently interpret it as an error message (an example from the Zeebe Forum), and we are refactoring it to either reword it — taking into account the user’s uncertainty — or just take it out completely (GitHub issue “Reword Blocking request DEBUG Message”).

This particular message in the Zeebe Node client — that every single connection to Camunda Cloud always emits an error — is terrible for UX.

New users have enough uncertainty already. This behaviour guarantees that even if they do everything correctly, they are still going to be thinking: “I’ve done something wrong, it’s not working.”

And if they have done something incorrectly and it is really not working, this message provides them with no clue as to what that is, and will lead them on a wild-goose chase.

It has to go.

THE CURRENT GRPC STATE MACHINE

The current gRPC connection state machine has evolved over time. It started off pretty simple, then evolved to support client-side retry logic for network-related gRPC Error 14 but not business errors, emit connection-level error events, a ready event for the downstream Node-RED client, and managing the gRPC Error 8 event for Broker back pressure response.

As these features were added, the responsibility for managing the representation of the current connection state (that’s a true variable in the code), debouncing transitions (the channel state can flap about wildly during retries when it does go down — and this requires managing more state: time), and attaching various different retry and user-defined event handlers (if you add one for worker job streaming to the same connection used for client commands, it gets called for both), got split between the ZBClient, ZBWorker, and GRPCClient classes. This happened because these various patches were made one after the other, with no refactoring.

Now, adding the concept of “connection characteristics” will exponentially complicate already fragile code. State management is the bane of program correctness and a source of interminable bugs and edge conditions. State needs to be coalesced as an encapsulated concern in an application to avoid leaking this complexity into the other components in an application, and making changes to state management a distributed concern that touches many components.

See the article “Avoiding Global Mutable State in Browser JS” and the “Valid use of variables” section in the article “Shun the mutant — the case for const” for more discussion about this.

That implementing connection characteristics requires changes to multiple components is a code smell that indicates there is a first-class concern with no, or an incorrectly bounded, materialised entity in the code.

It is time for a significant refactor. The time required for this refactor is not trivial, and I conveniently ignored the error message until it became a failing test suite, then looked long and hard at the code before committing to this.

This is a significant part of the Zeebe Node client — managing the state of the gRPC connection and providing an ergonomic API to signal and handle various failure conditions. However, you don’t have to be Einstein to realise that:

“We can not solve our problems with the same level of thinking that created them.” — Albert Einstein

In this case, adding the concept of “Connection Characteristics” with differential behaviours to a non-first class abstraction is asking for a million bug reports and an unending game of whack-a-mole where fixing one problem causes problems with other behaviours.

No thanks. I’ll suck up the refactor costs.

The code should model the domain. We now have in the domain “various gRPC connection classes with different business rules for handling their characteristics”, so we are going to refactor the code to model that bounded context.

Fortunately, I already have the behavioural specification written — my integration tests. If I change nothing in those, I just have to make Camunda Cloud’s integration test go green and keep the CircleCI one green, and I can be confident that it works as expected.

As Evan Burchard explains, in his excellent book “Refactoring JavaScript: Turning Bad Code into Good Code”:

When we have any code that lacks tests (or at least a documented way of executing it), we do not want to change this code. We can’t refactor it, because we won’t be able to verify that the behavior doesn’t change.

STEP ONE: EXPLAIN IT IN PLAIN LANGUAGE

The first step is to write a specification in a non-executable language — aka plain English.

This is the initial specification, and it should strive to be an unambiguous definition.

I created a GitHub issue “Manage that Camunda Cloud connections always error before eventually succeeding”, and described the motivation for the refactor in there.

Then, I added a design specification as a comment.

I am going to separate the functionality into two layers:

The underlying GRPCClient that manages the gRPC connection and emits state events.
An intermediary Business Rules component that models the connection characteristics and ensures that various connections behave the same at the API level.

There is one concern whose location I am not yet clear about. I am thinking that the GRPCClient will have a method to turn on a buffering mode, where it buffers state transition events, and then returns the eventual state with a list of intermediate transitions, after a specified period of time.

This allows me to turn this on for Camunda Cloud connections, and throw away those initial failures, reporting only the ultimate result.

This may complicate the GRPCClient state machine by adding an additional state machine to it (managing time, and the current operational mode), so I may move this into the Connection Characteristics component. We shall see. It’s a working hypothesis. Gotta start with some kind of vision, and keep re-evaluating along the way.

OK, let the fun begin!

About me : I’m a Developer Advocate at Camunda, working primarily on the Zeebe Workflow engine for Microservices Orchestration, and the maintainer of the Zeebe Node.js client. In my spare time, I build Magikcraft, a platform for programming with JavaScript in Minecraft.

Avoiding mutable global state in browser JS

Josh Wulf — Tue, 25 Feb 2020 09:18:02 +0000

This is part of a series of posts where I refactor code from StackOverflow questions, with a discussion of the changes. One of the great things about JavaScript is how scalable it is. You can start with a simple script, and there is nothing wrong with that. Usually these posts are about refactorings other than what the questioner asked about, and would be out of scope for the SO answer.

The accompanying GitHub repo for this article can be found here.

Global scope is a feature of browser JavaScript that is a source of application-spanning bugs (it is global). Global state doesn’t just impact the whole application — it creates a entire new surface area for bugs across the entire code base, that has to be managed. Bugs related to global state can happen anywhere. The number of potential bugs in every function increases as soon as you have global state.

Any local function can mess with the functioning of any other function by mutating global scope, and this can result in bugs that are hard to track down to their source.

In this refactoring we are not going to be able to completely eliminate global state — mostly because we don’t have enough information about how the state will be used in the rest of the application to make a recommendation for an alternative.

What we will do is reduce the bug surface area significantly. And along the way, you’ll be introduced to some of the concepts underlying React.setState and Redux.

THE QUESTION

Here is the code from StackOverflow:

global variable
var memArray =[];

//object   
function member(id, password){
  this.id = id; 
  this.pwd = password
}
var memObj1= **new** member("m001","123");
memArray.push(memObj1);

DISCUSSION

There is a lot going on this example that can be refactored, and we’ll look at a number of things in other articles. But for now, let’s look at global state.

MEMARRAY

The global memArray has two immediate issues - apart from being global.

First, it is declared as var, which means that it can be reassigned at runtime.

In fact, using var is a declaration to the machine and to other programmers that “I intend that the value of this assignment change over the course of execution".

It may be that the novice programmer misunderstands assignment of arrays in JS. Making this a var doesn't make the contents of the array mutable - you have to do real deliberate work to make them immutable. Rather, declaring this as var makes the assignment itself mutable. Meaning that memArray itself can be mutated by pointing it to something other than the array you just created and assigned to it.

Somewhere deep in the code, a function could do:

memArray = []

This could be because another programmer uses it as a local variable name with no declaration, in which case the runtime will use the previously declared global variable. You won’t get a warning from your tools about using an undeclared variable, because it is declared.

And this bug in one function somewhere, that maybe doesn’t even use this global state (it probably doesn’t, or the programmer wouldn’t have reused the variable name), just broke everything that does use it. And when you go to hunt it down, it is not in any of your functions that do use the global state.

The chances of this happening are increased because of the second issue:

Naming

See this article about the importance of naming.

In code examples on StackOverflow, I always name global variables like this: EvilGlobalMembersArray.

There is no way someone is accidentally reusing that in a local scope. At the very least, GlobalMembersArray is an unambiguous name that communicates what it is.

FIRST REFACTOR

const GlobalMembersArray = []

Make it a const so that it cannot be reassigned, and give it a meaningful and useful name. This is “naming by convention” that takes away cognitive load when reading the code.

If I find a reference to GlobalMembersArray in a function deep in the code, I immediately know what I am looking at, and I'm not using that name for a local variable.

MUTATION

The global is now not reassignable, and unambiguously named, which reduces the chances of someone accidentally reusing it. Since it is an array, they cannot change the reference to point to another array, object, or primitive, but they can still mutate the contents.

You want that, right? Presumably, we are going to want to add to, remove from, and update elements in this array.

No. By exposing just the array globally, we have devolved responsibility for mutating it to local functions in the application.

That concern, and hence the complexity of it, is now spread throughout the application. Bugs related to mutating the array values can appear anywhere in the application, at any time. And again, they can be hard to track down, because they will likely appear when a function uses the array and doesn’t find what it expects — rather than where the bug exists.

SECOND REFACTOR — IIFE

Rather than expose an array, we should expose an object that encapsulates the state, plus mutation methods. And we will not expose the actual state, because local functions can still and may be tempted to mutate it directly. Instead we will return a copy of the state, so that the only way to update it is via the object methods.

We can do this using an IIFE — an Immediately Invoked Function Expression, a JavaScript function that immediately executes and can return an object that has a private scope inside a closure.

In terms of ES6 classes, it is roughly analogous to creating an instance of a class that has private methods.

Here it is with no accessors:

const GlobalMemberStore = (() => {
  let _members = []
  return {}
})()

Note the enclosing () and the immediate invocation: (() => {})().

In this case, we will get back an Object with no properties. But what you want to know is that it also contains a hidden array — _members - that cannot be accessed by local functions.

But, but… aren’t you the “Just Say No to Variables” guy? What is that let statement doing there?!

Look, we can remove variables completely. But we don’t have enough information about the eventual application to do that. So what I’ve done here is take a global variable, and put inside a closure where it is invisible to the rest of the application.

All the complexity and bug surface area will be behind the singularity of the closure, with an immutable API. There will be no variables exposed to the rest of the application. And the resulting code is fully unit-testable.

IMPLEMENTING GETMEMBERS

Now we will provide a method to return a copy of the _members array:

const GlobalMemberStore = (() => {
  let _members = []
  return {
    getMembers: () => [..._members]
  }
})()

The ES6 spread syntax — [...members] - spreads the contents of the local members array into a new array, and returns that.

Local functions can add things to the array, or delete elements, but these operations do not affect the global state, because they have a copy of the global state, not a reference to the global state.

Note, however, that because the elements of the array are objects, local functions can still mutate members within the copy, and that will affect the global state — because the array elements are references to objects. The internal state array and the copy we just returned are different arrays, but they contain references to the same member objects

We can avoid that scenario like this:

const GlobalMemberStore = (() => {
  let _members = []
  return {
    getMembers: () => _members.map(m => ({...m}))
  }
})()

Array.map returns a new array, so the consumer has no reference to the global state array. The new array is populated by applying the predicate function to each value in the original array, and putting the return value in the new array.

It is “make a new array by applying this transform to each element in this other array”.

In the predicate function — m => ({...m}) - we return a copy of each member object from the _members array, again using the ES6 Spread syntax, this time on an object.

When you return an object in a one-liner arrow function, you need to put () around it so the interpreter doesn't interpret the contents of {} as function code, but knows that it is an object, so: m => ({...m}).

Now we have a new array, and new objects in the array.

Local functions now have access to the value of the global members state, but the actual global state is immutable by them, because they have no reference to it. They cannot update the global state from the copy that they get. For that, they will need to call an update method.

IMPLEMENTING SETMEMBERS

The first method we will implement is a hydration method that allows a local function to pass in an array of members.

I’ll take out getMembers for now to make it easier to read:

const GlobalMemberStore = (() => {
  let _members = []
  return {
    setMembers: members => _members = members.map(m => ({...m}))
  }
})()

Here we use the Spread syntax to copy the members to a new array, and this becomes the global members.

This means that a local function cannot set the global state by passing in an array of members, and then mutate the global state by mutating one of the members that it passed in.

If we did a naive assignment:

setMembers: members => _members = [...members]

Then the local function calling this method would have a local reference to the member objects that are now in the state store. By spreading them, we make a copy — another object in memory that the local function has no reference to.

IMPLEMENTING UPDATEMEMBER

It is likely that a business requirement for this application is that you can update a member.

So, we will implement an updateMember function. We will use Array.map to return a new array. A naive approach to this might be “let's iterate over the array using forEach and mutate the element we are updating". See the post “Just Say No to Loops and Variables” for an in-depth explanation of why you don't want to do that.

To implement the predicate function, let’s describe what we want it to do in plain language:

For each member in the state,

if the member id equals the id of the update, return the update;

otherwise return the member.

So, our predicate function looks like this:

member => member.id === update.id ? update : member

We are using the ternary operator here to implement if-then-else in a single expression.

We can probably shorten the name we use for member to m, because the context is sufficient to provide information about what it is:

const GlobalMemberStore = (() => {
  let _members = []
 return {
    updateMember: update => (_members = _members.map(m => m.id === update.id? update : m))
  }
})()

We enclose the assignment operation _members = in parens () to indicate that we did not forget to return a value, and intended only the side-effect. We could have put it in {}, but that will cause code formatters to turn our single line into three.

DESIGNING FOR FAILURE

20% of programming is getting it to work. The other 80% is programming for when it doesn’t work.

What happens if a local function requests to update a member who is not in the state? At the moment, the local function receives no information from the call to updateMember, and if you look at the code, what will happen is… nothing.

The predicate function will never match, and the new state will be a new copy of the existing state, unmodified.

We could throw an exception. This gives us the opportunity to figure out where the bug in the application is that it is trying to update a member that doesn’t exist. This is a good idea.

Let’s throw an exception so that the root cause can be debugged in the local function. To do this, we will need a getMember function that we can use. So, let's implement that.

IMPLEMENTING GETMEMBER

It’s likely that local functions will want only a single member. If we don’t implement it here, we will have local functions retrieving the entire state and filtering it. This leaks complexity into the application, because we can do that in “one place, and one place only” in the application: here.

Then we only have to test it in one place, and we only ever have to get it to work in one place. That reduces the surface area for bugs in the application.

We can use Array.filter to find elements in an array. Array.filter returns a new array containing only the elements from the original array for whom the predicate function returned true.

The predicate function is straight forward:

Return true if the member.id equals the requested id;

otherwise, return false

Reducing that down, we get:

Return member.id equals requested id

or:

m => m.id === id

So,

const GlobalMemberStore = (() => {
  let _members = []
  return {
    getMember: id => _members.filter(m => m.id === id)
  }
})()

The getMember array will now return an array with either zero (if no member with that id exists in the state) or one… hang on, what happens if there is more than one member in the array with the same id? In that case it will return more than one member.

Probably, the business requirement is that member id is unique. So we will take that into account when we write the addMember function.

So it will return an array with 0 or 1 members in it. Probably local functions want a member or undefined.

Although, we can provide a better API if we return an object like this:

{
  found: true
  member: Member
} |
{
  found: false
  member: undefined
}

Then consumers of this API using TypeScript can use a Type Guard to get safety against accessing an undefined value, and our API forces them to use it.

This reduces bugs. Otherwise, we are relying on every local function in the application remembering to test it for undefined before accessing it - another surface area for bugs.

So:

const GlobalMemberStore = (() => {
  let _members = []
  return {
    getMember: id => {
      const member = _members.filter(m => m.id === id)
      return member.length === 1 ? 
        { found: true, member: {...member[0]}} :
        { found: false, member: undefined }
    }
  }
})()

Remember to Spread the member to return a copy (I picked this up when the test case failed here).

Nice API.

THROWING ON IMPOSSIBLE UPDATE

Another significant advantage of this approach is that we put all our business validation rules about the data in a single place: in the store. They are not spread throughout the application, and the responsibility of everyone and no-one. They can be put in one place, tested automatically, updated in one place, and if a local function violates them, we will find out immediately when it tries to store the data, through an exception.

We can now consume getMember from our own API to guard against an update error.

How can we do that? We need to lift our API to its own context inside the closure, like this:

const GlobalMemberStore = (() => {
  let _members = []
  const Store = {
  }
  return Store
})()

Now we have a private reference to our own API, as Store. So we can use it to see if the member that the local function wants to update, actually exists - and if not, throw.

const GlobalMemberStore = (() => {
  let _members = []
  const Store = {
    updateMember: update => {
      const member = Store.getMember(update.id)
      if (!member.found) {
        throw new Error(`No member with id ${update.id} in the store!`)
      }
      _members = _members.map(m => m.id === update.id? update : m)
    }
  }
  return Store
})()

IMPLEMENTING PUTMEMBER

Probably, a business requirement of the application will be to put a new member in the store.

We have to make a decision here about the behaviour of the store. What happens if a local function attempts to put a member with an id that is already in the store?

That’s probably a bug somewhere further upstream in the application logic, so we will throw an exception to allow debugging to start.

So we can do this:

const GlobalMemberStore = (() => {
  let _members = []
  const Store = {
    putMember: member => {
      if (Store.getMember(member.id).found) {
        throw new Error(`${member.id} already exists!`)
      }
      _members = [..._members, {...member}]
    },
    updateMember: update => {
      const u = needsMember(needsArg(u))
      const member = Store.getMember(u.id)
      if(!member.found) {
        throw new Error(`No member with id ${u.id} in the store!`)
      }
      _members = _members.map(m => m.id === u.id? update : m)
    }
  }
  return Store
})()

DEALING WITH A UNDEFINED ID

Another potential bug that we can detect here is a local function passing in either undefined or a member with an id that is undefined.

We can write helper functions for this, and call them on all operations where it is a requirement:

const GlobalMemberStore = (() => {
  let _members = []
  const needsArg = arg => {
    if (!member) {
      throw new  Error (`Undefined passed as argument to Store!`)
    }
    return arg
  }
  const needsId = member => {
    if (!member.id) {
      throw new Error (`Undefined id on member passed **as** argument to Store!`)
    }
  return member
  }
})()

Here is how we use this:

const GlobalMemberStore = (() => {
  let _members = []
  const Store = {
    putMember: member => {
      const m = needsId(needsArg(member))
      if (Store.getMember(m.id).found) {
        throw new  Error(`${m.id} already exists!`)
      }
      _members = [..._members, {...m}]
    }
  }
  return Store
})()

FREEZE!

For our final touch, we are going to freeze the API object using Object.freeze:

return Object.freeze(Store)

This prevents anyone from overwriting or modifying the API methods themselves.

If you wanted, you could (deep) freeze all the return values from the API methods. That would deny local function consumers of the objects the ability to mutate the return values. They would need to use spread on them. We’re not going to do that right now.

Freezing objects has an impact on performance. Freezing the API is not going to make a huge difference, so the safety is worth it. The objects returned from the API are copies, so freezing them is overkill, IMHO.

PUTTING IT ALL TOGETHER

Here is the whole thing:

const GlobalMemberStore = (() => {
  let _members = []

  const needsArg = arg => {
  if(!arg) {
    throw new Error (`Undefined passed as argument to Store!`)
  }
  return arg
  }
  const needsId = member => {
  i (!member.id) {
    throw new Error (`Undefined id on member passed as argument to Store!`)
    }
  return member
  }

const Store = {
    setMembers: members => (_members = members.map(m => ({...m}))),
    getMembers: () => _members.map(m => ({...m})),
    getMember: id => {
      const member = _members.filter(m => m.id === id)
      return member.length === 1 ? 
        { found: true, member: {...member[0]}} :
        { found: false, member: undefined }
    },
    putMember: member => {
      const m = needsId(needsArg(member))
      if (Store.getMember(m.id).found) {
        throw new Error(`${m.id} already exists!`)
      }
      _members = [..._members, {...m}]
    },
    updateMember: update => {
      const u = needsId(needsArg(update))
      if(!Store.getMember(u.id).found) {
        throw new  Error(`${u.id} does not exists!`)
      }
      _members = _members.map(m => m.id === u.id? update : m)
    }
  }
  return Object.freeze(Store)
})()

This may seem like way more complexity than:

var memArray = []

However, this is the actual complexity involved in this data structure in the application. You will end up doing all of this anyway — but it will be spread throughout your application in manipulation and mutation of that array, and if statements, and fixing bugs in various places.

And it will be really hard to refactor in the future.

With this approach, the total technical complexity of this concern is now encapsulated in one place in your application. It is testable through automated tests — as demonstrated in the accompanying repo. There are 125 lines of test code for 40 lines of code. So 165 lines of code to replace var memArray = [].

However, business validation of the data now has a place to live, and the entire expected usage of this array is now implemented such that local functions cannot introduce bugs related to it — only their local use of it.

winning

FURTHER RESOURCES

This approach to state management has become popular in JS in recent years, and is the basis of the approach used by:

React setState
Redux
Flux
Immutable.JS
Nanoflux (My personal favorite)

If you grasped the concepts and rational for the refactorings that I made in this example, you will be well-placed to understand these mature, more sophisticated (and generalised) implementations.

Just Say No to Loops and Variables

Josh Wulf — Sun, 23 Feb 2020 07:46:04 +0000

StackOverflow — Where developers learn, share, and build their careers

Recently, I spent some time on StackOverflow, helping people with their school assignments — I mean, serious programming questions they had at work. (I’m pretty sure a fair whack of them were homework assignments).

One thing that came out of it — for me — was a pattern in the issues in the JavaScript programming tag (Discord bots are hot right now with the kids).

There are certain things that people struggle with when learning to program, and when learning to program in JS.

Asynchronicity is one. Callbacks not so much — mostly now people are struggling with Promises (with are a Monadic wrapper around an asynchronous operation), and with the subtle context impedance mismatch between async functions and non-async functions. Without TypeScript informing them of the type mismatch, they are baffled by code that is in a monadic async context interacting with code that is not. I mean: they look the same. At least with callbacks and Promises you have some clue in the indentation.

Naming is another one. The power of correctly naming entities in a program cannot be overestimated — I mean, it’s one of the two hardest problems in computer science: caching, naming things, and whether or not to move to San Francisco.

The impact of not correctly naming things cannot be overestimated either. Confusing messes of spaghetti code where the programmer had wound themselves up in a ball of yarn and not only gotten trapped inside it, but forgotten what they were trying to do in the first place. In the end, I started to have some fun with it, telling one questioner that “80% of programming is correctly naming things, and the other 20% is choosing the font for your IDE.” JetBrains Mono. (I solved his problem for him.) He had started with an entity named x and was now three levels deep trying to figure out how to iterate the data structure. The Tao becomes clear when you know that for each recipe we have an array of ingredients, and each ingredient has a set of attributes that characterize it.

As we read in the Analects of Confucius (“Confucius say”):

Tsze-lu said, “The ruler of Wei has been waiting for you, in order with you to administer the government. What will you consider the first thing to be done?”

The Master replied, “What is necessary is to rectify names.” “So! indeed!” said Tsze-lu. “You are wide of the mark! Why must there be such rectification?”

The Master said, “How uncultivated you are, Yu! A superior man, in regard to what he does not know, shows a cautious reserve.

“If names be not correct, language is not in accordance with the truth of things. If language be not in accordance with the truth of things, affairs cannot be carried on to success.

Computer programming is an exercise in applied linguistics. It is precisely specifying the execution of operations to transform matter through the utterance of magic spells. Say the wrong thing, and BOOM! you turn into a toad. The stakes are high.

Mixing concerns is another common one. A symptom of the confusion that arises from this — and I really do mean confusion here: the dosha, or philosophical error described in the Sanskrit logical system of Nyaya as bhranti darshana: a mistaken perception, literally: “an illusory vision”. For example: thinking that a rope is a snake. That is confused: two things are fused with each other in a way that they are no longer distinct, and one is mistaken for the other.

In Sanskrit philosophy, there is an entire school — Samkhya — dedicated to the study of separation of concerns. Samkhya is sometimes translated as “categorisation” or “distinction”.

According to Wikipedia:

Samkhya means “to reckon, count, enumerate, calculate, deliberate, reason, reasoning by numeric enumeration, relating to number, rational.” In the context of ancient Indian philosophies, Samkhya refers to the philosophical school in Hinduism based on systematic enumeration and rational examination.

It comes from two words: Sam meaning “the whole” or “totality” (from which the English word sum comes to us), and khya meaning to name. The founders of this philosophical system were totally into enumerating everything categorically, and describing the relationships between categories as an access to understanding the whole.

In modern software development separation of concerns is a widely accepted best practice for reducing complexity and technical debt.

MIXED CONCERNS LEAD TO EXPONENTIAL COMPLEXITY

One thing I noticed many novice programmers struggling with was the mixing of the concerns of data transformation — essentially a functional concern — with imperative flow control.

Nothing wrong with that, but it lead them into situations where they experienced overwhelming complexity. They couldn’t get the data transformation that they wanted, and they were struggling with building a custom state machine to produce it at the same time. The intersection of these two problems lead them to throw up their hands and turn to StackOverflow.

As I said to one questioner: “when you solve a problem using loops and variables, now you have three problems”. You have to build a custom state machine, track mutable state, and you still have the original problem you were trying to solve.

Now, seasoned programmers can often look at these trivial (to them) situations and see a clear way out of the scenario the new programmer has gotten themselves into, and guide the novice to correctly construct the state machine, correctly initialise and scope the variables, and get the desired data transformation with the resulting system — and you might think “what’s the big deal?”

It can even be a mark of pride to be able to tame it in this way. However, the novice programmer’s dilemma is a microcosm that simply scales up when they continue to code this way with more prowess.

SO, THE HOT TAKE

I overstepped the mark in one question. This particular week, I was in the top 0.11% of StackOverflow contributors worldwide, as recognized by the community on StackOverflow (thank you, you’re welcome), and on a burn.

To one question, I said:

I can see two problems with your code straight away:

Loops
Variables

I wrote a solution that used no custom state machine (no loops), and no mutable variables.

Another StackOverflow contributor wrote a functioning custom state machine with mutation that also solved the problem domain, and I commented:

Also, loops and variables! The more of those you do, the more complexity and moving parts you have to debug. Solid effort though. Pretty cool using a RegEx.

To which he took affront — fair enough. He said:

@josh Wulf — There is no reason not to use loops and variables in code. You are ridiculous. You should know that using certain convenience functions of Array are slower performing than using for loops. Filter is actually a loop that iterates through all elements of an array, and it performs slower than for loops across the board. I recommend keeping your comments constructive and avoid passive agressive remarks like “Solid effort though.” Its actually a viable solution for many cases. I even pointed out that another solution on the thread might be better. Maybe write a blog where you critique?

Ouch!

I apologised to him, because I did overstep the mark with that comment. It’s fine to have a perspective on something, but leaving a member of our professional community, who is giving their free time to contribute to others, with the experience of being disrespected is not what I am committed to.

So, I apologised, and accepted his request to write a blog article about it. Thank you to that member of our community for holding me to account to the level of professional courtesy and respect that you are due, and for the opportunity to write this blog.

Here we go:

STATE MUTATION AND COMPLEXITY

Mutable state in a program is additional complexity.

More variables means more moving parts: mo’ vars mo’ problems. If an assignment is declared as mutable, guarantees about that assignment are weakened. This means that reasoning about the eventual value of that assignment in other parts of the program is complex. With TypeScript (on a strong setting), the transpiler will make a strong assertion about the type of the value, to reduce complexity, but it cannot make any guarantees about its eventual value. Without TypeScript, neither is guaranteed. (And at run-time, all bets are off, so you are at the mercy of the accuracy and consistency of your typings).

Deliberately reducing complexity by choosing to eschew the mutant is a programming discipline, and one that I believe pays off.

Douglas Crockford wrote the famous book JavaScript: The Good Parts, where he shared his discovery that if he deliberately avoided certain parts of the language — artificially constraining himself to a subset of the language — his productivity improved.

I believe that variables belong in the category of “things to avoid”.

I took on programming without variables, and there has only been case where the word let has left my mouth in the past two years:

**let** result
**try** {
    result = **await** asyncOpThatMayThrow()
} **catch** (e) {
**return** handle(e)
}

**try** {
**await** useResult(result)
} **catch** (e) {
**return** handleThis(e)
}

This is something that I have grappled with, because it is at the intersection of another programming discipline that I adopted: striving for a single level of indentation. Memories of grappling with deeply nested code bases, trying to figure out which level got unbalanced, and ending up with code that would again lint and run, but that I wasn’t sure still produced the same effects, lead me to that.

I recently resolved this, and that let is no more for me - but that is another blog post.

I see novice programmers declaring variables as let and var, sometimes interchangably in the same code, with no reassignment of their value in the scope. Why would you do that? These declarations communicate your intent to the machine and other programmers: “I intend that the value of this assignment change over the course of execution". When you don't change it, why communicate that intent? You have incorrectly named a thing.

And when you do mutate it, you make it necessary for the machine and more importantly, for other programmers to then trace the flow of execution through the code base to reason about its value in different places.

And when you make a coding error, and accidentally mistype a variable name (because you gave them non-descriptive, or similar names), you just created a case of mistaken identity mutation bug in the program. And no reasoner can detect your unintended mistake and warn you of it, because variables.

Just say No to variables. Try it for one year (I know that seems like a long time if it represents a significant percentage of your programming career to date).

If you are a new programmer struggling to get your data transformation to work, reduce the complexity — take out one of the variables: variables.

CUSTOM STATE MACHINES: LOOPS

Loops are problematic in several ways.

Oftentimes, armed with the loop and an array of data to transform, a novice programmer will frame the problem as: “I have to transform every element in this array”. So they make a loop, with side effects. If those side-effects are asynchronous, now they are dealing with three problems.

That’s an explosion of complexity, and leads to complex and fragile constructions that are resistant to refactoring. As the novice (or maintenance) engineer iterates on the data transformation taking place in the loop, the coupling of the state machine with the data transformation can cause the state machine to break, or to require a change in the state machine to accomodate a change in the data transformation. This is especially problematic for the novice programmer who is trying to get both to work at the same time. Now they are solving a two variable problem when they started with one!

Niklaus Wirth’s classic work on programming distinguished two categories: Algorithms and Data Structures. A third concern in a program is control flow — in distributed systems it is processes — directed evolution of the program (system) state over time.

By using a loop, you are putting all three in one place. Many novice programmers (and experienced ones working on new systems) are operating without a clear picture of the eventual shape of the data that they need to model the state and the transformations required to achieve the outcome. When you put all three in one place, you now have a three-variable equation that you are trying to solve at once.

And you are doing it by building the machine that will apply the transformation to the data structure, manually.

This, I believe, is at the core of the breakdown for many of the novice programmers who loop themselves into a knot with these assignments — I mean, work problems. They end up going: “What the heck am I even doing???”

The complexity is too much.

And what got missed, right at the outset, is that the problem is not “apply a transformation to every element in this array”.

That is the automatic GOTO (sorry, couldn’t resist) of the programmer armed with a loop.

The problem is in fact, much, much simpler. It is: “apply a transformation to each element in this array”.

Once this is grasped, the separation of concerns becomes clearer:

“I need to write a data transformation function that takes one element and returns one transformed element.”

“And I need to apply this transformer function to each element in the array.”

The problem has suddenly reduced in both scope and intersectional complexity.

The state machine and the transformation are now separate concerns, whose complexity can be reduced independently.

Loops are imperative control flow constructs. They can be used well, but they are dangerous. They couple control flow with transformations / side effects. I believe they cause more harm than good, especially for novice programmers, because they obscure the separation of concern.

Compare this:

**function**  **countNumbers** (arr) {
**var** count = 0;
**for** (num **in** arr) {
**if** (Number(arr[num]) !== NaN) {
            count++;
        }
    }
**return** count;
}

With this:

**const** isNum = n => !isNaN(parseInt(n));
**const** countNumbers = arr => arr.filter(isNum).length;

In the second, the two concerns are separated and named. They are not intermixed.

The data transformation function can be unit tested with ease, and can be refactored without impact on the state machine. The code for each lives in a distinct location and isolated context.

The problem is much clearer.

Once the state machine is in place (Array.filter), the design of the data transformation can be iterated on with an automated test suite with ease, leaving the programmer to focus on one thing only.

A mistake in syntax while doing that can only break one thing. The programmer is not grappling with and changing code that affects two concerns at the same time.

There is no mutation to track (another surface area for bugs).

This is a microcosm, but I believe one that perfectly expresses as a koan the power and beauty of taking a vow to deliberately avoid using variables and loops.

I will not overstep my bounds by making an evangelical claim of some absolute truth, and I invite you to try it. Program without variables and loops, and observe how it changes the way that the problems in front of you present themselves, and what solutions emerge from taking on this discipline.

Easily run multiple apps with HTTPS using Docker and LetsEncrypt

Josh Wulf — Mon, 17 Feb 2020 07:01:43 +0000

I frequently deploy Web APIs in Docker. On a single VM I might have up to five or six services running.

All of them have TLS certificates and are accessible via port 443 using SNI — Server Name Indication, where the request is routed to the correct backend based on the domain name of the request.

This is very easy to set up using letsencrypt-nginx-sidecar, (which builds on this project and this project).

This approach is super-quick to set up and use, and reliable — I’ve been using it for a couple of years now, with no issues. It uses an nginx reverse proxy that listens to a Docker network. When a container joins the Docker network, the nginx reverse proxy adds an entry to route requests to it, and a companion container contacts LetsEncrypt to automatically provision a certificate for TLS connections to that domain.

Setting it up is easy.

Set up

Git clone the repo:

git clone [https://github.com/jwulf/letsencrypt-nginx-sidecar.git](https://github.com/jwulf/letsencrypt-nginx-sidecar.git)

Now create a Docker network for your containers:

docker network create letsencrypt

Start the nginx proxy and LetsEncrypt companion:

cd sidecar && docker-compose up -d

That’s it! You can now deploy multiple webapps to this host using docker-compose and have them automatically proxied with TLS.

You have a proxy running that can route requests to your webapps based on the domain name of the request, and which will automatically get (and renew) a certificate for TLS as you add new webapps.

Adding a web app

Any time you want to add another web app to this network:

First set up the DNS record for the domain to point to the host machine’s external interface.
In the docker-compose.yml file for the webapp, add the following environment variables:

environment:
  # Set to the port that your webapp listens on
  VIRTUAL\_PORT=3000

  # Can be comma-separated list
  VIRTUAL\_HOST=mysubdomain.mydomain.com 

  # The domain for the cert, can also be comma-separated
  LETSENCRYPT\_HOST=mysubdomain.mydomain.com

  # Your email for the cert
  LETSENCRYPT\_EMAIL=me@gmail.com

Add a networks entry to have your webapp join the letsencrypt network:

networks:
  default:
    external:
      name: letsencrypt

Thats it!

Your webapp container will join the network, and the proxy and companion containers will do the rest.

Your webapp container does not need to map any ports in the docker-compose.yml file. The nginx proxy handles all external communication and routes requests over the Docker network.

You can see an example of a complete docker-compose.yml file in this repo.

Complex multi-repo builds with GitHub Actions and Camunda Cloud

Josh Wulf — Wed, 12 Feb 2020 14:06:55 +0000

BLUF (Bottom-line Up-front): GitHub Actions are AWESOME and will change your life, but you risk losing yourself in a microservices architecture of repos, or have to go monolith once you get a few dependent projects or cross service provider boundaries — unless you orchestrate. I show you how I did it in this article.

Get access to the Camunda Cloud Public Access Beta here. Use the Zeebe GitHub Action to orchestrate multi-repo builds with Zeebe and Camunda Cloud.

In this article:

Preamble
What are GitHub Actions?
GitHub Actions: The Good
GitHub Actions: The Bad
GitHub Actions: The Ugly
Custom CI with GitHub Actions and Camunda Cloud
Zeebe GitHub Action
Modelling the problem / solution
Starting a Camunda Cloud workflow from GitHub
Passing around an encrypted GitHub token
Setting up Camunda Cloud to access the GitHub API
Passing in a secret from Worker Variables
Using workflow variables as string constants
Triggering a GitHub workflow from Camunda Cloud
Problems accessing the GitHub API
Communicating a GitHub Workflow outcome to Camunda Cloud
GitHub Matrix builds
Multiple causes for an action
Specialising HTTP Worker payloads
Specialising GitHub Workflows
Ad Infinitum: To Infinity, and Beyond!
Live Stream

Preamble

Over the weekend, I did some work on an side-project, Magikcraft. It’s a platform for teaching kids to code in JavaScript, and it builds on an excellent Open Source project called ScriptCraft, written by Walter Higgins.

Since Microsoft bought Minecraft, the pace of development has increased, and there have been a lot of breaking changes to the APIs. Oracle have also deprecated the JavaScript engine — Nashorn — that we have been using since JDK 8, and introduced their new polyglot engine GraalVM.

Testing on and supporting all the various combinations of Minecraft server versions and JS runtimes has stretched the capacity of the ScriptCraft community, but there are many passionate users, including other downstream kids coding programs, like Code Makers, that use it.

It got to a breaking point with the latest Minecraft release, when a workaround patch we’d been relying on finally stopped working, and Walter let us all know that he was strapped for time and attention to give to the project. So I chipped in and built an automated building and testing infrastructure using GitHub Actions.

What are GitHub Actions?

“GitHub Actions” is the overall name given to a workflow automation capability on GitHub. I suppose it is not called “GitHub Workflows” because that namespace is already taken by the git-flow kinda stuff.

You have workflows, which you define in YAML files in a .github/workflows folder in your GitHub repo. These workflows contain named jobs, which are made up of 1 or more named steps. The steps can use GitHub Actions, which are reusable modules of functionality, providing capabilities like installing build chains and runtimes, executing Docker commands, and so forth.

GitHub Actions: The Good

GitHub Actions let you define workflows in YAML that run in response to actions like a git push. You can use one of the many existing actions, or build your own.

The workflows run on Azure — no surprise there, since Microsoft bought GitHub. What was a surprise to me, however, is how fast they run. I created a matrix build that builds and publishes nine — yes, 9 — Docker images to Docker Hub. The equivalent build on Docker Hub takes up to an hour. On GitHub, it is done in minutes. There are some serious compute resources available on there.

It is so much faster that I moved the Docker image builds from Docker Hub’s automated builds to GitHub Actions using the Publish Docker GitHub Action.

GitHub Actions: The Bad

You can’t validate or test your workflows without pushing them to GitHub. There is no local executor like there is for CircleCI, and while there is a validator plugin for VS Code, I couldn’t get it to give me “required parameter” feedback for the Zeebe GitHub Action that I created.

This is offset by how fast GitHub Actions run, so the code-execute-debug cycle is bearable. If it weren’t this fast, that would be a serious problem — but I actually enjoyed the power of so much compute resources dedicated to Open Source while debugging, so it kept me going.

YAML — meh, as one product owner told me over dinner at the recent DevConf.CZ: “It’s the least worst thing we’ve come up with so far”, paraphrasing Churchill’s take on democracy. Of course, I was like: “Have you heard of this thing called BPMN?“

Sidenote: when I started teaching kids to code, many were using this thing called Scratch, which is a block-based programming language. I turned my nose up at it, saying: “That’s not real programming”, and proceded to teach them the difference between ) and } and where to find them on the keyboard - like that’s real programming. My “Road to Damascus” moment was when an 8-year old kid came in and I coached him through making a combination lock guessing game, and he built a working prototype in thirty minutes with Scratch - because he was dealing purely with the logic of the program, and not struggling with syntax errors. The trickiest thing for him was state management, not balancing the right type of brackets.

GitHub Actions: The Ugly

Part of my contribution upstream to ScriptCraft has been integration with NPM, the JavaScript packaging ecosystem. The purpose of that is to unlock the innovation of the community to create, share, and discover others’ work — an essential dynamic in a vibrant Open Source community.

So, once I got automated builds, unit testing JavaScript in a dockerized Minecraft server in a GitHub action (seriously — your workflow can run for up to six hours — you could play Minecraft in a GitHub action), and publishing test images to Docker Hub, the obvious next question is:

how do I trigger a test run for downstream dependent packages when I publish a new image of the core API?

Yes, you can trigger GitHub actions in a downstream repository via webhook configuration (there is even a GitHub Action for it), but now you enter the rabbit hole of peer-to-peer choreography, with an attendant loss of visibility.

Is everything wired up? When you walk away from it for a week or two, or a month, and come back — how do you remember how it is all hooked up? Where do you go for the single point of truth for the system?

Even with three layers (base image, release package, image with release), and one dependent package, I’m having trouble remembering what triggers what and how.

“I push to this GitHub repo, that triggers Docker Hub, when that is done it should trigger a retest and rebuild of this other layer on GitHub, and that should then publish to Docker Hub, which should trigger all downstream package tests.” Pretty straight-forward, no? No.

Documentation is one attempt to address this — but is that really going to stay up to date?

Devolving responsibility is another way — package maintainers are responsible for triggering their own rebuilds / retesting. But again, there is a tight coupling to automate that, and it is opaque. And I already know that when I walk away from the small part that is already implemented, I will forget how it works and curse the idiot who wrote the documentation because it doesn’t make sense.

This is not a problem with GitHub Actions really, it is a characteristic of any complex system with multiple stakeholders, integrating disparate systems, with limited resources.

Custom CI with GitHub Actions and Camunda Cloud

It was right about then — 2am in the morning — that I remembered Ruslan’s recent article “Quick Serverless start to Camunda Cloud Public Beta”.

In the list of example use cases at the start he included “Custom CI/CD”.

Now, I always thought the “custom CI” example was contrived — I mean, isn’t that what CI systems literally do? Don’t they have this stuff built-in?

Spoiler: Yes, they do — but often using peer-to-peer choreography, and without a dependency graph — especially when you start to scale across a community or an ecosystem, or even just multiple service providers.

Zeebe GitHub Action

To solve this multi-repo build automation with Camunda Cloud, I wrote a Zeebe GitHub Action. It bundles the Zeebe Node client into a single file, and allows you to use it with declarative YAML in GitHub workflows to create workflow instances and publish messages back to Camunda Cloud from within GitHub Actions.

Modelling the problem / solution

OK, so let’s do this.

We going to create executable documentation of the system architecture that cannot go out of date, and crystallise it with the minimum number of moving parts to scale to more dependent packages.

We can use the GitHub API v3 — it’s an old school REST API left over from the days before GraphQL. It will give the whole thing a retro vibe, and we can call it directly from Camunda Cloud using the built-in HTTP Worker. I’m joking! We’re going to build a custom GraphQL client. No, we are not. REST is fine, and perfectly suited for this task. We are not requesting arbitrary related data over multiple calls, or introspecting an unknown schema, so GraphQL has no benefit here.

We can trigger a GitHub workflow over the API with a repository dispatch event.

First step, I model how I want the system to behave in the Zeebe Modeler — a desktop application for creating BPMN diagrams (apparently it is XML in the background, but seriously — it’s 2020, who needs to see that stuff?):

These are three dependent repositories that should trigger a rebuild / test in all downstream repos when they are updated.

The first one gets triggered in response to a push to a git repo, and has no other cause. This repo has a GitHub workflow that builds and pushes the base images to Docker Hub. When this GitHub workflow is complete, it communicates this back to Camunda Cloud by using the Zeebe GitHub Action to publish a message to Camunda Cloud.

Starting a Camunda Cloud workflow from GitHub

We need to correlate this message back to the running process instance, so we need a unique correlation id. We can get one by starting the Camunda Cloud workflow from GitHub on repo push using the Zeebe GitHub Action to create a workflow instance. When we start it, we’ll pass in a unique buildid, which we will use to correlate messages to this particular workflow run (see this article for a great tutorial on message correlation in Zeebe).

Here is the GitHub workflow that is triggered by a push to the repo, and starts a workflow instance in Camunda Cloud.

name: Kick off Build Orchestration

on:
  push:
    branches:
      - "master"

jobs:
  startWorkflow:
    runs-on: ubuntu-latest
    steps:
      - name: Get current time
        uses: gerred/actions/current-time@master
        id: current-time
      - name: Create Zeebe Workflow
        uses: jwulf/zeebe-action@master
        with:
          zeebe\_address: ${{ secrets.ZEEBE\_ADDRESS }}
          zeebe\_client\_id: ${{ secrets.ZEEBE\_CLIENT\_ID }}
          zeebe\_authorization\_server\_url: ${{ secrets.ZEEBE\_AUTHORIZATION\_SERVER\_URL }}
          zeebe\_client\_secret: ${{ secrets.ZEEBE\_CLIENT\_SECRET }}
          operation: createWorkflowInstance
          bpmn\_process\_id: magikcraft-github-build
          variables: '{"buildid": "${{ github.sha }}-${{ steps.current-time.outputs.time }}", "gitHubToken": "Bearer {{GitHubToken}}", "TYPE\_TEST": "TYPE\_TEST", "TYPE\_PUBLISH": "TYPE\_PUBLISH" }'

The ZEEBE secrets come from the Camunda Cloud console client configuration, and are added to the GitHub repo as secrets, so that the GitHub Actions have access to them in the secrets context.

Passing around an encrypted GitHub token

We also need to get our GitHub Token into the workflow. We could put it into the variables of the Camunda Cloud workflow when we create it from the GitHub workflow (it is injected there as ${{ secrets.GITHUB_TOKEN }}). However, this will put it into the Camunda Cloud workflow variables in plaintext - which we would like to avoid if we can.

Luckily, we can. The Camunda Cloud HTTP Worker can store encrypted secrets and replace them at run-time from a template. So we can create a personal token on GitHub (make sure it has the repo scope). Then, add a GitHubToken secret in the HTTP Worker “Worker Variables” dialog in the Camunda Cloud Console, and put the GitHub Token in there.

Setting up Camunda Cloud to access the GitHub API

Log into the Camunda Cloud console.
Go to your cluster, and click on “Worker Variables”. These are like secrets in CI systems — you can inject them by name, rather than putting secrets in your BPMN or code.

Click on “Add Variable”, and create a new variable called GitHubToken. Paste the token in there as the value.

Passing in a secret from Worker Variables

When we pass in our authorization header that will use the secret Worker Variable, it looks like this in the variables payload for a createWorkflowInstance or publishMessage call:

{
  GitHubToken: "Bearer {{GitHubToken}}"
}

The Bearer part is important, so don’t forget it. The {{GitHubToken}} part will be templated with the value from the Worker Variables when the HTTP Worker uses this variable. I should have called this variable GitHubAuthHeader to make it clear what it is, and how it is distinct from the Worker Variable.

So you can think of it like this:

{
  GitHubAuthHeader: "Bearer {{GitHubToken}}"
}

The workflow’s GitHubAuthHeader variable now contains an auth header, and the HTTP Worker will template in the value of the GitHubToken secret from the Worker Variables.

So the on push workflow in the base image repo does nothing more than create a new workflow instance in Camunda Cloud, with a unique buildid for this instance, and an authorization header for our GitHub API calls, containing the name of the encrypted secret to template in at runtime, and some string constants to allow us to specialise HTTP payloads via I/O mapping. What??

Using workflow variables as string constants

We create the Camunda Cloud workflow instance with some variables that will act as string constants for us in the workflow:

{
  TYPE\_TEST: "TYPE\_TEST", 
  TYPE\_PUBLISH: "TYPE\_PUBLISH" 
}

We don’t have the ability to specialise the payload of HTTP Worker calls with string constants in a BPMN model — but we can through variable mapping if we start the workflow with a known “enum” dictionary. Then we can specify a string constant by mapping one of these variables into the body.

There is always a way.

You can view this initial workflow on GitHub.

Triggering a GitHub workflow from Camunda Cloud

The first thing our Camunda Cloud workflow does is trigger another GitHub workflow in the base image repo, to rebuild and publish the base images to Docker Hub.

We trigger the workflow by posting a repository_dispatch event to the base image repo via the GitHub API, using the Camunda Cloud HTTP Worker.

The service task that does this has the task type CAMUNDA-HTTP.

It has two custom headers that are semantic for this worker: method and url.

We set method to post for an HTTP POST request, and set the url to https://api.github.com/repos/Magikcraft/minecraft/dispatches for the repository dispatch event endpoint for this repo.

Two of the workflow variables are also semantic for the HTTP worker: body and authorization.

The value of body will be JSON parsed and sent as the body of the HTTP request, and the value of authorization will be sent as the authorization header.

So we create I/O mappings in the Zeebe modeler to map the variable gitHubToken (which contains our templated auth header string with GitHub token) to authorization, and the buildid (our unique correlation id) to body.client_payload.

The body.client_payload key can contain arbitrary data that will be available to the GitHub Actions that respond to this event. With access to the correlation key buildid, they can publish messages back to Camunda Cloud that are correlated to this running workflow instance.

The body.event_type key is required by the GitHub API, and here we can map one of our enum variables in: TYPE_PUBLISH. This repo does not have specialised behaviours that we need to orchestrate, so we don’t bother testing the event_type in the triggered workflow.

Problems accessing the GitHub API

If GitHub is responding with 404, if the URL is correct (double-check it!) — it means that you were not authenticated.

Typically, we send a 404 error when your client isn’t properly authenticated. You might expect to see a 403 Forbidden in these cases. However, since we don’t want to provide any information about private repositories, the API returns a 404 error instead.

from here

You can debug this by pointing the HTTP Worker at https://webhook.site/ to see the exact payload it is sending to the GitHub API.

Communicating a GitHub Workflow outcome to Camunda Cloud

We can publish a message to Camunda Cloud from the GitHub workflow after we push the base images to Docker Hub.

Here is the section of the base image repo’s publish workflow that communicates success back to Camunda Cloud:

notify\_Camunda\_Cloud:
    runs-on: ubuntu-latest
    needs: build\_and\_publish
    steps:
      - name: Tell Camunda Cloud What's up!
        uses: jwulf/zeebe-action@0.2.0
        with:
          zeebe\_address: ${{ secrets.ZEEBE\_ADDRESS }}
          zeebe\_client\_id: ${{ secrets.ZEEBE\_CLIENT\_ID }}
          zeebe\_authorization\_server\_url: ${{ secrets.ZEEBE\_AUTHORIZATION\_SERVER\_URL }}
          zeebe\_client\_secret: ${{ secrets.ZEEBE\_CLIENT\_SECRET }}
          operation: publishMessage
          message\_name: BASE\_IMAGE\_REBUILT
          correlationKey: ${{ github.event.client\_payload.buildid }}
          variables: '{"test\_passed": "false"}'

(View it on GitHub).

However, execution halts in the GitHub workflow if the task fails. One way to deal with failure is to wrap every step in a GitHub workflow and test for failure, but that is clunky.

Instead, we will use a boundary interrupting timer to detect failure by timing it out. We could publish a message from somewhere else by listening to the repo webhooks, but we want to do this with zero additional infrastructure.

Our timeout task could trigger any notification behaviour, but let’s keep it zero-infra and raise an incident in Operate. We can do this by creating a CAMUNDA-HTTP task with an I/O mapping that we know does not exist.

In the I/O mapping we map build_timed_out_set_true_to_retry to dummy. When the token enters this task, it will raise an incident because the variable build_timed_out_set_true_to_retry doesn’t exist. When you fix the issue with the build, you can set this variable to true in Operate and retry.

The I/O mapping for the task maps the build_timed_out_set_true_to_retry variable to dummy on output, so we can reuse this pattern elsewhere - the variable is still unset in the workflow.

This CAMUNDA-HTTP task, when it does run, just does a GET to Google. If that fails, then you have bigger problems.

GitHub Matrix builds

Matrix builds in GitHub workflows allow you to build or test multiple configurations with the same steps. By default the workflow halts on the first failure, but you can specify fail-fast: false to have the build execute all combinations before announcing failure.

name: Publish Docker images

on: [repository\_dispatch]

jobs:
  build\_and\_publish:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        baseimage:
          - { image: "oracle/graalvm-ce:19.3.1-java8", tag: graalvm }
          - { image: "openjdk:8u171-jdk-alpine3.8", tag: openjdk8 }
        minecraft:
          - { jar: paper-1.15.2, dockerfile: Dockerfile }
          - { jar: paper-1.14.4, dockerfile: Dockerfile }
          - { jar: nukkit-1.0, dockerfile: Dockerfile-nukkit }
          - { jar: nukkit-2.0, dockerfile: Dockerfile-nukkit }
    steps:
      - uses: actions/checkout@v2
      - name: Publish Docker image to Registry
        uses: elgohr/Publish-Docker-Github-Action@2.12
        env:
          BASEIMAGE: ${{ matrix.baseimage.image }}
          MINECRAFT\_JAR: ${{ matrix.minecraft.jar }}.jar
        with:
          name: magikcraft/minecraft
          username: ${{ secrets.DOCKER\_USERNAME }}
          password: ${{ secrets.DOCKER\_PASSWORD }}
          dockerfile: Dockerfile
          tags: ${{ matrix.minecraft.jar }}-${{ matrix.baseimage.tag }}

  notify\_Camunda\_Cloud:
    runs-on: ubuntu-latest
    needs: build\_and\_publish
    steps:
      - name: Tell Camunda Cloud What's up!
        uses: jwulf/zeebe-action@0.2.0
        with:
          zeebe\_address: ${{ secrets.ZEEBE\_ADDRESS }}
          zeebe\_client\_id: ${{ secrets.ZEEBE\_CLIENT\_ID }}
          zeebe\_authorization\_server\_url: ${{ secrets.ZEEBE\_AUTHORIZATION\_SERVER\_URL }}
          zeebe\_client\_secret: ${{ secrets.ZEEBE\_CLIENT\_SECRET }}
          operation: publishMessage
          message\_name: BASE\_IMAGE\_REBUILT
          correlationKey: ${{ github.event.client\_payload.buildid }}

We make the second step (_notify_Camunda_Cloud_) — where we publish the success message to Camunda Cloud, depend on the first step (_build_and_publish_) — the one where we publish the Docker images.

We do this by specifying needs: build_and_publish in the second step. This prevents GitHub from parallelizing the tasks and optimistically reporting success back to our workflow in Camunda Cloud.

This ends the first repository’s automation. If this succeeds, then our dependent repository needs to retest, rebuild and publish Docker images, then trigger further dependents.

Multiple causes for an action

Our second repository can be rebuilt in response to two different events:

A git push to its own code.
A rebuild of the base images.

We model this by creating a message start event, and then in the on push workflow in the repository, we publish this message with the metadata that our workflow needs — the templated authorization header to get our GitHub token injected into the HTTP Worker, a unique buildid, and the string constants.

name: Test

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Get current time
        uses: gerred/actions/current-time@master
        id: current-time
      - name: Kick off workflow
        uses: jwulf/zeebe-action@0.2.0
        with:
          zeebe\_address: ${{ secrets.ZEEBE\_ADDRESS }}
          zeebe\_client\_id: ${{ secrets.ZEEBE\_CLIENT\_ID }}
          zeebe\_authorization\_server\_url: ${{ secrets.ZEEBE\_AUTHORIZATION\_SERVER\_URL }}
          zeebe\_client\_secret: ${{ secrets.ZEEBE\_CLIENT\_SECRET }}
          operation: publishMessage
          message\_name: MAGIKCRAFT\_API\_PUSH
          variables: '{"buildid": "${{ github.sha }}-${{ steps.current-time.outputs.time }}", "gitHubToken": "Bearer {{GitHubToken}}", , "TYPE\_TEST": "TYPE\_TEST", "TYPE\_PUBLISH": "TYPE\_PUBLISH"}'

From that point forward, the workflow in Camunda Cloud is identical, regardless of how we got here.

We use the same timeout pattern to detect failure, and run a matrix test in the next workflow we trigger from the Cloud.

Specialising HTTP Worker payloads

In this subprocess we have two distinct GitHub workflows in the same repo that we are triggering from Camunda Cloud using the repository dispatch event.

Our tests may fail on the new base images, or due to the new code we just pushed, so we don’t want to publish test containers if it does. So we have a test workflow and a publish workflow.

To get specialised behaviour in response to the repository dispatch event, we need to specialise the HTTP Worker task that calls the GitHub API. We do this by mapping one of the string constants from our enum dictionary in the variables to body.event_type in the I/O Mappings for the task:

<bpmn:serviceTask id="Task\_0nvbnfg" name="Trigger matrix tests with base images">
  <bpmn:extensionElements>
    <zeebe:taskDefinition type="CAMUNDA-HTTP" />
    <zeebe:ioMapping>
      <zeebe:input source="gitHubToken" target="authorization" />
      <zeebe:input source="buildid" target="body.client\_payload.buildid" />
      <zeebe:input source="TYPE\_TEST" target="body.event\_type" />
      <zeebe:input source="test\_passed" target="body.client\_payload.test\_passed" />
    </zeebe:ioMapping>
    <zeebe:taskHeaders>
      <zeebe:header key="method" value="post" />
      <zeebe:header key="url" value="https://api.github.com/repos/Magikcraft/MagikCraft/dispatches" />
    </zeebe:taskHeaders>
  </bpmn:extensionElements>
  <bpmn:incoming>SequenceFlow\_0weo0p1</bpmn:incoming>
  <bpmn:outgoing>SequenceFlow\_0a48581</bpmn:outgoing>
</bpmn:serviceTask>

Specialising GitHub Workflows

We have two GitHub workflows that are triggered from the repository dispatch event. We can specialise the event_type that we send with the event from Camunda Cloud with the HTTP Worker, now we need to specialise the GitHub workflows.

We can do this with a conditional:

jobs:
  test:
    if: github.event.action == 'TYPE\_TEST'
    runs-on: ubuntu-latest

The event_type from the API call appears in the GitHub workflow as github.event.action.

You can debug repository_dispatch events sent to a repository by creating a workflow like this:

name: Debug

on:
  - repository_dispatch
jobs:
  echo:
    runs-on: ubuntu-latest
    steps:
      - name: Debug
        run: echo "${{ toJson(github.event) }}"

View the two workflows on GitHub: test and release-test.

Ad Infinitum: To Infinity, and Beyond!

The third subprocess is essentially the same thing repeated. To run unit tests inside Docker inside a GitHub Action, I took inspiration from this article: Github actions running javascript unit tests on docker. The Magikcraft and plugin unit tests actually run in JavaScript inside Minecraft inside Docker inside a GitHub Action — now orchestrated by Camunda Cloud.

It’s possible to extend this system further in either direction — for example, the base images are actually built in response to upstream dependencies updating, principally Minecraft server versions. That could be automated and added as a trigger to the whole process.

One thing that became apparent to me is the need to use a standard pattern, and as thin a shim as possible at each layer.

I discovered the idea of passing in string constants to specialise HTTP Worker payloads when I found myself replying to Camunda Cloud with a message that encoded knowledge in the GitHub workflow of where it was in the BPMN process flow — a clear symptom of tight coupling.

The Zeebe Action came about in order to lift a repeated pattern to a first-class concern — I started by creating a light-weight zbctl container and scripting it via BASH in GitHub Actions.

There is more lifting that can be done, and I am sure that as patterns emerge there will be less manual wiring of I/O mappings that will be required to accomplish this integration.

In the meantime, you can accomplish quite a lot with zero-infrastructure.

An important feature of designing systems like this is in their comprehensibility and responsiveness to change — their agility. It is not just “how easy is it to do?” it is “how easy is it to extend and modify the behaviour of the resulting system?“

For that, right now, I don’t have an answer. My intuition is that now that the basic components are in place and some patterns have emerged, it will be a flexible system.

Live Stream

Finally, here is a live stream of me working on this.

What is Zeebe?

Beginners Guide to Live Streaming Code on Twitch in 2020

Josh Wulf — Wed, 05 Feb 2020 16:19:42 +0000

I started live streaming code on Twitch last year, and at Linux Conf AU in January I gave a talk about live streaming “How to live stream on Twitch and YouTube” that was live streamed on my Twitch.tv channel.

In this talk, I cover a lot of the mechanics and technology of streaming: the software (OBS / Ecamm Live), streaming platforms, microphones, control surfaces, lighting, and using a green screen. I show both ends of the spectrum: the ultra-high end and the “get started now in your garage”.

Check out the archives of some of my live coding sessions in the archive in this YouTube playlist.

If you stream, let me know. I love checking out other people’s streams.Suz Hinton (noopkat) and MPJ inspired me to start streaming, and if this helps someone else get started, then I’d consider that paying it forward.

Microservices Operational Monitoring: Zeebe Cloud Canary

Josh Wulf — Sun, 03 Nov 2019 12:04:23 +0000

This post originally appeared on the Zeebe blog.

Designing a resilient microservices system means planning for, and alerting on various failure states. The Zeebe Cloud Canary npm package adds alerting to your Node.js Zeebe applications.

There are a few things that can go wrong in a Zeebe system that you definitely want to surface operationally. Your client applications might exception and halt. The broker might fail - whether due to a hardware failure or some edge-case condition that puts it in an infinite restart loop while recovering (it could be memory constrained, for example, and rescheduled by K8s before it can recover its state on boot up).

Both of these cases can be detected by probes. The broker has a readiness probe that can be monitored for this, and your application can have a periodic healthcheck using something like healthchecks.io.

Another case that is more subtle: when the broker is running, and your application is too - but does not have a connection to the broker. Maybe something has failed in the network. With software-defined networking, it no longer takes someone removing the cap at the end of a 10-base-T network, or unplugging a workstation in the middle of a Token Ring network to disrupt a connection.

In a development environment, for example, if you are forwarding ports to a broker in a Kubernetes cluster (maybe using bulk kubefwd), the forwarding may stop.

In this case, unless you are watching the logs, you may not notice that your application has lost its connection. It just looks like there is no work at the moment.

The Node.js client does transparent client-side retries by default, and if you don't write solid handling on the onDisconnect() handler, it will just keep trying to reconnect, and your application will report that it is alive.

Cloud Canary

I've written an npm package zeebe-cloud-canary, that deploys a canary workflow that chirps periodically. The canary worker pings a "chirp" endpoint whenever it gets the chirp task, and if it misses a chirp task by 50% of the heartbeat period, it can optionally ping a "squawk" endpoint.

If you are using healthchecks.io, then you don't need a squawk endpoint, because healthchecks.io can be configured to alert you after a missing ping.

In the initial implementation of this, I created a single, long-running workflow instance for the canary. This is problematic, because the workflow events are not reaped until the workflow completes. This causes disk space usage to increase over time, and broker recovery takes longer when a node is restarted (which can lead to those rebooting loops).

The new implementation starts a new workflow instance for each chirp, from the canary worker that chirps. Message correlation is used to make sure that you get only a single chirp, and not a chorus of tweets.

You can use this across multiple workers to prove that you have at least worker / application connected for the class.

Installing

To install the package to your application:

npm i zeebe-cloud-canary

Usage

Then in your application code, create a canary:

import { ZeebeCanary } from 'zeebe-cloud-canary';

// Uses the zeebe-node zero-conf constructor, either localhost or from ENV
const canary = new ZeebeCanary({
    ChirpUrl: `${healthchecks_io_url}`,
    CanaryId: 'some-canary-id',
    HeartbeatPeriodSeconds: 300
})

See the README for more configuration options, and take a look at the canary source code (it's just 108 lines).

The canary uses micromustache to template the CanaryId into the bpmn before deploying it (code here, bpmn example here), allowing you to namespace the canary by application, worker, application instance, worker instance, or any other resolution that makes sense.

At the moment, I'm using it per-application instance. My applications have multiple workers in them, so my hypothesis here is that as long as the application instance canary has connectivity, all the workers in the application have a connection.

To prevent race conditions, (for example, you name-space by application and spin up multiple instances at different times), when a worker services the chirp, it publishes a message to cancel any other instances of its name-spaced canary workflow, before starting another one.

Here is the bpm diagram:

Summary

This is one idea for operational monitoring, using message correlation, and templating of a generic workflow. There are many ways that you can do it, and exactly what you monitor and how you do it depends on your tech stack and your potential failure modes.

Note that you can't open the workflow in the Zeebe modeller - the process id with the template string in it doesn't validate. To create it, I edited the bpmn file in Visual Code after creating it in the modeller.

Zeebe Message Correlation

Josh Wulf — Thu, 15 Aug 2019 17:42:57 +0000

Zeebe is a new workflow engine for microservices orchestration. It has first-class support for clients written in JavaScript via zeebe-node.

This article was originally published on the Zeebe blog.

Message correlation is a powerful feature in Zeebe. It allows you to target a running workflow with a state update from an external system asynchronously.

This tutorial uses the JavaScript client, but it serves to illustrate message correlation concepts that are applicable to all language clients.

We will use Simple Monitor to inspect the running workflow state. Simple Monitor is a community-supported tool, and is not designed to be used in production - however, it useful during development.

Workflow

Here is the basic example from the Zeebe documentation:

Use the Zeebe Modeler to open the test-messaging file in this project.

Click on the intermediate message catch event to see how it is configured:

A crucial piece here is the Subscription Correlation Key. In a running instance of this workflow, an incoming "Money Collected" message will have a correlationKey property:

  zbc.publishMessage({
    correlationKey: "345",
    name: "Money Collected",
    variables: {
      paymentStatus: "paid"
    });

The concrete value of the message correlationKey is matched against running workflow instances, by comparing the supplied value against the orderId variable of running instances subscribed to this message. This is the relationship established by setting the correlationKey to orderId in the message catch event in the BPMN.

## Running the demonstration

Clone this repository.
Install dependencies:

 npm i && npm i -g ts-node typescript

In another terminal start the Zeebe Broker using the simple-monitor profile from the zeebe-docker-compose repo.
Deploy the workflow and start an instance:

 ts-node start-workflow.ts

This starts a workflow instance with the orderId set to 345:

await zbc.createWorkflowInstance("test-messaging", {
      orderId: "345",
      customerId: "110110",
      paymentStatus: "unpaid"
    })

Now open Simple Monitor at http://localhost:8082
Click on the workflow instance. You will see the current state of the workflow:

The numbers above the BPMN symbols indicate that no tokens are waiting at the start event, and one has passed through; and one token is waiting at the "Collect Money" task, and none have passed through.

Take a look at the "Variables" tab at the bottom of the screen. (If you don't see it you are probably looking at the workflow, rather than the instance. In that case, drill down into the instance):

You can see that this workflow instance has the variable orderId set to the value 345.

Now start the workers:

ts-node workers.ts

Refresh Simple Monitor to see the current state of the workflow:

Now the token is at the message catch event, waiting for a message to be correlated.

Take a look at the "Message Subscriptions" tab:

You can see that the broker has opened a message subscription for this workflow instance with the concrete value of the orderId 345. This was created when the token entered the message catch event.

Now send the message, in another terminal:

ts-node send-message.ts

Refresh Simple Monitor, and you see that the message has been correlated and the workflow has run to completion:

The "Message Subscriptions" tab now reports that the message was correlated:

Message Buffering

Messages are buffered on the broker, so your external systems can emit messages before your process arrives at the catch event. The amount of time that a message is buffered is configured when publishing the message from the client library.

For example, to send a message that is buffered for 10 minutes with the JavaScript client:

  zbc.publishMessage({
    correlationKey: "345",
    name: "Money Collected",
    variables: {
      paymentStatus: "paid"
    },
    timeToLive: 600000
  });

Here is how you can see it in action:

Keep the workers running.
Publish the message:

ts-node send-message.ts

Click on "Messages" at the top of the Simple Monitor page. You will see the message buffered on the broker:

Now start another instance of the workflow:

ts-node start-workflow.ts

You will see that the message is correlated to the workflow instance, even though it arrived before the workflow instance was started.

Common Mistakes

A couple of common gotchas:

The correlationKey in the BPMN message definition is the name of the workflow variable to match against. The correlationKey in the message is the concrete value to match against that variable in the workflow instance. Arguably, it might be more appropriately named correlationValue to make this distinction clearer. There is a GitHub issue around that - feel free to add your feedback.
- An important thing to know is that the message subscription is not updated after it is opened. That is not an issue in the case of a message catch event, however for boundary message events (both interrupting and non-interrupting) the subscription is opened as soon as the token enters the bounding subprocess. If any service task modifies the orderId value inside the subprocess, the subscription will not be updated.

For example, the interrupting boundary message event in the following example will not be correlated on the updated value, because the subscription is opened when the token enters the subprocess, using the value at that time:

If you need a boundary message event correlated on a value that is modified somewhere in your process, then put the boundary message event in a subprocess after the task that sets the variable. The message subscription for the boundary message event will be opened when the token enters the subprocess, with the current variable value.

Summary

Message Correlation is a powerful feature in Zeebe. Knowing how messages are correlated, and how and when the message subscription is created is important to design systems that perform as expected.

And Simple Monitor is a useful tool for inspecting the behavior of a Zeebe system to figure out what is happening during development.

How to select an Open Source Project to contribute to

Josh Wulf — Fri, 10 May 2019 12:48:07 +0000

I'm going to ask two questions. Just two questions.

It's a koan.

Meditating on these questions will reveal everything to you.

The answer to these questions is not the answer to which project to contribute to, but meditating on these questions will reveal the project to you when you encounter it.

Here are the questions:

1. How many commits does it take to become the #2 committer in the world on Immutable.js?
2. If you did that, where would you end up working?

About me: I spent three years as a recruiter analyzing GitHub to find developers. Also, I'm a professional software developer, and I've worked in open source for over a decade. Read about how I got my last job by committing to open source.

How to Get a Job Through Doing Open Source

Josh Wulf — Fri, 10 May 2019 12:09:41 +0000

In 2017, as a recruiter, I wrote an article about getting hired from GitHub contributions. Hacker News hated on it so hard that I quit my job as a recruiter, got a job as a developer, spent 14 months contributing on GitHub, and got myself hired from it to work on Zeebe.io. Turns out it does work!

My 16-year-old son, Prahlad, just walked into our apartment.

"What did he say???" I ask.

"He said 'Yes'."

Understated, playing it cool, like many teenagers do with their parents. But I know he's deeply excited, and probably a little bit scared.

He just got a gig working at the table-top and role-playing store in the building next to our apartment block in Brisbane, Australia.

I coached him how to get it. I saw that he was enthusiastic about the games they sell there. He plays Magic - The Gathering (MTG) there with his friends, and a couple of weeks ago I came home to twenty teenagers (yes - 20!) in our apartment playing the game. He went to the recent MTG pre-release and played it all day.

"Why don't you ask Sean if you can do some part-time work in the store?" I suggested. Sean is the owner of the store.

Prahlad asked Sean, and Sean said that he doesn't have any work that needs to be done there.

"Go back and tell him that you want to get work experience," I told him. "Tell him you'll do it for free. That way you can learn from him. He is running a successful business, and you can learn that. You can learn how to DM, how to run events, and how to do customer service. You might find ways to expand the business. And if it doesn't turn into a paying gig, when the time comes you can get a job at another store based on your experience."

So he had that conversation, and he starts tomorrow.

The Impact GitHub is Having on Your Software Career, Right Now...

In 2017, I wrote my (so-far) most popular article of all time: The Impact GitHub is Having on Your Software Career, Right Now….

In that article I cast the vision for how you can develop your career through open-source contribution.

It clearly struck a nerve - it got 382 points and 237 comments on Hacker News.

Many of the comments disagreed with my main premise, but I felt that they had missed the point.

They said that:

My experience at Red Hat was atypical. There are no other opportunities like that.
I didn't know what I was talking about because I was a recruiter (they must have skipped my ten years in engineering at Red Hat part)
It was unrealistic advice for developers with a day job in companies working in BitBucket.
It wasn't possible for devs who work in companies in the financial sector with high regulation and security.
It wasn't possible if you weren't willing to sacrifice your life outside work.
It just wasn't possible.

There is nothing that I love more than a challenge, so I went "deep cover".

I quit my job as a recruiter, and got a job as a software engineer in a pure closed-source company that uses BitBucket and has PCI-compliant security.

Fourteen months later, I got hired by Camunda to work as the Developer Advocate for Zeebe - a workflow engine for orchestrating microservices, purely based on my open-source contributions while working at that job. I just did everything that I advised readers to do in the comments of my original Medium article.

While I was doing that, I didn't sacrifice my hobbies or my family, in fact:

I kept building my side-hustle Magikcraft.io
I did a launch event for Minecraft for Type 1 Diabetes that was filmed for Mojang's YouTube channel.
I trained for and competed in three physique competitions with my wife - it was on her bucket list to win a bikini-modelling competition. She won her division, and I eventually placed 2nd in a State-level competition.
I trained to lead a personal development course.
I did a three-month long course with my son.

I don't say this to brag - just to show that it's not a zero-sum game, as some people paint it: "Oh that's easy to say, but it privileges people who are willing to work for free and sacrifice their time with their family."

False.

Of course, for the person who starts with the conclusion that "it's not possible", no amount of evidence is ever going to be enough - and to those I say: "What do you want to see now?"

Not because it will change their mind, but because it's a fertile source of inspiration for me.

Misconceptions and Strawman Arguments

Before I sketch out how to do it, I want to explicitly state a couple of objections that just miss the point:

You can't tell anything about a developer from their contribution graph on GitHub.
I've hired lots of developers and I've never looked at their GitHub profile.

Let me just say this: you aren't going to get hired through your open-source contributions by these people. That's all those things say. There is more I could say about that, but I'm not going to address them any further.

I also want to make it clear: I'm not saying that you have to do this. Just that you can.

How I did it

As I pointed out to commenters on the original Medium article, in your day job you either use open-source libraries, or may be able to introduce some to your stack on new projects.

If you don't have a day job, then you can introduce them to your stack easily.

While doing research for a new project at my day job, I tried a number of different projects - including a TypeScript gRPC server. I found a bug in it, and opened an issue. Then I wrote a patch for it, that got merged in.

We didn't use that project, but it was just part of being a member of a global community - like picking up a piece of paper in the hallway of our apartment building or on the street.

We eventually did use a gRPC library, and we needed it to support gRPC streams. So I wrote a patch for that, and it got merged. That contribution was enough to get me a mention in the contributors on npm.

I opened issues and submitted patches to Netflix Conductor (example). Nothing earth-shattering, just contributing to the environment I live in.

One of the things I explained to management was that a key decision-making factor in which technologies to adopt is how fast issues get addressed, and if we patch something for our own production use-case, how amenable the maintainers are to accepting our pull requests to the mainstream.

No-one involved in hiring me at Camunda looked at these (callback to the peeps who say they never look at people's contributions on GitHub - X-Factor judges don't watch you practicing in front of the mirror either - just sayin').

I will say, however, that smart recruiters at Google and Facebook would hit me up based on watching my activity. It's not enough to land a job, but it does have people come looking for you - especially as you build up a history of it.

The Big Break

The big break came when we landed on using Zeebe as the orchestration engine for our new microservices project.

I was over the moon, because Zeebe had an officially-supported Go client, and this was my chance to get our team coding in Go. I'd done some POCs and side-projects in it, but we coded in JavaScript.

No-one else on the team of six was keen to make the move, however, so we needed a JavaScript client.

I managed to get alignment on using TypeScript for the new project, and so I created a TypeScript client library.

I had logged some issues and contributed some small patches to Zeebe, as part of the civic duty / evaluation. Now, I pitched to management making the client library open-source.

My argument for it had two parts:

It means that our library has the opportunity to become the most widely-used one, which means it gets more eyeballs on it, and the possibility of patches from the wider community. Meaning that we don't end up maintaining something internally and finding out later on that there is a more widely-used/supported one that we have to rebase on.
It is a good developer-marketing tool to raise our profile, build our engineering brand, and differentiate us in the market when hiring.

I got alignment. However, the company had no GitHub presence. I got that by explaining that we needed a GitHub organisation in order to collaborate with the engineers at Camunda.

This is a company with no prior experience in Open Source. I had the advantage that I spent ten years working in it, so I knew both that it works, and how it works. Looking back at it, I probably wouldn't have stood for it the way I did if I had an ounce of doubt - if I "listened to the haters" in the comments.

Once that was in place, we pushed the library live, and published it to NPM.

One of the things that can you develop over time participating in open-source as a civic member, is a sixth sense of opportunities. I could see that this was an exciting technology that was poised to land on a rising wave - and we were about to ride that wave too. And there was no JS client.

It took over a year for the factors to align, but if I were sitting at my desk committing to BitBucket (yes, I was committing code to private BitBucket repos too) only, then I wouldn't see an opportunity like that in ten years.

I wrote an article on Medium.com announcing the library. I actually got in trouble for that. I jumped the gun and didn't get alignment for it before publishing. Lesson learned.

However, the library, the article, and my participation in issues, patches, and the Slack channel got me noticed by the folks at Camunda.

Enough so, that when I was looking for another role, they acted fast to hire their first employee in Australia - faster than local companies moved, even with their time zone and established infrastructure advantage.

On a call with Berd Rücker - one of Camunda's cofounders - I shared why I thought it was a good match: at Red Hat we would often hire or acqui-hire from the community (*acqui-hire means you bring an entire project in, people and tech - it's a portmanteau of "acquire" and "hire").

We would "find someone who was already doing it, and just pay them to do it for us". Normally when you hire, you find someone, pay them, and hope that they will enjoy what they are doing and be good at it. Hiring open-source contributors reduces the risk. They still might turn out to be terrible to work with, but you have a sense already of how you work together, and you know what their work is like.

It's a universal principle

So it's the same thing for my son. He has the opportunity to demonstrate value, build trust, and gain skills at the gaming store. And when the time is right, he will start getting paid - either there, or elsewhere.

At the careers fair for Red Hat in 2004, I brought along a 20-page paper I had written where I predicted the future of tech: open-source eating the world; devices shrinking to hand-helds; emergent network effects. I wrote it on breaks while working on an ISP help desk, between installing Linux on old computers I bought for $20 a pop.

In a recent Slack discussion, someone said: "The problem with Open Source contribution is that it privileges those who have time to do it"

Firstly, that's the opportunity of Open Source, not the problem. Secondly, everyone has the time - and I would say civic duty if you use it - to contribute something.

And lastly: yeah, that's right. Just like people who have the time to play clubs and pubs every Friday night, and practice their chops in their bedroom are privileged to be able to be discovered by an A&R talent scout and get a record deal.

The Beatles played for two years straight in Hamburg to hone their skills. As a professional software developer, if you are going to hone your craft anyway, you might as well do it in a way that contributes to a greater civic good and your own portable, public reputation.

Or not. Whatever.

I'm just saying that it is possible.