Forem: Mattermost

The Platformer #32: The Gift Of Wisdom

Zef Hemel — Fri, 13 Jan 2023 00:14:57 +0000

It was -12C (around 10F) when I dropped off my kids at school this morning. Mildly chilly, even for Polish standards. Time to put on a sweater before going outside. This can only one mean one thing: the holidays are near, and as I’ll largely take off from the middle of next week (to prepare for our festivities) this will be the last Platformer of 2022.(Editor's Note: This post was originally published in December 2022.)

We have to end this well, and therefore I’ll suggest a special holiday gift. This one will be for the specialest of someones. Not your significant other (if applicable), nor your kids (if applicable), nor your boss (if applicable). No, this one will one for the person we often forget about.

You.

Before we get to that, let’s get to the picklings of them fruits.

Cherry picks

One of the big decisions this week was that, due to instability of community (much better this week, thanks for asking), we’re skipping the 7.6 release (or more dramatically put: “we’re canceling 7.6!!!11one”).

We will move on right to 7.7 in January. Honestly, I always thought the .6 as a major release was a bit odd anyway, 7.7 at least looks symmetric. I think 7.6 simply had it coming.

As part of this skipping of the versions, we’re pushing mobile v2 GA along with it. So mobile v2 will also be released min-January now.

On the mobile platform end, we’re using this opportunity for a ground breaking idea. Extensive market research has shown that human kind is ok with a fair amount of things, but in a significant part of the world (that some biased voices would refer to as “the civilized world” although the level of its current civilization can be heavily debated) they’re not too keen on ingesting a lot of one specific animal species: bugs. Therefore, we try to eliminate them as much as possible from our customers’ diet. Yeah, we’re fixing bugs. Why don’t I just say that? Because it gets boring after a while. Gotta flower it up.

On the web platform end, we spent some time fixing bugs uncovered during our slew of outages over past weeks. We’re also looking at adding ESLint rules to avoid passing global state, and continuing work on menus (it never ends).

On the desktop platform end, we’re planning to invest in some more lower hanging performance fruits, like reducing the number of additional web views we use or at least making them much lighter weight.

On the server platform side, we are continuing to monitor and investigate issues with community instability. Specifically we’ve been working to run load tests on a clone of our community environment to see if we can replicate similar conditions that we saw in production. In addition, we have been supporting customers with their data migration (and fixes with tooling around this process), wrapping up the ability to revert data migrations and getting Perseus in a production ready state (CI, docs, tests, metrics). This ensures we’ll be the first in line to cause a fresh batch of outages on Community in the new year. I kid. That would never happen.

On the QA platform side, we’re heavily at work migrating macOS-based Rainforest tests to macOS Monterey, and cleaning up test suites. We’ve also been helping the server platform team with running the previously mentioned load tests.

HOWTO: Be Wise

There’s a poorly kept secret at Mattermost. In fact, it’s a fairly common thing in the industry: there is such a thing as a training budget.

Every year we’re allowed to spend a certain amount (in our case $500) on “professional development”:

Professional development: Work-related online courses and professional development certifications, not included in career path training or training for specialty roles, e.g. Information Security. Mattermost staff members are allotted 500 USD per fiscal year to spend on one or multiple training courses.

I’ve suggested to some people to translate “training courses” to “books.” Going on a training course is so ‘90s. I think nobody will take issue with that, and it opens up a fun holiday present opportunity: how about you buy yourself some nice books and let the company reimburse you? Disclaimer: please check with your manager on specific books before you do this, just in case.

Why would you invest in books? There’s roughly two reasons that I can see.

First: they can severely boost your meeting ethos.

Step 1: If you don’t have one in your Zoom video frame already, get yourself a bookshelf and make sure it’s visible on your video stream. Then, buy some impressive-looking books (suggestions later) and put them on that shelf.

Step 2: Hold back in meetings. Be the last to speak. Then, when the timing is right, raise your (virtual) hand dramatically. Turn around, pull one of the books from the shelf behind you, open it at a random page, and utter the words “As Marcel Proust would say...” Be sure to have a relevant quote here that just nails it. I have never read Proust myself, so I cannot help you with the specifics.

Boost in perceived wisdom guaranteed!

The second reason to buy books is that you may actually learn something from reading them. I hear you, this one’s a bit of a stretch, but hear me out.

When I was young and naive, I bought a whole bunch of books on fancy programming topics. My strategy was to buy the thickest book on a particular topic available in the store. Every time I’m at my parents I still see my “PHP4 for dummies” and “XSLT the ultimate guide” using up an impressive amount of shelf space. We didn’t have zoom calls yet back then to impress people, so let’s just say I was ahead of my time. Today’s reality is that by the time a programming-related book comes out, it’s already way out of date. I’m pretty sure there’s a PHP5 now, for instance, although I’m pretty sure XSLT is still pretty hot — some things are just evergreen.

So, what books to buy instead? Let’s use this as a fun “holiday books suggestions” thread. In order not to overwhelm, I’ll limit my suggestions, and make them somewhat self-serving.

While I may not have been doing the best job to push people into a managerial track with all my sarcasm about the value of management, Mattermost needs more people to take this path. Therefore I’ll include some books that give a better sense of what that path is like.

I’ll include Amazon links for convenience, usually Kindle links, but if you want to use them as shelf fillers, be sure to get physical, hard-cover copies.

A philosophy of software design by John Ousterhout: This is a great book on how to approach software design, both from a philosophical but also tactical perspective. I keep hearing recommendations from many sources, so there must be something here.

Then, a few “pick your career path adventure” suggestions.

If you’re interested in exploring the “IC path”: that is, to remain focused on the technology and growing there. There are two books on this topic for those farther into their career (senior and above):

Staff Engineer by Will Larson
The Staff Engineer’s Path by Tanya Reilly: this one’s very new. I only bought it this morning, but it’s looking promising.

If you’re interested in exploring the management path, the canonical book to get (even if you’re not really sure) is The Manager’s Path by Camille Fournier.

If you’re interested in exploring some more of these “people topics” (as you should), I’ll recommend three last books:

The Culture Map by Erin Meyer. This is extremely relevant to Mattermost where we have people working together from such a wide diversity of cultures.
Nonviolent Communication by Marshall Rosenberg. This completely changed my thinking about how we (ought to) communicate.
Punished By Rewards by Alfie Kohn, which really challenges the standard thinking on how we use rewards (I wrote about this topic at length based on reading this book).

One last important thing. I own hundreds of books. I’ve even read most of them. However, I made one mistake: for most, I didn’t take notes. And while this may be my age: I don’t remember all that much. Today, it often happens that somebody mentions “Oh yeah, I’m reading book X” and I’m thinking: yeah, I’m 99% sure I’ve read that — but I remember zip.

Don’t be like me. Make notes. I’ve made an agreement with myself some time ago that I will only claim to have read something if I have notes to show for it. Not only does writing notes help with retention, but they’re generally also easily retrievable.

If you’re looking for a tool to keep such notes and externalize more of your knowledge — I saw something on the front page of Hacker News and the Self-hosted Reddit last week, seems like a pretty cool tool. You should check it out. It’s open source and the author seems like a nice and very smart guy.

That’s all for this year folks.

Leave your own book recommendations in the thread.

I wish you all and your families happy holidays and see you next year!

The Platformer #31: Incidentally

Zef Hemel — Thu, 15 Dec 2022 19:04:03 +0000

I hope this post makes it to you safely. Let’s immediately address the mastodon in the room: these last two weeks haven’t been the greatest in terms of the stability of our community server. Even today, teams continue to firefight and figure out what the issues are. It seems to be an accumulation of many things. Sadly, reality is always more complex than we’d hope. Obviously, this needs to be fully, and confidently ironed out before releasing anything to customers.

For those unaware: we deploy the latest (and typically) greatest version of Mattermost to community.mattermost.com daily (I think around 10am CET). The reason for this is what some call drinking your own champagne — but I prefer the original, more disgusting version of eating your own dogfood (or “dogfooding” for short).

Of course, we test features and fixes as well as we can before merging them (and we’re constantly looking at ways of doing this more reliably). However, you only really know if something works by putting it into some sort of production environment. And we’re volunteering for this ourselves. In the highly hypothetical case that something doesn’t quite work, we’d notice it immediately. Also, due to our use of feature flags, we can enable certain features that are not fully baked yet just for this community environment for testing. The downside of this approach is what we’re currently witnessing, the hypothetical case where something doesn’t work quite right.

Trade-offs!

I could spend a few hundred words dynamic islanding the topic of outages, because there are potential long-term benefits to having production issues like this. However, a few things need to happen first:

We need to put out the fire.
We need to refresh, take a shower, eat something, and chill for a bit.
We need properly retrospect, and root cause all of this, extract learning, and come up with actional plans of how to improve. We’re still at stage 1. So one thing at a time. That said, I’ll mention that early in my industrial career, for a year or two I was responsible running a fairly large SaaS service. I was woken up at 2am, multiple times per week for many months. Those were tough times, but I learned almost everything I know today about building resilient software, monitoring, release strategies, and scaling during that time. There’s opportunity in everything. It just sucks while you’re in it.

This is all wisdom for later. Until then: HugOps!

While a lot of the issues are owned by the SRE and Boards team, people in the server platform teams are also helping out wherever possible, which changed the focus a bit this week.

On to this week’s kersen plukkerij (that’s Dutch, if I remember it still).

Cherry picks

Here are the notes for this week’s multi-product architecture sync.

On the mobile platform end, this week we have the code freeze planned for our v2 GA release. So besides getting certain Sentry changes in, the focus has been bug-fixing all the things. The end is nigh’.

On the web platform end, we now have an initial version of performance regression tests integrated into our pipelines, here’s an example. While a performance regression doesn’t block a PR’s merge yet, it will appear as a failing test. We’re still wrapping up the work on the unification of the post-component, which is now mostly about fixing broken end-to-end tests. Work continues on menus, and we’re figuring out some caching issue related to webpack federation.

On the desktop platform end, it seems Apple is equally as excited to upgrade the desktop app, and has therefore approved 5.2.2! Available at your neighborly App Store now!

On the server platform side, besides helping out with recent outages, we have made progress on additional fronts. Including running the regular release regression tests (no regressions found with the load test suite, but with current issues we may find there are gaps — likely because all load tests focus on Channels and we don’t have Playbooks nor Boards coverage yet — did I mention opportunity?). We also added some documentation with best practices on how to configure coverage frequency.

On the QA platform side, we continue where we left off last week: release testing, fixing flaky tests, but also preparing for some serious “winter cleaning” in January of all our tests. And... we are now getting iOS mobile v2 test runs reported to our test report channel 🥳

That is all for this week. The end of 2022 is nearing. Let’s go into the holiday season in as stable shape as we can.

Have a quiet weekend!

The Platformer #30: 22e82dfc

Zef Hemel — Thu, 15 Dec 2022 18:57:20 +0000

A big moment is within reach. It’s so close you can almost smell it.

How it started

commit 22e82dfc2a912f06425483f85eee78c90d99e603 Author: Elias Nahum Date: Sat Nov 14 11:06:43 2020 -0300 Barebones code for v2

This was the first commit on a clean, orphaned branch on our mattermost-mobile repo. This commit kicked off the work on mobile v2. At that time — some of you may not have been born yet (or not yet born into Mattermost — which would include yours truly) — it had become clear a bit of a reset was required. Mobile v1 had grown organically, as many products have, out of a proof-of-concept. Some fundamental design decisions (like only supporting a single server, or how state management works) were foundational limitations and effectively impossible to fix iteratively. A more hard cut had to be made. Not a full rewrite per se, but close. This work kicked off inside of the mobile team, but after about a year also started to include a whole slew of engineers across the organization, as well as product and UX.

While we all had hoped this would be in a shippable state earlier, we are now very close.

How it’s going

This week we have our feature freeze, meaning we will no longer be including any new features. The remaining time until GA (general availability) in the middle of December, will be spent fixing bugs. In the hypothetical case there are any :face_with_rolling_eyes:. If not, we’ll just chill, sit on the snowy beach chatting on Mattermost with our fancy new app.

A few weeks back we also changed our main branch to point to v2, so people won’t accidentally contribute to a legacy code base.

This is a major milestone and I’m very excited it’s here.

What do you all (who are not on the beta already) have to look forward to?

Multi-server support
Significantly improved performance (a lot of optimizations were done, and state management is more scalable by relying on an actual database)
A whole slew of UX improvements
The fresh scent of a new mobile app, can you smell it?

While that may seem like an unimpressive list for 2 years of work, do take into account that we effectively rewrote every single line of code in the process. To TypeScript no less. TypeScript is the hot “new” stuff if you hadn’t heard. All the cool kids are usin’ it. So we’re cool now too.

Thanks to everybody involved across a whole slew of teams and roles inside of Mattermost, as well as from the community, both as testers, designers, and code contributors.

Now, let’s just whack those last few (hypothetical) bugs, and push this baby to the stores full of Play and Apples.

In other fruity news...

Cherry picks

On the mobile platform end, we’re full-time, all the time, focused on getting v2 to feature ready. This means we will not hit many of our originally planned goals (besides supporting v2 GA and including Sentry in it).

On the web platform end, we merged the upgrade to React 17 (again) into master, and performed a bunch of (more minor) dependency updates while at it. We also continue our journey towards using MUI-based menus.

On the desktop platform end, we’re making progress on the diagnostics project and preparing for the Desktop 5.2 release.

On the server platform side, we continue working on improvements in mmctl regarding import and export (better error messages, better user experience). We also are close to wrapping up proper rollback support for our database migrations.

On the QA platform side, we’ve now updated Mattermod to include a link to the test dashboard (showing e2e results nicely) after requesting e2e tests. An example message can be seen here. In our capacity of temporarily stepping in to support the Calls team with their testing needs, we’re going to do some interesting stuff around testing actual calls. Months ago I was still joking about this, but now it’s going to happen: bots “talking” (likely producing tones at specific frequencies) to bots to make sure the signal transfers reliably. The future is near.

And that’s all for this week. Have a great weekend!

The Platformer #29: Extremely Hardcore

Zef Hemel — Tue, 22 Nov 2022 17:43:07 +0000

To get my Friday boost of inspiration last week, I took my usual detour home after dropping the kids at school. It took a solid a solid 45 minutes of detouring, during a pretty-damn cold November morning here in Poznań at -3C (about 26F).

What can I say, I’ll do whatever it takes!

I’m extremely hardcore.

Disclaimer: In this The Platformer, I will use “extremely hardcore” liberally, which is a reference to a demand Elon Musk made of his employees at Twitter this week (if you’re unaware). Long-time readers of The Platformer should already know this (I would hope), but I love irony and often joke. My references to being extremely hardcore are going to be ironic. Honestly, what’s happening at Twitter right now is concerning and I hope things end well — although it doesn’t look particularly good. Times are getting tougher everywhere, but Twitter is in a particularly bad place.

On to the better news with some zbieranie czereśni (that’s Polish, according to my wife).

Cherry picks

A large ongoing topic has been the Multi-Product Architecture. Here’s this week’s multi-product architecture update. This week we achieved a major milestone: Boards is now running as a product on community 🥳.

And as with every solid platform-related launch: :crickets:, you had no idea this happened. You just reloaded and everything... was still there (mostly). You hoped for something extremely hardcore but, got nuthin’. How disappointing.

The real action took place behind the scenes. Getting this done required some hardcore pipeline action. Some with good judgment may even assess it as extremely hardcore pipeline action.

On the mobile platform end, as we’re getting closer to the mobile v2 GA we’re trying to pull our weight to shipping this thing on time. Hardcore style. We’ve now added support for announcement banners (allowing us to broadcast our new levels of extreme hardcoreness), and we will soon allow you to share how extremely hardcore fans you are of v2, with an in-app review dialog. We have also integrated Sentry, so we will have more insight into how the app is performing and when it crashes, hardcore. And last but not least, the work on a nice piece of share sheet is heading toward Android (previously only implemented on iOS).

On the web platform end, we continue our work on integrating the new menus in more places. We participated in the extremely hardcore pipeline action required to get Boards-as-a-Product shipped on community. We continued our work on extremely hardcore performance testing, and... we upgraded more dependencies. Just random dependencies you ask? Hah, of course not — some, extremely hardcore ones.

On the desktop platform end, we’re in extremely hardcore negotiation with Apple to ship v5.2.1 of our desktop app to the Mac app store. While flexing that muscle, we’re also working on the desktop diagnostics UI. While not technically platform work: floating panels for Calls are coming to the desktop app, adding to the list of differentiators that make the desktop app The Best Client™️ for every extremely hardcore Mattermost user. And if you’re not one of them, you have no business being here, frankly.

On the server platform side, half of the team was involved with the aforementioned extremely hardcore pipeline action. If you have any questions related to Gitlab, ATA (Ask Them Anything)! The server platform team in general is a team with a high level of hardcoreness. We supported some extremely hardcore load testing of upcoming Channels features, for instance. And we got Perseus to the level where we can put some extremely hardcore strain on it, to it see if that will make it sweat. Writing a PgBouncer replacement in 6 weeks could be assessed as an extremely hardcore engineering effort.

On the QA platform side, we supported the hardcore Mattermost 7.5 release. We’re also going to temporarily support the Calls team with their end-to-end testing needs for a while. The goal is to make sure Calls sustains its high level of quality. Hmm, what would be a good name for this Mattermost-level of quality? Right, one could frame this as a extremely hardcore level of quality. Also soon, mattermod will gain the ability to automatically run end-to-end tests for mobile.

Opportunity Cost

“I hate what I've become! Trying to twist reality into making it appear rosy, when it's not. Everything sucks! 😛” — Anonymous

When you move into management one of the things you miss is the quick dopamine hit of delivering a feature, fixing a bug. Everything is long term and you don’t know what’s going to make an impact. “Observations” like this are the little things that make it all worth it.

So let’s double down on my journey to make everybody see that in fact, nothing sucks, and everything is an opportunity (or in other words: turning punch holes into dynamic islands).

As you may have noticed, a lot of our industry is struggling. Many companies seem to have been overly optimistic about growth, and now have to course correct. At Mattermost we seem to be in a stable place, but this is a time to be more mindful than before for everybody. The sky is less the limit than our industry thought (or that sky is positioned lower than we anticipated).

So, where is the opportunity in that?

There’s an invaluable skill that is particularly important when we have more constraints than usual: the ability to assess impact and opportunity cost.

We like to romanticize about “10x engineers” — engineers that can seemingly pull off amazing engineering feats in 10x less time than others. Great stuff. While we can ask a lot of questions about whether these 10x engineers exist, how they operate, if they’re desirable — the discussion misses a much more important point.

Even if you build a feature 10x as fast, if nobody ends up using it, the value is still 0. So should we be obsessed with the amount of code a single person can squirt out in a day, or only really care about the impact that work has on the user?

Yes, that’s a rhetorical question, Elon.

I’ve seen large teams build incredibly complex and technically impressive features for years that... solved no significant problem, ended up being used by hardly anybody, and were ultimately canned (if we were lucky).

I’ve also seen single-line commits that boosted revenue by 20% (yes, it was about the positioning of a button).

There’s always an infinite number of ideas of things we could do. Our backlog is never empty.

The gains don’t come from figuring out how to crunch through this backlog faster or with more people, but how to prioritize them by actual impact, and how to implement those selected items in the highest impact way possible (e.g. by applying the Pareto principle).

Since time is limited, every single thing we decide to spend time on implicitly means saying no to something else. This is what’s called opportunity cost and especially in “the sky is the limit” times we tend to ignore it. Saying yes is easy, especially if, from a technical perspective, something seems interesting. Saying yes to the right things and no to everything is much harder.

So, before blindly jumping onto something that’s at the top of the backlog, or that seems technically interesting we need to step back and reflect: Is this important? Will this make a difference? Is there something else I can do that achieves the same goal but is simpler and takes less effort?

Only when we’ve convinced ourselves this is the best way to spend our time, we should shift to doing the extremely hardcore engineering fun part.

Shouldn’t we always operate this way, though? Absolutely, however, times that ask us to do more with less, or more with the same, offer the opportunity to flex this muscle even more.

The Platformer #28: No More Bad Ideas

Zef Hemel — Fri, 11 Nov 2022 23:22:39 +0000

Happy Friday! What’s the worst idea you ever had, and how long did it take to realize it? Can we speed this process up? We’ll talk about that in a bit, but first this week’s череши (apparently Bulgarian has a single word for cherry picking, according to Google, efficient!)

Cherry picks

Platform channels of the week:

Feature: Gekidou & Mobile v2: for discussions around the upcoming mobile v2 app.
QA: Test Automation Reports: end-to-end tests are run for various PRs, and suites run on master overnight as well, results can be found here. Every week we have an SDET “on call” to fix or disable breaking tests.

Notes from the latest multi-product architecture sync-up.

On the mobile platform end, an early look at how we’re trying to measure performance and performance regressions in the future (video). We’re closer to integrating Sentry. Search just got merged and should go out with the next (internal) beta (which should release today) — the platform team is covering this functionality with end-to-end tests.

On the web platform end, we continue the work on menu components, found and fixed the issue that marked a whole lot of channels as unread last week (with GarphQL enabled). We’ve also been working on performance regression tests. And we’re upgrading moar dependencies.

On the desktop platform end, some bugs slipped into last week’s 5.2 release, which we’re addressing right now. A 5.2.1 release will happen soon.

On the server platform side, we continue to spend time on Hacktoberfest PRs. To better support customer migrations from alternatives-that-shall-not-be-named to Mattermost, we’re making usability improvements to our import tooling, such as better logging and error messages. On the Morph (our library for data migrations) side, we’ve started work on rollback support. We’re also making good progress on Perseus (our PgBouncer replacement). Bonus: we’ve released version 1.7.0 of our load testing tool.

On the QA platform side, progress continues on our QA infrastructure. As can be seen in the QA Test Automation reports channel linked previously, we’re not currently in a “green stable state” — however, this is being worked on as well.

Idea weeding

I get my best ideas while taking showers or taking walks.

If you feel you don’t get enough great ideas, start taking showers. People around you will appreciate it for other reasons as well. And take walks too. Exercise you know.

However, contrary to what you may have assumed — not all my ideas are pure gold. Most of my ideas crash and burn before anybody even finds out. And you should be thankful.

Misremembering a common phrase you might be tempted to encouragingly say: “There’s no such thing as a stupid idea!” That would be before you heard some of mine, in which case you’d have to concede: “yeah, except those.”

However, I hope you will agree that a good amount of ideas coming out of my mouth or keyboard seem reasonable. So how does that happen? What’s my secret? What’s my process? And how could you appear (although, obviously, never be) just as smart?

Let’s use this very article as a case study. It’s a bit meta, but we like meta.

As often happens, after dropping the kids at school this morning I took a detour home through a local park. A bit of exercise.

As I walked waiting for inspiration to kick in, I slowly started to accept that this week finally the inevitable had happened: I had zero inspiration for The Platformer. It had to happen some day, that time had finally come.

Then, all of a sudden something hit me.

Good ideas.

This topic had come up a few times over the past weeks chatting with people. How does one balance encouraging people to come up with creative ideas, but also not wasting too much time on ones that deep inside you know are... not going anywhere.

This could make a good topic for The Platformer, no? Good idea? 🤷

So how would I find out?

Let’s follow The 6 Step Plan.

Step 1: What problem does this solve?

Other than “I need a topic to write about.” What am I hoping to achieve writing about this topic? What’s the goal? Pure entertainment? Some good padding for dad jokes? No, while on Friday some level of entertainment is desirable (I have to find ways to get people to keep reading 1k-2k word pieces every week, and I found humor makes it bearable). People ideally get some value out of it as well. Helping people to figure out if their ideas are good or bad is likely something more people struggle with.

Alright, I’m convinced. This may be worth pursuing.

Step 2: Is this a new idea, or has it already been done, or tried and failed?

Of course we’re all uniquely brilliant and creative, occasionally. But most of the time we’re really not.

This is where we could start doing some research. We may Bing our idea. We may think of other places where similar issues may have arisen and see if they found solutions. If it didn’t apply or work there, does that mean it also won’t work in our particular context or are there differences that may make it work here? Or perhaps, this failed ten years ago, but things are different enough now to try again.

I decided to skip this step.

The reason: opportunity cost. I cannot be spending my whole Friday researching and writing. I have a (self imposed) deadline. Every hour I spend researching, I will not be able to spend on other stuff (like approving expenses). Rather than thinking, developing, scrutinizing, researching I could just do it and see what happens. Worst case would be some wasted hours. Although, honestly with 100+ people in this channel I could be dragging you along in this waste as well — so there’s still the question if I should hit “Send” on this one.

If you’re reading this, it means one of two things: I think it may provide value, or I just don’t value your time very much.

This may not apply to your idea. If your idea takes more than a few hours to implement, perhaps days, weeks, months or even years — it’ll be worth doing some research, and likely finding out that “yeah, people been there, done that, gotten the t-shirt.” And that’ll be that.

Step 3: What are the critical unknowns?

Usually ideas have a few key unknowns. Some we’ll not know about and will find later, but some we can identify upfront. Can we, in our mind, figure out what those critical unknowns are? And if it’s an idea that involves some code, can we perhaps just prototype super minimal versions of those critical unknowns to see if we can make them work? In other words: what’s the minimum amount of work we can do to detect and remove obvious blockers?

In this article, I knew I’d have to come up with some list of steps. Honestly, I had never thought to deeply about this topic at all. Neither do I have a Playbook ready for such cases. Could I come up with at least a few steps that seem reasonable and not too obvious? I got to three so far, so I’ll take that as a pass. Let’s continue.

Step 4: Can I provide a narrative?

Some people are talkers, some people are writers.

You’ll be shocked to learn I’m a writer. For me the ability to write down an idea in a coherent way is a pretty good test for the quality of an idea.

The format would be like a “pitch,” which largely replicates the previous steps — but now in written form:

What problem are we trying to solve?
What solutions exist? Some solutions may already out there ready made. And it’s rare that there’s just one possible solution to a problem — so what are the alternatives?
What are the trade-offs between solutions? No solution is perfect, there’s always costs and benefits.
Given the above, why pick this particular idea?

It happens a lot that an idea sounds great in my head, but when I force myself to write it down I discover tons of holes. And that’ll be that. Companies like Amazon have famously institutionalized this idea in their “6 page memos.”

If writing scares you. No worries. Find a rubber duck victim (an actual rubber duck may actually work).

“Hey, can I run a crazy idea by you?”

And then you start pitching. Mid-way you may run into trouble. And that will be that.

For this particular article: the narrative is the product, so I just went for it.

Step 5: Build a proof of concept

Still here? Alright, we may be on to something at this stage. What is the least amount of time you can spend to show your idea actually works? This is where we may use spare time, open source Fridays, hackathons and the like. Now we start building stuff, but with the goal to prove that the concept actually works.

Our first filter: us. Now that we see this come alive, are we still excited? Do we still believe it’s a good idea? Maybe not, and that will be that.

If we do, it’s time to bring more people in. Demo it to people live, or record a demo and share it and see what they think. Don’t forget to include the pitch, so that people understand what you’re trying to do here.

For this particular article: I’m typing these words at 10:42am as a first pass. After rushing home after the idea hit me, I have now spent about 2 hours in the first 5 steps. I think all the content is there, but it’s still in a somewhat rough state. I’ll likely need to spend another hour or two polishing, but at this stage I think it will be worth it. And to be honest, for what it is there’s also a bit of the sunken cost fallacy going on, but don’t tell anybody.

Step 6: Change the world

What happened when you saw your idea come alive? Did it fall flat? That happens, a lot. That’ll be that.

Or did it blow some serious mind? Time to iterate, share more broadly, create buy-in and iterate some more. And after that, you know, change the world.

Want to take a closer look at the inner workings of the Platform team? Join the Community server to be the first to read The Platformer every Friday.

The Platformer #27: The Most Wonderful Time Of The Year

Zef Hemel — Mon, 31 Oct 2022 17:52:36 +0000

Back in July, I revealed the start of financial quarters to be a universal celebration. It’s great to have universal days of celebration, but can we top that? Wouldn’t it be cool if we’d share seasons as well?

“How could we make that happen, though, Zef!? In some places in the world, we’re heading into the winter, in others into the summer. You can attempt to define reality all you want, but there are limits!”

And this is where you’re wrong. There are no limits.

Even though you may not have realized, or more likely: you may not have appreciated it — we do share at least one season in common, worldwide, universally.

And the amazing news is that at Mattermost this season starts today.

It’s “Performance Review” season!

Indeed, judgment is in the air.

More on this most wonderful time of the year, and how to get the most out of it, after this week’s picking of the κεράσια (that’s Greek, according to Google).

Cherry picks

Platform-related channel picks (in case you find yourself thinking “hey, I have FOMO, perhaps I’m not in the right channels.”):

Client Performance Automation — where pan-Platform discussions happen on how to best approach measuring performance.
Multi-Product Architecture — where discussions and status updates on the multi-product architecture happen. Here is this week’s summary.

On the mobile platform end, we’re making progress on the performance regression testing. Some of us are learning Go to be able to contribute to the one-command install project. We also had a good meeting with the Boards team to discuss what’s in stock for Boards on mobile 2024. It looks like a solid first step.

On the web platform end, multi-product architecture for the front end is now merged into master on webapp. We’ll enable running Boards in “product” mode on Community server soon. Beside this we spent time debugging why we regressed on Community last week (marking a lot of channels as read-only for a short amount of time). We also continued our effort to upgrade dependencies.

On the desktop platform end, we primarily focused on the last fixes to prepare for the 5.2 release next week.

On the server platform side, on the load testing end, we’re performing some low-hanging fruit maintenance work like upgrading the version of terraform and doing better cleanup after running tests. To better support the ever-increasing cloud load, we’re making progress on Perseus (the PgBouncer replacement we’re working on). Regarding the drastic refactoring of the server repo we’ve been doing, we’re starting to hit a point of diminishing returns, so we will likely stop and shift our effort to other, more high-impact areas after wrapping up ongoing things.

On the QA platform side, we’re still in the process of setting up Percy for visual regression testing.
We’re also feature testing Boards 7.4 with the community, and asking for user feedback on recording in calls. And continuing work on the observability of our test infrastructure.

Mirror, mirror

Some time ago I wrote about the challenge to turn punch holes into dynamic islands. A fun subset of this problem space is to frame tasks that may appear as chores as life-altering experiences.

At some point, I’ll do a world tour asking audiences to yell out a chore, and in real-time I’d come up with a framing that shows it’s the best thing to do, ever. It’ll be fun.

Until then, let’s look at one timely chore: writing your self-reflection!

Part of this most wonderful season of the year, writing your self-reflection is the first step.

Annoying right?

Wronk!

In fact, this an exercise we should all do regularly anyway. Including it in The Season™️ is just a life hack to do it at least twice per year.

Here’s how I suggest you approach it. It means answering just two questions to yourself. How hard can it be? Or rather: how much more fun can it get?
We love talking to ourselves!

The first question: When did I feel awesome this year?

When did I feel king or queen (or some gender-neutral variant of royalty) of the world? Or, if that is too high of a bar: what do I feel were my biggest wins this year? When did that dopamine blast me in the face? What were the circumstances? Can I think of more cases? What were the circumstances then? Is there a pattern?

Now, as a bonus question (that doesn’t count to the 2), what could “the company” do to create more instances of awesomeness for me? (Let your manager know, you may be surprised about what’s actually possible.)

And in case I really cannot come up with something here — this is a bit of a flag. Am I in the right role, the right team? Do I have to make some changes? When I mentally peruse the company’s org chart, is there any place, any role, any team that makes my blood flow faster? In other words: is there a place where I could be more awesome? Ideally, in the company, but you know, that’s not always the answer. I hear Twitter has some recent openings (or maybe wait a bit).

Second question: Have I been giving the company my very best this year?

Let me nuance this question: have I been giving the best I can do, while still maintaining a reasonable work-life balance?

Some sanity-check questions around this nuanced version:

Does my significant other still recognize me?
Does my cat still come to me when hungry?

So, have I been giving my reasonable best? If yes — amazing.

Really, though?

Am I sure? Was there really nothing that impacted my performance this year?

Really?

Was it all rainbows and unicorns?

Really?

Seriously though, really?

I thought so.

(I call this technique the “5 really’s” — the recpipe is to ask “really?” until the target realizes they’re wrong — usually 5 times suffices.)

If not, why not? What was blocking me?

Was it skill? Motivation? Was it unclear where I’m heading, what I was supposed to do? Is it related to my team? My manager? Was the work not interesting enough, or seem valuable enough? Was I distracted? Did I not feel appreciated? Was I not able to do the work I feel I should have been doing? Are my skills not used to their full potential? Or, was it life getting in the way and we need to rebalance a few things?

Bad news: the answers to these questions might not be one-to-one copyable to the form you are going to be asked to fill in for your self reflection, but they should provide you with enough content to provide solid answers there.

Side note: if you’re (un)lucky enough to report directly to me, you are hereby granted permission to cheat the system, ignore the questions and just dump in the answers to the two questions above in any of the fields — if you’re comfortable with that, of course.

But the form, nor the process, nor system is the goal. It’s the opportunity. Self reflection is not a punch hole, it’s a dynamic island in the island group of self discovery.

Listen to me, attempting to sound all poetic and shit. Let me make a note for my own self reflection.

The questions should help you discover and reflect on your strengths and weaknesses and figure out where you want to go next. Not because the company is nagging you, but because — hopefully — you want to discover this yourself.

And as Yogi Berra supposedly said: “If you don’t know where you’re going, you may not get there.”

Our ability to step back and self reflect is likely one of the most effective ways to improve as people. Peer feedback will come, your manager may have an opinion, sure. Ultimately, it’s you who will have to consolidate it all and turn it into something useful.

Who cares what other people think anyway, we know ourselves and what we’re capable of. We make our own decisions. Amirite? Power to us, the people!

This post was adapted from its original version on the Mattermost Community Server. Want to take a closer look at the inner workings of the Platform team? Join the Community server to be the first to read The Platformer every Friday.

The Platformer #26: The Subtle Art Of Letting Things Blow Up In Your Face

Zef Hemel — Thu, 27 Oct 2022 18:29:08 +0000

Sorry for the long title on this one, but it’s just too good of a tease not to use. Before we go there, let’s start with the pickings de la cereza — that’s Spanish (according to Google).

Cherry picks

Hacktoberfest continues, so we’re still spending a lot of time on reviewing all those incoming PRs. Thanks for all your contributions!

On the mobile platform end, we’re going to try something new: a weekly mobile hangout (in the public ~Developers: Mobile channel of course, using Calls) on Mondays. The goal is to have a casual “drop in and out as you like” type of environment. Some of our mobile engineers will be there, if you like to talk about mobile development, are stuck somewhere, or would like to share your love of dynamic islands — this will be the opportunity. We’re also pushing three initiatives forward: automated performance testing (channel: ~Client Performance Automation) in collaboration with the web platform team, Sentry Integration and the “single command” installer for development environments. On the latter topic, we polled people last week. We’ll create a prototype of the “single binary” (written in Go) option.

On the web platform end, the team has been working on unifying post components (and cleaning up afterwards), more fighting with Webpack and friends in the context of the front-end part of the multi-product architecture, testing out a prototype of menus based on Material UI, and the regular cycle of dependency updates.

On the desktop platform end, working on native notifications and fixing bugs in the download list. We’ve now upgraded to Electron v21! In addition, during last week’s web guild meeting: we gave some details on the plan to webapp/desktop module integration. Feedback is welcome!

On the server platform side, already some time ago (but I somehow missed it), we posted a thorough
investigation of performance regression you may be interested in. Last week I mentioned “community on demand” (a way to deploy duplicates of our community server for testing purposes) and it is now available to Mattermost employees. We haven’t broken community this week, therefore we’re continuing our refactoring streak.

On the QA platform side, we've been doing a lot of work stabilizing our QA infrastructure, adding observability (collecting metrics), and automating more things (such as building AMIs for iOS-based testing). We have now also added Percy to our stack (a tool to support visual regression testing). On the test case management topic, we are currently witnessing a piece of history in the making — the first QA-Engineering collaboration on test cases, happening in a PR (because now we can) 🤯.

Quality Utopia

Picture this:

As an engineer, you get a new feature to build. The spec is clear, the designs are done. And... you have an acceptance test suite that will only succeed when the feature is implemented correctly.

Energized by the new assignment you start typin’ the codez.

Tap, tap, tap.

In the background, all sorts of quality magic is happening. Safety nets activate.

Your compiler is checking types. Your compiler is happy, a green checkmark lights up.
Then, your unit tests run. Another green checkmark.

A suite of integration tests run. Another checkmark.

At this point we’ve reached the limits of what your whiny laptop can handle without spinning up fans too loud (for those who haven’t yet transferred to ARM nirvana), so your code is making its way to the QA Cloud.

The QA Cloud is spinning up your code and runs elaborate suites of end-to-end tests covering all of your product’s functionality. Of course, this coverage is extended with the acceptance test of your particular feature. Check. Check.

When these pass, the code is deployed to various setups that mirror actual production setups our customers use, as well as representative sets of their data. Here, too, an elaborate functional test set is run. Check. Check. Check.

Then, a slew of load tests run on these deployments. Your code is exercised as if 5k users, 10k, 20k, 50k, 100k are concurrently typin’ the messagez, makin’ the callz, draggin’ the cardz, and checkin’ the boxez. Check, check.

Then, for a while, things go quiet.

As your mind drifts, all of a sudden you receive a message on Mattermost from a colleague.

“Nice job on that new feature, I’ve been playing with it for a few minutes and it works great!”

Your code is live on community.

A few hours later, a phone call.

“Ahoy-hoy! KeyCustomer Inc. just called and they just love the new feature, and our turn-around on delivering it. They expanded their contract and added 5k more seats!”

While this may sound utopian. It’s not as far out of reach as you may think. In fact, it’s what we’ve been building towards (perhaps implicitly) for quite a while. Let’s consider it a kind of a “quality north star.”

If I could have this tomorrow, I would take it, wouldn’t you?

Next week is also fine.

The good news is: we’re moving there. Step by step. However, there are still plenty of gaps.

The annoying thing with gaps is: where are they, how big are they, and how should we prioritize them? Also: what about the gaps we don’t know about?

One obvious gap we have is that we don’t have 100% test coverage at any level. I hope nobody is shocked by this. Not at the unit test, integration test, or end-to-end test level.

Doing this would be an excessive amount of work, we could create a virtually infinite backlog of tests to write. So how do we prioritize?

Less obvious gaps are... well, unknown. We know they’re there (we can smell them), but we don’t really know.

So, what now?

For this, let me introduce you to a surprisingly simple, yet powerful art form:

The subtle art of letting things blow up in your face.
...and then learn from it.

Right, I shouldn’t forget about that last part, it’s kinda critical.

This technique helps in prioritization because it gives us a realistic idea of the things that actually tend to break in real life.

For instance, reading The Platformer, you may naively think it’s pretty important to have solid test coverage for excessively long posts. However, it’s possible that you’ll find that in practice there are only a handful of nutcases in the world that write >1k word posts (most of them in platform teams), and this is not really blowing up all that often, and therefore not really worth putting a lot of effort into.

Letting things blow up “by design” should give you a whole different perspective on what others may refer to as “failure.” Because it’s not failure, it’s opportunity! We identified a gap, w00t! Now let’s learn from it and adjust our priorities. Good times.

And if things don’t blow up — equally good times. We keep typin’ the codez.

The “subtle art” part is to contain the blowing up to reasonable levels. We can only YOLO our way through life to some reasonable level. The earlier things blow up, the smaller the impact, the better. We want to increasingly have things blow up more and more upstream, if you will. This is why typed languages are valuable, they give you hints of you doing crazy stuff in milliseconds. You don’t really want a customer to have to call you to learn that “undefined is not a function.”

What do we do when we’re “successful” and our face needs some fixing up, post-blowout? What do we do when your half-baked feature, somehow ends up being deployed to customers in a broken state?

We retrospect.

We investigate where things went wrong. Already today we have a lot of layers of safety nets in place: we have types, unit tests, integration tests, end-to-end-tests, we do code reviews, we deploy to the Community server, we have Rainforest tests, (some) manual QA test suites and explicit QA sign-offs.

We need to pinpoint where in this line of layer after layer of safety nets this should have been caught and fix it, or if we need even more layers. And while at it, look around a bit more and see if there are other obvious gaps we can close as we’re cleaning up anyway.

That’s all.

Then: rinse and repeat.

And before you know it — give it 20-30 years— it’s here: Quality Utopia.

The Platformer #25: High-Impact Hacking

Zef Hemel — Mon, 17 Oct 2022 20:10:11 +0000

We’re well into Hacktoberfest and like every year, Mattermost is participating. While we all knew this was going to happen, we perhaps didn’t all anticipate how much time it would take to review the overwhelming number of incoming pull requests. This, of course, is a luxury problem that we’re happy to have. It is slowing our planned work down a bit — but what can you do?

As I’ll argue later in this post, supporting community contributions may be the highest-impact work we’ll ever do. A whole lot of new people are flooding into Mattermost projects to contribute and if their experience is good, they may stick around.

Cherry picks

Agniva posted another update on the multi-product architecture. These updates mentioned the emerging Event Bus project that we worked on during Mattercon. Of course, #ThereIsAChannelForThat: ~Contribute: Event Bus on which yours sincerely posted a 3-minute demo video of the current state of things.

On the mobile platform end, a big topic for Q4 will be to make progress on the “one command to set up your environment” project. While we figure out what the initial scope of this project should be, one important decision to make here is the tech stack to build this on — taking into account we will likely want to expand this command beyond the mobile use case, e.g. set up the server, web, and QA environments, as well as use it to manage the many git repos that we have. Poll is here.

On the web platform end, we are overwhelmed with Hacktoberfest pull requests. Again, an amazing problem to have, but it takes time. Q4 focus-wise, we’ve picked up work on the menus again, focusing primarily on places where it’s used in the sidebar. We also did a bit of a retrospective on the React 17 upgrade (and rollback) and what we learned from this. During the Developer Meeting we also discussed a plan to perform the upgrade once and for all.

On the desktop platform end, we are working on OS-level do-not-disturb integration. We’re also upgrading to Electron 21, and diving back into the topic of combining the webapp and desktop app more closely.

On the server platform side, we merged a huge refactoring PR last week and have now dealt with the aftermath. While intended as primarily refactoring (which by definition doesn’t change behavior), it also changed behavior — as it turned out — especially due to initialization logic. We ran all our test suites, did manual QA, and still ended up blowing up deployment to Community. This was quickly fixed, but still, it shouldn’t happen. The issue was related to the very specific community setup we have, but that doesn’t mean we simply have hope for the best 🤞 next time. What came out of this is that for the future we now have a way to avoid this problem: “community on demand” — the ability to deploy a replica (in terms of setup + data) of our community server on demand, for testing purposes. Details on how other teams can leverage this will be shared soon.

On the QA platform side, we had another week, which means: another release to run QA for. Due to Hacktoberfest, a lot of review work is on the QA guild, with a large number of submitted tickets porting our Cypress test suite to Typescript.

High Impact through Open Source

“It was this project that inspired me to finally learn programming a year ago. And I’m happy to share: I just landed my first programming job!” — Parham

I had a conversation yesterday about how we gauge our impact when it comes to the part of the job where we’re primarily hoping to grow and support other people, rather than a codebase.

Code bases are easy in that sense. You clone it, make changes, maneuver through review, and merge: bug fixed; feature shipped; years of cursing from people when they git blame and find this code.

I made a difference. I was here.

However, in more senior roles we’re no longer just expected to ship stuff (or as I call it: “typin’ the codez”), we’re also expected to support other people in their growth. The pointer I’m giving to the most senior people is “Figure out ways to pull other people to your level.”

That’s easy to say (that’s why I do it), but even if we figure out strategies that may achieve this, how do we know if we’re successful — how do we measure it? And realistically how long does it take to know for sure?

About 20 years ago I started my first open source project. It was forum software written in Perl. That’s right, Perl — you’re welcome, world. It’s what we had back in those days. Don’t judge.

This project, called YaBB — Yet another Bulletin Board (I’ve always been good at naming things) — is pretty much dead. A few months ago the website was still up, but it appears to be down now. Yes, the project was forked into YaBB SE (a direct PHP port), and ultimately evolved into Simple Machines Forum, which still seems to be under development today (and powers Bitcoin Talk — if you’re still into the crypto thing).

However, all those many thousands lines of Perl code I wrote 20 years ago — pretty much dead. Which, honestly, is probably for the best — you would not trust a word I say if you had seen the stuff I pulled in that code.

So what remains? What was the long term impact of this project?

That is really, really fuzzy.

However, here is what I’d like to believe: some of the good that came out of this project are notes I received, like the one I started this section with. Stories like Parham’s. Apparently, it got Parham into software development (despite Perl) about 20 years ago. We’re connected on LinkedIn, he is now the VP of Engineering somewhere. We can debate whether you would consider that a success story, but I’d like to think it is. Was that all due to this project I started 20 years ago? Of course not, but likely it contributed, this is where the fuzziness comes in.

It also provides a clue on how we may make a lasting impact on people’s lives through software, even long after all the code you wrote has (finally) been deleted. Which, not to be dramatic or anything, is going to happen. That lasting impact is all about what people learned from you while being involved in a project. Many of these learnings should transcend technical stacks, project, companies, and time.

Ask anybody to “root cause” how they got into their particular profession. Chances are high that they will lead it back to a particular teacher they had in school, decades earlier. By the way, if this is the case for you, reach out to that teacher (if they’re still around) and drop them a note, they’ll appreciate it. Being a teacher is likely one of the highest-impact professions in the world. That’s why we compensate them so well. cough.

However, we don’t need to quit our jobs and go teach in primary and high schools to make a difference, we can play a similar role at work. Open source companies make this even more accessible.

How? Here’s the reality: most software companies tend to decide who they hire based on background and experience. You should already be able to do software development coming in, it’s rare to hire people with no experience at all.

At Mattercon, we heard a talk from a community member who shared he had used Mattermost as a training ground. He had no background in software development whatsoever, and learned it all through self-study and open source contribution. He’d make a change, get feedback on his pull request, and learn. Rinse and repeat.

He’s a core contributor now.

As my earlier example showed, this is not a rarity. This is a great way to get into the industry. We hire a lot from the community, and many of these people have “non-traditional” backgrounds.

So, how do we measure our impact long term? How will we know that we “pulled people up”?

I don’t know. Let’s evaluate in 20 years. Drop me a note.

Until then — if you’re a maintainer for an open source project — go review pull requests from community members, during Hacktoberfest and beyond. It may be some of the highest-impact work you’ll ever do. And tragically, you may never find out.

The Platformer #24: Regressions

Ashley Dotterweich — Thu, 13 Oct 2022 18:40:05 +0000

It’s the first week of the month, and not just any month. It’s October! And we all know what that means: the start of a new quarter — Q4 no less. Time to look back and forward.

A common theme for the next quarter for many platform teams is going to be performance monitoring and regression detection.

To explain why this matters, let’s look back to the biggest performance regression we witnessed in the last year: the introduction of collapsed-reply threads (CRT). CRT is a shockingly complicated and large project, with many complicated aspects. Sadly, one unanticipated effect was a significant regression in performance. We should have caught this before shipping it to customers. And we could have — if only we had the right infrastructure and focus on this topic in place. While we had the infrastructure (our load testing tool), we didn’t properly leverage it.

We try to learn from our mistakes, so after some heroic efforts to fix the performance issues, we now have renewed focus on this topic. On the server platform team, we have Alejandro García dedicated to this topic. And guess what? It works. Even yesterday we detected a performance regression using our tool and reverted the change before it impacted anybody. Not a CRT-scale regression, but a regression nonetheless.

Performance testing is one of the highest impact topics we can work on in platform teams, so we want to “spread the love” beyond just server, specifically to mobile, web, and ideally, desktop as well, leveraging the pipeline and tooling from QA. Therefore, this will be a shared topic among all four platform teams. Stay tuned.

OKRy picks

So, what did Q3 bring for the platform teams, and what will Q4 look like? Let’s dive in.

On the mobile platform end, both in Q3 and Q4, the big effort remains to support the planned release of v2 by the end of the year. In addition, we managed to spend some time on platform topics as well. Specifically, this quarter we:

Built a logging framework;
Are now using GraphQL for the initial load (on servers that have GraphQL enabled);
Made good progress on integrating Sentry;
Vastly expanded our end-to-end test coverage.

In Q4, we will ship this Sentry integration and keep expanding our test coverage. In addition, we’ll be laying the groundwork to have the first Boards and Playbook functionality be integrated into the mobile app next year. As part of this, we’ll invest heavily in further smoothening the developer experience. The goal is to have a one-line command to set up your development environment end-to-end. This should remove a big barrier to entry to get started on mobile.

On the web platform end, we made solid progress on the multi-product architecture for front-end in Q3 and will continue this effort in Q4. In addition, we are now using GraphQL (for servers that have it enabled) in more places. A few big refactor and cleanup efforts are still in progress and will be continued in Q4, including the menu component and post components. In Q4, we will spec out our new front-end Suite API, implement the data manager, and avoid DoS-ing ourselves (avoid the thundering herd problem in more places). In terms of focus on performance, we intend to improve channel and team switching performance by 20% and contribute to the platform-wide effort to detect performance regressions.

On the desktop platform end, we internationalized the desktop app! We also implemented the new download list UI. Both of these major features will ship as part of desktop 5.2, which will ship later this month. In addition, we now have support for native node modules, which will allow us to do more operating-specific cool stuff like native notifications, which we hope to leverage in Q4. In Q4, we intend to further invest in reliability by making it easier to run our E2E test pipelines and reduce the number of flaky tests. We also want to make technical strides by attempting to compile-in some parts of the webapp into the desktop as a proof of concept, starting with the top bar. Last but not least, we want to invest in supportability, another big platform topic, by giving customers a simple screen to “sanity check” their installation to make sure everything is set up right.

On the server platform side, we did a few huge lifts with big diffs in Q3. In addition to supporting the mobile and web teams with GraphQL work, we landed a few huge refactors reorganizing how code is structured as part of the multi-product architecture. The biggest one was merged yesterday, which resulted in some issues that we’re still figuring out as we speak.

However, refactoring is not all we did. We also significantly improved our error handling in many places (resulting in more massive PRs). This means we’re ready for our next big lift: supporting Playbooks in their move to the multi-product architecture as well in Q4. Other things on the docket for Q4: automatic continuous profiling, rollback support for database migrations, mmctl import/export improvements, tons of improvements to our load testing tool, and scope of the load tests. In addition, we want to look at effective ways we can use Go’s generics to improve our code base, and will hopefully introduce the Event Bus.

On the QA platform side, we made major strides on our testing pipelines. Not only supporting the current ones, but also creating new ones for our desktop and mobile apps. They are now operational but can still be optimized significantly in terms of run time, which we’ll work on in Q4. Further in Q3, we kept iterating on our Rainforest tests, supported weekly releases, and worked the infrastructure to allow us to engage the QA community through writing test cases in the future. In Q4, our work on pipelines and infrastructure will continue. We also hope to continue growing the QA community again with five new people, publish best practices on how to write test cases, and continue to expand the scope and robustness of our end-to-end tests in Cypress and Rainforest.

And that is all for this week. Somehow, I’m not super how high on the dad joke humor today. So, to compensate, I’ll end this with a dad joke I pulled from the Twitters:

I was walking past a farm and a sign said: Duck, eggs!
I thought: That's an unnecessary comma — and then it hit me.

Want to take a closer look at the inner workings of the Platform team? Join the Community server to be the first to read The Platformer every Friday.

Is it time to give up GitHub? August Open Source News

Ben Lloyd Pearson — Mon, 08 Aug 2022 13:40:00 +0000

Overwhelmed by Anticipation in The Cloud — The Mattermost Platformer #18

Zef Hemel — Fri, 05 Aug 2022 10:58:55 +0000

This post is part of a series previously posted in the public "The Platformer" channel on Mattermost's Community server. "The Platformer" is published on a weekly cadence sharing the latest updates from the Mattermost platform teams (web, server, mobile and QA). It's an insider look into how these teams operate, and what they're working on.

August is a month of onboarding for us. This Monday Mario joined the QA Platform team as a Senior Software Design Engineer (we don’t develop software at Mattermost, we design it). Next Monday we will have Tasos and Pantelis join Web Platform (as SDE II and Engineering Lead, respectively). I’m happy to have so many people join the party, especially in times where there is not so much “partying” going on across the industry. I’ve been following these industry lay-off and hiring freeze developments quite closely (I also have a bunch of former colleagues affected). My reading of the situation is that in most cases this seems to be correcting for over-optimistic “the sky is the limit” thinking for the last year or so. There seemed to be no end to explosive growth, an infinite pool of VC funding, and an infinite number of new coins to mine. But guess what? Reality is harsh. So, it’s time to cut back and focus on what matters most 🥁 Padum tss.

Before continuing our journey towards platform teams, let’s pick some of the fruits of our (in many parts of the world) sweaty labor.

Cherry picks

Joram posted another monthly update on the multi-product architecture.

On the mobile platform end, we released another beta build, squashed more bugs and are filling more gaps in functionality. GraphQL support is still eagerly awaiting to be activated. However, re-enabling GraphQL on community is still awaiting a few fixes on the web platform end. Speaking of which…

On the web platform end, beside being overwhelmed by anticipation of the upcoming new joiners, we’re making good progress in getting webpack federation ready to be merged. This will be a solid step towards the multi-product architecture on the front-end. The upgrade to React 17 is also almost there. Beyond that, as mentioned, we’re fixing a few issues in GraphQL before we enable it again on community.

On the desktop platform end, we’re making progress on building the desktop app in The Cloud ☁️. We also finally got native node modules working, squashing bugs, and are overwhelmed by anticipation of doubling the size of the desktop team next week.

On the server platform side, we’re still moving fast, but breaking fewer things (compared to last week). One could say we’re moving fast on stable infra. Refactoring of the multi-product architecture into a few smaller services continue, flushing out the “app” package bit by bit. By the end of Q3 you’ll all be confused Travoltas, you’ll see.

On the QA platform side, beside onboarding Mario, we’re also entering the wrap-up stage of running end-to-end tests of mobile v2 in The (Mac) Cloud ☁️. We’re running another weekly The Cloud ☁️ release. As well as making plans to move test plans over to a git repository (hosted in The Cloud ☁️), to make it more accessible to The Community ☁️ and version control it.

The Road to Platform Teams Episode II

I have to apologize to you all. I had not anticipated the amount of sleepless nights and marital problems I would cause to many of you by leaving you hanging with a cliffhanger like that, last week.

A few cleaned up responses:

WHAT HAPPENS WHEN WE GO VERTICAL!? I NEED TO KNOW! I CANNOT WAAAIT A WEEK TO FIND OUT GAAAH!!!!111one

And:

Dude, with all due respect, you cannot play with minds like this. My family needs me during the weekend. I was completely absent, drawing org charts on white boards in our basement for days and nights trying to figure out what would happen, and especially WHAT THE TRADE OFFS WOULD BE. Man! My wife is upset. My kids kept asking “where is daddy?” Stop toying with our minds.

I hear you. However, a disclaimer upfront: we’re not going to wrap this journey up this week either. My family also needs me, and I cannot be sitting here typing up absurd, made-up reader feedback day and night either. Just setting expectations here.

Before we proceed, and for those who have not been losing sleep over this, a quick TL;DR recap: Last week we started to look at how to structure an engineering organization effectively. We looked at structuring teams “horizontally” that is: per functional area, per role. A mobile team, a web team, a server team, QA team etc. We saw this has advantages, but comes at the cost of communication and coordination cost. Our hypothetical four-week project, turned into a hypothetical twelve-week project.

Then I wrote:

So next week, let’s flip this whole structure on its side, and go vertical. Sounds kinda random? This is management, we just throw stuff at the wall and see what sticks. YOLO! No seriously, there’s good reason to believe this may actually work, you’ll see.

So, let’s do this. Let’s take that radical turn and no longer organize around what people do (“Look at me, I’m a HTML developer!”), but what they work on (“Look at me, I build the Boards product!”).

So once again: reorg time!

Poof! gone are our web, mobile, server and QA teams. No more! We shall now have “product teams” internally staffed as follows:

Frontend engineers
Backend engineers
Mobile engineers
QA
Product Manager
UX
Engineering Lead/Manager

That’s right. Everybody cosy put together. In our previous round we didn’t cover the product roles (product manager and UX) specifically, but I’ve listed them here. If we want to go all in, verticalization can even go deeper. You can also embed the “business people” into the team, such as sales and marketing. We’ll look at one company later that does this.

Why would you do this? To exactly solve the problem we observed in our previous setup: no more external communication required. In an organization, any time we have to reach across team boundaries and depend on them, we incur a cost. It’s expensive. By having all functions sit together in one (virtual) room, we severely limit the number of interactions we need with other teams, so we can go fast.

And guess what, this works! When you form autonomous units like this, each controlling their own destiny without much outside direction, you are effectively creating mini startups. And guess what startups are good at? Going fast, breaking things (without stable infra), and being innovative!

Wooh!

So, we got there, organizational nirvana, no? The silver bullet of company organization, yes? As we all know, there is no silver bullet (except for Silver Bullet), so what’s the catch this time?

To answer this question, let’s bring back an often forgotten old family-game classic. It’s a game that was highly popular for generations, but then brutally destroyed by uncle Marv, who we shall get to in a bit.

The game: guess the org chart!

This is how it works:

You’ll all be given a piece of paper and a pencil. Then, I’m going to call out the name of a company, and you’re all going to draw the organization chart.

Who gets closest to reality wins!

I can see the excitement in your eyes.

Since you will likely have limited experience with this game (for obvious reasons which, again, we’ll get to in a moment) let’s start with the “baby edition” version of this game. Don’t worry about the label, I sometimes play this with my five-year old twins as well. In this version you don’t have to draw the entire org chart, you just have to pick one of two options: horizontal or vertical? Is this company organized by function, or by product?

Alright. Ready?

We’ll start with two companies you will all know (and maybe even love).

Amazon
Apple

Think. Think some more. Got your answers? Perhaps you even got some org chart drawing you want to show? Great drawing!

Ok cool.

Before getting to the answers, let me tell you the story of why “guess the org chart” dropped significantly in popularity about 60 years ago.

In the 1960’s there was a fellow who was exceptionally good at this game. He’d win every time. You’d tell him the name of the company, show him their products, and he’d draw you an org chart. He was good. Annoyingly so. I think we all know people like this that just spoil it for everybody else, right?

The name of this party pooper was Melvin Conway.

What is the ultimate sign of success when you’re good at something? They name a law after you. In Melvin’s case: Conway’s law:

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.

What does this mean in non-Wikipedian? You ship your org chart. Your products betray how you are organized. They’re a tell, as they’d say in poker (although “guess the org chart” predates poker by about a decade).

Once you figure this out, “guess the org chart” is no fun anymore. Thanks Uncle Melv!

Alright, back to Amazon and Apple.

Amazon is a massive company, but the amount of stuff you put out doesn’t generally scale with the number of people you employ. However, at Amazon, and this is especially visible in AWS, there seems to be a much stronger correlation between number of employees and features shipped than in other places. If you ever witnessed their yearly re:Invent conference, or ever checked the AWS services page your mind will have been blown. AWS had a bit of a head start in the cloud computing world, but the amount of stuff they keep adding to their offering is shocking. They are moving super fast and only seem to be accelerating.

Their secret? Amazon is probably the most extreme example in the tech industry that organizes vertically. While us, mere mortals are pretty braggy about “doing DevOps.” AWS has implemented BusProdDevOps, integrating not just development and operations (DevOps), but adding Product and Business as well into a single units with a large degree of autonomy. Whenever AWS sees an opportunity to add another product, they just “instantiate” another BusProdDevOps team and off they go.

So what’s the catch? There are at least two obvious ones that come to mind:

The first you’ll hear about the second you first talk to any senior person at Amazon. “Oh my, there’s so much duplication here, you have to learn to accept it.” Since each team is autonomous, they get to pick their own tech stack. Of course, teams use other team’s services. AWS Lambda runs on AWS EC2, they wouldn’t build their own data centers again. However, in many cases teams pick their own path, facing similar problems and solving them their own way, over and over again. People tend to talk within their teams, less so with people outside. So if a frontend engineer needs a button, they’d ask the team: “Do we have one? No, cool, I’ll just build it.” “But that other team has a button!” Yeah, but it doesn’t do x, y or z, and we don’t want to create a feature request and be blocked on waiting for it. We’ll just build our own. How many implementations of a button are there at Amazon? I don’t want to know. This is also externally visible, which brings us to the second catch.

Incoherence in product line integration and user experience. If Melvin Conway would be around today, he’d able to pretty accurately draw you an org chart simply by clicking through the Amazon website. You often find yourself on a page wondering “Why does this look so different? Why is this listed here, whereas that other thing is all the way over there?” Org structure! Two different teams that don’t coordinate.

Are AWS services seamlessly integrated? Is it super clear when you would use this service over that other one, because they have cleanly cut boundaries? No spoilers, ask any AWS customer to get the answer.

I think this may have recently improved, but until recently the AWS Console was also a massive tell on AWS’s internal organization. The console for each service looked ridiculously different. Some seemed to have been implemented with jQuery UI, where others looked pretty modern. There was no consistency in UX at all. Why? Again: team boundaries. I’m sure there’s an AWS Console team, but what they offer is likely just the skeleton that other teams can plug their iframe UIs into.

Now, let’s turn to Apple. When Steve Jobs returned to Apple in 1997 he made a big change:

When Jobs arrived back at Apple, it had a conventional structure for a company of its size and scope. It was divided into business units, each with its own P&L responsibilities. General managers ran the Macintosh products group, the information appliances division, and the server products division, among others.

In other words: Apple was organized primarily vertically before.

Believing that conventional management had stifled innovation, Jobs, in his first year returning as CEO, laid off the general managers of all the business units (in a single day), put the entire company under one P&L, and combined the disparate functional departments of the business units into one functional organization.

Apple (apparently ever since) has primarily been organized horizontally: by functional area.

Alright, so let’s have a look at Apple’s products, can we tell this is the case? I would say so. Even to the casual observer, this is actually quite visible.

Apple’s products are in fact a remarkable feat in consistency. They look and feel similar, and do not overlap all that much. Also if you look at the tech level that underlies it, they share chip sets and CPUs and manifacturing pipelines. At the software level, they’re effectively all powered by the roughly the same operating system (ranging from Macbooks to iPads to iPhones to Apple TVs to Apple Watches and Homepods) and same underlying development frameworks (UI Kit, C, Objective C, Swift).

When you think Apple, you will also think “design” and indeed, you’ll also find that in the Jobs era, Jonathan Ive (Apple’s lead designer) reported directly to Jobs.

Oh Conway...

So what’s the catch here, then? Apple is the most valuable company in the world, right?

Not surprisingly, I’d say the catch is speed.

Apple is also a massive company, and they sure do a lot, but my sense is they don’t put out the same amount of “features per employee” that Amazon does. Knowing what the challenges are of a horizontal organization like Apple it’s not hard to understand why. If you’re working on the next gen Airpods and need some changes in your design, you may have to wait in line, because designers are currently working on the box design of the next iPhone.

The way Apple managed to still be successful is through relentless focus. Given their size, their product line is actually rather small.

So let’s sum this up:

Horizontal organizations are great at producing consistent, coherent products, with a lot of reuse of knowledge and software/hardware components. However, this comes at the cost of speed.
Vertical organizations are great at moving fast and scaling up. This comes at the cost of replication of work and a _less _coherent suite of products.

Oh my. None of them are perfect. Why can we not have nice things? Welcome to life. However, there’s something we can do that strikes a reasonable balance.

And you will never guess what is is... More next week!

Have a great weekend! Try to get some sleep over this.

The Road to Platform Teams ep. I — The Mattermost Platformer #17

Zef Hemel — Fri, 29 Jul 2022 12:34:00 +0000

“But we have a QA team, right. Why aren’t they writing the end-to-end tests for this?”

“We have a mobile team, right? Why don’t they build the mobile version of this?”

“We have a server/performance team, right. Why don’t they write the load tests for this?”

Excellent questions! Unfortunately, asking me these questions, you stepped into a trap.

Perhaps you were expecting “because I said so” as an answer, but I’m not your dad (let’s assume, because that would be quite the tangent).

No, I don’t like brushing important question off like that. I’d like to give some context of “why the QA team can’t just do QA for us.” One reason is we don’t have a QA team, we have a QA platform team, but we’ll get to this seemingly subtle difference eventually (in a few weeks by current estimates).

So yeah, sorry, you’re on the hook for a multiple week, episodic drama entitled “The Road to Platform Teams.”

But no worries, we’ll laugh (a bit), we’ll cry (most of the time), it will be an experience like no other. And all this for the low-low price of an estimated ten minutes of your time (for this first one, estimated by Silver Bullet).

And at the end of it, perhaps you’ll have learned to stop asking me questions.

But before that, let’s start with our usual...

Cherry picks

On the mobile platform end, the full team is back into action this week, catching up on things and reviewing changes. The biggest incoming features (beside the usual batch of bug fixes) for the v2 mobile app are support for Calls (capital “C”) and search (low-case “s”).

On the web platform end, we are upgrading major dependencies, including React (to version 17 first, later to 18). This is resulting in some cascading other upgrade we need to do at the same time, such as upgrading react-bootstrap (where we were stuck at an old pre-release fork). Working towards the multi-product architecture for the front-end, we’re working more on enabling module federation. We’re also wrapping up some of the performance optimizations we worked on in last week’s performance theme week.

On the desktop platform end, we are finally getting native node_modules to work, allowing for deeper integration into each of the OSes we support. As an initial demo of this ability, we will support picking up the “do not disturb” mode from the OS. On recruitment news, we will have the desktop team double in size on August 8, when Tasos will join us 🎉 Which is, incidentally, the same day that Pantelis will join as the web and desktop engineering lead (again: 🎉). That’s right, we’re onboarding half of Greece in a single day (the other half was already working for us in the SRE team).

On the server platform side, we’re moving fast and breaking things. As we’re doing deep refactoring, moving a lot of code around as part of our Q3 effort to refactoring things around the multi product architecture we’re also discovering that a lot of this code was not properly covered by tests. Surprise! Of course, we are using these “discoveries” as opportunities to fix the gaps in our test suite.

On the QA platform side, we’re now starting to report some interesting stats for a release, including: the number of commits part of the release, how many of commits have been (automatically) determined to require tests, how many of those commits in fact have tests, how many commits have been verified by QA, for how many the suite of end-to-end tests have been run. For now these are just numbers without interpretation, but as we are monitoring these trends over time we’ll be able to flag issues and give recommendations.

Going horizontal

How do you structure an engineering organization once it gets too big to be just #oneteam? There’s many ways to do this, and it’s usually a bit of a journey for a company to find a model that works for them.

I quickly described the setup of engineering at Mattermost all the way back in The Platformer #1. However, this is rarely the setup companies land on from day 1, so let’s relive such a journey in the compressed time of just a few weeks. Let’s see where various setups succeed and fail, and iterate from there. Then, ultimately, we’ll hopefully have a better understanding why Mattermost is structured the way it is. While not perfect, it makes sense for us right now. We can always reorg tomorrow (but no spoilers).

Are you ready?

The most obvious first iteration in any organization is “let’s go horizontal!”

At the architectural level, we often slice our product in horizontal slices: we got a frontend layer, a backend layer, a cloud/operational layer, and QA to make sure it all kinda works. That makes sense, let’s split our teams up that way as well! People in these layers all have the same job anyway, more or less, so why not? YOLO!

So, we’re going to put web frontend engineers in a web team, all mobile people in a mobile team, all server people in a backend team, and all QA people in a QA team.

There is plenty of upside to this. It brings people doing the same job together. Working together daily, they share knowledge, can mentor each other, talk about their problems with their technology stack and fix them. Their manager is likely an expert in their field so fully understands their every day problems. The team recruits its own members so all is nice self contained. Suite!

Let’s do it.

Alright, now we have to ship a feature. This one’s a bit bigger, it doesn’t just affect moving a button from one side of the page to another (as is usually the case), it also requires some new endpoints in the backend. It also needs to be implemented in the mobile app, and because it’s critical, we have to make sure that it works properly end-to-end — so we’ll need end to end tests, QA team!

This will require a bit of coordination, so in comes 🥁 management.

Hello! 👋

Management says: “This looks fun, we’re going to turn this into a project!”

We’re going to bring together the leads and architects of all teams and we’re going to carefully plan and spec this whole thing out. Since communication and coordination between teams is rather expensive, we are going to have to nail down these specs quite precisely. Then, we’re going to create a dependency map and going to do some planning. Have you heard about the magic of the Gantt chart?

Both the frontend teams (web and mobile) rely on back-end endpoints being present. So in the first week, the backend team is going to do its magic. Then the next week, this will be ready for the frontend teams. They will work on this in parallel (efficient!) for the next two weeks, and then finally after that, the QA team jumps in to test this thing and automate that testing, which cannot take more than a week.

And voila (as the French Canadians say), in four weeks we’ll have ourselves a feature!

Off we go!

Sadly, things go wrong pretty much immediately. There is a major incident in cloud, requiring the attention of half the back-end team for pretty much a week. There’s still progress made, but because we also have five other projects running at the same time, some senior (code for old) manager person needs to step in to decide which of these take priority. Sadly, our feature is low on the list, so we’re delayed by a week. The web, mobile and QA teams who had planned their work based on the assumption that the backend team would “do their job,” now have to overhaul their planning. Those backend people, they just cannot seem to estimate their way out of a paper bag...

The next week, luckily, the backend team delivers its endpoints. Yay! The frontend and mobile team are unblocked and can go, go, go (ironically not using Go)! Sadly, as these teams build out their frontends, they discover some problems with the endpoints.

“But it’s exactly done according to the spec!” the backend team yells across the team fence with some delay. “You agreed to this!”

“Yeah, well, we didn’t catch this until we actually got to use it!” the frontend teams yell back.

“Oh my,” the backend people think, “these frontend people cannot anticipate their way out of a paper bag. How hard can this stuff possibly be with just a single user interacting with their code?”

Ok fine, next week we’ll have time to make these changes, we got actual hard scaling problems to solve, we can’t context switch all the time.

In the mobile team, they decide not to wait, and, for now, mock the endpoints as they will (hopefully) work down the line. In the web team they decide to just pause development and move to other projects for now.

Another week, more incidents. The backend team doesn’t have time to implement the endpoint changes. More replanning is happening across all other teams. The web team learns about the mobile team’s mocking approach and starts to implement their own mock using a different framework that’s more “webby” than the one the mobile team used.

Another week, and the backend team delivers the updated endpoints. The mobile team removes the mocks to make sure everything works as expected. It doesn’t fully, but with a few days of work, manages to fix everything. The mobile version is ready to go.

The web team had to schedule other projects in the mean time, so they won’t get to it this week. QA is preparing to write tests, but likely will start with just covering mobile, because web isn’t nearly ready yet.

Another week, the web team throws away the half-finished mock implementation they started to work on, because the actual implementation is now ready. They implement the feature and they’re done!

Another week, and now it’s the QA’s team to shine. The feature is ready, and other teams have moved to other things. But guess what, the QA team uncovers issues. Indeed, there are bugs. Both in the web and mobile clients. They create tickets for both the mobile and web teams.

The next week, the mobile and web teams have their triage and planning session. They noticed the reported bugs. “This is probably a server issue!” they say, and reassign it to the backend team.

The backend team has their planning session. “What was this project about again? Ah right, I remember this. But we implemented this according to the spec, so it must be a client issue. Let me rerun our test suite. Yep, all good. Let’s assign it back.”

The next week, management (in one of their various fancy JIRA dashboards) notices the bouncing back and forth of the bug tickets and sets up a meeting with all involved to resolve the issue. They resolve it. Fixes are scheduled for the next week.

The next week, the fixes are implemented.

The week after that week, the QA retests things and all seems ok.

And voila (as the French Canadians say) after twelve weeks, we have ourselves a feature!

While a lot of what happened here (and this greatly simplifies what happens in real software development) can be attributed to the inherently unpredictability of software development (external factors, naive estimations), the glaring issue here is communication and coordination cost. We have 4 teams with their own planning cycles, priorities, habits and mini cultures. They cannot deliver a feature without heavy coordination. Because they are organizationally distant, they communicate through lossy and stringent “interfaces” (specs, JIRA tickets) interpreted through often cynical lenses. Obviously, if they’d all be using Mattermost none of this would have happened, but we cannot all be so lucky (wink wink).

There must be a better way!

Indeed, there may be. So next week, let’s flip this whole structure on its side, and go vertical. Sounds kinda random? This is management, we just throw stuff at the wall and see what sticks. YOLO! No seriously, there’s good reason to believe this may actually work, you’ll see.

How’s that for a cliffhanger 🤯

Have a great weekend!