Forem: Standup & Prosper

Measuring team success

Warren Parad — Tue, 28 Jun 2022 12:27:25 +0000

It should be a inevitable conclusion that you to need to measure the success of your teams. After all, we get what we measure. And having successful teams sounds like a good thing.

Although what do we mean by a successful team? Do we mean high-performing? Do we mean happy? We can’t measure a team by a metric to find out of if they are successful, that’s a catch-22, we need to start with what it means that a team is successful, and then we can create metrics to match that.

You may be able to fill in the blanks a bit, but it’s worth diving into, what is a successful team?

An ability to learn and adapt as the business and technology change
New features can be delivered quickly
Low number of bugs reported from users in production
Team members are motivated and potentially even happy
When there is a problem the team steps up to deliver a fix
The team is a leader in their domain, they innovate
New team members onboard quickly and effectively
Low turn over rate for team members
[Bonus] They work on the right things at the right time

Have something else, add it to the list, what’s important is clearly state what you think a successful team means by looking at outcomes. Don’t say something dumb like “prioritizes backlog items”, “has retros”, “estimates correctly”, or “does sprints”. These are the activity, instead focus on outcomes => “delivers meaningful business impact”, “team morale improves over time”, “delivers frequently”, or “applies the right structure at the right time”.

The worst metrics

Now that we have a shared understanding of what a successful team looks like, what metrics should we use to capture this? The right metrics should be focused on only the things we said, and the explicitly exclude anything that isn’t related. The critical fault would be having a metric that the team can’t control the outcome. Here I’ll start with what metrics should never ever be used, and why.

Objects & Key Results (also known as OKRs) tracking and key results completion
Sprint goal completion
Velocity — Number tickets or story points completed
Lines of code code committed

Why?

I tend not use sprint goals nor OKRs. While I think OKRs are fantastic for driving alignment, transparency, and correct prioritization among teams and the business, their “delivery/status/value” is coupled to the two things that are completly out of the teams control:

The correctness of the OKRs
Dependencies outside the team

We actually don’t want OKRs to be at a 100% nor do we want them to be at 10%. OKRs success is our ability to drive motivation by trying to get to 100%. If we never get to 50% then we know we critical overestimate our abilities. If we get to a 100% before the quarter is over, we’ve underestimated ourselves. The next OKRs adjust for this, which means we will never be successful by an OKR metric.

Additionally OKRs exist outside of one team, they involve deliveries and collaboration with other teams and users/customers. And also are evaluated based on changes in Product visions and Business strategies. You cannot under any circumstance measure success of a team by something they can’t control. That’s just unfair, even the perfect team can’t sustain problems caused by faulty expectations on metrics. One extreme example that proves that point is a critical pivot that invalidates an OKR would means that team failed, without room for discussion.

Further, successfully matching the Objective’s KR, tells us only about the ability to predict the business success than actually measuring if the team is responsible for it. If we set a goal and we hit it, that only means that we are great at setting goals, not at hitting them. If you want to run an unhappy team, keep setting bad goals. The OKRs won’t be hit, and your outcome will be blaming the team. Team had nothing to do with it.

Sprint goals and velocity go the same way. They are great for alignment, but they are absolutely terrible for determining success. Again, because both of them look at estimating what you will do before hand. We couple our ability to estimate to our definition of success. A team being able to effectively estimate their work is useless for determining success. And a real successful team doesn’t need to estimate. Even if estimates had a value, we are going to get them wrong, by a lot. And most importantly meeting these goals just means we estimated well, not that the team is performing or successful. If we want to call a team that estimates well — successful, go ahead, but that’s the only thing they’ll be good at, and we know in the software this has very little value.

And I hope that I don’t need to get into Lines Of Code or anything that is hands on technical related, these don’t tell us anything about the value of team delivery, just the raw output. Don’t use output, use outcomes.

One more problem!

Not only does estimation contaminate the ability to use OKRs, there’s a second issue. Since you’ll be evaluating team success and performance based on this, your teams will by design interfere with the process. Your teams are already deciding what the OKRs will be, and if they are to be judged on them as well, then you’ve set yourself up for a critical failure mode. The reinforcing system design problem is then building OKRs to make the team look good or bad. If you can intentionally make a team look good or bad by a process then there is something wrong, it should be as objective as possible.

As a result, a common organizational antipattern is to take the team being judged out of the process and assign them sprint goals and OKRs. Doing so renders the OKRs worthless, since the whole point is alignment; assigning them prevents teams from being on-board. Further, itprevents them from growing to the point where they know what makes a good OKR in the first place.

Usually this happens in places where there is a lack of understanding of what effective teams look like. And OKRs that not sourced from the team, created by the team, measured and record by the team, is indicative of toxic leadership.

So what are the right metrics

We’ve uncovered an important criterion in the wrong metrics listed above, and that’s anything related to estimation and planning. We should never measure a team on estimation or following a plan, but instead on outcomes. Since it is really hard to measure “successful customers” or “value delivered” we need proxies for this. Some companies like to use “revenue”, but that’s conflated with the business and Product-Market-Fit which may be non-existent, or the company is accidentally successful, like most out there.

When I started engineering a long time ago, there was no consensus on what was important, we could talk about it, and get pretty far, but we were only touching on the boundaries of what that is.

We knew it had to do something like speed of delivery, not wasting time, and “quality”. We also knew concepts from the Toyota Way, would help introduce Lean to software in a good way. But talking about removing wastes is different from measuring success. Sure we want the wastes to be zero in the team, and we see how things like hand-offs are wrong, because they create transportation waste. Or doing work that isn’t delivery directly to users is wrong, because we stock on feature inventory waste.

Now that we have some understanding of what makes good for measuring team success and what’s atrocious we can land on the following (also known as DORA metrics, and heavily discussed in the book Accelerate):

Lead time for changes— How long it takes the team to start working on something after they know they want to work on it.
Deployment frequency — How often a team deploys to production, visible changes for users, getting those changes out, not just merging.
Change failure rate — How often after a deployment there is a new bug.
Mean time to recovery — When there is a bug how long does it take to recover.

And that’s basically it. We know these are also the right metrics, because they are measurable, and they don’t represent bringing another process, which in itself could be flawed. There is zero estimation here, other than the cases where we don’t have precise measurements for them.

And that isn’t to say that you have to perfectly measure these, just that you should be thinking about these when measuring the success of a team.

[Bonus] The advanced metrics

Those are a great start, and I actually don’t stop there, there are two other metrics I like to add to this list for my teams, which prevent other standard team disfunctions. One common disfunction pattern is when a team is highly praised for always working on the right thing and all their customers are happy, but every team that works with them hates the way they work (i.e. the output of their work, not the team members themselves). Other teams in the same org could even think that team is doing a good job, and that’s because the team is spending all their time fighting fires.

Long term any team that spends an increasing amount of time fighting fires is creating a run-away problem that will eventually cripple a company. If more and more time has to be spent on resolving issues with already created solutions, then you will run out of time. And usually teams realize this already when they have no time left. That’s why one of the most important metrics I focus on is:

1. Percent time spent on fighting fires

This also solves the problem of having great values for the above metrics, but your organization thinks the team never works on anything valuable. Teams that are quick to release, but have a poor design process, tend to have high positive optics in an org but usually end up being a critical bottleneck. This is a frequent problem pattern.

The other angle from this is value delivery. So far most of the metrics are internal facing, with good reason, external facing metrics are contaminated with problems like is the product the right one , do we have the right users, does the business strategy make sense? If we have problems in one of these areas we can’t let those affect our judgment of the team’s success. Nonetheless, I feel the need to have a metric that tracks if the team’s work has value. Because, if a team isn’t completing valuable work, we’ll know they won’t be happy. So, my bonus second metric is:

2. The number of support/user/customer requests

Along with this comes the accuracy and time to answer, along with the number of people required to answer. We want a level distribution across the whole team, quick responses, with low involvement. Any team member can and should answer, and they should be quick and obvious answers.

We can use this as a driver to make sure our documentation (from external facing docs, to internal ones, and api specifications) makes sense and is relevant. Using this metric tells us what we are building, or how we are thinking about it isn’t right. User Confusion is a real problem, and usually the source is that instead of the team owning the product, the product is owned by someone outside the team. Ownership may look like a Product Manager that is incorrectly focused on features rather than product vision, or a Business Analysts focused on customer insights rather than capturing the potential value for the business. Product ownership is aligned at the team level, no where else.

Tracking these over time are ways I know that the team is spending their work in the right area. There are some longer running aspects such as engineering retention which can be hard to link to a specific team, and also team member growth within the team. But for hard metrics, I usually stop here, but these other aspects are great indicators, so let’s talk about them as well.

Leading indicators

That wraps up the quantitative objective metrics. However, it can be a challenge to jump in and introduce these and an obvious followup is Do we need to track them? The answer is almost always, a resounding NO. It is almost always enough to talk about them, without actually putting up numbers. There is no small number of SaaS products attempting to track them for you, and some graphs look nice, and you may be tempted to set them up and share them externally to prove that your team is successful. However, doing the tracking these metrics successfully, is actually a metric in itself, and not one that is listed above. Being a successful team is very different from proving it, and unless there is a concrete problem you want to fix, unnecessarily tracking metrics to prove the team is successful is another indicator of toxic leadership.

So, if we don’t actually track these, what are some good indicators for knowing that the right things are happening? You can start by looking with your eyes, it isn’t hard to know the direction to make things better, but there are also obvious signals that can be utilized:

Are engineers growing/being promoted (obviously with the caveat of being promoted for the right reasons.) Here I use a “gut check” based on who is attending/meeting/sharing/talking/moderating/leading. If it is always the team lead, then I know there is a problem. If other people are taking initiative, that’s a good sign.

Are the teams innovating either from pulling in/testing new technology or leading/experimenting with new product ideas. These are more subjective, but having a feel is a requirement to being able to judge the success of a team. If you don’t know what a team innovating looks like, then you probably aren’t the right person to be judging if a team is successful.

Free capacity is the ability to burn down tech debt at the right time, as well as working on “important things”. If teams are moving slower than new features are needed, then their process isn’t scalable. Since I expect software teams to automate their tasks, not just deliver, their responsibility is to put themselves out of a job. With goal of working on more important things over time.

If they don’t have the ability to turn around new features or prioritize them, we are either making too big changes and not being agile, or have problems with our previous work. If previous work is getting in our way, it tells us the teams aren’t solving problems in a way that matches the expectations of the business or how the product is conceived/how our users think.

For additional indicators, the Spotify Health Check is a great place to start, with the caveat, obviously you need to adjust for the specific culture you want to cultivate.

Wrapping up

Absolutely don’t use metrics that they themselves are flawed in creation. If we depend on measuring success by a flawed mechanism (estimation for instance), then we couple the success of a team to a meaningless and often wrong metric. You won’t get a successful team, you won’t actually know if your team is successful, and worst of all worlds, no one will be happy.

The Required team meetings

Warren Parad — Fri, 24 Jun 2022 13:09:03 +0000

The required team meetings

There is no shortage of advice out there on knowing when to call a meeting. And in the remote-first world async communication is now the norm, every meeting we have is up for review. There’s already come great advice on when to call a meeting, so let’s dive into what this actually looks like in practice.

It’s critical to note that any meeting can be effective if run correctly, and running any of these poorly will result in a huge waste of time. If run correctly, these are a great list of meetings to have. That being said, while there can be other meetings, usually having them is an indication that there is likely a critical fault somewhere else. You can’t fix a problem with clarity or transparency by applying the patch of sometimes sharing information late.

Categories of meetings

The discussion here places importance on a few different categories of meetings. I’ll go into details for each of these categories and share the importance of each one. Further meetings not listed here are great ones to challenge their usefulness.

Team meetings

Let start with team meetings. I’m assuming you are working on a team. However this isn’t something that should be glossed over. Many groups of people are called a team, even though there isn’t anything about them that is team oriented. It’s important to realize first if a group of people is a team. That means they share direction, purpose, values, identity, and responsibility. And to achieve that, they share alignment, meaning-meeting frequently is required at the team level. The necessary meetings for the team are:

Weekly team sync (~1 hour)

The weekly sync is where you and your team come together to create and affirm alignment. What are we working, are the priorities still the priorities, are there any changes to what the work is, or new things to consider. Using this time effectively means having the four quadrants answered:

What are the top work priorities — These priorities must align to your OKRs or business objectives
Are there service support issues — What are, if any, urgent production related issues. Are there reliability metrics trending downwards?
The 6 week initiatives — Are these still the same, they give clarity to what happens after the current work
Team health — How is the team doing, are there any red flags, callouts, problems that should be immediately discussed
[Bonus] — Are there any initiatives that individual team members should volunteer for and take ownership over. Usually there aren’t any of these, but one may come up time to time. Such as fixing a huge portion of documentation or training another team.

I usually recommend starting with service reliability, then navigating to team health, then 6 week alignment, and last to suggesting and voting on priorities. Everyone can suggest, and then use some sort of dot voting to pick only the top One or Two things. The Team Lead has the deciding vote. Everything else should Not be worked on. If the team didn’t prioritize it, then it doesn’t get worked on.

The point of this again is alignment. Team Health, then reliability, then current work priorities is the order. Never start new work when the team or existing services can’t support maintenance. This builds up more fires that need to be fought. You’ll notice here that talk about issues or tickets is strictly not part of this meetings. And with good reason. If the team is aligned on the work, then anyone can prioritize tickets into your “prioritized” or “ready to be worked on” column on your kanban board. In most cases the right tickets are already there.

Prioritization can done by the Team Lead or by another member on the team. You don’t need to spend time hashing out what a ticket means, that’s a separate meeting. If a ticket isn’t already clear when you get to this meeting, then it isn’t ready to be worked on, and that means shouldn’t be discussed here. If you can’t walk away from the meeting knowing which tickets need to be created or prioritized, improve the moderation of the meeting.

If you run out of work during the week, hold another meeting, don’t ever create more initiatives to avoid another meeting, that destroys alignment and creates confusion on priorities.

This structure closely matches the one used in Radical Focus, the premier book on creating and utilizing OKRs.

Bragging Session (~1 hour per month)

Our CEO created this concept to help everyone identify personal wins. Talking and discussing about what you’ve done great is a fantastic way for you and everyone to realize all the hard work that has been put in. The format is that once a month one person will recount all the great work they’ve done. We recommend taking credit for anything and everything. It doesn’t matter if you had help, it doesn’t matter if you weren’t the key player. You contributed to the success, time to talk about it.

Retrospective (~1 hour per month)

Retrospectives are the way to review everything related to your work. Everything is up for review. Discuss what works great that you want to continue and what isn’t working so you can focus on changing it. If the same topic comes up more than one or two retros in a row, it’s a sign you have too many retrospectives.

In person Standups (~1 hour per week)

Standups or Dailies is the practice of sharing the One most critical or important piece of information with the rest of the team. It isn’t a discussion board, and it never is about mentioning tickets. If you are mentioning tickets, you aren’t getting the value out of your standups. Also standups take two forms-async and in person (i.e. face-to-face over the camera). Twice a week for a face-to-face for 30 minutes each is a great opportunity to potentially align, discuss the need to dive deeper into other topics, or bring up something critical. You really want to limit this to one topic per person. If you feel like you have a lot of things, try to figure out what’s the most important for the team, are you blocked?

Async standups

The other three-to-five days of the week, have a daily async standup. Where team members are encouraged to fill out or engage with async written context from other team members. Again, these have the same importance and expectations as in-person standups, but are done async. Of course we recommend our favorite Slack standup app Standup & Prosper which we love because it is totally free and has everything we could ever need. The caveat here is, if you have team members with active times more ~6 hours apart. Your best bet is to have separate teams, and therefore separating the timezones. If there is overhead to collaboration it won’t happen effectively, and you’ll optimize for the wrong solutions.

[Optional] Innovation Day (~8 hours per week / month)

Every couple of months I recommend a full day (~8 hours) dedicated to working on anything that isn’t a priority. Being mindful that work can’t increase the load on the rest of team. Work that generates Pull Requests for outside of innovation time is a clear distraction. In general, leaving 20% of your time as free capacity dedicated to innovation is the right amount. However, not everyone innovates spontaneously. Maybe you already do something different every day, but if you are a person that needs scheduled time, having a single day can be the right approach for you and your team. If used effectively, this time may be indistinguishable from the work that your team is normally doing. But if you aren’t finding opportunities to innovate, improve your tech debt, or fix that really bad but just needed to be fixed bug, this meeting could be for your team.

Career and Team growth meetings

Direct report meetings (~3 hours per week)

Using Dunbar’s number, we know that the maximum number of highly trusted relationships we can have in a given context is ~5. So as a team lead that’s 5 hours with direct reports, and 1 hour with your career manager, making this 3 hours per week. There are lots of details out there on how to hold effective career meetings, the point of them, and strategies for what to focus on. The overall goal is for you to become a better version of yourself. Frequently, that means figuring out what you want to do and how you want to change in the next year+ timeframe.

Skip level meeting (1 hour per month / ~2 hours a week)

As an engineering director or Lead of Leads, you manage the team leads. With Dunbar’s number again, you likely have ~5 teams. While you can support up to 15, you’ll end up with way too many meetings. In the case you have a 5 teams, each with 5 people that’s 20 skip level meetings. Quarterly is usually a great cadence for this, remember this isn’t the only opportunity to get feedback, but it is the one meeting you are explicitly making time just for those that you lead.

360 Feedback (~1 hour per month)

Every month on a rotating basis, one team member gets the opportunity to hear back from their team about their successes and opportunities for improvement. A pattern that works is everyone keeps track of feedback that they should have already shared, and then during the meeting, the team points out topics in the form of Stop, Start, Continue. By grouping these together in a joint session we can see what everyone in the team cares about, and where someone has the opportunity to improve.

Lead-to-Lead meetings (~1 hour per month)

In the case you are Chief in Charge, meeting and driving alignment between yourself and other teams is critical for the organization and company success. Feedback on your own opportunities for growth as well as opportunities for your team to improve should be on your radar. In some cases, these may be direct mentorship, from inside or outside your org/company, in other cases they can be direct alignment to smooth over issues cross-team/cross-org. These apply in orgs at larger scales, for example if there are multiple 5-team orgs in your company/division.

Organization/company wide alignment meetings

There are of course organization wide or company wide meetings that are useful, while time is flexible on these, what’s always important is that in every meeting there should be a clear agenda as well as value for every member in attendance. If there is someone that doesn’t get any value by what is being shared, then the meeting doesn’t make sense. So we should always be concrete with the value participants will get. In this regard, all these meetings are 100% optional. If you come-you participate, if you don’t participate, you don’t get to have an opinion later.

Organization Direction (~1 hour per month)

After the team leads have completed designing the business initiatives (or OKRs if you are using that framework) and they have been signed off by the Engineering Director (who is a proxy for Product, Business, and Technology strategies), then they have to be published and announced. This happens at most four times a year, and during the times when this isn’t happening, review of the objectives and their results is the critical factor here. Are we on track, are we still doing the right thing? Do our OKRs still make sense, or is there is a mistake. This is the opportunity to question the OKRs and identify any problems with the current approach. This meeting should not present new information, because everything should already have been shared, but should seek for feedback and potential discussion on improvements. Think of this as the planning and/or retro at the organization level.

Book club or guild (~1 hour per week)

Guilds are a collection of people that meet on a weekly or bi-weekly cadence to improve their understanding of a topic. They don’t come together to solve a problem, they come together to learn. Anyone can join any guild or start one. Common guilds that I have started in my past include Quality , Deployment Automation , Lean software development , Building and using agile , Customer Support strategies , but guilds can be about almost anything. They don’t have a leader, it’s just a group of people deciding to meet.

Lightning Talks (~1 hour per month)

Dedicated time set aside for short 5–20 minute presentations on any topic. They are a great opportunity for cross-org collaboration and knowledge sharing. It doesn’t matter what the topic is, but remember that the goal is value for your audience. Don’t just talk about yourself for twenty minutes :).

Ad-hoc meetings

Ad-hoc meetings are everything that is not recurring, and every one of these should have a strict agenda, a plan on how to have the meeting, and a concrete goal of the meeting. The outcome of every meeting doesn’t need to be action, but it does need to have a followup. If the goal of the meeting is to make a decision, then the decision needs to happen, and time allocated for it. Don’t schedule a second meeting.

Establishing a team identity
Working through a pull request review real-time
Designing a new architecture
Identifying new work from support requests, customer feedback, or technical investigation
Discuss how to solve a customer problem or new feature and creating the necessary tickets
Creating/hashing out/reviewing an internal RFC
Establishing new OKRs for the OKR cycle
Interviewing and handling candidate debriefs

Almost anything is an ad-hoc meeting, focus squarely on the value, and targeted agenda. These are just examples above. Anything that isn’t listed in the recurring meetings above, shouldn’t be happening on a recurring schedule. There are of course rare exceptions when two teams are working very closely together, but these are rare, and time-bound for instance, getting customer feedback from one large customer on a monthly basis. Time bound this to 6 months. For something longer and ongoing, you’ll want to discuss concrete ways to more effectively get that feedback, there isn’t enough time to meet with everyone all the time.

Meetings we shouldn’t have

There is also great advice out there on indicators that a meeting is problematic. There’s the disfunction of the types of Meeting Creatures present, but also the question if the meeting itself is the right one. We know we shouldn’t have these types of meetings:

One person talking most of the time — This is a broadcast, if feedback isn’t an important aspect, this doesn’t need to be real time. Send out an update before hand, and then meet to discuss.
Meetings longer than an hour — The human attention span doesn’t last that long, after getting about ~30 minutes of new information, we need to process it. If we are spending more than this time, we aren’t getting to the processing step.
Meetings back to back — Same goes for the above, there has to be processing time built in. Take a second, sit down, and process the information before going to the next one. Having some back to back meetings during the day is unavoidable, but if you get done with all your meetings and you only have 1 hour of time remaining to get the processing in, you’re in too many meetings.
Recurring meetings that always get cancelled
Recurring meetings that people don’t know why they are there — Everyone who came should know why, if people are going to a meeting because they were in invited, you may have a toxic culture. Everyone should come to a meeting because they either have relevant value to provide, for instance decisions that affect them. If someone can’t provide input and get feedback, then they can read or listen to the meeting at 2x speed after it is done.
Meetings where people bring laptops, or performing a second activity — If people aren’t paying attention, or don’t think they have anything to gain, this is a sign that the meeting is a bad one.
The meetings mentioned above at higher frequencies — For instance it is relatively common to have a retro every week or every other week. Anything more frequent than once every month, means an extra 1 hour per person in the team spent on potentially rehashing already identified problems.

Conclusion

Take a hard look at your current meetings you have to see if there is an opportunity to shuffling them up. If you have multiple backlog grooming sessions, you might have identified a new failure mode that could be addressed. In some cases, people secretly declare meeting bankruptcy and stop going to anything. This also isn’t a great pattern. The first step to change is identifying that you and your team are in control of the meetings you take and how you interact and collaborate with outside experts, like product managers, leadership coaches, vendor account managers, etc… Once the team settles on a shared understanding, broadcast that, get feedback, and adjust.

Originally published at https://blog.teaminator.io.

The Punishment of Building a Slack App

Warren Parad — Sat, 21 Aug 2021 14:47:27 +0000

Three years ago, we set out at Rhosys to build the perfect tool for team performance and growth. Turns out lot’s of companies were interested in saying they want to help their teams and team members grow, but but were much less interested in doing anything about it. What they were interested in however was an app that they could force status tracking via standups. We’ve gotten a lot of requests for our Slack bot — Standup & Prosper, to make standup questions mandatory, but since that isn’t core to standups or productive teams, it’s one feature that will never be implemented.

With COVID and a huge switch to remote working environments, teams and more specifically managers took for granted how they were tracking performance. Frequently, we saw companies using “engineer in chair” as the effective metric for “working” and “sync online standups” as the mechanism for reporting “I did my work that was assigned.” This was terrible for so many reasons, however in an attempt to regain control and micromanagement of their teams, these managers wanted to do status tracking over Slack. Slack doesn’t remotely offer the ability to do status tracking in this way, but there are no shortage of tools available in the Slack App Marketplace for them.

At Rhosys, being a remote first company, we also were interested in tools to enable high productivity, and because Discord and Microsoft Teams were unusable (one of them still is today, but I won’t say which) we invested in Slack. Multiple times a week we would hold async standups, and wanted an automated solution for that, we thought to ourselves this is simple and easy, it ought to be free. And it was for a time, we experimented with five different standup apps looking for the best one that fit our small and simple use case. And we found one, and then used it effectively for six months.

One day, they said: Pay us or we’ll shut down your account. Well that was the end of it. For an app that does one thing, and doesn’t even need to do it that well, it shouldn’t cost anything. Like usual, as an engineering team we thought we could it better. There are some important qualifiers here:

At the time we were focused on building team tools, so this was our core competence.
We calculated the TCO and it was huge, and we used that to evaluate doing this in the first place.
We wanted to provide the community with a free standup app, because it was a great marketing opportunity for our main product at the time: Teaminator. (We now solely focus on user security SaaS with Authress.)

Our app

We set out and started building our standup app, one that was simple, cleverly named: Standup & Prosper, and with our core values provided a conversational experience with an AI to record your async standup.

Like all great microservices, it was done it under a month, but took another month to get released to the Slack marketplace. We were able to accomplish it quickly because we already invested in build/integration with:

User Management and Permission — Authress
External Messaging & Notifications — Our notification scheduler (almost open source)
UI experience for Slack integration — Requires some OAuth flow setup
Platform for deploying microservices — Gitlab + AWS Architect

You start out small and make it easy to handle integration events from Slack and post messages to users and channels.

Handle events from Slack
Send messages as DMs to users
Send messages to standup channels

And you’re done?

We’ll almost, now I’m share about all the issues we’ve found and how we went about dealing with them.

1. Slack Status API Lacks CORS support

Frequently, and I mean a lot, Slack is down, or otherwise unreliable, messages take too long to be sent:

(Don’t trust the 100% number, there are frequently reports that don’t make it into the calculation) It’s a terrible user experience, but it’s understandable for realtime messaging app. However, we still aim to make it better. When Slack is having a problem and a specific problem such as: Slack Conversation APIs don’t work, everything may seem fine to your users, but it still looks like your app is broken.

So at the top of our portal, now you see a link to the status:

How do we know Slack is having an incident? Well there is the Slack Status API. Great! Except that it doesn’t support CORS, because the reason was:

However, I’m afraid that we can’t enable CORS headers for our status.slack.com API.

Can’t seems a bit strong for this, of course it should be possible, but it would help me if you shared why you would choose to not enable this feature. There shouldn’t be any security concerns given the nature of this app.

We actually don’t support CORS for any of our API endpoints. Whilst it may not seem security related, we do have policy concerns about how this data is being retrieved.

Great, so there’s an API, but I have to write a one line wrapper to fetch this data myself to expose to our UI.

2. Non-Standard API and Inconsistent Payloads

Everything is a status code 200 returned from Slack except when it isn’t. It also frequently doesn’t even accept json requests, but will return application/json except when it doesn’t and the whole body is re-encoded into another string. But then Slack API will also fail, so that’s great. You have to explicitly check for an ok: true , in the response, but sometimes it’s ok: false , because you queried for a list and the list was empty…Oops.

You also might be getting lots of different types of events, turns out you will get all the following problems:

Events with undocumented fields
Events with three fields all called team_id but with different IDs
Events of different types have the same property in different named fields
Events for messages don’t have a timestamp
Events arriving over 10 seconds late
Events from workspaces that you don’t have an app token for any more

3. Undocumented Errors

Welcome to the land of undocumented errors. While Slack has tons of error codes documented for every single endpoint, things like user_not_found or inactive_account. You know you did something wrong in those cases. However, what will you do when you get back an error such as admin_only or org_login_required. Turns out you get these when other ridiculous things are happening such as a Slack account migration or Enterprise grid conversion. Why you can’t use the api at that time is beyond me, but you can’t.

Oh yeah, turns out the issue was there was a bug in the Slack source:

For this admin_only error, that was the result of an error in the Slack core code where we were supposed to show a proper error to the installer who happens to be a guest user.

4. User Login returns incomplete information

Welcome to the world of confusion. You want to log the user in with Slack Login. There are tons of pages dedicated to it. But it turns out that there are two login options:

Log user in — identity.user
Install App — oauth.exchange

When a user installs the app, you have no idea which workspace they installed it in, who the user is, among other missing details. Of these include missing EnterpriseId (if it is an enterprise). Further if the user installs the app, but your site fails to save the token, too bad. Slack doesn’t give your app permissions, it generates a unique token for the workspace. Even though you already have a clientId and secret, now you have to save additional information for every workspace.

Further, since you can’t know if they have installed the app yet, you always want to send them to the app install flow, send them to the user login, and you’ll end up with a user in your app, but no ability to call any of the Slack APIs.

When the user logs in, you don’t get to know simple information about the workspace, even if the app is already installed, and that information was available during installation. Why? I don’t know.

This means if you want to grant a great experience to your users, they have to log in twice. Same flow, once to install the app, once to just log in.

5. Lack of attention to critical features

Slack apps are lacking in security (until recently there was no support even for refresh tokens). They force the unnecessary generation of tokens for the app to save and then reuse later. If you accidentally expose one of these tokens, there is no way to rotate it. Slack was working on the improvement, and even released it, then told everyone it was the new thing called Workspace Apps. But after telling everyone to use it, they shut it down:

We had a prior app framework called “workspace apps” which was a separate project that required the creation of a new type of app. However, due to the fact that it didn’t take advantage of the current app framework, we decided to deprecate that in favor of porting the features into the existing app framework. We had announced the deprecation of workspace apps in this blog article: https://medium.com/@SlackAPI/aabc9e42a98b

6. Chat.postMessage is not idempotent!

You would think for a messaging service there would be a way to guarantee delivery of messages, every single message without repeating messages to a user. There isn’t. That’s really the end of the story. It would be so trivial on Slack’s slide to store idempotent tokens and then ensure not to deliver the message to the user twice, or even do the delivery, but hide duplicate messages on the UI either by code or by hash. There is no such thing, since Slack apps need to handle multiple retries from Slack on new message events (more on this in a second), idempotent message sending needs to exist in every single Slack app.

So we have a database table specifically to track and handle it. But as everyone knows that isn’t the right way to handle idempotency (you can’t as effectively do it on the client side), so it is sub-optimal. For us there is one edge-case that doesn’t work — Message hasn’t been sent yet, failed to deliver message to slack, failed to setup async processing in AWS, failed to delete the message indicator from our database. It’s 4 failures, and at that point we are confidence there is probably something wrong with the world, so failing here results in an CRITICAL exception sent to AWS CloudWatch and then forwarded to our on-call handlers. Not totally sure what we would do in that situation, other than /sigh in our slack workspace.

7. Fetching messages in a DM does not work with UserId

You can send all the messages you want to a user, using the user’s ID, but you cannot get the list of messages sent there using that same ID. You have to first convert it to a channel ID, using conversations.open , and then pass the result as the parameter. However, even if you manage to get this working, don’t. It turns out the conversations.history endpoint isn’t well supported:

75-second peaks on response time

I wouldn’t recommend using this endpoint as a critical usage in your app. We made the mistake of doing that, and while most calls are around ~100ms, every 10th or 20th one can be as high as ~5000ms. When you are making a realtime conversation app on top of slack to handle seamless AI conversations, not a good story. This is where all the performance goes (well here and GCP DialogFlow which is a different sort of trainwreck).

8. The unfortunate iOS experience

It’s no secret that I’m not a fan of Apple products, but it seams that Slack might also not be (although I would have ventured a guess they would be as, they are a lot alike). Frequently messages sent by bots are not displayable on iOS devices, or they do display but incorrectly. The experience is inconsistent. Instead of acknowledging “Oh there are messages displaying wrong — That’s a critical problem”, it’s been much quicker for us to change the type of message we are sending. Why? Because we can move much, much faster than Slack can, with our small team. It’s also something we’ve focused a lot on because we are in the security business, so when we find potential security vulnerabilities, it’s critical to assess and resolve as soon as possible.

iOS missing messages

You can see where it says (edited), there should be the team’s standup responses there.

9. Unexpected behavior

Ever wonder what happens if you do something not documented? The spoiler is only ever do documented things. Standup & Prosper APIs (and Authress’s APIs) are fully documented via an OpenAPI specification. If you do something not explicitly handled, you get an error, do something not explicitly handled by Slack, get weird results, 6 months later when something hidden changes.

If your app has the legacy old style bot permissions and not the new hotness granular permission scopes , when you send a message to the user with the flag: as_user=true , you get what you want. But if you have the new permissions, then you get an error. So start checking your permissions before posting to APIs, because the word deprecated doesn’t mean what Slack thinks it means. Deprecated to Slack means removed/disabled/deleted.

10. Slack message event delivery retries

Similar to almost all systems that offer webhooks for message delivery, Slack provides 3 retries for every message. One immediately, at 1 minute, and finally 5 minutes later. I don’t know the situation where our bot would still care 1 minute later, let alone 5. But you only have 3 seconds to respond to Slack with a “ success” — because Slack automatically times out and retries. Given that some of Slack’s APIs take on average more than 3 seconds, you are going to be fighting with this problem until the end of time. Further, 3 seconds isn’t even enough for our lambda’s to cold start in some rare cases, but usually the cold start is in 1 second, and then we wait for 3 seconds for slack to respond.

That’s not the worst, what is the worst is however for action events , such as button pushes and response modals, the request isn’t retried and a trigger token is passed with the event. This token is MAGIC. You must have the token to create a dialog with the user, and you want to create dialogs lots of times, because realtime messaging as I’ve shared above, is really challenging to get right. But that token also only lasts 3 seconds. And here’s the kicker, the timer threshold is when Slack responds OK to your request. So even if you manage to deliver the request to open a dialog in 100ms, if Slack has trouble processing it for another 3 seconds, sorry, it’s expired.

The solution we’ve added is to inform the user that they should try again, why not provide the retry, no idea. Why not just extend the time to 5 seconds, with the requirement that p99 of all requests have to be under 5 seconds. Why not let the users of the app decide for themselves if they like the experience, getting message at 5 seconds, is much better than getting an error message at 3 seconds that says “Oops something went wrong”.

11. Challenges with shared channels

Did you know there could be users talking to your bot, or mentioned by other users in messages, that your bot isn’t allowed to know about? For a long time, Standup & Prosper had received a confusing error message user_not_found. The user should have been found, someone in some channel added that user to a standup. The reason is that users could be in a channel from a different workspace, or guest users. These users could be in the same enterprise account, or totally different accounts. All I can say here is: Good Luck.

Alternatively you might get something like message_not_found , and not because the user deleted it, but because the user that posted the message is no longer in a channel that Standup & Prosper had access to, even though the bot was still in the shared channel, the user was not, and they were from a different workspace, so messages in the shared DM with the user are now no longer accessible.

You were able to follow that, right?

I’ll say that again, you can’t access DM messages from a user a bot has a conversation with if:

That user is from another workspace
That user was in a shared channel, that the bot was also in
And now the user is no longer in the channel
Even if that other workspace has your bot installed (these are separate workspaces with different user IDs after all)

12. Global Uniqueness of user IDs

The introduction of reusable user IDs across workspaces in the same enterprise slack account is necessary. So thank you Slack! However these user IDs are not unique across workspaces. It’s impossible to uniquely identify users in Slack, end of story. You can’t join the user ID with the team ID, because a user has the same user ID but different team IDs for different workspaces in the same enterprise account, and different user IDs in different workspaces. The only solution we’ve found is joining the user ID with the DM channel ID, these seem consistent, but I’m sure that will bite us later.

This something that Discord and every other messaging platform has figured out, you only need one user, but need multiple identifies one for each workspace. Instead of doing the sane thing, Slack has different users for each workspace, unless those workspaces just happen to be in the same enterprise account, in which case here’s a magic exception.

13. Breaking changes

The list of breaking changes just goes on and on. Let’s say you have a small bug in your configuration and you want to start listening to a new event type. It’s a critical problem now, with a simple fix, but you can’t easily deploy this. You have to wait for Slack to review your app in its entirety. Here’s the kicker, if Slack changed something in the meantime, now you are also required to make those updates as well. The UI for app management will have changed, and so have the requirements. One time for us — OAuth configuration update, “deprecation” of bot scope permissions, and security compliance were blocking us from a production fix.

When migrations are necessary, Slack assumes you can just do them, have done them, did them, and never offers ways to help migrate, keeping everything working before fully removing the deprecated functionality. You are either on the new version or you aren’t, features aren’t backwards compatible. There is such a thing as a migration path, but not with Slack apps.

14. You will hit Slacks outages before they know about them

I’ve already mentioned a bit about the outages, but what’s even more fundamental is, as an app developer you will hit these before Slack’s users do, because your users are doing more ridiculous things with your app. For example, this was clearly a bug, but we got it:

Then it became a 503, although that was supposedly a different problem.

Our team has been investigating and from our logs it seems like calls are being successful, and we not quite sure why you’re getting a 404. We’re gonna need yet another entry to double-check something on our end.

Why they don’t have logging in place, why they need logs from us, is beyond me. Our support requests involve a lot of time giving Slack their own data back.

15. Incomplete APIs

You will soon find out that it isn't just one or two of the APIs that are a bit problematic, but most are missing something critical from their interfaces. APIs that should have better paging, endpoints returning lists instead of objects, or just generally missing a critical query parameter to make them usable. For instance, want to get a list of messages sent to a channel, no problem, but you have to be in the channel. Makes sense, right? But, you can't even get messages that your bot sent. That means you need to duplicate the info about all message data on the bot side, and o that after you sent the message by reading the response (because remember, the Slack message api doesn't support idempotency). Want to know if a user is in a channel, welcome to paging through the list of all the users in that channel, even if there are 20k, and your pages only return 1k at a time. At small scale these things can work, but when you have a product available for an extended time, small little missing features cause big problems. Are you going to be able to get your 20 API calls in before 3 seconds are over, definitely not.

16. Misguided technical advice

Throughout all your mess in trying to figure out what is going on, you will end up contacting support. If you are lucky that they provide you with non-canned responses, you'll want to start questioning if the solutions to working with their apis are the correct ones. In the situations where events from Slack are taking over 20 seconds to be delivered, you are probably wondering why reliability of their delivery queue wouldn't be improved. But instead of taking that as a point of possible improvement, you'll get the advice like:

You'd call the conversations.history API for a specific time-frame at a rate of one request every 2 seconds, and compare the message details returned in the response with what you already collected from the event callbacks that you received/collected.

Sure that's reasonable on the surface, let's use the conversations.history endpoint, which was already untrustworthy, and call it 30 times a minute per user per workspace. For us that amounts to 10 times a second, however you'll notice that this is definitely higher than the allowed tier 3 rate limit of 50+/min, instead it is 600+.

This does show that the support is much better than tech companies with outsourced customer support. And further much better than companies like Google, where the first response is "No you're wrong, read the manual first", but it comes no where close to support for highly technical products such as AWS.

17. Tokens and configuration will overwhelm you

Even if Slack doesn’t make any changes, and they will, you will make mistakes because of confusion and complexity. The App UI and API documentation is so poor that when there is a change, you won’t be sure if clicking one little small button in the UI will cause you to lose all your users.

Your code will become confusing to maintain, there are some critical design choices that needed to be fixed as soon as possible, but instead of having this migration, Slack leaves those problems in place for the next app developer.

For example, assume you lost a bot token or worse, the app workspace or Slack breaks your token for that workspace, there is no way to force uninstall the app nor get a new token. That's right your installation stays stuck in broken mode until the workspace admin uninstall and then reinstall your app. And no, Slack support doesn't really help resolve this problem.

Wrapping up

I hope these sorts of issues help future engineers of bots and services better realize what Total Cost of Ownership really means when they build technology themselves. It isn’t just the features and feedback from users, but the platform you build on is just as important to be simple and polished. It’s a critical part of what you are building, how you are building, and what you spend your time on.

These sorts of problems all exist in the world of the developer experience. And experience like these for our team at Rhosys has led us to not only avoid them but ensure that the developer experience in security tools such as Authress works out of the box. We know if it isn’t simple, straightforward, and easy, then users will be unhappy, developers will be unhappy, and personally I will be unhappy. Although I don’t always need an excuse for that last one.

The Wolf Pack Team

Warren Parad — Thu, 04 Feb 2021 14:29:36 +0000

It’s already old news at this point to be talking about two-pizza teams. You can find a myriad of articles on Google discussing what they are and why you want them. I’m going to be talking about the innovation in team management that supersedes two-pizza teams and that’s the wolf-pack team.

(Just a quick note here, I know that not all wolves form packs, nor are the packs they form most optimal in every situation, however…)

Having two-pizza team amounts to somewhere between 3 and 7 team members, where all aspects of the team are handled by that team. It is an autonomous team with responsibility, ownership, and most importantly accountability. If the team does not deliver nor succeed, it will fail and fall apart. No one else should be involved in the success or failure, but that doesn’t mean the team can’t nor shouldn’t rely on outside resources.

All these things which are true of the two-pizza team are true of the wolf-pack team. However the wolf-pack team takes the expectations further. The two-pizza team fails to convey the roles and responsibilities clearly and results in the fundamental failure that is responding to change. While two-pizza teams are supposed to be agile, they nature of the team is static.

A wolf-pack team is dynamic.

New team members join as more experienced ones leave. Some may find more fitting causes for them to follow. Leadership is passed down through the team, and more essentially it is acknowledged that there are many levels of experience.

A wolf-pack team responds to all of these changes. With this type of team of 2 to 10 team members, your leaders or Alphas are responsible for training up the juniors or cubs for the future in the team. Seniors look for opportunities to become future alphas or find another pack. These are all things that happen in our existing teams, but the two-pizza teams don’t acknowledge these possibilities, so let’s solve that by doing so explicitly.

Your leaders are responsible for one day leaving the pack and having your juniors become the new Alphas. Will they be ready one day, or do things have to change? Change is natural in technology teams but also in all sorts of teams. The only constant anywhere is change. Planning for it is the only successful strategy. The Law of Legacy from The 21 Irrefutable Laws of Leadership by John Maxwell explains this well:

A leader’s lasting value is measured by succession.

A leader’s lasting value is measured by what he leaves behind. This in turn requires a leader to be highly intentional about his legacy, including knowing the legacy he wants to leave, living the legacy, choosing who will carry on the legacy, and deliberately passing the baton.

Knowing that your leaders will leave your team puts you ahead of your competition and starts your team a path to success in whatever you want to acheive. You start to focus on growing your cubs and not just leaving them there. There’s a realization that they are the future of your pack and that your Alphas will take them there. You can’t make your cubs grow by interfering with the pack, the pack takes care of itself.

In the case that there are some Betas, they might find themselves an opportunity one day to become the new leader of the pack, or find a new pack elsewhere that better fits them. This happens today with all our teams, but without the attention to the possibility or desire to support these migrations, we attempt to ignore a critical aspect of our teams.

Over time, your teams change just as your tech does. It’s important to recognize these changes and embrace them. The two-pizza team is a great start, but when you start to evaluate effective leadership, team growth, and individual responsibilities, the two-pizza team lacks critical understanding of these. Think about the pack you want to raise, and how to train them for their next hunt.