Forem: Dietrich Moerman

Effective Metrics for Development Teams

Dietrich Moerman — Mon, 23 May 2022 19:19:59 +0000

Today, collecting data produced by the software we use daily is pretty straightforward. Reporting is getting increasingly simpler as well. What often goes up the wrong alley is knowing which data to collect and how data and reports are interpreted and used.

This article outlines the pitfalls with commonly used software development metrics and what we may use as a more impactful alternative.

Vanity metrics: misleading numbers

In software development, you probably saw these metrics pass by at least a few times:

Number of story points per sprint (”velocity”)
Average amount of git commits per day
Number of pull requests or reviews per week
Percentage of code (test) coverage and its trend
Number of resolved tickets or bug reports per month
etc.

These are “traditional”, output-based metrics, easily deducible from the tools and software development teams use daily: Jira, Trello, or their choice of issue tracker, GitHub or GitLab, and language-specific code coverage tools such as PHPUnit, pytest-coverage or go test.

All of these metrics can be interesting in their regard, given they are used in the correct context and with the proper interpretation. Unfortunately, teams or managers, often due to pressure and lack of time, can be tempted to use these (and only these) metrics to draw the wrong conclusions about the performance of individuals or teams or how they compare to others.

The issues with measuring output

With data and output based metrics used to measure and track people’s performance, I believe there are at least two significant issues:

When aware of these metrics, people can influence them to make themselves look good.
You never have enough measurable data to shed a complete, trustworthy view.

The effect of people “gaming the system” is also called Goodhart's law, coined by British economist Charles Goodhart in 1975. It is defined as:

Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes. [source]

In essence, people can (un)consciously influence the outcomes of the metrics in ways that don’t have the intended effect.

For example, an easy way to increase the percentage of code coverage is to add tests that perform little to no assertions. Code coverage may go up, but you gain next to nothing as the application’s behaviour or results are unchecked. If your goal as a manager was to increase quality and avoid regressions, you are in for a treat.

“But what if we combine many metrics to get a complete overview that cannot be gamed?”

While sounding tempting, not all aspects of software development are measurable with quantifiable data or metrics. Even more so when working with teams, software development is a complex socio-technical system in large part based on human interactions.

Traditional output-based metrics, incorrectly equating “quantity of work” to value, also fail to encapsulate actual business value, impact, and client happiness. Hence, they are unfit for reporting back to investors, clients and other stakeholders.

Then there is the ethical aspect of whether you should at all measure and track individual people’s output. I believe tracking (and rewarding) people individually brings more bad things: decreased cooperation, declining psychological safety, more gossiping and politics, and higher risks of burn-out.

So, what does work, then?

A possible alternative is twofold. First and foremost, measure and optimise the development workflow rather than people or teams’ output. Then, start assessing the team’s business impact.

Workflow effectiveness and resilience

The efficiency and effectiveness of our development workflow are huge enablers to making an actual business impact. That’s why I always focus on this one first.

The easier a team can experiment, adapt to changes, fix issues and resolve incidents, the better they can focus on building cooperation and team spirit and creating that value and impact, positively affecting team stress levels and happiness.

In the Lean sense of Kaizen and continuous improvement, let’s look for some core metrics that we can use to assess our engineering workflows. Luckily for us, Google, in their DORA research, already defined a few base metrics called the Four Keys that modern DevOps teams can use:

Deployment Frequency: how frequently do we release to production successfully?
Lead Time for Changes: what is the lead time for changes to hit production?
Change Failure Rate: what percentage of production deployments cause a failure?
Time to Restore Service: how long does it take to recover from such failure?

Notice how these metrics do not measure individual output nor incentivise teams to get more of the same (bar number of deployments). Instead, they give us an idea of waiting times and bottlenecks when building and deploying solutions, how often broken builds are delivered, and the rate of and how quick we handle regressions after a production deployment.

To collect this data, you may go with Google’s proper software tool or start collecting them MVP wise:

Deployment Frequency: check your CI/CD tools to see if you can read or export the number of deploys. If not, you may start by manually posting a message in a Slack/Teams/… channel after you deploy and track the number of these messages. I’ve found this also helps with “leading by example” to perform deploys if your team does these manually multiple times a day. Don’t forget to measure production deployments only (internal or testing environments don’t count) and subtract failed ones.
Lead Time for Changes: the median time between the production deployment date and the dates of the (git) commits therein. If your CI/CD tools do not provide this out of the box or through a plug-in or script, this is considerably harder to keep track of manually. As an alternative, you may measure the cycle and lead times of stories instead (see Intermezzo below).
Change Failure Rate: tracking the occurrence of production deployments or feature releases (i.e. enabling a feature toggle) that require remediation through reverts, fail-forward fixes, etc.; what percentage of releases fail? Here I recommend logging incidents in an issue tracker or purpose-built tool such as Opsgenie.
Time to Restore Service: closely related to the previous, how long does it take to restore the application to a fully working state? Aside from issues triggered by deployments, consider other incidents such as degraded performance, service downtimes, etc.

Having collected these numbers, you can compare your team's performance to Google standards.

Intermezzo: Lean’s cycle and lead time

You might have heard about lead time in a different context: Lean. These terms are so valuable that they, too, deserve to be mentioned here.

In short, we can define lead time as the total time it takes for a (customer) request to pass through the entire processes and value stream up until it is released (to the customer). While DORA metrics measure commit lead times, measuring total production lead time gives a more broad indication of the performance of our product & engineering team’s processes.

The cycle time of a change, on the other hand, can be measured by comparing the release (to the customer) of the change with the moment the development team started working on it.

What constitutes a “request” or “change” is up for interpretation. Most likely, it will translate to a user story or change request that you add to your sprint backlog or Kanban board. Measuring cycle time becomes pretty easy, given the end state of the board equals “released to the user”, and the tool at hand allows to visualise or export the time differences between states.

Assessing business impact

As mentioned before, assessing and communicating on business impact is much more valuable to company leaders, investors and other stakeholders than technical/operational metrics but also more challenging to measure than the latter.

Working towards outcomes (over fixed output), evaluate with the team whether they meet their goals and if they align with the company’s strategic goals and mission. How you may determine or measure this value and impact dramatically depends on the company, kind of client, and more:

Communicate with your clients or users! Are they happy with the product? Do their business needs and expectations get met? Is their feedback asked and taken into account?
Does the team define an outcome-oriented product roadmap? If so, evaluate whether the team met the initiatives or goals.
Does the company define company-wide, outcome-oriented objectives and key results (OKR)? If so, this may also help with making that assessment. (Do beware that implementing OKRs comes with its pitfalls.)
…

If a team is struggling, overwhelmed, or otherwise fails to make the desired impact, the first step is to talk to and (as a manager) coach them into resolving any issues.

You guessed it. Having open conversations — within a safe environment — is more valuable than (just) measuring with data and comparing teams or individuals. And by all means, continue to focus on their happiness and well-being as a whole. Happy employees lead to increased productivity, which leads to more satisfied clients and contributes to a more durable way of scaling the company.

Development teams are no silos

Designing, developing, and releasing software applications is a complex undertaking. To succeed in this, we cannot afford to treat a team of engineers — or any team — as a silo. Doing so severely limits the quality and effectiveness of the solutions built and might even set up a development team for failure from the start.

This implies that — even with the improved approach to metrics and value assessment — the interplay with and influence of other teams can severely impact the success of a development team.

To start with, UX and UI analysts and experts, designers, QA engineers and testers, product managers, system engineers, and product and data analysts all contribute to the success of a solution. They should be internal to the team as much as possible.

Additionally, the other departments or teams of the company also have a significant impact, even if only for their influence on company culture: sales and marketing, customer success, finance, etc. And, of course, not to forget the quality and impact of the mission, vision, and strategies set out by the leadership team.

When setting up any initiative for improvement, even within a single team, take the entire system into account. Is there a deeper root cause? Do we require a more considerable culture shift? Or is this a purely operational issue that a team can fix themselves?

Resources

The Fallacy of the 100% Code Coverage, an article where Thierry de Pauw describes how the test coverage metric can be misused.
Google Cloud’s DevOps Research and Assessment (DORA) brings reports on the state of DevOps and is the origin of the “four keys” metrics.
GoogleCloudPlatform/fourkeys is a software tool built by Google to generate insights from data based on the four core metrics mentioned above.
Definitions of cycle and lead time as provided by Lean Enterprise Institute.
A Two-Person Agile Project (and what it teaches us) explains the core elements of an effective product or project development team.

A Two-Person Agile Project (and what it teaches us)

Dietrich Moerman — Thu, 24 Jun 2021 15:57:08 +0000

Reading about "being agile", "agility", or one of the famous agile frameworks, we often think of growing startups, scaleups and corporations seeking to adopt more efficient ways of working, building their products better and quicker, and making their clients happier.

We might easily forget how agile small companies and software projects with only a couple of people involved can be, and how these can teach us a lot about the essence of agility.

Let me briefly introduce you to a past project.

The two-person agile project

Several years ago, I was involved in building a new software product - a tailor-made web-based platform solving a number of our client's business problems. While I will spare you the details of the client, the product and its functionality, I want to focus on some approaches we took back then.

First of all (you guessed it): our team was small, very small: there was the client, and there was me. That means I was doing pretty much all of the (technical) work: setting up the infrastructure, building the system's back-end, implementing the front-end, caring about security, and deploying and releasing functionality to the end-user. Nowadays, we would call this full-stack development, and this scenario resembles typical freelance projects.

I was lucky to have a single spokesperson on the client's side, and we were working relatively close together. We didn't spend time planning months ahead but instead discussed in a (bi)weekly meeting what to focus on next as a direct result of the most recently delivered features.

We were both happy: the client saw the product evolve, supporting business cases they had discovered along the way and refining existing features that were just not there yet. While myself, I was able to stay focused primarily on developing and didn't lose time with things that didn't matter.

This story will not sound very groundbreaking to some, while for others, it may be reminiscent of simpler times or your 90s hobby or school software project. No corporate silliness, only happiness!

Identifying the success factors

Frankly, I needed a few years to truly understand and appreciate the elements that made this project so rewarding. It only started to dawn on me after learning more about the core values of agility (beyond your typical Scrum, Kanban, etc.) through later jobs, workshops and interactions with mentors and peers.

Of course, it wasn't the perfect software project -- does that even exist? Yet, the necessary factors to make this a success with a happy client were in place.

1. Uncomplicated team structure

It doesn't come any simpler than that: I was the sole developer working on this project, and I did all the talking with the client.

By eliminating more complex managerial structures, hierarchies or third parties to align with, we also:

mitigated waiting time as much as possible;
avoided handoffs between business analysts and the team or even between a product manager and the developer(s);
and (perhaps most importantly) allowed me to be self-managing.

2. Close and frequent collaboration

Through our (bi)weekly meetings, we presented, tested and talked about what was finished and delivered since the previous session and what was in progress, and we identified what we can or should focus on next.

In this process, receiving and processing feedback was at the core of our approach. We didn't hold back to give honest feedback, think through budgetary or technical implications, and build mutual trust.

3. Don't stick to The Plan™

Infamous for many large organisations and corporations, elaborate plans, analyses, and specifications are defined months upfront and carefully tracked in their progress. Ah, ye olde waterfall approach to project management.

Throwing this overboard is a staple of agility. Embracing change.

By incrementally letting the product evolve, the following steps became obvious during each meeting, and so were the basic functionalities to implement. This approach made us avoid overhead work and allowed the client to experiment with the next evolution of the platform by the end of the next iteration.

4. Maintain a consistent flow

Working alone and mono-focusing on a single change made it pretty easy to get into the zone. I didn't need to split development work among colleagues, wait for their tasks to finish, review pull requests or solve merge conflicts to wrap up user stories.

Without the need to sidetrack all too often and lose time over said activities, there was a relatively fast but most importantly consistent flow.

5. Prioritize bugs

Mono-focusing made us prioritise bugs more efficiently and get them fixed "first thing in the morning".

This approach contributed to a certain level of quality that we wanted to maintain. There is less to no value in delivering broken functionality or leaving subpar, confusing or unused features in place.

Applying to typical teams

You may think this approach is both naive and unfitting for building products in larger teams or organisations. Surely, in the real world of complex SaaS products, it can't be this simple, right?

And you're (partially) right! Most products cannot and should not be built by a two-person team having a single developer. Aside from the bad bus factor, complex business domain(s) or combinations of technologies used are better carried by T-shaped teams of around five people (YMMV). Eventually, applying concepts of team topologies may help to tackle more considerable complexities.

Still, the success factors mentioned above can be projected onto larger teams, contributing to the success of more complex projects and software companies.

Let's dive into some aspects of lean software development.

"Speed is the absence of waste"

This quote is from Mary and Tom Poppendieck's book Implementing Lean Software Development: From Concept to Cash, a primer on applying lean principles in building software products.

I particularly like this quote because it stresses that high speed or velocity is achieved not by pushing the team to work harder and faster (often with disastrous long term effects) but rather by getting rid of wasteful work ("Muda") as much as possible.

Wasteful work, work that does not add value for the customer, has many forms and definitions, and I'd like to expand on a few of them that we have touched in our story:

Creating elaborate plans and specifications (user stories, documents) too far in advance is part of a waste called Partially Done Work and a threat for the Extra Features waste that Poppendieck identified in their book. These wastes are sometimes (originally) referred to as "inventory" and "overproduction".

Making analyses and writing out specifications should -- if at all -- be done as late as possible. Otherwise, there is a potential risk of ending up with obsolete plans due to new insights, changes in the market or newly identified pains by the client, and risking building low impact functionality (= Extra Features; also a form of feature creep).

It is no exception for stakeholders to request and hold on to early defined plans and roadmaps only to end up missing the boat either on what is needed and how long it would take to achieve the desired goal or outcome. Even though this behaviour is understandable from a psychological point of view, seeking to control risks through waterfall project management instead of evolutionary product design and development does pose a threat to the success of the product or project.

Instead, continuously poll for the next most important step to take using market research, product analytics and data, test or beta users, team insights, or simply by conversing with your client after delivering your latest iterations and work in small steps. Of course, you may still define a long term strategy and goals (with deadlines if you must); ensure they focus on outcomes rather than outputs.
Our dramatically simple team structure avoided some other forms of waste: Handoffs and Delays (also called "waiting").

Our mini team was self-managing: in coordination with the client, I defined the pace, wrote user stories and defined technical approaches without relying or waiting on people outside the team.

With bigger teams, you want just that: as a company owner, CTO, VP/Head or manager, build up mutual trust, provide a psychologically safe environment where it is okay to speak up, and allow the team to work autonomously. Ensure teams comprise of all the skills they need to fulfil their goals: T-shaped people having at least some notion of a wide range of skills with more profound knowledge on a few specialist skills.

Attempt not to limit this mindset to traditional development skills; introduce UX and UI expertise, design, QA and testing, product management, operations, product and data analytics, and any other skills required within the same team as much as possible.

As the names of the wastes give away, this helps minimise handoffs of tasks to people outside the team, focusing on working together as often as possible and avoiding delays in execution. In the next section, I will dive a little deeper into these matters and show how we can prevent handoffs inside the team.
And last but not least, focusing on solving known bugs first helps to mitigate the obviously named Defects waste.

Minimising these wastes as good as we can, we limit excess costs on activities that do not add value for the customer, keep team morale high, and improve the overall return on investment (ROI) of development.

Pair and Ensemble programming

I remember what I thought when I first heard about pair programming many years ago: "How is sitting with two at one computer going to make the work go better or faster? Just leave me alone and I will do fine." Some years later, I had to admit I was wrong.

Many -- if not most -- software development teams are managed to maximise resource utilisation. Concretely, this means individual contributors are expected to be busy and productive as much as possible. When everyone is productive, a lot of work is getting done, so a lot of value (and money) is generated. Right?

Unfortunately, the practice of keeping everyone busy all the time kills flow because it keeps our previously mentioned handoffs and delays in place, and it does not spread knowledge across the team that well either.

Think about how frequently typical teams are indulged in planning and dividing work, waiting for absent or busy people, regaining focus after task switching, etc. In other words, trying to deliver more work units in the same timespan yields the opposite effect.

Ensemble programming (also known as "mob programming") is a valuable practice to increase flow by avoiding context switching and minimising handoffs and delays while facilitating knowledge sharing among team members.

Popularised by many agilists and podcasts, the essence of ensembling is an entire team working together in a so-called mob or ensemble, doing a single task (remember mono-focusing?), taking turns in being the "driver" (which is the person typing), while the rest of the team navigates, brainstorms and contributes.

Running ensembles, the T-shaped team focuses on one-piece flow, just as I did in the story's agile project. As a result, their cycle time, a metric for the time required for an item to move from "in progress" to "done", decreases drastically.

Doing this with a team ensures that whenever someone takes a break, moves to a private meeting, has a sick day off, etc., the remainder of the group retains focus and keeps the flow going. This format is hence also an ideal setting for onboarding new people; they can join the ensemble, ask questions and make minor contributions at any time right from the start.

Pull requests vs flow

A perfect example of an accepted practice killing flow that ensemble programming fixes is the cycle of code reviews.

Many of us have been there. As an individual contributor, you wrap up your task, create a pull or merge request and eventually request reviews from your mentors or team members. Then the waiting game begins.

Said reviewers are busy with other tasks, helping others, sitting in meetings, or (worse 😉) having a day off.
Luckily, soon enough, your code change is being reviewed, but you're not getting approvals yet! Questions pop up, potential problems arise, and missing tests unveil themselves.
Time to get those fixed! At least, as soon as you wrap up or abandon the task you started in the meanwhile.
When you've finished processing feedback, it's time to push those changes (hopefully without a lot of conflicts against your base branch) and let the cycle begin anew until everyone is satisfied and you're allowed to merge.

Sounds great, doesn't it? This unproductive cycle kills people's sense of accomplishment, focus, and in the end, the throughput of the team. Over time, people might put less effort into code reviews because they want to get their work done as quick as possible, with adverse effects on quality. In other words, this team is suffering from the handoffs and delays wastes within the group.

The good news is that this entire cycle of gathering ideas and technical approaches, writing tests and code, and approving and deploying the changes is part of the normal flow during ensemble programming. As a result, delays and waiting are reduced to the minimum, and development speed increases, especially when we already avoid many other wasteful activities.

All in all, having this steady flow of creating value in a supportive and lean environment, people become happier and more motivated.

Getting started with pair programming

Now, logging into work the following day and inviting your entire team in an ensemble is likely going to feel off, scary, unproductive or a combination of those. Ensemble programming is a practice that takes persistence, rehearsal and guidance, especially since it is radically different from how most people are used to perform their day to day job.

My advice would be to start with pair programming first, limiting the "mob" to two people, and allow people to get accommodated to working together more frequently. Many of the benefits will already be there: less waiting and focus switching, and more knowledge sharing.

Don't stop here, though! There are likely still handoffs and delays between pairs and individuals, and hence wasteful activities are still present!

Summary

In this article, we took a quick dive into applying the success factors of a simple, agile team to larger teams. As a result, we can learn and improve how larger teams work through concepts of lean software development while not forgetting fundamental agile values such as those of psychological safety, experimentation, and continuously delivering value.

When attempting to optimise the happiness and flow of a development team, ponder whether your team is spending too much time with wasteful activities, trying to make up for it individually in between meetings and distractions. Teach people to spot and alleviate waste in a continuous improvement cycle and help them make the company and the clients more successful.

And don't forget to think about that old project you did back in the 90s and what made that project awesome. Perhaps you may apply some practices from it as well!

Resources

If you want to read more about lean, I recommend Poppendieck's book linked above, as well as the (free) book 2 Second Lean by Paul Akers (thanks Ashley and Joshua of Industrial Logic for the recommendation!).

For more information on ensemble programming, check out the Mob Mentality Show, offering videos with common patterns, insights, tips, etc.

Elisabeth Hocke (@lisihocke) also compiled an extensive list of resources on Working as an Ensemble and created a talk on the practice.