Forem: Philipp Giese

Why Building Software is Hard

Philipp Giese — Tue, 23 Aug 2022 17:44:47 +0000

Every day someone new decides that writing code will be the thing they'd like to do for a living. Or maybe just for fun. And thanks to incredible frameworks and Stack Overflow, getting your first results has become somewhat easy. This is great but has misled some people into thinking that building software (i.e., larger apps that people pay for) is easy. I believe this is because some practices that lead to great software aren't prominent when you first encounter them. This post will be my ever-growing list of activities and misconceptions of the software world.

Before we start

What I don't want this post to be is discouraging. As I said, you can start small and get results. You don't need to know or follow everything I'm describing from day one. Pick something, try it out, and see for yourself. Some points might work better for you (and your team) than others.

What you think works	What actually works
Work on as many tasks as possible in parallel	Focus on one task at a time
Code reviews	Pair or swarm on tasks
Design and QA as separate steps	Co-create and involve people at every stage
Do rigorous planning up-front	Experiment and work in small increments
Few releases with extensive manual tests	Automate releases and do them frequently
Never change a running system	Continuously adapt
Build the big thing that handles everything	Build many small blocks that do exactly one thing
Have someone manage releases	Deliver continuously
Keep people busy all the time	Provide room for self-improvement
Writing software is easy	That sh*it ain't easy

Why you are faster the fewer tasks are open in parallel

When five people work on five tasks simultaneously, you'll get five results faster, right? Wrong! Many people (want to) believe this is true because then every additional developer will make the team faster. Unfortunately, the opposite is true. How come? This sentiment assumes that you can define and work on five tasks in complete isolation. Maybe on separate feature branches because this would be required to make the above somewhat true. In real life, the five people will probably need to talk to each other because their work is interdependent in one way or the other. They will also most likely change the code in ways that affect the work of the other developers. That means these developers will need to sync from time to time to check whether their work is still compatible. And with the word sync your parallel dream is over.

It can get worse. When person A works on something that person B needs and person B relies on the fact that A finishes their task before B picks up the next task, there is much more risk involved. Any interruption of A (vacation, sickness, manager casually swinging by their desk) will become an issue for B. Imagine there is a person C who is also waiting for B. This style of working builds a massive house of cards that will come down, eventually.

A system that optimizes for parallel work ultimately optimizes for busyness over productivity. People will always have something to do, but nothing will come out of it. Lean and Kanban teach us that single item flow (i.e., working on one thing at a time) is desirable when you want to get things done. That does not mean that one person works on one thing at a time (even though that might already be an improvement in some companies). It means that the whole team works on only one thing at a time. This seems counter-intuitive at first, but this way of working has some key advantages:

No need for reviews — When the whole team works on a task, the review happens continuously. With that in place, there is no need for another gateway step that speeds up the whole process.
No handovers — When designers and QA folks work with everyone, you don't need a design handover or a QA handover. The designers figure out the design with developers, and the QA folks help build the right thing. This removes a lot of back and forth, shortens feedback cycles, and speeds up the process.
No knowledge silos — As everyone is working on the same thing at the same time, everyone also shares the same knowledge. With that, it is absolutely no problem when people get sick or go on vacation. Everyone else is still on board, and when someone comes back, they are immediately on-boarded into the changes.

The methods I just described optimize the time not spent on wasteful activities. Doing so automatically increases the time you spend on activities that create value.

Why code reviews aren't helping you

Code reviews are a low bandwidth form of communication with a high chance for misunderstandings (Peter Hilton). When the reviewer does not share the same context as the person who wants the review, chances are high that the reviewer will miss or not fully understand the most significant bits.

The quality of a review is indirectly proportional to how much the code change is in need of a proper review. This means that a 15-line change will get a lot of comments because it is easy to understand. But because it is easy to understand, the chance that the reviewer will find something groundbreaking is also low. A 1500-line change will likely get no comments because the change is just too hard to understand.

Reviews are also an interruption to flow. What does the person who wants the review do now? Best case: nothing. Worst case: start the next task already (see why you are faster the fewer tasks are open in parallel).

You can make code reviews obsolete with mob programming plus test driven development. A code review happens all the time when the whole team works together. TDD is a forcing function to have the most important discussions at the start because everyone needs to be on the same page about what the mob is currently trying to achieve (get the current test green). It also helps to ensure you're not accidentally breakings something as you move along.

Why you should avoid handoffs

Handoffs happen when people work in parallel. This means that anytime the deliverable that got handed to you isn't perfect, you're going to interrupt the flow of someone else because they need to look at something again that they thought was finished. If you're lucky, they can do that right away. Otherwise, you'll need to wait until they finish their current task. You might be able to see how this quickly adds a lot of waste to your process. The more handoffs you have, the higher the chance for waste.

Now you're in a situation where everything takes forever, and you can't progress because everyone is waiting for something from someone else. This means deadlines will approach, and you're unable to deliver. Since people know that fixing issues takes forever, they tend to just not do this anymore, which leads to crap products being released. Crap products lead to bad customer feedback, leading to more rigorous handoffs. A vicious cycle.

Some people will claim that you can fix this situation with better planning. That won't work. Certain things are impossible to know up-front, and you'll only discover them when you start working on your task.

Again, the solution is to work together. Eliminate handoffs by co-creating with designers, developers, QA folks, and customers. Then make sure you work in small increments. When designers need to watch developers code for weeks on end, they're less likely to enjoy working with them. When you ship something every other day, and you can rapidly prototype and change the software as you get new insights, the process will motivate everyone involved.

Why you should not spend too much time planning

Some people think they are smarter than everybody else. The same people think you can plan a large software project in detail. A missed deadline? A better plan could have avoided that.

These people forget that our job is not to execute plans perfectly. Our job is to build the right thing and build it right. Nothing is worse than perfectly executing the wrong plan.

A big plan will ultimately lead to large releases. Why? Because frequent small releases will likely result in feedback, and that feedback may not match your plan. This might not sound bad to you, but for people who think their job is to come up with perfect plans, this is a threat. Mostly because it can expose that what they sold as knowledge was simply an assumption. Plans also fix many things in place because to create a plan, you need to make decisions. When you decide on aspects that might only become important in a couple of months early on, it is easy to pigeonhole yourself into a fixed mindset. And fixed mindsets don't like change at all.

Instead of planning too much, you should spend time experimenting. Thanks to concepts like feature toggles, you can test changes or new features with a subset of select users before you roll them out for everyone. This helps to mitigate the risk of releasing something that nobody wants.

You should also be clear about what you think is true and what you know to be true. Tracking your hypothesis can help you determine which experiments to run first because they have the most significant impact. With that, you can avoid surprises late in the process, which would otherwise lead to delays.

Why you will never find all the bugs

No one likes to introduce bugs. This is why some insist that rigorous testing must happen before each release. With that, they usually mean manual, exploratory testing. While this kind of testing definitively adds some value, I would argue that it shouldn't be done for every change.

When you try to find every edge case, you'll never ship something because you're never done searching. But when you never ship to production, you'll never discover all edge cases. Testing too much can lead to situations where people think they should plan more, outsource testing, release less often, and parallelize more work.

What to do? Accept that bugs will exist no matter what you do. Then work on reducing the risk for high impact bugs.

Work in small increments — This ensures that the code changes are small when a bug appears, making it easier to identify and fix. Also, working in small increments generally reduces the risk for bugs because small changes are easier to comprehend than large ones.
Better automated tests — Write tests that help you find defects faster so that even while developing, you can identify issues earlier and be more confident about your changes.
Test Driven Development — Admittedly, that's my personal bias. TDD makes you think about your changes before you do them. This usually forces you to have the most important discussions first, leading to better code and fewer defects.
Focus on MTTR — MTTR, or Mean Time to Resolve is my favorite metric. It measures how fast you are at fixing bugs. I think it is an excellent forcing function to help teams optimize their way of working to ensure that when a bug happens, it is fixed in the fasted way possible.
Use a zero-bug policy — A zero-bug policy does not mean that there are no bugs but that all bugs must be resolved before regular work happens. That's great because it means you'll never (knowingly) build features on top of broken code. This further reduces the risk of future bugs.

Why you should refactor

What habits did you change in the last year? Did you stop doing certain things and maybe start doing some others? Did you learn something new? Did the language you use get an update that makes it easier to achieve a certain task? All these things happen, so we need to refactor our code constantly.

The moment you write code you put an expiry date on it. Even if the code was crafted according to the current best practices and standards, the sheer fact of time would make it erode. Because your tooling gets better, and you'll learn new things.

When you never refactor code, it will eventually expire. This means that the cost of a large-scale refactor will equal the cost of a rewrite. It is a bit like housekeeping. When you clean up a little every day, your house will be in an acceptable state all the time. However, when you neglect cleaning for too long, things go from dirty to broken.

I always like to work according to the rule that you should always leave a place a little bit better than when you found it. Refactors don't have to be big. I'd even say they should never be big. Make small improvements every day, and they will amount to something big over time. When you get into this habit, tech debt related to outdated code won't be on your list of problems anymore.

Why that huge component might not be a good idea

When people learn to code, they inevitably encounter certain design patterns. One pretty much everyone knows (maybe because it is so simple) is "don't repeat yourself" or DRY. Oh, boy, do I not like this pattern. For some reason, people think that any form of repetition is bad design. What this leads to are premature and very leaky abstractions. Because they are premature, these abstractions leak all over the place and lead to high coupling and low cohesion in your code base.

One symptom of that is either huge classes or long functions. For functions, you can easily spot this when people start passing around flags that are used to control the flow. In most cases, you'd want two or more functions focusing on one thing. However, that would lead to some duplication, and when people think this is a bad design, they end up with huge functions that combine many use cases.

The goal of DRY was to keep things re-usable. Instead of duplicating logic into multiple places, it should be well encapsulated and re-used. That's a great idea! So why doesn't it work?

In a lot of cases, duplication is a coincidence and nothing more. If you mistake this coincidence for something more and create an abstraction out of it, then your abstraction also only works coincidentally. I always write the same code at least three times before I start thinking about extracting it. Only then will I have understood the use cases well enough to develop an abstraction that makes sense, is re-usable, and doesn't leak implementation details.

Think of abstractions and methods as a toolbox. You don't have one screwdriver that fits all different screws in all different sizes. Such a tool would be utterly useless. But multiple full-blown screwdrivers are also over the top. They would take up way too much space. So we took the re-usable part - the handle - and created a set of different bits that all fit on it. All bits still have some duplication: the metal, the part needed to make them fit into the handle. However, this duplication helps, and no one would think about removing it because that would make using the different bits much harder.

Why releasing should not be a big deal

Have you ever heard of No Release Fridays? It's a saying that teams should not release on Fridays because the release might break, and someone needs to spend their weekend fixing it. This sounds reasonable until you ask yourself why releasing is scary in the first place. When you build software so that releasing it is scary, change how you build it!

When something is hard, do it more often. That statement is true for working out and for almost anything else in life. Don't be afraid of bugs and release as soon as something hits your main branch.

When you're building a library or framework tools like semantic release can help you not only release but also figure out the correct version bump and automatically create change logs.

If you're interested in a real-world example of how to change your release process, then have a look at this article from my co-worker Tobias where he describes how we pulled it off at BRYTER.

Why people should not work all the time

We've already talked about wasteful activities such as working in parallel, code reviews, and handoffs. If you avoid all these pitfalls and crank out code using TDD in a mob all the time, you should be good, right? Unfortunately, not. In fact, you shouldn't be working 100% of your time.

That might be the point where you think I'm crazy. We're all knowledge workers, and if we're busy all the time, we're giving up a huge potential. The potential for innovation.

Think about it. When you're working head down all the time, you'll never have time to reflect. You will not read that book that gives you a great idea or watch that talk with some crazy new insight. It also removes the possibility of trying out the refactoring you thought about but never got to. I would argue that people in our busyness should only spend up to 70% of their work week with planned activities and leave the last 30% for their minds to wander.

Why writing software is hard

This has become the longest article I've written so far. I don't want to gate keep anyone from getting into the field of software development. But I also wanted to highlight that writing high-quality software and doing it in a way that produces good results in the fastest way possible is hard work.

This article merely scratches the surface of what is required of software developers every day. Of course, you won't be able to follow all these rules daily, which is exactly why continuous improvement is part of the list. The goal isn't to be perfect but to do something good every day. Over time these small bits will amount to something big.

I've mentioned TDD a couple of times. This practice alone takes so much time to get right. Now, not only do you have to get the code for the product right, but also the tests. Because tests should be written in a way so that they don't need to change every time a detail in your product changes but they need to make sure that you don't accidentally break stuff while you move along.

The best advice is to work in baby steps and keep iterations as small as possible. Get early feedback and improve. If you want to read more on the topic, Ron Jeffries has written a great article about it!

That's it for now. I'm happy to hear your feedback on Twitter.

The Rewrite Trap

Philipp Giese — Fri, 15 Jul 2022 08:25:44 +0000

This would all be way easier if we would just throw everything away and build it ourselves.

When was the last time you've heard someone say this or even said it yourself? I definitively heard this at least a thousand times. What I observed, though, is that this sentiment usually comes from people who recently joined a new team or a new company. Do these people bring fresh ideas that make them see issues where others are already blinded? Or are they the ones who are missing something and, therefore, think that a rewrite will produce a better result?

TL;DR

We'll learn why rewriting software will most likely not produce a better result. To not fall for this fallacy yourself, you need to understand how to:

move out of your comfort zone,
acknowledge that you don't have all the answers, and
what you might want to do instead

Why it is so tempting to start something from scratch

There are multiple reasons why starting from scratch seems like a promising approach. I will talk about the ones I have encountered most often and advise on what to do instead.

You don't need to wrap your head around new concepts

One key reason is that people like to stick to what they already know. When you start all over, you can choose the conditions you want to work in (e.g., the programming language or a framework). No matter what was already there, you don't need to try to learn something new. The freedom to use the tools that you already know can feel like a huge productivity boost.

Why is this a bad thing

Software is much more than a framework or a language. Most complexity is not found in the code that renders something on screen or accesses the database. You'll find the complexity inside the business logic of the software. I would argue that although some languages or frameworks will remove some boilerplate code, the gains you get from replacing one language with another are minimal in many software products. What you'll end up with is the same complex logic expressed in a slightly different way. Will that new way feel better to you? Maybe. However, it will only feel better because you could stick to what you already know.

People also usually forget that the current implementation won't just disappear. While working on the rewrite, your users will still use the old product and find issues that need fixing. The fixes will now take much more time because next to the time it takes to produce the fix, you will also add the time it needs to context switch between the old and the new world. Even worse, when the rewrite is not done by the team that built the original version, this can mean that the rewrite will reintroduce the issues. A co-worker told me about a situation where the old team saw the rewrite as a challenge to compete with the new team because they took pride in what had already been built.

What you can do instead

Be aware of why you like specific frameworks, languages, or concepts and talk to people about it. There is a good chance that you can apply some of the same methodologies also in another language. Also, you might learn that what the team is currently using has some nice features you have never heard of before. See this as a learning opportunity for both the team and you.

When you figure out that you want to refactor parts of the product, do it in the right way.

Follow your mental model

When we build software, we also create a mental model to guide the architecture. Getting into a mental model of someone else can be very hard. When you rewrite a product, you can build your mental model, meaning it is much easier to understand and follow architectural choices when encountering them. This lowers the cognitive load on your end and makes you feel more productive as you move along because the likelihood that you misunderstood a concept is lower (because you invented it).

Why is this a bad thing

Models change and evolve. It is likely that even though the mental modal you find seems confusing, there is a good reason for it being this way. When you create a new mental model, it will likely miss certain parts. However, you won't know this yet. By building up a new mental model that disregards what is already there, you set yourself up for failure.

What you can do instead

Talk to people. When something seems odd, this is a very good start for a meaningful conversation. Be clear about why something seems odd to you and suggest what you think would fit better. Even though there might be good reasons for the current model, this does not mean it is perfect. With your input, the team can iron out inconsistencies and improve.

Don't do this only once. Make sure to adapt the terms you use to build the product as the product evolves. This will make it easier for the next person that joins the team to get going.

Work the way you want

With new tools and new architecture, likely, you'll also choose new processes for your work. You might want to start with continuous delivery right from the start. After all, the product is small again, so changes to the process are easier.

Why is this a bad thing

How you work is only loosely coupled to what you build. If the system you're working in leads to 3-month release cycles, it will most likely lead to 3-month release cycles even when you start from scratch.

What you can do instead

Talk to people. Figure out why things happen in a certain way. Maybe people aren't aware that early integration creates value or that there is a better way of doing things. Make sure to figure out why things are the way they are before you change them.

You don't know what you don't know

Maybe you have noticed already that talking to people is the best way to overcome the initial urge to rewrite everything. Unfortunately, communication is hard and takes a lot of time, and that's probably why we sometimes try to avoid it as much as possible. However, when something is hard, do it more often to strengthen that muscle. Talking with people is no exception to this rule. It would be best if you built relationships to understand where people are coming from and their needs.

Another thing is not to underestimate how much you don't know. When you're new to a project, it is easy to underestimate the problem's complexity. Many problems seem small from the outside but are huge on the inside. Acknowledge that there is a ton of stuff you don't know yet. People will usually react nicer to your questions when you start from this assumption. Because now you're not the person who wants to tell people how to do their jobs, but you're the novice who likes to understand how we got to where we are.

Last but not least: time. We don't want to estimate how long it will take to develop a specific feature because we know our estimate will be incorrect. If we can't estimate how long it will take to build one feature , how can we be confident that a rewrite will be any good? We can't. We want to believe it is faster because of all the reasons I've outlined above. When we allow ourselves to believe in these lies, we're no better than any manager who believes that development estimates will align with reality. And we don't want to be this person. We need to acknowledge that even though we'd like something to be better once we've rewritten it, we don't have any proof that this will be the case. Even worse, we know that all things considered, it will probably take longer, and it won't be better but just different and come with a new set of problems.

The graph above illustrates what causes us to choose rewrites nonetheless. Initially, we can move fast. This makes us feel productive and gives us lots of small dopamine rushes because we can ship the first features quickly. We must be on the right track! We're missing that we're not fast because we are incredible software engineers but because the new project is small. Over time the complexity increases, and we learn that some of the assumptions we made (explicitly or implicitly) were wrong. Even worse, we've probably made other mistakes along the way that we now need to also fix. That leads to a decline in productivity and maybe to someone new coming along who wants to rewrite the software again.

Rebuild the parts that need rebuilding

Once you have overcome the initial urge to throw everything away and start over and have talked to people you can start to change the parts that matter.

Is there waste in your processes? Can you, for instance, replace a review step with pair programming? Do this!

Are there parts in your software that are re-purposed instead of re-used? Adapt their names, pull them apart, and improve the code step by step.

There will be real shortcomings (i.e., technical debt) that might be more obvious to you than to the existing team. Be vocal about these parts and look out not to make things worse when you add new features. Build appropriate extension points and maybe devise a plan to incrementally get the software into a better shape. A new pair of eyes with fresh ideas is usually appreciated. The one idea that is not: let's start from scratch.

Have you been part of a big rewrite? Did you cause it, or did you have to live through it? I'm interested in hearing about your experiences. Good and bad! Tell me on Twitter: @philgiese.

Set your project up for success

Philipp Giese — Mon, 17 Jan 2022 11:23:04 +0000

This article is loaded with my personal opinion on how to do things. The baseline I'm going to describe is derived from my personal experience. If you have a different view on some topics, please let me know. I'm not claiming that what I'm doing is perfect!

When I'm talking about a project in this post, I'm thinking of a larger, commercial project. Not something you'd do on the weekend. When I try out stuff on my own, I also choose not to do some of these steps. However, once you're working with a team on something you want to sell to customers, I'd go with the list below.

TL;DR

That's the first time the TL;DR section is straightforward. Have a look at the buzzwords and decide for yourself whether you want to read more or not.

Define the terms you use,
Automate your tasks,
Format your code automatically,
Zero tolerance for lint errors and warnings,
Check for cyclic dependencies,
Tests!, and
Automate deployments

A common language

This step is often forgotten or declared not crucial because it isn't so much fun. You can't play around with technology, and you have to sit down and talk with your teammates. Probably for a couple of hours, even, and I still think it is precious.

You should - as a team - come up with definitions for the words you use. Did you think "bug" means the same to everyone? I got news for you: It doesn't.

That's why it is essential to come to a shared understanding of what these terms mean for your team. The good news is that you can then publicly share the definitions so that other people know them. Especially the fine line between a bug and an improvement is crucial.

Here is a non-exhaustive list of things you might want to define.

Ready (how must a ticket look before you start to work on it)
Done (what needs to happen so that you consider a task done)
Bug
Improvement
Epic
User Story
Idea (e.g., what do we expect to gain from it)

A tool to automate your tasks

If a step in your process is essential, you should look into automating it. It shouldn't be necessary for people to know the process in and out to follow it. All of the activities I describe work best when they run once you push code to your repository.

It doesn't matter which platform you pick. Here are some that I've worked with previously.

Jenkins

Your company might already use Jenkins. I'd guess the likelihood increases with the age of the company you're working in. I must say I've had more bad experiences with it than good. That's not because Jenkins is terrible but because it was always self-hosted. Unless you're in the business of task automation for developers, you probably shouldn't host these things yourself. Pay a company to do it for you.

CircleCI

CircleCI is great. I've never had any issues with it. They have a nice GitHub integration, and their support is always responsive.

The only disadvantage I could think of is that all your jobs and all your workflows need to go into the same file. So, this file will grow over time, and it will become harder to maintain it.

GitHub Actions

GitHub Actions are extremely powerful. You'll need to host your code on GitHub to use them.

The most irritating thing that I encountered was that you don't want to build what GitHub calls an "action" in most cases. You want to create a workflow. Creating your action will probably be a sporadic activity. More likely, you'll use actions that others have built already.

It took me way longer than I'd like to admit to figure this out. And if you don't figure it out, you're set up for a pretty complicated ride.

In contrast to CircleCI, each workflow will live in its file, which makes it easier to create workflows with concise concerns. However, you'll need to redefine tasks like install that might be the same for each workflow (or you'll create an install action).

Automated code formatting

It would be best if you didn't argue about code formatting. Use a tool like prettier and never speak about the topic again. I don't put any effort whatsoever into formatting my code anymore. When I save, and nothing happens, this is a sign that there is a syntax error somewhere.

Also, don't trust the "But we all use the prettier plugin for [IDE]." Someone won't, or someone will have it configured in the wrong way. The last project where I got this answer had violations in 1500 files, and everyone was sure all code was formatted correctly.

You should have this check in your pipeline for when the tool itself is updated. They might have changed a rule. In this case, you should probably reformat the whole codebase inside the PR that introduces the update.

Zero lint errors or warnings

Most people agree that you shouldn't have any lint errors, and I'd say you also should not have any warnings. If you allow warnings, you're going to end up with pointless discussions around whether an error should be a warning or a warning should be an error, or which warnings are safe to ignore. Save yourself time and fix all warnings and errors.

I would also suggest that you agree on a set of rules that make sense in a way that there is a good reason for them to be there. I'll give you an example.

Good Rule

Every link that has target="_blank" set must also define rel="noreferrer noopener"

I consider this a good rule because it statically checks for a genuine threat. If you don't set these attributes, you open up your page to one of the most prominent attacks on the internet.

Bad Rule

Imports must be sorted alphabetically

The order of imports does not have any effect on your code. Rules like this one are the personal preferences of some individuals. If you want to enforce them, look at the section about automatic code formatting. Put this stuff in there. Don't make developers actively think about these things.

🙏 This. 🙏 Is. 🙏 Not. 🙏 Important.

A check for cyclic dependencies

I had to learn the hard way that cyclic dependencies can cause real trouble. Some of the weirdest issues arise when you introduce cyclic dependencies into your codebase. You have a cyclic dependency when two modules depend on each other. Sometimes this is easy to spot, but it becomes more challenging the larger your codebase gets. A cycle of A > B > C > D > A is tough to track down but can cause you real trouble.

So far, I've always used a tool called madge, which saved my ass countless times.

The biggest issue with cyclic dependencies is that they don't always immediately lead to trouble, and they are ticking time bombs in your codebase. And who likes these?

Automated tests

Here are some sentences I don't want to hear.

This is just a prototype. We don't need tests.
We need to move fast. Tests would only slow us down.
This code is too complicated. You can't test it.

Yuck. Yuck! YUCK!

I've blogged about this here, here, and here so I'll keep this short. Testing means understanding. If you think you don't need to understand what you're doing before doing it, go ahead. I really doubt that this makes anyone any faster. The only thing you get is the illusion of speed because you don't know what you're missing.

I don't think there is any reason not to start with tests right away. It doesn't harm a prototype to work reliably. If your software doesn't break all the time, that makes you fast. If your code is that complex, please, add these tests!

Automated deploys

To me, something is only done when users can use it. As long as a feature isn't deployed or released, it can't create value. Therefore, the step to get something out of the door and in front of users must be as easy as possible.

Way too often, it is the case that only a few people can actually deploy and then have to follow a ton of manual steps. I don't get it. We're engineers. We just made the computer do a thousand things, and then we stop at the end?

Sometimes I get the feeling that deploying is seen as something special. As if the ability to deploy is something you have to earn. I think that's total BS.

If you behave like that, it is no wonder if deploys fail often. Deploying your software should be on the same level as running your tests. While there is more to this than merely automating the process, automation is a good start.

That's my list. Is there more? Absolutely. But that's for another post.

Do you have anything to add to this list? Do you agree or disagree? Tell me on Twitter: @philgiese.

Using GitHub actions and Vercel for end-to-end tests

Philipp Giese — Fri, 04 Jun 2021 14:22:10 +0000

End-to-end (E2E) tests are the tip of the test pyramid.
They are supposedly the hardest to write and take the longest to run.
But they are also valuable as they are the tests that "use" your app as your users do.

When we first started to use E2E tests at Signavio, we were able to run them after changes got merged to our master branch but not on feature branches.
That is problematic because it had already made it to our mainline and possibly to production when they uncovered a defect.
Another obstacle to not run them on each PR was that spinning up an environment was not an easy task and would take considerable time.
Since each minute the pipeline runs prolongs the feedback loop for the individual developer, we had a problem.

When GitHub launched actions we discovered that actions can run when a deployment happens.
By then, we already had set up Vercel to deploy our client on each PR.
I asked myself whether I could connect the Vercel deployments with our E2E tests.

TL;DR

No, shortlist this time.
Sorry! Suppose you are solely interested in connecting cypress to a Vercel deployment.
In that case, you can skip ahead to the Creating the workflow section.
If this is the first action you write, then I would recommend the full article.
If you don't have time for that, definitively read Caveats.
That might save you some time.

Vercel deployments

Setting up your project in Vercel is (and I don't say this lightly) easy if you're working with solutions like create react app or static site generators like Gatsby.
Vercel can import and run your project without you needing to do any custom configuration.
But even if you didn't bootstrap your page like that, the configuration is straightforward.

Another reason I'm pointing at Vercel is that they already integrate with GitHub and register themselves as a deployment which is essential for the next step.

GitHub actions

GitHub actions are a powerful tool.
But I also find them a bit hard to get started with.
I struggled a lot with wording and how to achieve a simple goal.
This article is in no form a complete guide to writing good actions.
It's merely a description of what I needed to do to get the job done.

Let's write an action! Wrong.
The first thing I had to learn was that what I needed to do is add a workflow.
I got fooled by the "Actions" tab on GitHub and assumed it would list actions.
But what you will see there are workflows.

An action is one step in a workflow.
A workflow composes actions so that you can achieve a more complex goal.
You also need the workflow to express when you want these actions to run.
Workflows are co-located with the code inside one repository.
For GitHub to run a workflow, you need to create a YAML file inside the .github/workflows/ folder.

name: E2E tests
on: [deployment_status]

Why does this say deployment_status and not deployment?Because we wanted to improve the developer experience.
To give our developers fast feedback on their PRs, we set a custom status.
For that, we need some more information about the deployments.

Suppose you are not interested in building a custom action.
In that case, you can skip ahead to Creating the workflow.

As we will need to work with the GitHub API to set a status on a PR, we will now create a custom action for that.
GitHub offers two ways to define actions - JavaScript and Docker.
In this example, we will create the action using Docker since this is how I know to do it.
We will also use the actions-toolkit to make building custom actions easier.

An action to set a PR status

Since we do not intend to release this action for other developers, we can put all code into a .github/actions/set-pr-status folder inside our repository.
This folder will contain three files.

a package.json because we want to use the actions-toolkit package,
a DOCKERFILE that describes the container to run our code, and
an action.js file that contains the code of our action

Let's start with the package.json.
Since this will not be a package you want to release on NPM, we can keep it short.

{
  "name": "set-pr-status",
  "private": true,
  "main": "action.js",
  "dependencies": {
    "actions-toolkit": "5.0.0"
  }
}

Next, we need to set up the container for our action.
If you have never worked with Docker before, this might look a bit confusing.
An advantage of Docker is that other people have done the heavy lifting for us already and provided base images for containers (i.e., execution environments) that we can re-use.
In our case, what we need to do is:

use a base node image (because we want to execute JavaScript),
copy all files we need into the container,
install the dependencies, and
run the action

FROM node:slim

# The * after package is there also to copy the
# package-lock.json that should is created when
# you run `npm install.`
COPY package*.json ./

RUN npm ci

COPY action.js /action.js

ENTRYPOINT ["node", "/action.js"]

We have defined the dependencies and made sure that an environment can run our action.
Since we want to set a different PR status based on the workflow, the action needs two inputs.

The kind of status we want to set (pending, success, or failure)
A description that gives our developers some context

const { Toolkit } = require("actions-toolkit")

const tools = new Toolkit()

// You could also hard-code these as this
// action is bound to one repo but this makes
// copy-pasting this code easier
const { owner, repo } = tools.context.repo

// The SHA of the commit that triggered this action.
// Makes sure this status is associated with the
// correct commit.
const { sha } = tools.context

// These are inputs that we define. You can extend those
// or change their names if you like
const { state, description } = tools.inputs

tools.github.repos
  .createStatus({
    owner,
    repo,
    sha,
    state,
    description,
    target_url: `https://www.github.com/${owner}/${repo}/commit/${sha}/checks`,
  })
  .then(() => {
    tools.exit.success()
  })
  .catch((error) => {
    tools.log.fatal(error)
    tools.exit.failure()
  })

You might have noticed that we don't need to do any authentication with GitHub.
The actions-toolkit library takes care of this for us as long as it finds a GITHUB_TOKEN environment variable with a valid token—more on this in the next section.

Creating the workflow

The primary goal of our workflow is to run our E2E tests.
If you have skipped the section about adding a custom action, you can skip ahead to Pointing the test runner at the deployment

But it must also make sure that the correct states are shown in the PR.
Here's what we want to achieve.

Show a pending state while the deployment isn't ready or the tests are still running
Show a success state when the tests have finished without errors
Show a failure state when the tests failed

Let's start with setting the state to pending.

Registering a new status check

We have defined our workflow to run whenever there is a deployment_status event.
However, we solely want to set the PR status to pending as long as the deployment is also pending.
Good thing jobs inside a workflow can be conditional.

jobs:
  set_pending:
    name: Register pending E2E tests state
    # This is the place where we define this job to only
    # run when the deployment state is still "pending".
    if: github.event.deployment_status.state == 'pending'
    runs-on: ubuntu-latest

    steps:
      # This checks out the code of this repository.
      # We need this because this is where our action
      # lives.
      - uses: actions/checkout@v1

      - name: Set status to "pending."
        uses: ./.github/actions/set-pr-status
        env:
          # You don't need to configure any secrets for this
          # to work. GitHub injects the GITHUB_TOKEN automatically.
          # We need it in this step so that our action
          # can talk to the GitHub API.
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          # This is where we define the inputs
          # for this action.
          state: pending
          description: Waiting for E2E results

If we now open up a PR, we'll see a new status that reports on the status of our E2E tests.

Pointing the test runner at the deployment

We're using cypress to run E2E tests.
In this scenario, we're using the baseUrl configuration to point cypress to the location of the Vercel deployment.
We also make sure that this job only runs when the deployment was successful.

jobs:
  run_e2e_tests:
    name: Run E2E tests
    # This statement makes sure that this job is only
    # executed when the deployment to Vercel was
    # successful
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    # Thank you cypress for providing a container
    # that works out-of-the-box
    container: cypress/browsers:node11.13.0-chrome73
    env:
      TERM: xterm

    steps:
      - uses: actions/checkout@v1

      - name: Run E2E tests
        run: cypress --config baseUrl=${{ github.event.deployment_status.target_url }}

Reporting the result of the tests back

The last thing we need to do is to set the status to success when the tests passed and to failure when they did not pass.
We can use the exit code of cypress for that.
When a test does not succeed, cypress will exit with a non-zero exit code and mark the job as a failure.

jobs:
  run_e2e_tests:
  # See above for the complete definition of this job

  steps:
    - name: Set status to "success"
      # The "if" underneath makes sure that this step
      # runs solely when the step before was successful
      if: success()
      uses: ./.github/actions/set-pr-status
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        state: success
        description: All tests passed

    - name: Set status to "failure"
      # The "if" underneath makes sure that this step
      # runs solely when the step before was *not* successful
      if: failure()
      uses: ./.github/actions/set-pr-status
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        state: failure
        description: Some tests failed

That's it! We have created a custom action that we can use to set a status check on a PR and a workflow that will run our end-to-end tests when a deployment is ready.

Caveats

I like to mention one thing that probably cost me an hour right at the start.
Because I was thinking in the context of a PR I always looked for the deployment action in the "Checks" tab of a single PR.
But that is not how this works.
Since deployments are not necessarily coupled to a PR, GitHub lists them in the "Actions" tab for the whole repository.
I hope this piece of information saves you some time!

Toggle wisely - A story about feature toggles

Philipp Giese — Fri, 04 Dec 2020 12:12:06 +0000

I've written about early integration and why it leads to better features. While this is true you might not want to roll out changes to all users at once. For instance, you might be working on a feature for a very specific target group and want to make sure that once you release it for everyone there are no major hick-ups. At the same time, you don't want the source code to diverge and rot on a long-lived branch. Enter stage: feature toggles.

Feature toggles allow us to reintegrate new features early but only activate them for certain customers at a time. This can be used for beta testing, A/B testing, and more.

That's great until some of our POs required every new feature to be behind a toggle. This might sound like a reasonable idea but let me explain why I think you should try to keep the number of feature toggles to a minimum.

TL;DR

This post is not anti-feature-toggles. It's about being aware that feature toggles aren't the silver bullet that people want them to be. While they can be extremely useful they can also be used to amplify bad behavior in a broken system.

When you work with features toggles you'll need to be aware that:

Feature toggles introduce another layer of complexity into your system,
they can hide that more work is in progress than you think, and
that you should favor incremental changes over big features

Feature toggles are if-statements on steroids

Have you heard about cyclomatic complexity (also known as McCabe complexity)? In a nutshell, this metric tells you how many distinct paths there are in your code¹. For instance, if you have one if-statement then there are two possible paths.

The if-condition is true
The if-condition is false (i.e. the else-path)

I think the most basic lecture I give to less experienced engineers is how to reduce the amount of if-statements in code. Because the more conditionals there are the harder it is to reason about the code.

Now, if you're on the lookout for this kind of complexity on a method level, then you should definitely be on the lookout on the feature level. Each feature you're toggling adds a condition to your application. One path where the feature is active and another one where it isn't.

What this means is that you'll need to add integration tests for all possible scenarios where different features overlap. This can add significant effort to your development process. It can also increase the chance of defects that are harder to trace because your production system might have many different feature configurations.

When everything is toggled nothing is done

When you're not releasing a feature for everyone this usually means that it is not completely done. Maybe you want to evaluate whether a certain hypothesis holds or you need to check whether the performance of the feature is at a level that you're confident with.

Unfortunately, toggled features do not take up space² in most development boards I've seen teams use. That means that there is no WIP limit on non-GA features. Oof. If that's the case then we might have removed bottlenecks from our development process but we've added a monster of a bottleneck at the end. This is problematic because we're doing a lot of work and not all users can see it. If not all users can use it you're going to miss edge cases. When you're missing edge cases you create defects.

Long story short: a feature that is toggled cannot be considered done. When a feature is not done this should prevent the team from starting something else. If this isn't the case you have a problem.

Favor small changes over a feature toggle

Before you add a feature toggle so that you can hide a new, big feature behind it consider the following:

Could you also add a smaller version of the feature without a feature toggle?

You probably should not use feature toggles to hide the fact that you're doing big bang releases³. Yes, you're already integrating the feature code back into your codebase. But: if no user can use it and, therefore, run the code then how much integration is really happening?

Use a feature toggle when you're unsure whether to add the feature at all. Then build a minimal version and test it with a dedicated user group. Based on that result you either throw it away or continue (in which case you can probably already remove the initial toggle). If you're simply hiding regular development behind a toggle then I'd say you're doing it wrong.

Do you expect a negative effect when your users see the new feature?

Is the feature disruptive? Are you, for instance, altering a core user flow inside your application? Then that's a good reason to not just roll it out to everybody but to do a small test run (see the section above).

However, if you're adding a new feature which does not impact the general user flow then why are you hiding it? You might prevent your users from being more productive faster. If there's nothing to test then don't hide it.

Are you not sure whether the feature makes sense at all? Then why are you already writing code for it? That's a pretty expensive experiment you're running! Paper prototypes or tools like Figma for high fidelity mock-ups might be a better alternative.

Do you add the feature toggle to have fewer bugs in production?

Oh boy. I guess this combines both previous sections. Where should I start?

🚨 Feature toggles will not fix a broken development process! 🚨

That felt good... If you have a problem with too many bugs in production then I'd recommend doing one or more of the following:

define a zero-bug policy,
do TDD (not preaching, read the article 😉),
make sure that continuous improvement is part of your process,
fix problems instead of symptoms,
integrate as early as possible

I'm running out of posts to link here but that list is not exhaustive. The key takeaway is that releasing features (either by not having the code online or hidden behind a toggle) will not solve your bug problem.

What do you think? Am I crazy? have you seen feature toggles being used for the wrong reasons? I'd love to hear your stories! Reach out to @philgiese on Twitter!

A certain someone added a longer explanation to his master's thesis ↩
Once again, I'm talking about teams using Kanban. ↩
Oh hi, waterfall ↩

Violent communication

Philipp Giese — Mon, 30 Nov 2020 09:32:55 +0000

This post is brought to you by Jonathan Soifer. Well, at least without him I wouldn't be able to write it. Jonathan was my manager at Signavio and when I received the feedback that I sometimes come across as too aggressive he recommended reading the nonviolent communication (NVC) book. At first, it sounded like one of these mushy books that I definitively do not want to read. But I trust Jonathan and so I straight up ordered it.

TL;DR

Read it. Sometimes you need another person to tell you something so that you can reflect better. This is what this book does. And it does it in a great way.

After you've read it you're probably never going to communicate in the same way again. Also, this book reduced my personal stress level by an order of magnitude at least. It does this by teaching you to approach situations where someone else is aggressive towards you slightly differently. But it's that slightly different approach that creates a huge difference in your reaction.

In this post I'd like to focus on my top takeaways so far:

being able to describe a situation without judging it,
clearly formulating what I need and expect from others, and
understanding when "No" is a perfectly sensible answer

Describing without judging

Imagine you're living with someone else. You go into the kitchen and there is a stench in the air. It can't be that your housemate did not take out the garbage again, can it? You open up the drawer and flies come swarming out.

Now, you might have already heard that it doesn't help if you snap at the other person (even though this might be what you want to do). Instead, you should describe the situation so that the two of you can talk about it and not get into a fight. You call your housemate into the kitchen, pull yourself together, and say...

You never take out the garbage!

But to your surprise, your housemate does not acknowledge their mistake. You make out a flash of surprise on their face before they reply...

So? You never put down the toilet seat!

This didn't quite go as planned, didn't it? What happened?

You did not solely describe the situation, you were judgemental. When you exclaimed that they never do something (and probably they took out the garbage at least once) you put them into a corner. In this situation, all they could do is become defensive. From there on it's much harder to have a sensible discussion with them.

Here's an alternative.

For the last 3 weeks you have not taken out the garbage.

Now you're stating facts. This doesn't drive them into a corner and they can respond better. For instance, your housemate could now respond with:

You're right. The last couple of weeks were really stressful for me because I had to study for a very important test. I probably have neglected a lot of things because of that.

With this extra information, you can explain that while you acknowledge that this is a stressful situation for them and you're happy to help them you also need them to take out the trash from time to time because you need the kitchen to be clean.

Being able to state what we're experiencing without judging the other side is an important step when you embark on the journey to nonviolent communication.

Say what you need

Being able to clearly express what you need is important. I had to learn that I didn't always state my needs and feelings in a way that was helping the conversation. You might have already heard that when you're in a stressful situation you should describe your emotions to help the other party understand what you're going through. If done properly this can be a valuable tool. But as so often in life things are a bit more complicated than they might seem at first.

Let's look at an example.

I feel frustrated when you're on the phone while we pair-program.

Sounds good, right? You described the situation, and you mentioned your feelings. The other person should now be able to change their behavior in a way that defuses the situation.

Unfortunately not. Imagine the other person now puts away their phone but starts reading a magazine instead. Would this make you less frustrated? My guess is it wouldn't. Right now you imply that the actions of the other person are solely responsible for your feelings. But in most cases, this is not true. Usually, there is a need hidden somewhere.

I'm gonna rephrase the above statement a little.

I feel frustrated when you're on the phone while pair-programming because I want to avoid rework through a second pair of eyes.

Now the statement contains a clear need. By stating that you want to avoid rework the other person knows your intentions. When you communicate your needs and connect them to your feelings you enable the other person to act towards your needs. For instance, your pair programming partner could help you write tests so that you build a safety net while you code.

Demands and requests

This one is my favorite. Because a lot of people (in my experience often times managers) demand something but disguise it as a request.I'll show you what I mean.

Can you deliver this feature until next Friday?

What do you think? Is this a request or a demand? The statement is phrased like a request. Why? Because it leaves the option to say "No". If I don't think I can deliver that particular feature until next Friday I need to say "No" to this question. When the person who asked the question then gets angry they actually made a demand. The way they should have phrased this is:

Deliver this feature until next Friday.

You could still say "No" but you would know from the start that if you do this will become a bigger discussion.

In our day-to-day communication, we need to be careful to not issue demands as requests. Even more so we should learn to not be disappointed when people reject our requests. Every time you formulate something as a request ask yourself "Am I ok with when this request is rejected?". If the answer to this question is no then you need to think about whether you actually demand that something is being done. When this is the case you need to make sure that you are clear about your needs and not judgemental.

I reflected a lot on how I talk to people and I've found myself requesting things when I was actually demanding them.NVC helped me understand the issues with this approach. It's a tool I can use every day to make my life and the life of my co-workers less stressful.

Did this article help you reflect on how you communicate every day? Have you also not been clear about your needs and sometimes judged others when you should have described a situation? Do you already use nonviolent communication? I'm curious to hear about your experiences. Send a tweet to @philgiese on Twitter.

The productivity paradox

Philipp Giese — Sat, 21 Nov 2020 20:40:28 +0000

Here are two sentences that I sometimes hear from developers.

"We are always starting new features but we never finish what we have already started", and
"Because of the WIP limit I feel so unproductive. I can't start work on the next important feature."

Right now it might seem obvious that these statements contradict each other. But during day-to-day work, this might not be as obvious as it seems. When we adhere to WIP limits we make sure that we finish tasks before we start new ones. This solves complaint (1) but also generates complaint (2). If a developer cannot start another topic right after she finishes what she worked on then this can make her feel unproductive.

TL;DR

In a world of deadlines and never-ending backlogs of tasks, it's easy to think you're being productive when you're merely busy. What is important to understand is that:

Being busy neither means that you're doing the right thing nor that you're doing it right,
The goals you set for the team must not contradict the goals you set for individuals, and
That making sure people don't have something to do all the time will get more tasks done faster

Busy ≠ Productive

A common mistake people make is to think that being busy is the same as being productive. I can relate to that because I have fallen into that trap as well in the past. Crunching through work for hours and hours can make me really happy. It gets problematic though when none of the work we do relates to a goal that we or our team aims to achieve. If all that you are doing is keeping your hands busy typing then what good does that do?

Being productive means doing something meaningful. This isn't related to how complex a task is or how much time it will consume. You can be productive working on a task for a week but you can also be productive when that task is done within an hour. The important part is that you make sure that every action you take will get you closer to the goal you want to achieve.

You cannot be productive without setting goals

When you don't know where you are going, how will you make sure you arrive at the right place?

I understand that defining a clear goal can be a hard task. Why? Because the goal might not turn out to be what we initially thought it would be. Since we're all human we tend to not like to do the things that force us out of our comfort zone. We can achieve this by not setting goals. If there is no goal then we can claim we've reached it at any point. While this might make us feel good about ourselves we need to understand that it won't help us in the long run.

Different kinds of goals

If you think about your last day at work, what would you consider goals? Working on a user story?Doing a refactoring?Ticking off the action item from the last retrospective?Implementing one test case?

You might have never thought about these items as goals. Maybe you have even wondered why they exist in the first place? We're doing this exercise of slicing problems into smaller pieces to have items we can work on. Frozen II featured a whole song dedicated to this topic! That is, for instance, why a user story needs to have acceptance criteria. Without them, how would we know when the story is done (i.e. the goal is achieved)?

One of the smallest goals I could think of is a test case. That is probably also why I like TDD so much. Whenever a test goes from red to green I know that I've achieved this particular goal. If I've written and implemented all the tests that make up a user story I've achieved the next larger goal.

When we work like this we focus most of our energy on tasks that are essential. This means we're wasting as little time as possible on tasks that do not help us achieve a goal. And this means we are productive!

I encourage you to use this argument any time someone claims that writing tests make you slower. That's a fallacy for another post though.

Team productivity / Individual productivity

If your organization correlates promotions with personal goals then you might be in trouble. Because personal goals do not always align with what a product or the business needs to achieve. In this respect, developers might work against any change to your development process that prioritizes the team over the individual. While I can understand that managers need some way to evaluate developers there need to be betters ways.

This is not an unsolvable problem. What is important is that you are aware that you might be motivating people to do the thing you don't want them to do. How can you fix this?

Individual goals need to be in alignment with team goals

I believe what is missing in a lot of scenarios are clear goals for the team as a whole. Team goals must be about more than story counts, cycle times, or the number of bugs. They should directly connect the work of the team with the value it creates for the customer. Don't get me wrong. Counting stories and bugs might help to measure the success of a certain goal but they should not be the goal. In the end, what does it mean to achieve super-fast cycle times with no bugs if nothing of what you have built helps your customers?

OKRs might be one way to achieve a hierarchy of goals while also aligning them with each other. You'll need to choose what works best for your team. The important exercise is to be aware of the goals and make them visible. Because after you have done that it becomes easier to see when a goal you set for a person contradicts a team goal.

Provide people with the space to work on individual goals

Now that we have acknowledged that there are different kinds of goals we need to make sure that there is room to achieve them. If your development process is optimized to solely contribute to team goals then you're sooner or later going to run into problems when people work on their personal goals. Some agile methods have introduced the concept of Gold Cards for this purpose. Developers can use them to opt-out of day-to-day work and work on a personal topic instead. My personal experience tells me they are rubbish, though.

When your development process aims at keeping everyone busy all the time it can be hard for a developer to use one of her gold cards. I've noticed that this is sometimes caused by a feeling of "letting the team down".Because when there is so much to do, aren't I selfish if I'd use a gold card?

If this happens it might be a sign of over-utilization which I'll explain in the next section.

The cost of over-utilization

The last two sections were mainly about the point (1). The feeling of not being able to finish a feature can be resolved by creating and communicating clear goals that the whole team agrees upon. While there might still be some frustration it can help the developer understand why the team is acting in a certain way.

Now, we're going to talk about all the work you should not do. If I would be an evil manager I would claim that "A team only performs well when everyone works on something all the time."This YouTube video does a great job demonstrating why this is not true. To avoid people being busy all the time Kanban introduced WIP limits.

However, if not everyone is busy then some people will have nothing to do from time to time. This can make them think that they are unproductive because they confuse being busy with being productive. The book The Phoenix Project provides an explanation that I find useful¹.

Imagine a developer that works on planned work 50% of her time. This means there is a ratio of 50/50 between the time where she is busy and the time where she has nothing to do. We are going to use this as our baseline of 1 (because 50 divided by 50 is 1). If we assume that she is busy 90% of her time then the ratio changes to 90/10. What does this mean? This means that if something unexpected happens (e.g. a critical bug on production) it will take her 9 times as long to work on that task. In other words, by keeping everyone busy all the time we're leaving no room for the unexpected. When we don't expect the unexpected then every bug, security issue, or other kinds of incidents will disrupt the team's productivity. The extra cost of context switches will then make the scheduled tasks also take longer. I think we can agree that this is not a situation we would like to be in.

WIP limits

I've mentioned WIP limits a couple of times in this article. They are a tool you can use to prevent the team from being over-utilized.In my team, when we set WIP limits we tend to pick a number that makes us feel a little uncomfortable. We do this because raising the limit is easier than reducing it. We also make sure that we make this a topic in the next retrospective to check whether the WIP limit works or is harmful. So far, we have never raised a WIP limit.

At this point, you have made sure that your team can react to unplanned activities without disrupting the overall flow. But you've also created a void. Free time is something that can feel weird in the beginning. It might even lead to some frustration as shown in the statement (2). That is why I would recommend that you and your team come up with a list of activities developers can take up when there is nothing to do.

Slack time as a forcing function

One thing I've learned is to give developers some example options of what they could do when the WIP limit prevents them from starting something new. They could:

Pair program with a peer to get their work finished faster,
Do a refactoring that wasn't planned but would be nice to have, or
Write a blog post (guess who's doing this right now)

The important part is that any of the other activities, while they are nice, aren't critical. When a critical bug happens then a developer can exit a pairing session, pause that refactoring, or stop blogging at any time without harm. By not being busy all the time we gain flexibility (dare I say agility). This might seem like a super power from the outside. Hey, this team is working so ~~hard~~ smart that they can react to bugs and requests at any given time and they still deliver on their deadlines.

Even though this might sound awesome in theory you need to make this visible to every single developer.Because in the end there will be situations where the best thing to do is to watch a video on YouTube.This can be frustrating for some developers. When they bring that frustration up, hear them out, show them the numbers, and never be too tired to explain that "the needs of the many outweigh the needs of the few, or the one".

Doing less as an individual so that the team can achieve more is what I call the productivity paradox.

If you want to learn more about this then I recommend reading The DevOps Handbook. ↩

What does a Tech Lead do?

Philipp Giese — Mon, 16 Nov 2020 20:18:34 +0000

I'm transitioning out of my current job as Tech Lead at Signavio. One action item that came out of our last retrospective was that I should compile a list of my responsibilities. That does make sense given that I won't be around come December. However, I don"t really have a list of recurring activities. So, this blog post isn't necessarily about what any Tech Lead does, but what I as a Tech Lead do.

TL;DR

This time there won't be a TL;DR. If you think there is a shortcut to becoming a good leader then you've opened the wrong article.

Do not become a bottleneck

I've heard that some people think that they need to be involved in everything when they are a leader. I couldn't disagree more! You need to trust your people and let them do their jobs. Otherwise, you'll always be busy which means you probably will never be productive.

On a high level, I need to be aware of everything that is happening around me.

What projects are in progress?
What's up next?
Is the team blocked in any way?
Is the infrastructure up and running?
Do we need to work on any critical updates?

Because I have to make sure that everything runs smoothly, I can't get deeply involved with one topic in particular. Even if I'm the one working on the most important feature, I cannot completely neglect all other aspects of my job. The result would be that I will be in a situation where I'm responsible for the most important feature but I'm doing something else. That's not good!

I'm trying¹ to avoid this situation by not working on user stories. Well, at least I don't pull them into the "In progress" column². By doing so I can still pair program with other developers but the chances that someone is waiting for me to finish a story are pretty low.

Help others to grow and learn

If you're not helping others become better at what they do then I would not consider you a leader. Unfortunately, I also had to learn that doing so is easier said than done. Here are some mistakes I've made.

Handing out solutions like free candy

When someone asks you a question your first intention might be to answer it. That's normal. But when you do this you rob them of an opportunity to learn. Because solely when you get to a solution yourself and you understand all the steps it took to get there then you learn something. Otherwise, you might just learn to ask me again the next time you face a similar challenge. And that doesn't scale!

I've come up with my own set of rules to solve problems and I'm now trying to teach them. I'll ask questions like "What did you try already?" or "What do you know? What do you not know?". These questions should help your brain figure out the problem on its own.

If one person needs to learn something everybody can learn something

Pair programming is nice. Mob programming is better!

At first, when we ventured into a new area the new knowledge was restricted to the one or two people who worked on the first user story. It's better than nothing but we still needed to do the initial on-boarding into the topic a couple of times. This led to different people understanding the problem in different ways and lots of rework. Now, when a new topic is on the horizon we have a team session upfront to make sure everyone is on the same page and understands the problem. We (usually) also start working on the topic together. This means the whole team works on the first user story. Stuff like this sounds like we're wasting a lot of time but I'd argue that we're saving a lot of time because there are fewer misunderstandings and rework.

Not being aware of how I'm perceived

Not only do the things you say matter but also how you say them. I had to learn that sometimes even exhaling too loudly could stress the other person out and would prevent them from learning anything. What helped me the most was learning about non-violent communication and how I can use it to make people more comfortable around me. It also helped me understand that I, too, need to be clear about my needs because it also does not help if I get frustrated.

Also, do not disguise requests as demands.

Help people make smaller mistakes

Bugs happen. Mistakes happen. However, you can actively help your team reduce the severity of them.

When shit hits the fan the most important thing to do is to stay calm. It never helps to start rushing work in the middle of a crisis. As soon as the crisis is over you need to make sure it can't happen again. In order for that to work out two things need to happen:

Everyone needs to understand what happened and why it happened.
You need to check whether a change to your team's processes would prevent the thing from happening again.

Your job is to make sure both actions are taken. You don't have to do all the explaining or propose adjustments. However, you should be the one who makes sure that these activities are taken care of.

Here is a non-exhaustive list of things our team does (and I blogged about) to avoid large scale defects.

We do TDD as much as possible,
we write tests that help us find and fix defects faster,
we always get to the bottom of problems, and
we continuously improve how we work.

Make others more productive

I've seen some people interpret this as "make any other person, inside your team, with a technical focus more productive". Unfortunately, that excludes a lot of other people that you might interact with each day. Here are two groups of people you might be excluding even though you shouldn't.

Do you help your product owner?

OK, you can ignore this part if you don't have a product owner (PO).

My team has an outstanding PO. She tries her best to be always approachable and get as much structured feedback from our customers. In the past, she also wrote most of our user stories. The issue is that writing user stories isn't necessarily what she should be doing most of her time. Why? Because she needs to both write the story and then explain it when a developer picks it up. That can be really cumbersome.

We've also faced situations where stories were split in a way that makes sense for the end-user but didn't take our system design into consideration. That sometimes left us with spending more work on two stories than would have been needed if it would have been only one.

We've talked about this and restructured the process so that developers write stories themselves after we've had a session in which we made sure everyone understands the problem we're trying to solve. Our PO is the facilitator for these sessions. This way questions are asked and answered in a larger group.

Do you integrate design into the development flow?

Admittedly, I haven't figured this one out yet.

I try to remove hand-offs as much as possible. I find the idea of a hand-off somewhat appalling because it implies that some people can think everything through and then are done. I've never seen this work.

Our designers are doing a great job. However, they are also humans and, therefore, forget things or make mistakes. when they work on a design for two weeks then hand it over and an engineer discovers that this will never work they've just wasted a lot of time. What I'd love to see is designers and developers collaborate and work on these features together.

But... I haven't found the magic spell to make this work. What I try is to lead by example.

I try to get our designer involved early in the process,
I make it clear that nothing is ever set in stone and we'll always be able to change things, and
I try to instill a sense of incremental development in designers (I've found that some of them are somewhat resistant to that idea)

By doing the above I hope that, over time, both designers and developers will treat each other as equals and collaborate more closely.

Reserve some room for self-reflection

Even though I might risk repeating myself, let me say that again: You should not be busy all the time. In your role as Tech Lead, you also need to reserve the room to think and reflect.

I sometimes re-read comments I made on PRs when I had a stressful day. Chances are high that my personal stress level leaked into my tone while writing³. If others feel bad because I had a bad day then this sucks. Since I don't want to be that person I try to make sure to right my wrongs whenever possible.

Also, blogging helped me a lot this year. To write a post I need to get my thoughts in order first. Sometimes I also need to visualize certain parts of my thought process. That gives others the opportunity to get a look into my head but it also helps me think about the person I'd like to be. I think while writing about goals, processes, and communication I've also reflected on what I did wrong in the past and would like to do in the future.

This time I'd like to emphasize even more that I appreciate any form of feedback in regards to this article. Please reach out to @philgiese with questions, remarks, and ideas to try out when leading development teams.

Of course, I work on the occasional user story. There is always an exception to the rule. The most important aspect is that I'm aware of what's going on around me. ↩
We're working with Kanban which means tickets progress through different stages. When someone picks up a ticket to work on it, it will move into the "In progress" column so that everyone sees that work is happening. ↩
Tools like Grammarly can help you with that. ↩

The religion of test-driven development

Philipp Giese — Tue, 10 Nov 2020 20:14:10 +0000

I recently talked to another developer and mentioned that I use TDD. The other person then asked me whether I'm also one of that religious TDD followers? I had never thought about this. Do people think I'm preaching to them when I encourage them to use TDD? Are they only doing it to not hurt my feelings because they think I'm a believer? Did I create an atmosphere where TDD is a dogma that cannot be questioned?

Let's talk about TDD, why it's not a religion, and why I think it's so much more than a practice to write code.

TL;DR

If you're treating TDD like a religion you should probably reconsider your life choices. Because following something based on dogma won't do you any good. This article focuses on:

why TDD is not a religion,
why I, personally, like to work with TDD, and
that TDD is really not only about testing

Why TDD is no religion

Just because.Because you shouldn't blindly follow any practice to write code. If you think you need to preach anything work-related please stop. Life is too short to get into fights about whether yours or someone else approach is better. You're probably both wrong. At least to some degree. There simply is no perfect, one-fits-all solution to anything out there.

Keep your mind open. Assume that no one gets up in the morning to do a bad job. Be mindful and your stress level will decrease a lot.

Why I write tests first

If you don't like my reasons then you might want to read what Kent Beck has to say on the matter. This section is purely my personal opinion. If you have strong feelings about any of the claims I make I encourage you to reach out to me (find out how in the last paragraph). I'm more than interested to hear your thoughts.

Create APIs from the perspective of the consumer

When you start with nothing you can free yourself from thinking about the details. I know that the test will be failing in the beginning. Even more so, I want the test to fail initially. When I create an API that does not exist yet I can dream up anything I want. By doing this I can make sure that the DX is great. Because when you start from the other side you often end up with APIs that leak details about your implementation. However, when there is no implementation yet you cannot leak anything. This can help you to build minimal APIs that are easier to consume.

Keep me from doing too much

When I start on a new and exciting topic I can get ahead of myself. If I directly start coding and already have a couple of use cases in my head I can end up in a situation where there is a lot of code and the borders between the use cases start to get blurry. Sometimes they even start to influence each other. Tests help me to stick to one use case because I can follow some simple rules. When I write a test first I know that when the test goes from red to green I'm done. If I think that I'm not done yet this means that I need to add another test case. Tests help me split a large task into small units of work.

Make sure the test points to the correct problem

This is one, if not the most important reason. When I write the test first I'm forced to see it fail. I then need to verify that it fails for the correct reason. While doing this I not only make sure that the test targets the correct problem but also that the error message is descriptive enough so that others in the future have enough context should their changes break the test.

You might lose all that when you write your tests after you added the implementation. I've seen it too often that developers tend to write tests that replay what they have implemented. That does not prove anything. Even worse, these kinds of tests fail for any number of reasons that have nothing to do with what they should verify.

Help future me pick up the work tomorrow

Context switches suck.

Even when I continue with the exact same task the next morning I need some time to figure out where I left off and what I need to do next. What helps me a lot is to end the day with a failing test. This way past me sets a pointer for future me for where to pick up the work. I can get right into coding again and make that test green. After that first test, I'm usually back on track.

Give me serotonin rushes every couple of minutes

Each time I finish something I'm happy. Ending a day with the feeling that I haven't accomplished anything is super frustrating. When I code the whole day but can't be sure that what I've done actually works then this makes me sad. But when I TDD myself through the day I have that feeling of accomplishment every couple of minutes. Even better, I can be sure that everything I've done so far works as expected.

This feeling of getting sh*t done and the serotonin rush that follows is probably also the cause why I got addicted to TDD. Right now it's more like a drug (in the most positive way) than anything else.

TDD is much more than writing tests

Here's the kicker. For me, the biggest advantage of TDD is not the tests.I see TDD as a great forcing function to help developers solve the right problems.

When a developer sits down starts with a test she needs to understand the problem first. I would claim that it's close to impossible to write an assertion when you don't know the problem you're trying to solve. At least, it will become much harder to that. If our developer realizes that she lacks some information this happens before she starts building the new functionality.

I'd like to point out that the most important aspect here is that all this happens at the start of the task. This means:

She hasn't spend days writing code she might need to delete later
She is not frustrated because she has not invested much time yet
The person who helps her will not need to understand any implementation details

Not having a possibly wrong implementation frees you from a whole set of issues as well. Without code there you will not have to rework anything. Also, you can't get attached to anything you've already built and try to justify keeping it in regardless.

If you figure out that the problem is not entirely clear then you can clarify it. That's also great because there cannot be blame at this point. Why? Because when the resources you work with aren't enough for you to figure out the problem then it cannot be your fault. At some point, someone has probably made some assumptions and you merely discovered these shortcuts. By asking for clarification you are the hero in this story because you are now making sure that you're working on the right thing.

My personal experience working like this is that it improves the atmosphere a lot. It's easy to get mad at someone for spending days of work but not solving the actual problem. I have caught myself asking Why didn't you ask me earlier?. But then I also realized that the person didn't realize that he was going in the wrong direction. With TDD in place, people stop to ask for help and start to ask for clarification. That's much nicer. Because then the person who asks doesn't try to undo a mistake but actively seeks out clarification to avoid doing that. In my experience, this leads to solely positive reactions.

To end this post on an even more meta-level I'd like to propose a name change. I believe that by working with TDD we will end up with something that I would call CDD or communication driven development. Because this is what I believe is what helps development teams the most. To talk to each other and to do this as early and as often as possible. We talk a lot about feedback loops. But I think people, especially developers tend to think of feedback as something that comes from customers or managers and that means more work. Making sure you're working on the right problem is another form of feedback that is cheaper to get and will already help us a great deal.

What is your experience with TDD?Do you use it for your company? If yes, would you agree with my observations or not? If no, does this article encourage you to try it out? Tweet @philgiese with your ideas and questions.

Undo-redo and its effects on your architecture

Philipp Giese — Tue, 20 Oct 2020 08:03:07 +0000

It's a running joke at Signavio that the product I'm working on does not support undo and redo. And even though we built the tool with redux we're still struggling to get that functionality in. I'm not going to bore you with the details of why it's hard for us but over the years(!) I got an idea of why it might be hard, in general, to get features like undo-redo into an existing app. The good news is that I believe there are steps you can take to make it easier.

TL;DR

I've built a library called react-undo-redo. Check it out on GitHub. 😄

However, this post is more about architecture than it is about undo-redo. The topic is a good example to showcase how certain choices for your architecture can help you built new features faster. Also, you'll see how small decisions can have large impacts. So, go ahead and learn about:

The principles behind undo-redo,
that your components should not care about how they get their data, and
that some concerns are better handled in isolation.

The basics of undo and redo

Let's imagine an application that processes state updates. If there is no notion of undoing or redoing then any update will create a new state which represents the current present. If you happen to use reducers then you might have encountered this as f(state, action) => state*.

Every time something happens inside our app we create a new present state. By doing this we also disregard any information about what came before. It becomes clear that to undo an action we'll need to keep track of our past and also our future (that's redo).

The most basic implementation I know to built undo-redo is to introduce two stacks next to the present state. One stack keeps track of what happened before and the other stack tracks what would happen next. I like to use stacks because they best fit my mental model. This way we're coming from the past, move over to the present, and into the future. You could also use arrays and always append to the end but this would make popping items just that tiny bit harder.

With the above data structure in place, we need to establish some ground rules for how we want to work with it.

Progressing the present

Whenever an action happens that is neither "undo" nor "redo" we need to first move the current present into the past and then create a new present which is our updated application state.

With every user action, we also need to clear the future stack. That's mostly to save ourselves from having some bad headaches. If we would not do that then with every "redo" we would need to figure out which of our possible futures we're going to choose. Everyone who has ever watched Back to the future knows that you don't want to do that.

By the way, I'm not talking about redux actions or reducers here. When I'm talking about actions in this article I mean "the user has done something and our application state is somehow updated". It's important to understand this to not get tricked into thinking undo-redo can solely be built with a redux based architecture.

Undo

You might have guessed already what is going to happen when we undo an action. When this happens we're going to take the most recent item that's in the past stack and make it our present. Also, we're pushing the current present into the future stack (that's important for redo).

Redo

When we "redo" an action we're making the immediate future our current present and the current present becomes the past again. Are you still with me? This is usually the point where I've made sure I've lost everyone in the room.

Now that I've confused you let's move on to the actual topic of this blog post: the effect this has on your application architecture.

How your state access might cause you trouble

For this section let's assume you have an architecture that uses a reducer and actions for state handling. Also, I'm going to use the classic "counter" example to keep it simple in here even though we all know the real world can be much harder. Without any notion of undo nor redo we have come up with the following code.

import React, { useReducer } from 'react'

function counterReducer(state, action) {
  switch(action.type) {
    case 'increment': return state + 1
    case 'decrement': return state - 1
  }
}

function Counter() {
  const [count, dispatch] = useReducer(counterReducer, 0)

  return (
    <>
      Count: {count}

      <button onClick={() => dispatch({ type: 'decrement' })}>
        -
      <button>

      <button onClick={() => dispatch({ type: 'increment' })}>
        +
      <button>
    </>
  )
}

What implications would it have if we would add undo-redo functionality to our application? Quite a lot I would say. We need to ask ourselves a couple of questions.

Do we want the counterReducer to "know" that there is a past, present, and future state?
If the count state would be used in multiple places in our application, would we want all consumers to know about the past and future bits as well?
What impact would it have if we later decide to use a different approach to get undo-redo working?

Personally, the last question got me thinking the most. You might have read my post about cohesion and the section about different kinds of coupling. If we would decide to make the application aware of the past and the present then this would mean a large value for afferent coupling. In other words, undo-redo then will be coupled to everything else in our application. Stuff like this keeps me awake in the evenings. It just sounds wrong.

I pondered some time about this topic and why it bothered me. After a while, it struck me. Of course, it had nothing to do with undo-redo. What I believe we often times get "wrong" (at least to a certain degree) is to move logic too close to the primitives (e.g. useReducer) that manage our state.

Alright, I'll explain.

Given its name, the Counter component should solely concern itself with counting. However, by directly using useReducer it also deals with state management on a very detailed level. The component "knows" that state is handled by a reducer. By coupling the view with how the state is managed we make it harder to change any one of these concerns. Luckily, every problem in software can be solved by adding a layer of indirection.

import React, { useReducer } from 'react'

function counterReducer(state, action) {
  switch(action.type) {
    case 'increment': return state + 1
    case 'decrement': return state -1
  }
}

function useCounter(initialCount) { 
  const [count, dispatch] = useReducer(counterReducer, initialCount) 

  return [count, dispatch]
}

function Counter() {
  const [count, dispatch] = useCounter(0)
  return (
    <>
      Count: {count}

      <button onClick={() => dispatch({ type: 'increment' })}>+<button>
      <button onClick={() => dispatch({ type: 'decrement' })}>-<button>
    </>
  )
}

The new useCounter hook creates an abstraction that helps us to decouple the component from the details of the sate. With this change, we can make changes to how the state and reducers work without breaking the component. In other words, the Counter component does no longer care. However, the reducer still cares. A lot.

Extract concerns to make your life easier

In the last section, we decoupled our component code from the state. This way we could add undo-redo functionality without needing to make the component aware of that. However, the reducer would need quite some adjustments.

function counterReducer(state, action) {
  switch (action.type) {
    case "increment":
      return {
        past: [state.present, ...state.past], 
        present: state.present + 1, 
        future: [], 
      }

    case "decrement":
      return {
        past: [state.present, ...state.past], 
        present: state.present - 1, 
        future: [], 
      }

    case "undo": {
      const [present, ...past] = state.past

      return {
        past,
        present,
        future: [state.present, ...state.future],
      }
    }

    case "redo": {
      const [present, ...future] = state.future

      return {
        past: [state.present, ...state.past],
        present,
        future,
      }
    }
  }
}

If you'd ask me that is a lot of overhead in there. Have a look at the handlers for increment and decrement. Except for the present bit they are the same. By making our reducer aware of the undo-redo feature we're going to have to do a lot of extra typing in the future. Unless, of course, we have the final realization in this blog post. Your reducers probably also do not need to care about undo-redo.

Your reducers "live" in the present. So, why should they care about the past or the future?past and future are concepts that only undo-redo should be aware of. Otherwise, we're back to needing to change all reducers whenever we change something around undo-redo.

function createUndoRedo(presentReducer) {
  return function undoRedoReducer(state, action) {
    switch (action.type) {
      case "undo": {
        const [present, ...past] = state.past

        return {
          past,
          present,
          future: [state.present, ...state.future],
        }
      }
      case "redo": {
        const [present, ...future] = state.future

        return {
          past: [state.present, ...state.past],
          present,
          future,
        }
      }
      default: {
        return {
          past: [state.present, ...state.past],
          present: presentReducer(state.present, action),
          future: [],
        }
      }
    }
  }
}

Using the above createUndoRedo method we can enhance any other reducer and give it undo-redo powers. All without the need to make the reducer (or reducers) aware of this feature. How would this change our application code?

import React, { useReducer } from 'react'

function counterReducer(state, action) {
  switch(action.type) {
    case 'increment': return state + 1
    case 'decrement': return state -1
  }
}

function useCounter(initialCount) {
  const [{ present: count }, dispatch] = useReducer(
    createUndoRedo(counterReducer), 
    { 
      past: [], 
      present: initialCount, 
      future: [] 
    }
  )

  return [count, dispatch]
}

function Counter() {
  const [count, dispatch] = useCounter(0)

  return (
    <>
      Count: {count}

      <button onClick={() => dispatch({ type: 'increment' })}>+<button>
      <button onClick={() => dispatch({ type: 'decrement' })}>-<button>
    </>
  )
}

By extracting the concern of the undo-redo feature we can limit its effect on our implementation. The whole notion that we want something to be undoable is encapsulated inside the useCounter hook. This makes sense since, ultimately, this hook defines where the state lives and what shape it has. Since this is the place where we use createUndoRedo it's also reasonable that this hooks "knows" about the present part of the state. The important part still is that with this abstraction none of our reducers and none of our components need to know about this.

While I was writing this post I thought this would make a nice and small side-project. Enter stage: react-undo-redo, available today.

When do you use abstractions to better design your applications? Do you think I've made reasonable decisions or would you have done things differently? Let know on Twitter.

Thoughts on technical debt and continuous improvement

Philipp Giese — Mon, 12 Oct 2020 08:59:19 +0000

Lately, I've come across a couple of blog posts like "Most Technical Debt Is Just Bullshit" and "Technical Debt Isn’t Real". They're discussing whether we twisted the initial definition of technical debt to mean anything and thus nothing. People have since then asked me about my opinion on the topic. So, here it is.

Before I start I'd like to state one thing I particularly don't like about any of those articles: Their catchy but aggressive titles. I've written about non-violent communication and I believe that if you want to have an earnest discussion you can't use such titles. This article contains my thoughts on the topic and I'm trying to keep emotion out of it.

TL;DR

I believe that when people argue about the definition of technical debt this is a symptom of a larger organizational problem. If engineering teams are not empowered to make decisions on their own this can lead to large-scale problems that no product manager will be able to solve. We need to acknowledge that it's impossible to avoid technical debt and we should, therefore, design our development process in a way to reduce it every day. One way of doing this is through continuous improvement which can come in all sorts and flavors. In the end, it doesn't matter so much what we improve but that we're improving the system because the alternative is doing nothing and this will always hurt us.

What is technical debt?

If you build a house today you'll have to renew the paint roughly ten years from now. This does neither mean that you were sloppy today nor that the paint you picked is bad. The paint wears off over time and at some point, you will need to redo it. Not the people or the tools are your enemies; time is.

Peter Hilton once told me "The moment you write a line of code you also put an expiry date on it". For me, this is what technical debt is all about. Making sure to refactor code before it reaches its expiry date. Because when this happens you end up in a situation where:

developers aren't familiar with the used technology anymore,
the system around the code has changed so drastically that it might be impossible to adjust, or
it seems easier to rewrite the code instead of refactoring it (which means duplicate work).

In short, by not making sure that you update your code before it expires you create waste.

Taking on technical debt

During my career, I got to see software development both from the perspective of an engineer and of a product manager. I've used a language that I would not use today and I've also heard other people make statements that sounded wrong in the past and ring all my alarms today. Let me give you an example.

As a product manager, you need to know when to take up technical debt

This single sentence might summarize everything that can be wrong in a company. I believe that as a product manager you should:

make sure that the team(s) you're working with understand the problems they're trying to solve,
that they have access to the customers who use their product, and
that you are able to answer all the questions around problems and features they might have.

Should you be telling them how to solve these problems? Even more, that they should take on technical debt? I doubt it.

I now think that the real meaning of the statement above is

As a product manager, you need to make sure the team cuts corners to get those features delivered faster.

But this doesn't sound so good if you'd actually say it out loud. Even worse, this has nothing to do with technical debt. This is asking the team to ignore defects in the software to gain the illusion of speed. It's a bit like stating that breaks make a car heavier, and, therefore, slower so why not remove them?

Engaged teams

Even if a product manager makes such statements this shouldn't affect the product. Why? Because it's still the engineers who build the product. They decide what to type into their editor. So, what might be the reason that teams still struggle with the topic?

Product management never prioritizes work to reduce technical debt.

And they don't have to. If you'd ask me I'd like the product manager to stay focussed on the customer. When an engineer says something like that I think in most scenarios it's because the team isn't empowered to make its own decisions. They might be used to a process where managers make decisions and the team then does the work. The company might even like this way of working. However, if you don't make sure that you listen to your engineers then it is only a matter of time until your software rushes past its expiry date.

The issue with refactoring is that the benefits are not as obvious as product managers would like them to be. You try to improve your code before it causes issues and this is hard to report on. Because you would need to measure the time not spent. That's an impossible task.

You will not be able to fix this with any spreadsheet. The only way out of this is trust.

Trust your engineers that they want to do a great job,
trust the teams that they prioritize the right work at the right time, and
make sure the team has access to all information it needs to make these decisions.

Avoiding technical debt

I don't think there is a way to avoid technical debt. However, you can avoid bad code. Here's how I suggest doing that.

Reduce scope not quality

There will probably never be enough time to build all the features. So, when faced with a deadline I'd suggest to rather reduce the scope of the feature. Never "save" time by reducing quality. Because when you reduce the scope of the feature you still ship software that works. This means that when you continue with the next feature you don't have to spend a lot of time fixing what is already out there. In other words, you can continue to move at a steady pace.

When you reduce the quality of your software it means that you're most like busy fixing bugs you don't know about after you have shipped the initial version. This takes away time from any further feature development. If this means that the next deadline is also at risk you find yourself on a downward spiral where you spend most of your time fixing bugs and not building features.

Zero-bug policy

One way of getting ahead of the management game can be a zero-bug policy. I would guess that it's pretty hard to argue against such a policy. It also gives the teams some leverage as long as they are the ones who decide what a bug is and what isn't. By immediately fixing bugs when you find them you make sure that you're always working on the best foundation possible at any given time. This usually leads to better software quality in general.

Also, this can lead to more time for feature development even though this might seem counter-intuitive at the beginning. Because when you make sure there are no unresolved known bugs then you also reduce the risk of "surprises". Any unresolved bug comes with a risk that something larger and unknown isn't working properly. However, if all bugs are resolved you reduce this risk as much as possible.

Continuous improvement

Everything I wrote above boils down to one basic rule. Never stop improving your software. Technical improvements are not projects or initiatives. They are something the team has to constantly do. Compare it to cleaning your bathroom. You wouldn't do that only once a quarter (at least I wouldn't).

A couple of techniques I've used in the past are:

Boy Scout rules

I've constantly used this rule. Essentially it means that you should always leave a place a bit tidier than you found it. While this is also a good rule in the real-world it also helps to keep a software project up-to-date.

For instance, we have the rule to remove lodash methods for filter, map, reduce, etc, and replace them with their respective native counterparts. I did not want to turn this into a full-blown project because there is no big immediate benefit. In the long term, this means we're getting rid of a library that is no longer needed and can reduce our bundle size. Also, while lodash might be known amongst developers right now this does not have to be true in the future so we're making sure that we're slowly but steadily getting rid of it.

Funny Fridays

On any given Friday the members of my team are free to work on any experiments they'd like to. This means they can test out a new library, spike on a refactoring, or try to build a feature they think is awesome. If they choose to not use the time that's also fine.

What I'm trying to do is give them a dedicated space to explore new ideas. Whether they're doing this is up to them. However, I'm also trying to lead by example and pitch ideas I have for possible new "Friday projects".

Reduce team utilization

I've previously blogged about the difference between being busy and being productive. If the team is busy all the time then there is no time for reflection and improvement. That's why I believe you need to create an environment that prevents people from being busy all the time. Similar to "funny Fridays" this helps to create an environment where improvements can happen more naturally.

How is technical debt handled at your company? Do you have more rituals you'd like to see listed in this article? I'm interested in hearing about them!

You're a product manager and disagree with what I said? Also, please reach out and let's talk. I'm always interested in hearing other opinions.

What is cohesion and why should you care?

Philipp Giese — Fri, 02 Oct 2020 08:06:04 +0000

Cohesion means you make sure that the parts that belong together are close to each other. 💥 Bam! Shortest blog post ever!

On a more serious note let's talk about why this is important. If you would have asked me ten years ago I probably would have known about principles like DRY but not about cohesion and its evil twin coupling. I asked my team. Everyone knew and could explain the DRY principle. Some had heard about cohesion but no one could come up with a good explanation. Why is the one principle easier to learn and understand than the other? I think principles like DRY are easier to apply because they come with a set of distinct rules. If you find the same code block multiple times then you're code probably isn't DRY. In contrast, cohesion does not have such a set of rules you can follow. You rather need something we call "Fingerspitzengefühl" in Germany.

TL;DR

This is a tough article and I recommend to read it top-to-bottom. But as always here's the list of key takeaways. To achieve high cohesion you need to:

understand the difference between re-use and re-purpose,
be aware of the two kinds of coupling that exist, and
know about the influence this has on your application design

Re-use what can be re-used but not more

I've mentioned the DRY principle before. That's because I believe that especially less experienced developers can be tricked into bad application design decisions while trying to not duplicate code.

Let's assume you're building an application where users have a profile. That's why you implemented a UserProfile component.

function UserProfile({ user }) {
  return <div>{user.fullName}</div>
}

One day you get a ticket that your customer support department needs a new feature to block certain users. To give support some context when they perform this action (so they do not block the wrong users) you put the button into the UserProfile component. After all, you have already built that component and you want to keep your application DRY.

import DisableUser from "./DisableUser"

function UserProfile({ user }) {
  return (
    <div>
      {user.fullName}

      <DisableUser user={user} />
    </div>
  )
}

But now everyone sees this button even though solely the people from support should. Coincidentally, another developer already implemented a useIsSupportTeamMember hook that tells you whether the current user is from your support team. Great, let's use the hook!

import DisableUser from "./DisableUser"
import { useIsSupportTeamMember } from "./hooks"

function UserProfile({ user }) {
  const isSupport = useIsSupportTeamMember()

  return (
    <div>
      {user.fullName}

      {isSupport && <DisableUser user={user} />}
    </div>
  )
}

While what we have done might be fine for this use case we can get into trouble if we keep on adding features like this. Next, product management asks for a feature to control feature toggles. Then, marketing asks for a feature to get tracking information about a particular user. Now, the component looks like this.

import DisableUser from "./DisableUser"
import ManageFeatureToggles from "./ManageFeatureToggles"
import ManageTracking from "./ManageTracking"
import {
  useIsSupportTeamMember,
  useIsMarketingTeamMember,
  useIsProductTeamMember,
} from "./hooks"

function UserProfile({ user }) {
  const isSupport = useIsSupportTeamMember()
  const isMarketing = useIsMarketingTeamMember()
  const isProduct = useIsProductTeamMember()

  return (
    <div>
      {user.fullName}

      {isSupport && <DisableUser user={user} />}

      {isMarketing && <ManageTracking user={user} />}

      {isProduct && <ManageFeatureToggles user={user} />}
    </div>
  )
}

The tricky part is that the code above isn't necessarily bad. However, I would claim that while we intended to re-use the UserProfile component we ended up re-purposing it. Why? We took four different use cases for a "user profile" and assumed that the overlap in functionality is large enough to build everything into one UserProfile component. This "master" component now contains code for our end-users, support, product management, and marketing. Since there is little overlap between the use cases most of the code of the component is never executed when we render it.

Initially, this might not cause you trouble. But when requirements change and, for instance, marketing wants more features that don't relate to a user anymore it becomes harder to squeeze them into the existing code. That's when things tend to get ugly and people start to talk about "historic reasons" for certain design decisions.

While you might have heard about coupling before I'm guessing that you haven't heard of afferent and efferent coupling. As I'm the author and I do believe it is worth talking about them we're going to have a closer look in the next section.

Afferent and efferent coupling

A-What? Another tool you can use to check whether your software design goes in the right direction is to check for afferent and efferent coupling. Afferent coupling describes how many components will be affected by a change. Efferent coupling then tells us about how many other components have an effect on the component we're looking at. A trick to remember this is to look at the first letter of the words. Afferent starts with an a like affected and efferent with an e like effect.

The picture above illustrates our current design. Notice that the switches we've put into our business logic do not matter here. Since any component can potentially show up for each group the connection needs to be there. This means that, for instance, the ManageFeatureToggles component has an afferent coupling value of 4 (because it's used inside the UserProfile component but also indirectly in ProductAdmin, SupportAdmin, and MarketingAdmin). If we make a change in that component we can potentially affect four others.

Also, the UserProfile has an efferent coupling value of 3 since it uses three components (ManageFeatureToggles, ManageTracking, and DisableUser). A change in any of the three components could have an effect on the user profile itself.

Again, none of these values are inherently good or bad. But they give us something that we can compare against. Let's look at a different way to structure the code. This time we're going to build dedicated components for each kind of user profile.

In this design, the ProductProfile has an afferent and efferent coupling value of 1. If we anticipate more changes to the product profile then this design might be more favorable. Why? Because changes are local to the product use case and cannot influence other parts of our application. We have also introduced a new UserDetails component. This component contains the code that we actually intended to re-use. I would argue that extracting the user details into a small component is the better way to structure our code in this scenario. The new component is small and has no switches which means that all the code (the user information) adds value in every use case where it is rendered. UserDetails has an afferent coupling value of 6 which isn't bad because we've designed this component to be re-used.

What does this have to do with cohesion? In the first design changes to one use case (e.g. product) would have an effect on other use cases (e.g. marketing). This was reflected in high afferent coupling values on components where we would have expected low values. Since these use cases should not overlap but the components are connected I would argue that we did not achieve cohesion because two components that do not share a use case might influence each other.

The second design approach is different. If we wanted to add another component to the product profile we could implement that without affecting, for instance, any marketing component.

This means that you can achieve cohesion by making sure that components that serve a specific use case have an afferent coupling value which represents this. If you have three components that handle product use-cases but one of those has an afferent coupling value of 10 then this is a smell that you have not achieved cohesion.

Impact on software design

In my experience, the most important thing is to be aware of the concepts I've described above. There are no good and bad values for afferent and efferent coupling per se. For instance, a Button component will probably have an enormous afferent coupling value but that is totally fine. What is important is to compare the value with the use case you are trying to solve. Is the component meant to be re-used? Then go for it. But make sure that the component covers only one specific use case. A button should handle clicks and a label. You probably don't want to add functionality to do a video call to it because the use case you are currently working on demands it. Especially at the beginning of your career, this can be hard because this requires you to always keep the bigger picture in mind. This is a skill no one will ever master. I still make these kinds of mistakes all the time. While you can point at some part of your application in hindsight and proclaim "This is where we did it wrong" this can be close to impossible when there are other problems to figure out or a deadline is around the corner. My best advice is to stay aware of the issues and constantly challenge your design.

Design to compose

A benefit of keeping use cases separate and making sure that the coupling values represent what we want our components to do will lead to a more composable application design. This is great because it lets you plug together functionality without worrying too much about the reach of your actions. Subsequently, bugs will tend to stay more local. While a bug in a button might still make that button not clickable it won't corrupt your video conferencing. I refer to these as bugs that make sense. The bugs stick to their respective domain and don't lead to a whack-a-mole kind of situation.

Another big benefit is that high cohesion makes your software easier to maintain. If you discontinue a part of your application you can be sure that you might have to fix the occasional bug but you will not need to adjust the legacy code each time you add a new feature.

Even though this means that we need to spend a little more time thinking about our software design it can save ourselves time and trouble in the future. And that's why you should care about cohesion.

Update 2020-09-01

It got pointed out to me that it might be helpful to include the changed code for the specific profile components as well. Here we go. What you will end up with are more but smaller components without conditionals in them.

Product profile

import ManageFeatureToggles from "./ManageFeatureToggles"
import UserDetails from "./UserDetails"

function ProductProfile({ user }) {
  return (
    <>
      <UserDetails user={user} />
      <ManageFeatureToggles user={user} />
    </>
  )
}

Support profile

import DisableUser from "./DisableUser"
import UserDetails from "./UserDetails"

function SupportProfile({ user }) {
  return (
    <>
      <UserDetails user={user} />
      <DisableUser user={user} />
    </>
  )
}

Marketing profile

import ManageTracking from "./ManageTracking"
import UserDetails from "./UserDetails"

function MarketingProfile({ user }) {
  return (
    <>
      <UserDetails user={user} />
      <ManageTracking user={user} />
    </>
  )
}