Forem: Dawid Kędzierski

Why do projects become worse over time?

Dawid Kędzierski — Wed, 18 May 2022 19:50:07 +0000

In the Clean Agile book, Robert C. Martin wrote:

“It is the single greatest indictment of the software industry, the most obvious evidence of our failure as professionals, that we make things worse with time. The fact that we developers expect our systems to get messier, cruftier, and more brittle and fragile with time is, perhaps, the most irresponsible attitude possible.”

But is it really?

Don’t get me wrong, some developers tend to think that it is the nature of projects’ lifecycle to become worse over time, which is very dangerous. So, if the future seems to be so fatalistic, what’s the point in trying, right? I don’t need to explain to which pathologies it leads.

Nevertheless, I want to believe that in the majority of cases it is not so obvious that the developers are to blame and putting all the responsibility on them could be deeply unfair.

Let’s start with Murphy's Law: “Anything that can go wrong will go wrong.” Or to paraphrase: shit happens. No matter what you do: cry, scream or have a tantrum, you are sometimes powerless against the odds. You can try with all your might, but some things won’t work with you. It’s not a matter of opinion, it is just stating a fact. It’s a rule of thumb based on hundreds of developers’ experience.

Do you know The Harvard Law of Animal Behavior? It states “Under controlled experimental conditions of temperature, time, lighting, feeding, and training, the organism will behave as it damn well pleases.” Perhaps it’s not limited to the living organism only?

It will not come as a surprise that the more complex system is, the easier it is for unpredictable behaviors. The Cyclomatic Complexity increases over time together with the number of new, often thoughtless, functionalities. Hence, the number of cases to cover by automated tests increases as well. It quickly turns out that this number exceeds the team’s capabilities. In extreme cases, for most complex systems, the truth is they work in constant failure mode. The situation happens so often that new terms were coined. You can read e.g. about Crash-only Systems in another of my articles. There’s also a nice article about Chaos Engineering.

However, the project problems aren’t often purely technical. “The major problems of our work are not so much technological as sociological in nature.” [Peopleware: Productive Projects and Teams.] Talking with various stakeholders I repeatedly say - from the technical point of view almost anything is possible. The problem often touches different aspects, and one of them is management quality within a team, a project and a company. A great example has been described in “Extreme Ownership, How U.S. Navy SEALs Lead and Win” by Leif Babin and Jocko Willink: there were two teams, one the best and the other the worst, they were having a race in pontoons, in the same unfavorable conditions - cold, wet, with extreme physical exhaustion. It was enough to switch the leaders of the teams in order to make the worst one the best, and the best one the worst. It’s a true story. The same team, with the same budget, with the same goals and project scope, within the same time, will provide completely different results depending on the quality of management. Having said that, how often is the quality of management taken into account when making technical decisions?

So, why do projects become worse over time? Is it really due to developers’ lack of professionalism? Or maybe, it’s everything else, their surroundings which force them to make questionable compromises, with themselves, with projects, with team mates, or managers? Are we coming back to the nature vs nurture discussion in the context of developers? Or are we, as human beings, just not able to evaluate complex systems due to limited cognitive capacity? It’s easy to blame one group of people, but there are just too many factors to make such a simplified reduction.

The quick story of Crash-only System

Dawid Kędzierski — Thu, 12 May 2022 08:56:18 +0000

Some time ago I was working on a project that checked all the boxes for being absolutely awful. One of the biggest obstacles was emitting hundreds of events with every single user action. Obviously, it caused many unpredictable issues. Issues which should have been handled properly. Or eliminated. Both of which mean laborious effort, often costly and threatening for the project. What if there was a third path?

In the mentioned project we had diagnosed that one functionality accounted for 20% of the overall number of bugs. Just one! Moreover, almost any attempt to use it ended up with a bigger or smaller issue. A lot was done to improve it. Many talented developers had to admit defeat. And, what’s even worse, every single try, every single developer, left various artifacts behind in the source code which added fuel to the fire by increasing its complexity.

Still, we couldn’t give up. It was producing too many issues.

The functionality was characterized by refreshing the whole application, almost rendering it anew, among other things. Following that lead, we measured the average cycle time and adding to it initial bootstrapping time during the first application run. It turned out that it was 500ms. It’s a lot, and I mean a lot a lot, so we started looking for possible improvements. Our goal was an ambitious one, it assumed going down below 30ms which stands for maximal permissible macro-task time in Chrome. We reached it a few weeks later. Then it was time for phase two - each time an error occurred in the functionality, we wanted to turn off and on the system, restarting the whole state of the application including all the errors. We could do it, because the application didn’t require the persistent state. If it takes less than 30ms, a user won’t even notice it, right? Quickly turned out that 30ms is long enough to notice a “blink”, that’s why we used a trick known from OpenGL among others - we loaded next frame in the background of the old one and then switched them. In other words, the new instance of the application hid behind the old one until it was ready to be displayed. The whole operation took slightly more than 30ms.

A few weeks later it appeared that we implemented Crash-only System totally unconsciously and unintentionally. Quoting Wikipedia: “Crash-only software refers to computer programs that handle failures by simply restarting, without attempting any sophisticated recovery.” Isn’t it mind-blowing? Instead of spending hundreds of hours trying to develop proper error handling, never being sure if all the cases were covered, we can approach the problem from a different angle - assume that if there’s an error the service would be able to reboot quickly enough without interfering the proper functioning of the system as a whole.

But we went one step further! Based on statistical data, we quickly calculated the probability of a bug occurrence within the functionality. The probability was so high that we decided to further simplify the new approach to the problem - instead of maintaining the source code responsible for diagnosing the moment of bug occurrence and restarting the system, we assumed that the problem will occur for sure so we reboot the application immediately.

I know what you’re thinking. Everything here is one big lie, because in fact we didn’t fix anything, we only hid the problem which we couldn’t solve, right? Having said that, the fact is we: (1) have significantly boosted application performance; (2) from the technical point of view the system works properly right now, because it meets the users’ expectations; (3) over a course of a few weeks, we have reduced the overall number of bugs by 20%. And the solution, although it might seem surprising at first, it turned out to be trivial and well-known for everyone - the only thing we needed was just to turn it off and on.

Estimation - money loss?

Dawid Kędzierski — Tue, 19 Apr 2022 11:23:51 +0000

The way of thinking when we try to bring a whole complicated process of software delivery down to a single number, to a single date, to a single measurement is a post-industrial inheritance. If you have a factory, you are more or less able to determine the velocity and capacity of an assembly line or a robot in a given period of time. Creative branch, fortunately or unfortunately, eludes such simplifications. Is a writer able to tell when the inspiration will come or the muses show up? Highly doubt it. Luckily, IT branch is somewhere in the middle. Most of the time we don’t produce highbrow or lowbrow art, but we’re not an assembly line either.

Some people think that Agility has been created to speed up development. Well, it may seem so, but not because things are happening faster. Each project has four basic characteristics: cost, time, scope and quality. It’s implausible to satisfy all of them at the same time. Successful project management requires skillful balancing between these four elements. And here Agility kicks in, as it is a powerful tool thanks to which we know the team’s velocity and the current state of the project. It helps to make well-informed and data-driven decisions. There’s no magical way in which impossible becomes possible; for instance, if a project requires a given amount of time, Agility will not suddenly make it shorter. However, when you know that half of the time has passed and you’ve accomplished only 1/3 of the work, then you can make a suitable decision. Tough as it may be, it’ll save the project as long as the situation isn’t desperate.

Putting that aside, let’s focus on the crux of the matter - estimation. First of all, you need to answer the fundamental question how important the estimation in the context of your project and organization is. Second, an equally important issue is the required accuracy of the estimation.

Think about this: you work for a company which is about to sign a new contract, with a fixed timeframe and budget which will be based on your estimate. Thus, your estimate is key as it can either sink or swim the project. Or, you work for a product-based company which is crazy about quality and can even accept delays and/or exceeded budgets in order to keep its high reputation.

Each project new context, and so different needs and expectations.

The same goes for accuracy. In some cases, simple expert estimate is enough. A developer who knows a project reads a draft of requirements and 5 seconds later provides an estimate. It is probably one of the most inaccurate ways, but in many cases it may be good enough. For many projects the margin of error is quite high. At the same time, it is one of the cheapest estimating methods. Having said that, it is important to emphasize that expert estimate works well for smaller projects, more granular tasks, and not so well for big items with many question marks.

In another situation, a much more accurate estimate is necessary. It could require much more time and money, but even more money is at stake. Sacrificing four weeks to prepare a blueprint, to collect a list of detailed requirements, to create spikes and to do necessary R&D, all of that performed by good - but pricey - professionals, can be worthwhile. Mind so it doesn’t turn into Waterfall!

No matter what, there’s always a margin of error. The question is how much of this margin is acceptable?

Remember, any practice taken to the extreme can become harmful. And, we shouldn’t be doing anything without fully understanding why we’re doing it. Ask yourself, what kind of problems you’re trying to resolve by doing estimations, and what way of doing it is good enough? Any other will just be a money loss.

What is the system supposed to do?

Dawid Kędzierski — Tue, 05 Apr 2022 13:09:11 +0000

In nearly every legacy system, what the system does is more important than what it is supposed to do. [Working effectively with Legacy Code]

Let me describe you a real-life example.

Some time ago my team was struggling with a tricky issue. A typical situation without good solution. Imagine an application which allows users to freely modify layout settings, like color of the button background. Unfortunately, there was a bug in the system - instead of changing background color, the setting was changing the border color.

The bug occurred a while ago, at least a couple of months, and users, whether consciously or not, got used to it in the meantime. So, when the issue was found and resolved, so that the setting started to change background color once more, many clients treated it as... a new bug.

How is that possible that a behavior, which works as it was originally intended, was perceived as an incorrect one?

Well, the incorrect behavior was occurring so long, that it became a new and desired system function. The baseline has changed. It is more important that over the last dozens of versions, the program worked in a non-intended way, despite the original specification. So, in this particular case, instead of changing the system behavior, it would be enough to change the name of the setting, e.g. background color to border color.

The key to understanding the situation is to focus on the phrase “was resolved”. What does it even mean? Is resolving an issue supposed to mean returning to the original intention of a designer, or is it about reflecting how a system actually behaves? So maybe it is about stabilizing, legitimizing or accepting rather than fighting?

What is the system supposed to do? That’s the most fundamental and the most difficult question while dealing with Legacy Code. It’s hard to find proper criteria for this evaluation. No matter what you have decided: communicate clearly with the users, keep backward compatibility and perform impact analysis. No one likes not to know what is happening, no one likes it when things brake without their fault, and everyone should be doing deterministic analysis of changes to prevent the previous two from happening.

Improve Product Quality by focusing on Developer Experience

Dawid Kędzierski — Tue, 08 Feb 2022 21:53:39 +0000

Some folks use QA as the bug catchers. They send them code that they haven't thoroughly checked. [The Clean Coder]

Everything starts with the source code, and developers are the only people capable to change it - to better, or to worse. It’s obvious that the QAs don’t create the bugs, they just find them! The bugs, and hence the quality, starts with the source code.

Having said that, if the project DX is really poor, then it is extremely hard to improve the overall Product Quality. Regardless of the chosen measures.

Consider an example - you’re going through the legacy project in order to fix a bug. The mess is everywhere, you see many places worth refactoring, but instead of doing anything like improving it, you don’t even touch it, you do everything to touch as few lines of code as possible, because any change may break something, and you don’t want to be the one responsible.

This is what I call poor Developer Experience. You noticed the problem, but didn’t fix it. You violated your professional ethics, because you were so afraid of making a mistake. Naturally, you don’t want to ruin your professional reputation, but in that particular case, it also means that you prefer to leave the problem to fester and exacerbate rather than be perceived as the one who touched and broke something.

In the long-term, it’s impossible to work like that, to drive the project like that.

In a perfect world, you would have tools and processes to help you, like Tests (Unit, E2E, Integration), Static Typings, Linter, other CI quality gates, and so on and so forth. All of them would confirm that your changes are safe, so you would be proud that you made the project a little bit better. And for sure, you wouldn’t be having any ethical issues.

When a developer isn’t afraid to change the code then they change it all the time. This is simply the nature of software delivery - to make constant adjustments to the new reality, to the new business request. It is not dangerous to make constant changes, incremental improvements to the code, but leaving the code static is.

Trust developers. There’s nothing more devastating than micro-management. Let them be free and responsible. It will generate a feeling of ownership and commitment.

Give developers all the tools they need and ask for. If they hate testing library - they won’t use it, or only as little as required. If they hate slow CI workflow - give them time to improve it. If they want to refactor something - at least consider it! The idea is pretty straightforward - when someone doesn’t like what they do, feels forced or disrespected, then we cannot talk about efficiency, thus we cannot talk about Product Quality.

22 rules to follow to become professional developer

Dawid Kędzierski — Mon, 16 Nov 2020 22:00:15 +0000

Inspired by The Clean Coder of Robert C. Martin.

1. Being professional means being responsible.

Non-professionals leave responsibility for the job they do to their managers. Professionals take responsibility, if they did a mess - they will clean it up.

2. Professionals don't create bugs.

Professionals take responsibility for their imperfection. "The fact that bugs will certainly occur in your code does not mean you aren't responsible for them". Professionals error rate should decrease over time and get closer and closer to zero, even if zero is virtually impossible to achieve.

3. Professionals don't release code to QA if they are not sure it works.

Releasing code to QA if we don't know whether it works is "desperately expensive behavior" and "ruins schedules and undermines the confidence of the enterprise in the development team".

4. Professionals test their code.

That's the only way to be sure that their code works. The only proper way. 100% test coverage is their goal.

5. Professionals know The Boy Scout Rule and use it every day.

"Leave the campground cleaner than you found it". Leave source code you found, a little bit better, each time.

6. Professionals read, learn and practice after working hours.

It is not an employer responsibility to train developers, give them time to learn and make them marketable. Some of them do that, but that is a favor, not responsibility. "True professionals work hard to keep their skills sharp and ready. It is not enough to simply do your daily job and call that practice".

7. Professionals have the courage to say no to their managers.

Good managers expect that from them.

8. Professionals don't promise to try.

Trying usually means putting extra effort. But if you don't have reserve of energy, don't have a new plan and original estimates are still reasonable, then dishonest is promising to try.

9. Professionals don't hope.

Hope destroys projects and their plans. If something needs 10 man days, it won't take 8 man days. Professionals make sure their stakeholders are aware of the situation, they don't let anyone else have hope.

10. Professionals deliver what really needs to be delivered.

It is common that provided requirements are not the actual needs of the customers. Usually, clients don't know what they want or need, before they see it. They don't know how to describe and precise their expectations. Professional developers know that and proactively confirm the requirements they got with the real needs of the clients.

11. Professionals do not code when they are tired or distracted.

They know that otherwise created code will be incorrect and have to be redone.

12. Professionals know that development is a marathon, not a sprint.

They conserve and care about their energy and creativity. They know how easy is to miss elegant solution just because of being tired.

13. Professionals know long-term consequences of overtime work.

They agree to work overtime only if it is not a common long-term situation and their managers have fall-back plan in case overtime work won't help.

14. Professionals know how to estimate based on low precision requirements.

Business needs to authorize projects based upon estimates. Professionals know that they are not sure exactly what they are supposed to deliver, so they make sure that their stakeholders understand the uncertainty by including error bars to their estimates.

15. Professionals have a single definition of done - done is done.

Done does not mean something is waiting for code review, or waiting for QA review, or is in any other status than actually DONE.

16. Professionals have the courage to leave boring meetings.

But they remember that there's no need to be rude and say "This is boring!".

17. Professionals are not passive-aggressive.

They do not agree just to end a conversation and then sabotage a result by refusing to engage in a solution. They never think or say things like "you've got what you wanted, that's your problem".

18. Professionals know priorities.

"Professionals evaluate the priority of each task, disregarding their personal fears and desires, and execute those tasks in priority order". By doing this, they avoid priority inversion.

19. Professionals don't tolerate mess.

"Professionals do not succumb to the temptation to create a mess in order to move quickly. Professionals realize that "quick and dirty" is an oxymoron. Dirty always means slow! We know that messes will slow us down, causing us to miss dates and break commitments."

20. Professionals manage their stress.

By trusting their disciplines. Hard time is not the reason to abandon them, quite the opposite - that's the time to pay special attention to them. The only way to go through the pressure is to rely on what we already know works, so e.g. if you usually follow TDD, don't stop it. Professional developers don't rush - that's the easiest way to go into a deeper hole and create a bigger mess. Remember point 19th?

21. Professionals pair.

That's the most efficient way to solve most problems. That's also the best way to share knowledge with each other. Professionals teach young developers discipline and skill of being a craftsman, they guide next batch of software engineers to maturity.

22. Professionals are team players.

They keep an eye out for teammates and help them when they are in need.