Forem: Lucas Medeiros Reis

The Single Most Important Driver Of Software Quality

Lucas Medeiros Reis — Fri, 02 Jun 2017 13:21:30 +0000

So, there I was, ready to press the button. All the while three hundred people were looking at me, thinking "is this guy going to make my work more difficult?". I had nowhere to hide, and the button needed to be pushed.

My claim: developers having skin in the game is the main driver of software quality. If your life depends on the software you are building, you can bet that the software will be of a much higher quality than any software that your life does not depend on. That being said, there are also "second best" scenarios, like being near people whose lives depend on the quality of your code, or being an actual user of the product yourself. I cannot imagine any better factor than skin in the game to drive quality.

But let me start from the beginning.

I work at a large online retail company, with millions and millions of users. In addition to the millions of users using from home, a modified version of the website is leveraged by a telemarketing sector within my company. As you may have guessed, these are the three hundred people who were not very comfortable with my presence.

I can't say that I blame them for being uncomfortable, because they do earn a share of what they sell for the company. In other words, they have a lot of skin in the game. They don't use the website as a normal user, to buy something; they use it as a selling tool. If the checkout doesn't work, they don't sell, and they don't make money. Needless to say some of them are thinking, "I'd kill the person who broke this thing!" when the site goes down.

When I pitched my idea to refactor/rewrite a part of our checkout to upper management, they were a little unsure of the project. First, I tried to sell them on it with a promise of an increase in code quality and better performance, and when they still seemed hesitant, I said, "I'll deploy the changes to production in the middle of the telemarketers". At that moment they realized I was serious about idea, and they agreed to sign off on the project.

As soon as I got back to my work station, I felt something different. As always we were very serious about our metrics and processes, and we took every action necessary to be sure the website would not have any problems for our millions of users. But just the thought of deploying the changes to production while at the telemarketing headquarters, next to people who depend on the site to make a living, gave us a different feeling. From early on, everyone on the team worked hard to avert potential disasters down the road.

Not only were we super careful with our code, we also got very engaged with UX, UI, and with the infrastructure of the project. From the very beginning, our team came up with strategies to deploy the application while causing the least impact possible. We created a "panic" link for the telemarketers to go to the previous application. We even developed a new telemarketing-only feature to help them sell additional services and insurance, as an incentive not to press that panic link.

Our website is really large, with lots of different micro-services and different clients interacting, and at this level of complexity, it's almost inevitable that problems will occur in production, no matter how much testing you do, or how careful you are in your process. With that in mind, it wasn't a huge surprise that on deploy day, in the first ten minutes, we had a small problem affecting the sale of gift wrap.

Everybody had been cordial to us from the beginning, but I could feel the tension in the air during those ten minutes. And then, after rolling back and fixing the issue, I could still see that the telemarketers were not very confident. What other errors would affect the sales?

But by the end of the day, everything had gone well. We had anticipated almost every scenario, and we were ready to fix all the problems that were raised. The application started to run smoothly, and no one was using the panic link anymore, success! By the end of a couple of weeks, we learned that services and insurance sales had risen by 20% for telemarketing, and we raised the overall conversion of the website by about 2%.

So, when I see that Basecamp uses Basecamp, and the VS Code team uses VS Code to build VS Code itself, I understand why those products have really high quality. They have skin in the game.

Final notes: of course this is not a scientific claim :) It is a hypothesis raised from our own experience, and we continue to observe this property in different projects. The effects of skin in the game are usually very well described by some economists and philosophers, and especially by Nassim Taleb. He's actually writing a book called Skin In The Game, and a lot of really interesting chapters are online.

This post was originally published in my personal blog.

Solve All Bugs Or Implement New Features?

Lucas Medeiros Reis — Thu, 06 Apr 2017 14:39:02 +0000

This post was originally published on my personal blog.

Most projects have moments in which they have new features to implement and, at the same time, bugs to solve. A choice has to be made on whether to focus more time on solving those bugs and making sure they don't appear again - let's call this option quality - or implementing new features - scope.

A lot of arguments have been made in favor of quality. One of my favorites is an argument presented by Joel Spolsky in this already classic text.

In this post I'll make the argument that, even from business and financial points of view, focus on raising quality should always come a before focus on implementing new features.

The concepts

First of all, let me explain better what I mean when I talk about quality and scope. Let's say there's a demand for X new features to be implemented in a given period and we know there are currently Y open bugs. One could say:

"We are going to implement all X features and solve as many bugs as we can."

Or you could say:

"We are going to solve all Y bugs in their root causes, and we'll implement as many features as we can."

See the difference? It's a matter of explicitly stating what your priorities are. In the first example, we'll make sure everything we implement is well tested and well designed, but some bugs will not be considered due to time constraints. In most cases, bugs will be listed in a spreadsheet or on a board with a "priority" property, and the business will choose which bugs should be solved per sprint, based on a "financial" analysis (which I'll talk about in the next section).

In the second case, a team will focus all of its energy on, at least, investigating every bug that appears. Whenever there's a decision not to solve a bug right away, it's because the developers diagnosed that the bug is not a manifestation of something that would stop the business, or incur a big loss.

Those are what I'm calling in this post "a focus on scope" and "a focus on quality", respectively.

Losses due to lack of quality and scope

One of the main criteria needed in order to choose between focusing on solving a bug or implementing a new feature is an analysis of the financial loss that the business will incur by not doing these things.

It's impossible to know for sure the loss a business could have in the future, so we'll need to estimate it. And to do the estimation we'll need to deal with uncertainty. My point is that the uncertainty of a loss due to not solving a bug is qualitatively different from the uncertainty of a loss due to not implementing a new feature.

To explain this difference, I'll briefly present two concepts introduced by author Nassim Taleb in his book Black Swan. I suggest this book to anyone interested in having their mind turned inside out. :)

Mediocristan and Extremistan

Taleb argues that uncertainties can "come from two places". The first place is Mediocristan: observations do not vary very much, or vary in a somewhat predictable way. An example of an Mediocristan uncertainty is "the average height of a random group of a hundred people". After you measure five or six groups, you would have a very nice idea of what other averages will be. Even if one person in the group is the tallest man on Earth!

Then there is Extremistan, a place where one observation can be very, very different from the others, which makes predictions very difficult, sometimes even impossible. An example is "the average net worth of a random group of a hundred people". Every group will have a very different average. We could have a run of thirty, or even fifty groups with a very similar average, and then suddenly Bill Gates is in a group and "BAM".

So, here's my first claim:

Losses due to not implementing a new feature belong in Mediocristan.

A new feature is something that in the past did not exist in your business (duh). That means that not having a new feature, in the majority of cases, will not cause a big loss. Or at least it will not make the losses you already have any bigger.

Let's consider for example an e-commerce business that wants their users to be able to pay with a new payment option, let's say Bitcoins. It could be difficult to estimate how much the company would make, but it's relatively easy to see that there won't be a big loss if the Bitcoin feature is not implemented. Current users are buying with current payment options and will continue doing so even without a Bitcoin option.

And that leads to my second claim:

Losses due to not solving a bug belong in Extremistan.

Every bug, no matter how small, can be the tip of the iceberg to a much deeper problem. That's why it's so difficult to estimate the effort needed to solve a bug. And deep problems can stop a business in it's tracks.

A bug in the error feedback of a credit card form could explain the drop in conversion rates. Also, that bug could be a manifestation of a much deeper problem with server side validation, which, in turn, could start to appear in other forms.

Some errors can be solved quickly by "patching" them. But that means that errors could reoccur if someone touches the code, or uses those same modules elsewhere. A lot of times, a more complex refactoring has to be done carefully, and must be fully backed by unit, integration and end to end tests. With these tests, we can diminish the chance of a bug creeping back up on you.

But is the whole team always going to be working on bugs?

What about all the new features that we need to implement??? When will that happen?

By solving every bug at its root cause, we diminish the chances of new bugs appearing. That means that in the not-so-long-term, almost everybody in the team will be focusing on implementing new features. The project gains traction, and will deliver a larger scope with a higher quality. And that's what everyone wants, isn't it?

I'm not saying that all bugs have to be solved right away. I'm saying that all bugs should be investigated, and the developers should diagnose whether or not the root cause could give rise to other new, more dangerous problems.

A script for dealing with a bug

Every time a new bug is reported, and comes to the developers to be solved, ideally we should follow the following steps:

Investigate the root cause of the problem. Ideally this should be done in pairs, with at least one of the developers having already worked on that part of the code. That would make things quicker.
Choose whether or not to solve the bug right away. The answer to this question should be "yes" in most cases.
Write tests to reproduce the bug. Unit tests preferably, but don't be afraid to write new integration and/or end to end tests, if needed. By creating new tests, we try to guarantee that if anything brings the bug back, it will be caught by tests before going to production.
Solve the root cause as well as you can.

This approach is not new. It's actually one of the main components of Toyota's "Lean Manufacturing" revolution, called Jidoka.

Summing up

Losses due to not implementing a new feature are much more predictable than losses due to bugs not being solved. Every bug is a potential business-stopper. And that's why we should always give quality a higher priority than scope: to keep our businesses alive.