Forem: Blaine Osepchuk

How to be more successful at software maintenance

Blaine Osepchuk — Wed, 18 Nov 2020 15:41:17 +0000

In this post I want to explore the kind of software maintenance that happens after active maintenance ends. This is the stage of the software life cycle where the software has been in production for some time and the initial wave of bug fixes and enhancement requests ends. It's also at this point that managers may cut the number of developers assigned to the project and turn their attention elsewhere.

Keeping this kind of project running safely, securely, and profitability as the years tick by--even if you're not adding or changing functionality--is especially challenging.

I've spent more than a decade in this situation. And I'd like share some of my observations and advice about maintaining this kind of software with you.

The 8 laws of software maintenance

If your software lives long enough you'll experience one or more of the following:

1. You'll have trouble getting enough resources to maintain your project

Keeping old software running isn't sexy. Managers are more interested in acquiring new functionality than maintaining older systems. And, very few programmers want to spend their careers doing maintenance so both talent and money may be in short supply.

2. Your hardware and software dependencies will become out-of-date

The languages, frameworks, libraries, APIs, development tools, build systems, and hardware used by your project are constantly shifting under your feet. Some stacks are shifting faster than others but nothing is standing still. And the more dependencies you have, the harder your project will be to maintain.

3. Your code base will be a mess

I can say this with a fair bit of confidence because most code bases are a mess. But very old code bases can be especially problematic. They might have been developed without unit tests, documentation, static analysis, coding standards, style guides, code reviews, or modern development methods.

Very old code bases have frequently been worked on by several programmers with different programming styles over time. And if your software has been underfunded or developed with speed as the top priority, it probably contains heaps of low quality code.

4. Your memory will fade

Your understanding the requirements, design decisions, and how your code works degrades rapidly after active development stops. People forget the details or take valuable knowledge with them when they move on to other projects. And your documentation and test coverage will almost certainly be inadequate to make up for that missing knowledge.

5. You'll lose something important

Information, software, or hardware critical maintaining, building, testing, documenting, or deploying your software may be lost. Think config files, compilers, compiler switches, hardware configurations, encryption keys, source code for dependencies, drivers, backup scripts, development environments, test environments, etc.

Losing any single critical component of your project could mean that it cannot continue to operate without heroic effort on your part. And some problems--like losing the contents of your database--can leave you with no choice but to shutdown your project.

6. You'll need to change your software

You may not want to change it but, if your software lives long enough, you'll likely discover a good reason to change it.

Reasons include:

new requirements
defects
obsolete dependencies
security vulnerabilities
new legislation or regulations (privacy, PCI DSS, etc.)
hardware failures
hardware obsolescence

Always assume you'll have to make a change at some point in the future no matter what anybody tells you.

7. Maintenance is difficult and expensive

This is the stage of your project where your chickens come home to roost. The decisions you and your predecessors made earlier in the software development life cycle will determine how painful and expensive keeping this system running will be. But, no matter how much care and foresight everyone exercised, maintaining old software is almost always difficult and expensive.

Cleaning up the code base to make it easier to maintain (aka refactoring) is tempting to most programmers, but it is often a terrible investment.

Large sums of money are wasted on ill-advised code cleanup projects that will never be paid back by faster future development. For example, I currently maintain tens of thousands of lines of code that haven't been meaningfully modified in years and may never be modified again. So any refactoring or cleanup I do on that code is almost certainly wasted. It doesn't matter how ugly I think that code is.

You might be able to make economically advantageous, highly targeted changes if you look hard enough. But you are largely stuck with whatever you have at this point.

8. Your software will stay in production much longer than anybody anticipated

This is certainly not true of all software but if you restrict yourself to the types of projects we are discussing in this post (software in the late stages of maintenance) you'll find that this type of software tends to be hard to kill for various reasons:

replacing it never gets to the top of anybody's to-do list (regardless of what anybody says)
it's just too expensive to replace
the replacement project is late or fails to meet user needs
the replacement project never delivers working software (because the project failed or was cancelled)
your users can't or won't upgrade (for various reasons)

How to make software maintenance better

Like I said earlier, keeping this kind of project running safely, securely, and profitability as the years tick by--even if you're not adding or changing functionality--is especially challenging.

Your biggest risks are that:

you waste your maintenance budget on low value activities
an unexcepted event cripples the value of your project and/or brings it to a premature end

There are no one-size-fits-all solutions to this kind of software maintenance but there are some things you can do to make your life better. Your first step is to do some analysis.

How and when will you retire this software?

What's the retirement plan for your software? Is it expected to live for another year? 5? 10? 50? Do you plan to develop a replacement product and migrate your users to it? Does this software need to be changed to enable the migration? How realistic is the delivery date? Or will you retire the software without replacement? Or do something else?

What do your stakeholders want from this software between now and the time you retire it?

Don't assume you know what's important to them. Ask them. You might be surprised by what they say.

I did just that with the owners of a project I had been maintaining for over 10 years. I thought I knew their preferences very well. But I setup a meeting to ask them (just to be safe). I knew they didn't care about technical debt or code quality but I was surprised by their views on security, up-time, disaster recovery, and several other issues.

What is the value of your software?

How much money will your software make (or save) you each month or year until its planned retirement? There are several ways to do this but, if you are looking for a recommendation, I suggest you start with a net present value analysis.

You should also account for non-monetary factors in the value of your software. For example, if you promised your biggest customers that you would support the software for another 5 years, it would be wise to take that into account in your analysis.

You may want to get the business types involved in this analysis to help you figure things out.

What will it cost to keep your software in production until its planned retirement date?

What does it cost now? Is it reasonable to project that number into the future with some sort of inflation factor? What unavoidable maintenance tasks will you have to perform? For example, you might expect to migrate to new servers every 5-7 years and migrate your code to a new version of your language every x years. There are also one-off concerns like the year 2038 problem. Or maybe you know you'll need to replace one of your dependencies because its end of life date has been announced.

There are also costs that are harder to pin down. For example, is your project written in a language that is no longer popular? How will you find people with the skills to maintain it all the way to its retirement? Are you on old hardware? How will you keep that running? Or what's the probability that you have a security breach that costs you money or damages your reputation?

It's natural for people to underestimate how much effort it takes to keep software running. So any analysis you do will be more realistic than just throwing a number out there.

How risky is your "maintenance debt"?

Maintenance debt is a relatively new term. It is described in this paper as follows:

We introduce the term "maintenance debt" for maintenance needs generated by an implementation’s dependence on external IT factors such as libraries, platforms and tools, that have become obsolescent. The application continues to run, and the IT department forgets this theoretical liability, focussing on more urgent requirements and problems elsewhere.

Such debt accumulates over time, silently eating away at the value of the software asset. Eventually something happens that makes system change unavoidable. The owner may then discover that the system can no longer be modified--it is literally unmaintainable. Less dramatically, it may take too long, or cost too much, for maintenance to solve the business problem, and an alternative solution must be found. The software has suddenly crashed to £0 value.

Is it worth keeping your software in production until its planned retirement date?

It might become clear to you that your planned retirement date is uneconomical. Maybe you need to move your retirement date up? Or reduce your maintenance budget? Or increase how much you charge your users?

Perhaps you can purchase off-the-shelf software that largely does what your software does for a fraction of the cost and migrate to that.

Or maybe you realize that your software is extremely valuable and it makes sense to devote more resources to its maintenance to make it even more valuable and/or push back the retirement date.

Yet another option is that your software is a money pit and you need to figure out if you should rewrite or refactor it. Or perhaps the best thing to do is shutdown the project immediately without having a replacement (as Google is fond of doing) and work on something completely different.

The important thing is to do the analysis. The business types respond to data. And, even if there's quite a large level of uncertainly in your estimates of cost and value, it's a good starting point to having an intelligent conversation about the best way to maximize the profitability of your software.

It's all about risk management

If you've made it to this point and you've discovered your software is worth maintaining in some way, you'll have to decide how you are going to spend your maintenance budget.

You'll need to assign some of it to your "must-do" maintenance tasks. Then you can take whatever's left and use it to reduce the risk that something bad happens to your software.

For example, if you are worried about a hardware failure, maybe you work on your backup, replication, or redundancy. If you are worried about key people leaving the project, then you might focus on employee retention, letting your developers work on their software pain points, and documenting the system.

My point is that it's unlikely that you'll have enough resources to do everything you might want to do to maintain your software so you'll need to prioritize your efforts.

Additional things you can do

In addition to everything I described above, I think it's wise to do the following two steps periodically.

1. Revisit your analysis from time to time

The value of your software will change over time--often in unanticipated ways--so it's wise to redo your analysis periodically and make adjustments to your maintenance plans as necessary.

2. Build and deploy, even if you have no changes

You might consider updating your dependencies to recent compatible versions, building, testing, and deploying periodically. Even if you have no changes to integrate it's worth something to demonstrate that you can still do it successfully. It's also a good opportunity to look at your hardware and documentation to evaluate how well it's meeting your needs.

How often you build and deploy is up to you but every six months or so seems reasonable to me.

The scenario you want to avoid is that you need to make some small change and discover that its impossible or nearly impossible because you don't have a critical piece of information, software, or hardware required to complete the process. At that point all your options are bad (and probably expensive).

Wrapping up

If you find yourself responsible for maintaining a software system after active maintenance ends I recommend you focus on striking a good balance between squeezing as much profit out of it as you can and taking steps to reduce the most serious risks of something catastrophic occurring to the system.

Following the steps outlined in this post will help get you going on the right path. And, remember, it's your stakeholders who decide what makes your software valuable, not you. Good luck!

Additional resources:

Software Engineering at Google (book)
Working Effectively with Legacy Code (book)
Why governments can't migrate away from cobol (article)
Maintenance Debt (research paper)

Have a comment, question or a story to share? Let me have it in the comments.

Enjoy this post? Please "like" it.

Safety-Critical Software: 15 things every developer should know

Blaine Osepchuk — Sun, 01 Mar 2020 20:29:37 +0000

Despite being all around us, safety-critical software isn't on the average developer's radar. But recent failures of safety-critical software systems have brought one of these companies and their software development practices to the attention of the public. I am, of course, referring to Boeing's two 737 Max crashes, the subsequent grounding of all 737 Max aircraft, and its failed Starliner test flight.

How could such a distinguished company get it so wrong? Weren't the safety standards and certification process for safety-critical systems supposed to prevent this kind of thing from happening? Where was the FAA when the Max was being certified? These questions raised my curiosity to the point that I decided to discover what this specialized field of software development is all about.

In this post I'm going to share what I learned about safety-critical software development and how a little knowledge of it might be useful to "normal" programmers like you and me.

1. What is a safety-critical system?

Safety-critical systems are those systems whose failure could result in loss of life, significant property damage, or damage to the environment.

Aircraft, cars, weapons systems, medical devices, and nuclear power plants are the traditional examples of safety-critical software systems.

But there are other kinds of safety-critical systems:

code libraries, compilers, simulators, test rigs, requirements traceability tools, and other tools used to manage and build the kinds of systems we traditionally think of as safety-critical systems are safety-critical themselves
software that provides data used to make decisions in safety-critical systems is likely safety-critical. For example, an app that calculates how much fuel to load onto a commercial jet is safety critical because an incorrect result from that app may lead to a life threatening situation
other software running on the same system as safety-critical software may also be classified as safety-critical if it might adversely affect the safety functions of the safety-critical system (by blocking I/O, interrupts, or CPU, using up RAM, overwriting memory, etc.)

2. What do safety-critical software projects look like?

Safety-critical software development is a very specialized, expensive, methodical, slow, process-driven field of software development.

Safety-critical software systems are often embedded, distributed systems

Typically, the software monitors its environments with various kinds of sensors and manipulates that environment by controlling actuators such as valves, hydraulics, solenoids, motors, etc. with or without the assistance of human operators.

These systems are usually embedded because they often have timing deadlines that can't be missed. Plus, the creators of these systems are responsible every aspect of the hardware, software, and electronics in the system, whether they created it or not. So the more stuff they put in the system, the more difficult and expensive it is for the company get it certified.

Safety-critical software systems are often distributed systems because:

they are often large
subsystems may be developed by different teams or companies
many of these systems require redundancy and/or the separation of safety-critical code from non-safety critical code

All these factors push the system design towards multiple cooperating processors/subsystems.

There's more safety-critical software than you realize

There's a lot of this software being created. But it doesn't get nearly as much attention as consumer-focused software. For example, just about every story in Military & Aerospace Electronics discusses systems that contain safety-critical software.

It's difficult and expensive to create safety-critical software

It takes lots of resources and money to assemble the cross-functional expertise to develop, certify, sell, and support such complicated and expensive systems. So, while a few small companies might produce small safety-critical software systems, it's more common for these systems to be produced by larger companies.

3. Safety-critical software is almost certainly the most dependable software in the world at any given size and complexity

Software developed and certified as safety-critical is almost certainly the most dependable software in the world. The allowable failure rate for the most critical systems is absurdly low.

And these numbers are for the whole system (hardware and software combined). So the allowable failure rate for software at any given SIL level is even lower!

At SIL 4 (roughly equivalent to NASA's "maximum" category) we are talking about less than one failure per billion hours of continuous operation. That's 114,000 years!!!

To put that in perspective, if your smart phone were built to that standard its unlikely that you or any of your friends and family would have ever experienced a dangerous failure (dead phone, broken camera, random reboot, malware, data leakage, data loss, loss of connectivity, etc.), nor would you expect to encounter one for the entire span of your life! That's an incredibly high bar. And it just shows you how different this kind of product and software development is from the kind you and I do on a daily basis.

Here's my impression of where safety-critical software is supposed to fit in the software quality landscape:

Safety-critical software is designed, built, and tested to ensure it has ultra-low defect rates and ultra-high dependability.

Nothing beats the space shuttle software's quality

The shuttle software is probably NASA's best example of software done right. At the time the article (linked above) was written, the shuttle software was 420 KLOC and had just one error in each of the last three versions. That's orders of magnitude fewer defects than your average software project. Read this article if you want to know what it takes to write the lowest defect software in the world.

4. The focus is on safety, not features or speed

(For the remainder of this post, I'm going to use the NASA Software Safety Guidebook as an example safety standard.)

In most companies, software developers are primarily concerned with getting the software to "work", then going faster, shipping more features, and delivering more value. But software developers involved in the creation of safety-critical systems must be concerned primarily with creating safe systems. And that is a very different task.

They must find all the realistic ways the system could create harm. And then develop ways to mitigate that harm. They also need to make sure the system, as built, functions as specified. They must convince themselves and their auditors that they've done an adequate job of building a safe system before their product can be certified.

We're way beyond unit testing, code reviews, and filling out some forms. We're talking about painstaking checking and re-checking, mandated processes, multiple rounds of analyses, and piles of documentation.

Build it as if you know you are going to be sued

The image that came to my mind as I read through the NASA Software Safety Guidebook is that you have to jump through all these hoops as if you are preparing for a future court case where your product has killed a number of people. And your company and you personally are being sued for tens of millions of dollars. You need to be able to place your hand on a bible and swear that you and everyone working on your project:

was properly trained
followed best practices set out in the safety standard you followed
minimized the risks of harm associated with your product as far as practicable
earnestly worked to get your product certified as meeting your chosen safety standard by a competent certification body

And then you need to be able to point to a stack of boxes full of your documentation as proof that what you're saying is true, knowing that the plaintiffs' experts and lawyers are going to try to tear you apart.

If you feel like you might not win that hypothetical court case, then your product should not be released.

5. Safety and hazard mitigation requirements will dramatically increase the complexity of your product (and there will be a ton of them)

In addition to software engineering tasks associated with creating software for a normal product, you are going to have a lot of extra requirements related to safety and hazard mitigation. And, as we all know, as requirements interact code volume and complexity skyrocket. For example, we know that doubling the number of requirements will more than double effort required to complete most projects.

Plus, safety and hazard mitigation requirements are often difficult to implement both because they aren't "normal" things software developers do but also because they are inherently difficult. Let's look at an example.

Fail-safe design

Your system must fail safely. That's one of the fundamental principles of safety critical systems design. This means that under any reasonable scenario where the system is being used in accordance with the operating instructions, it must not cause a dangerous situation if something goes wrong.

For many systems that means stopping all actuators and reporting an error. For example, the blade in my food processor will stop immediately if I remove the lid while it is spinning. That's a simple case but for other systems, just figuring out how to fail safely is really difficult.

Heart bypass machine example

What does fail safely mean in the context of a heart bypass machine? The system can't just turn itself off if the power goes out because that will kill the patient.

Battery backup?

So maybe it needs a battery backup to keep it going until the hospital's generator can kick in. And a charging sub-system to keep the battery charged. And a battery health monitoring subsystem to alert the maintainers of the system that the battery won't hold a charge any more. And there should probably be a condition in the code that will not let you start the bypass machine if the battery is defective. Or missing. Or discharged. And some kind of change to the display so the operator can be made aware of status of the machine's power.

By the way, adding a lithium ion battery to a safety-critical system isn't as straightforward as you would think. Even after extensive testing, Boeing's 787 still had battery problems in the field, which required an extensive investigation and changes to the design of the system.

Multiple backups?

Actually, after looking more closely at the requirements, a battery backup alone isn't enough because this system is required to continue to operate with two failures. So maybe we need two independent battery systems? Hmm... do we need to source each system from a different supplier so that a fault in one battery system doesn't bring them both down at the same time?

Or maybe we should install a hand-crank for the nurse to turn a dynamo, which can power the heart bypass machine if the power fails and the battery fails or is exhausted? And we'll need a subsystem to ensure the dynamo works and generates enough power to run the bypass machine. But even a new dynamo won't produce enough power to run all the machine's functions. So maybe we need a low-power mode where the machine runs essential functions only? Or maybe we need to redesign the whole system to use less power? And so on.

So that one safety requirement ("patient must not die if power goes out") ballooned into several explicit requirements and a bunch of questions.

Other failure modes to consider?

And that's just one failure mode. We need to cover all of the realistic failure modes. What about:

fluctuating power?
dead power supply?
damaged RAM?
memory corruption errors?
an exception coming out of nowhere?
sensor failure?
sensor disagreement?
hard drive failure?
a random hard shutdown during a surgery?
pump failure?
display failure?
the case where the operator attempts to input settings appropriate for an elephant instead of a human?
the case where the operator accidentally turns the machine off?
a blown capacitor on one of the circuit boards?
and... and... and...

It's pretty easy to see how the number of safety requirements could easily dwarf the functional requirements in a safety-critical system. And that the implementation of those requirements would drastically increase the complexity of your system for both hardware and software.

6. Some systems are so dangerous that their construction is prohibited

I was surprised to learn that NASA won't let you build anything you want as long as you take appropriate precautions and get it certified (pages 27 - 28).

NASA prohibits the construction of the following kinds of systems no matter how many software precautions you are willing to take:

systems with catastrophic hazards whose occurrence is likely or probable
systems with critical hazards whose occurrence is likely

Where:

catastrophic is defined as: loss of human life or permanent disability; loss of entire system; loss of ground facility; severe environmental damage
critical is defined as: severe injury or temporary disability; major system or environmental damage
likely is defined as: the event will occur frequently, such as greater than 1 out of 10 times
probable is defined as: the event will occur several times in the life of an item

A prohibited system example

A spacecraft propulsion system that will blow up the spacecraft unless a computer continuously makes adjustments to keep the system stable would almost certainly be rejected by NASA as too dangerous.

Even if the system designers believe the software reduces the risk of killing the crew down to 1 incident per 5,000 flight years, the system would likely be classified as "prohibited" because NASA knows that software is never perfect.

7. Safety-critical software is about as far from agile as you can get

While there's some talk of using agile methods for safety-critical software development to improve speed and drive down costs, I don't think it's very realistic to think that agile will get very far in safety-critical circles. The Agile Manifesto defines 4 development values and 12 development principles and most of them are in direct opposition to the requirements of safety-critical software development.

"Welcome changing requirements, even late in development" seems to be a particularly problematic agile principle in the context of safety-critical software development. A single changed requirement could trigger a complete re-analysis of your entire project. If the change introduces a new hazard, it will have to investigated, analyzed, and possibly mitigated. The mitigation may result in multiple design changes. Design changes (hardware and/or software) may result in changes to the documents, safety manual, user manual, operator training requirements, test cases, test environments, simulators, test equipment, and code. Code changes require targeted retesting around the change. Plus, all your safety-related tests are supposed to be re-run for every code change. And, finally, you'll have to get your product recertified.

And that's the easy case. Imagine what would happen if your requirements change pushes your software to a higher safety level (say "minimum" to "moderate" for a NASA project or SIL 2 to SIL 3 for a IEC 61508 project)? Now you have to re-execute your project using the prescribed processes appropriate to your new safety level!

Waterfall or spiral development models dominate the industry

Most safety-critical software appears to be developed using the waterfall or spiral development models. NASA specifically recommends against using agile methods for the safety-critical elements of your software (page 87).

I've seen lots of examples where the V-model is mentioned as a specialized version of waterfall appropriate for safety-critical software development. In the V-model, the steps on the same horizontal level are loosely related. So, for example, you should be thinking about unit and integration tests when you are doing detailed design. Some sources even recommend writing the unit and integration tests during detailed design (before coding begins).

8. "Normal" software is sometimes used for safety-critical purposes

There's considerable controversy about what should be considered safety-critical. If a doctor uses a calculator app on her smartphone to calculate an IV drug dose for a patient on a very dangerous drug is the app safety-critical? What about the math library used by the calculator? What about the phone's OS and the all the other software on the phone? The answer is unclear.

Should doctors be forced to use certified calculators to determine IV drug dosages? Are calculators certified for safety-critical applications even available? I know this sounds like nitpicking; it's obvious that calculators work correctly, right? Actually, no. Harold Thimbleby has made the study of medical mistakes involving technology his life's work. And I found this video of him demonstrating numerous errors in the functioning of regular calculators deeply unsettling. It's not just one or two calculators that have errors. He calls them all "crap."

But it's not just calculators that we have to worry about. How many spreadsheets out there contain safety-critical data and calculations?

Looking in from the outside, authorities don't seem too concerned about the use of normal software for safety-critical functions. However, as a "regular" developer you might want to be aware that your software could potentially be used this way. Unless you are encouraging this use of your software, you are probably in the clear from a legal liability point of view but the ethics of it might be another story.

9. Insecure software cannot be considered safe

Many people criticize the safety standards for doing a poor job of addressing security (see the criticisms section below). So it shouldn't surprise you to learn that security researchers have found glaring security vulnerabilities in many classes of safety-critical products.

Because your entire safety case relies on your software behaving exactly as specified, you have a major problem if someone can make your system misbehave.

10. There are no silver bullets

I was hoping that my investigation into safety-critical software development would introduce me to some secrets of fast, high quality software development beyond what I've already discovered. I was looking for some tools or techniques to give me an edge in my day job. But I found nothing useful.

Safety-critical software development succeeds, for the most part, by throwing large quantities of money and people at the problem of quality. I think there's definitely an element of "do it right the first time" that's applicable to all software development efforts where high quality is desirable but there's nothing magical happening here. For the most part it's process, process, and more process.

The closest thing I found to "magic" is automatic code generation from simulation models and correctness by construction techniques. But neither of these approaches are going to benefit the average developer working on a non-safety-critical project.

11. How developing safety-critical software systems is supposed to work

There are a bunch of safety standards out there. If you are developing your own system, the standard you use may be dictated by your industry. Or, under certain circumstances, you can choose the standard you intend to meet. Then you assemble a competent team, build your product, and attempt to get your product certified by a certification body or you might chose to self-declare that your product meets the standard, if that's an option in your industry.

If you're building a product for someone else they may dictate the standard you must meet as part of your contract. If your customer is NASA then you are expected to follow the NASA Software Safety Guidebook for software development (and possibly some others, depending on the nature of your system).

Determining your safety level

The rules are very complicated but the basic idea is that you have to evaluate how dangerous your product is, the probability that the danger might occur, and the extent to which your software is expected to mitigate that danger (as opposed to hardware controls or human intervention).

Software that ensures a consumer pressure cooker doesn't over-pressure and explode would be certified to a lower level than the software that autonomously controls the safety functions of a nuclear power plant. So you'd have to follow a more rigorous process and do more work to certify the software for the nuclear power plant.

NASA has 4 levels of certification for safety-critical software: "full", "moderate", "minimum", and "none" (page 28). The NASA Software Safety Guidebook will help you figure out which level your proposed system is at.

It's also possible to have a mixed level system. For example one sub-system might be at the "full" level but another sub-system might be at the "minimum" level.

Adopting the required processes, analyses, and documentation

Once you know what level(s) your product is at, you can look up which processes, analyses, and documentation, you need to follow, perform, and create to achieve that level. Each development phase has a table that tells you what you need to do (other standards have similar tables).

(I think these tables are interesting enough that I included the tables for each phase of the development from the NASA Software Safety Guidebook below. But you don't have to read them to understand the remainder of this post.)

Conception (the stage before requirements where you are defining what kind of system you need) (page 102):

Requirements (page 107):

Design (page 136):

Implementation (coding) (page 167):

Testing overview (page 181):

Unit or integration testing (pages 181 - 182):

System testing (page 182):

Operations and maintenance (page 199):

As you can see, even at the "minimal" safety level NASA wants you to do many, many things that most software projects never do.

You are expected to go to extreme lengths to ensure your software is safe

Let's take a closer look at the dynamic testing table to illustrate my point. Even if you already routinely unit test your code for non-safety critical projects, I bet you aren't that thorough. Maybe you skip things that are particularly difficult to test and either not test them at all or "sort of" test them manually? Well, that's not good enough for safety-critical software. And getting to "good enough" might take more time than all the easy testing combined.

NASA wants to see the following tested for the "minimal" safety level:

typical sensor values
extreme values of inputs
all modes of each sensor
every statement, branch, and path executed
every predicate term tested
every loop executed 0, 1, many, max -1, max, and max + 1 times

It doesn't take much imagination on your part to realize a few things are going to happen if you attempt to follow these testing requirements to the letter:

you need to write all production code for testability, which will negatively impact other properties of your code
the volume of your test code is going to be larger than the volume of your production code (probably by a wide margin)
it's going to be difficult for you to write your test cases in such a way that you can track each of them back to the dynamic testing category it covers as well as the specific requirement it satisfies
it's going to be even more difficult for you to know that you've created enough test cases to satisfy the dynamic testing requirements for every subroutine in your code base
because you are testing so deeply in your system (not just the public API like most unit testers recommend for non-safety-critical systems), you are going to find it nearly impossible to refactor your code without causing chaos in your test suite
if QA discovers a problem with your product, it's going to be a nightmare to modify your code and test cases without making any mistakes

Getting your product certified

There are two ways to go here.

Self declare you meet a standard

In some industries you can simply declare that your product meets all the requirements of some standard, possibly with the assistance of an independent company to confirm the declaration.

While this might seem like an easier route to certification, your customers might not accept it. Plus, I'm not actually clear on when or under what circumstances you can self-declare that your product meets a certain standard. NASA doesn't allow it. And, based on what I've read, it seems unlikely that the FAA and the FDA allow it either. But other industries might be different? Please leave comment on this post if you know how self-declaration works.

Have your product certified by a certification body

The other way to go is to hire a certification body to independently evaluate your product and issue a certification if your product meets the standard you seek. This is the more rigorous of the two routes.

If you are going to work with an external company to help certify your product, they should be brought into the development process as early in your project life cycle as possible. It is not uncommon for an entire product to be developed and then fail to receive certification because of a mistake made very early in the development process. Adding missing requirements, processes, or documentation to a product after it has been constructed can be virtually impossible.

Certification is not guaranteed. Only 25% of companies seeking certification received it in this paper. I'm sure the rigor of certification varies from industry to industry and SIL to SIL but, as far as I can tell, it's a serious process everywhere.

Independent Verification and Validation

If you are working on a NASA project that is particularly high risk or high value, NASA may assign an Independent Verification and Validation (IV&V) team to your project (page 102). This team is tasked with ensuring that your project is on track to deliver the quality and functionality required for a safe and successful system/mission. The work the IV&V team does is over and above all the work you are expected to do for the project, not a replacement for it. The IV&V team can also act as a resource to answer your questions and help you tailor your effort appropriately to the risks of your project.

Tracking your product in operation

You are responsible to tracking usage and defects in your product as it operates in the field. The scope of this activity varies depending on the kind of product you are creating but the goal is to gather data to show how dependable your system is in actual use.

For example, Rolls-Royce jet engines use satellite communications to "phone home" with engine telemetry. Rolls-Royce collects data on the number of hours of use, along all kinds of performance, fault, and failure data for their engines. You probably can't get better tracking than that.

On other other end of the spectrum, something like an airbag in a car might rely on estimates of deploy events and consumer reports of malfunctions. This is much lower quality tracking data than Rolls-Royce collects but it's still better than nothing.

13. How developing safety-critical software systems actually works

I've described how safety-critical software systems are supposed to be built. But the reality of the situation appears to be quite different. It didn't take me long to realize that some companies approach safety-critical software development with little regard for the standards or the seriousness of what could happen if their systems fail.

The Barr Group's 2017 Embedded Safety & Security Survey

The Barr Group surveyed embedded developers around the world and found a shocking lack of best practices on safety-critical projects.

28% of all respondents work on safety-critical projects.

Only 67% of the safety-critical subset meet a safety standard!

Coding standards, code reviews, and static analysis are all mandatory at my e-commerce job because they are cost-effective, proven ways to improve quality. But a significant portion of the safety-critical subset don't do them.

And it just gets worse. Only 59% of the safety-critical subset do regression testing! Less than 100% do system level testing!

These numbers are hurting my brain!

And there's much more to be concerned about in this survey. I encourage you to watch the CEO of the Barr Group go over the survey findings in this YouTube talk.

Anyway, if you look at these results and compare them to all those tables I reproduced from the NASA Software Safety Guidebook, its pretty easy to imagine how many of the other tasks are not being done on safety-critical projects in the real world.

Toyota Unintended Acceleration Investigations

If you haven't heard Toyota's unintended acceleration problems I suggest you read a brief summary on wikipedia to get yourself up to speed. This is definitely a safety-critical software system and it is a mess.

Let's set the hardware and safety culture problems aside for a moment and just look at some of the software findings for the Electronic Throttle Control System. It's the system responsible for reading the pedal position and controlling the engine throttle settings:

over 250 KLOC of C (not including comments)
2,272 global variables
64 instances of multiple declaration of the same global
333 casts alter value
22 uninitialized variables
67 functions with complexity over 50
one function was found to have a complexity of 146. It contained 1,300 lines of code and had no unit tests
bugs that can cause unintended acceleration
gaps and defects in the throttle failsafes
blackbox found to record incorrect information
stack overflow and defects that can lead to memory corruption
and more...

Anyone who has made it this far in this post knows this is not the kind of code you want to see in a safety-critical system.

For more details please watch Philip Koopman point out multiple problems with Toyota's hardware, software, and safety culture (slides). It's an eye-opening talk.

Patients hacking their insulin pumps and posting the plans online

I ran across a video several years ago where a non-programmer was describing how he glued together a bunch of tech to allow his continuous glucose monitoring device to send commands to his insulin pump to automatically adjust the amount of insulin it delivered to his body to control his diabetes.

As a professional software developer, who is very aware how difficult it is to write correct software, I was very alarmed by what this guy was doing. He knew basically nothing about programming. He was building this system from tutorials and trading information with other non-programmers who were also working on the project in their spare time. I believe it was literally the first thing he ever programmed and it is definitely safety-critical--too little or too much insulin can definitely kill you.

Now, if you want to take your life into your own hands and build something like this for your personal use, I say go for it. But these people were sharing the design and code on a website and making it available for anyone to use, which is not cool in my opinion. The FDA issued a public warning not to build your own "do-it-yourself" pancreas after watching from the sidelines for several years.

Anyway, this is another example of how the theory is radically different than the reality of safety-critical software development.

The Boeing 737 Max and Starliner software issues

Two Boeing 737 Max crashes and a failed Starliner test flight are what inspired me to write this post. But as I dug deeper and deeper into safety-critical software development and safety-critical software development at Boeing in particular I've realized that this topic deserves its own post. I'll link to that post here when I publish it.

14. The safety standards are widely criticized

The standards are designed and approved by committees. They are based on what the drafters could get approved, instead of what is most appropriate or what is backed by evidence. While there appears to be broad agreement that coding standards, design reviews, code reviews, unit testing, and the like are good things, the standards disagree on specific practices and approaches.

Industry doesn't want to be told what to do

One criticism I read suggested that industry wants to minimize the work prescribed by the standards so it lobbies against adding more prescriptions to them. In some cases it is because they want to minimize their work to get their products certified. In other cases, it is so they can pursue the most effective techniques as they become available instead of being forced to do something they know is ineffective compared to the state of the art, just because the standard says they have to do it.

There's little evidence that following the standards produces safe software

There is actually little evidence that following the prescribed activities will result in the expected quality/failure rates experienced in the field. I think page 12 of the NASA Software Safety Guidebook describes the situation very well:

Opinions differ widely concerning the validity of some of the various techniques, and this guidebook attempts to present these opinions without prejudging their validity. In most cases, there are few or no metrics as of yet, to quantitatively evaluate or compare the techniques.

Security is ignored

Other people criticize the standards because they are mostly silent on security. I think that's a valid concern. If your product isn't secure, it's hard to argue that it's safe.

Here's another slide from the Barr Group's 2017 Embedded Safety & Security Survey. It shows that only 78% of the respondents who are working on safety-critical projects that are connected to the internet even have security as a requirement. It's not even on the radar of 22% of projects! Now, how many of the 78% of respondents who do have security requirements do you suppose are doing security well?

Here's the compliance with coding standards, code reviews, and static analysis for the safety-critical and connected to the internet subset. The ratios are about the same as the safety-critical only subset I showed you previously in this post. If you're not following a coding standard, doing code reviews, or using static analysis, I have a hard time believing you're somehow writing ultra-secure code for your internet connect, safety-critical product.

The standards don't address component interaction failures among others

Nancy Leveson shows many examples where following the standards isn't enough. Just as we acknowledge that all software contains defects and plan accordingly, we should also acknowledge the possibility that our software may also contain/experience:

component interaction failures
requirements errors
unanticipated human behavior
system design errors
non-linear or indirect interactions between components or systems
etc.

Leveson argues that we need to build safety-critical systems that can cope with these factors. She proposes STAMP as an additional approach to improve system safety for these types of issues. She has a great talk on YouTube that covers the basics.

By the way, she's not an odd-ball spouting BS from the sidelines. She's the Professor of Aeronautics and Astronautics at MIT and she works with the DOD, NASA, and others on safety-critical systems.

15. Thoughts on the future of safety-critical software development

Here are some of my ideas for where safety-critical software development is going and some of the challenges we'll face in getting there.

I anticipate sky-rocketing demand for this kind of software in the following areas:

autonomous cars and driver assist features
internet of things (including the industrial internet of things)
robotics and industrial automation
military systems
medical devices and healthcare related applications
space-based applications

Challenges:

demand for larger, more complex systems surpasses our ability to build them to the quality standards this kind of software requires
we need a way to drastically reduce the cost and time required to produce this kind of software
how can we incorporate AI and machine learning into safety-critical systems?
how do we allow safety-critical systems to communicate with each other and with non-safety-critical systems without degrading safety?
security is going to continue to be a thorn in our sides
more and more software is going to be used for safety-critical purposes without having been designed and built to safety-critical standards (like the doctor calculating IV drug doses on her smartphone)
how do we attract sufficient numbers of talented software developers to work on safety-critical projects?

Resources

Wrapping up

What do I want you to take away from this?

The quantity and complexity of safety-critical software is growing rapidly

Safety-critical software is all around us and we can only expect more of it as we connect more of the world to the Internet, make "dumb" devices "smart", and invent entirely new products to make our lives better.

Developing this kind of software is hard

Developers of safety-critical software systems feel the pain of the lack of maturity of the software engineering field more than most of software developers. Much to my disappointment, they don't appear to be hoarding any magic tools or tricks to get around the problems we all face in trying to develop complex products. Aside from doing more up-front planning than the average project, they just throw money and process at their problems just like the rest of us.

Some safety-critical software is developed with a shocking disregard for best practices

There's a wide gap between how safety-critical software systems are supposed to be developed and certified and what actually happens. I was shocked to learn that many safety-critical projects don't use coding standards, static analysis, code reviews, automated testing, or follow any safety standard at all!

If you find yourself working on such a project, maybe take some time to consider the potential consequences of what you're doing. There are plenty of jobs for talented developers where you aren't asked to endanger people's lives.

Most safety-critical software is relatively safe

Despite the many challenges of safety-critical software development, most safety-critical software systems appear to be, more or less, safe enough for their intended purpose. That doesn't mean that it doesn't contain errors, or even that it doesn't kill people (because it almost certainly does). What I mean is that most safety-critical software is trustworthy enough to be used for its intended purpose as evidenced by the public's willingness to fly on planes, live near nuclear power plants, and have medical devices implanted in their bodies.

Until Boeing's recent problems with the 737 Max, the go-to example of a safety-critical software error resulting in death was malfunction of the Therac-25 radiation therapy machines. An investigation revealed that the machines were pretty much a disaster from top to bottom. And between 1985 and 1987 a handful of people received harmful doses of radiation from Therac-25 machines and a couple of people died.

That the Therac-25 was the go-to example of a failure of safety-critical software causing death says something about just how well most of these systems perform in the real world. After all, safety-critical software is controlling nuclear power plants, medical devices, rail switching, elevators, aircraft, traffic lights, spacecraft, weapons systems, safety features in cars, and so on. But when is the last time you heard about a software error causing a safety-critical system to fail and kill people (besides the Boeing 737 Max and Tesla autopilot fatalities)?

Have a comment, question or a story to share? Let me have it in the comments.

Enjoy this post? Please "like" it below.

Checkout the world's safest open source sumobot (with video)

Blaine Osepchuk — Thu, 16 Jan 2020 19:43:25 +0000

I created the High Integrity Sumobot using Ada/SPARK and high integrity software engineering techniques. I wanted to make it easy for people interested in Ada/SPARK to see how all the pieces fit together in a small embedded project.

I did my best to write simple, clean, and maintainable code throughout. It's extensively commented.

System overview

I started with a Zumo 32U4 sumobot. But instead of programming it directly as a microcontroller like most people do, I turned it into an I2C slave device, which I control with a microbit I programmed in Ada/SPARK.

The zumo continuously collects sensor data and stores it in a buffer. The microbit periodically requests sensor data from the zumo over I2C and the zumo sends the data it most recently collected. The microbit validates the sensor data, decides how the zumo should move, and sends motor commands back to the zumo (also over I2C). The zumo validates the motor commands and then executes them. And then the process repeats (at least 50 times per second).

More information and build instructions

You can find the full documentation and build instructions for this project on my website. I've included everything you need to create your own High Integrity Sumobot:

written requirements
full parts list
wiring diagram
source code
detailed build instructions (with photos)
detailed software installation instructions (with screenshots)
instructions for how to run the automated unit tests
compile/build and upload/flashing instructions (with screenshots)
end-to-end testing plan linked to each requirement

Wrapping up

I welcome any feedback on my project. If you have a way to make it better I'd love to hear from you.

I'm programming a sumobot with the world's safest programming language

Blaine Osepchuk — Thu, 14 Nov 2019 18:38:10 +0000

The Make with Ada contest is offering $8,000+ in prizes for "cool embedded applications programmed in the Ada and SPARK programming languages."

I've always wanted to build a robot and I've been looking for excuses to play around with Ada so I thought this would be a great time to kill two birds with one stone.

Project overview

I'm building a robot that can compete in the mini-sumo class. I bought a Zumo 32U4 robot online because it's packed with sensors, has a great software library, and I didn't want to build a robot from scratch. It's meant to be programmed entirely as an arduino microcontroller but I'm going a different route.

I'm collecting all the sensor data from the zumo and sending it to my microbit microcontroller over I2C. I'm using the Ada drivers library to program the microbit in Ada (and I'll add some SPARK too if I have time). The microbit will receive the I2C sensor data, decide what to do, and directly command the robot motors from the GPIO lines on the microbit. All of the critical code will be written in Ada/SPARK on the microbit, which I hope will impress the judges.

My robot should be quite a capable sumobot when I'm done.

Current status

I have a working proof of concept. I'm collecting sensor data from the front proximity sensors, transferring it to the microbit over I2C, validating and processing it, and commanding the motors to turn the robot towards the detected object. My next task is to get all the electronics securely mounted on the zumo. Then I'll continue adding capabilities to my sumobot.

Any ideas for features with "buzz effect" I should add to the project?

One of the judging criteria is buzz effect:

one of the main goals of the Competition is to give more exposure to Ada, SPARK and related technologies. Projects will be judged in part with regards to how we think the project can have an impact in that regard: Will the project give more exposure to Ada? Does it have a “wow effect” that will appeal to the technology community in general? Will it show the benefits of Ada and related technologies?

I imagine a sumobot like mine would have more "buzz effect" than a lithium ion battery charger written in Ada. But, beyond that, I'm not entirely sure what they are looking for. I'd welcome any suggestions for improving my project's "buzz effect."

More about the make with Ada contest

If you have any interest in embedded software development or Ada you should definitely consider entering the make with Ada contest. The number of serious entries in previous years has been quite low. So you have a good chance of winning one of the top ten finalist prizes of $600, if not the 1st place prize of $2000 if you put a decent amount of effort into your entry.

The final day of the contest is January 31, 2020. So you have lots of time.

Wrapping up

I welcome any feedback on my project. If you have a way to make it better I'd love to hear from you. And I hope I can inspire at least one other person to join the contest. So let me know if you plan to enter the contest in the comments.

The ONE chart every developer MUST understand

Blaine Osepchuk — Sat, 07 Sep 2019 16:59:21 +0000

Our industry is famous for delivering projects late and over budget. Many projects are cancelled outright and many others never deliver anything near the value we promised our customers. And yet, there is a subset of software development organizations that consistently deliver excellent results. And they've known how to do it since the 1970s. In this post I'll tell you their secret.

It all starts with understanding this one chart from Steve McConnell. It describes the relationship between defect rates and development time.

This chart says that most teams could deliver their software projects sooner if they focused more effort on defect prevention, early defect removal, and other quality issues.

But is this chart true?

Steve McConnell published this chart in a blog post titled Software Quality at Top Speed in 1996. This chart (and the blog post) summarizes some of the data in his excellent book Rapid Development. And that book is based, in part, on the research of Capers Jones up to the early 1990s. I'm throwing in all these dates because I want you to know just how long we've known that:

In software, higher quality (in the form of lower defect rates) and reduced development time go hand in hand.

Anyway, Capers Jones kept doing his research and he released another book in 2011 with co-author, Olivier Bonsignour titled The Economics of Software Quality. They analyzed over 13,000 software projects from over 660 organizations between 1973 and 2010 and collected even more evidence that:

... high quality levels are invariably associated with shorter-than-average development schedules and lower than average development costs.

In other words, Steve McConnell's chart is true.

So what's the problem then?

There are three problems.

Problem 1: we're ignoring the research

The majority of projects are run as if this chart isn't true. Hardly a day goes by when I don't hear of some project or someone exercising poor judgement and then predictably getting smacked down by the universe for it. Literally billions of dollars are lost every year to this foolishness. It's been going on since we started programming computers. Every developer has experienced it. And there's no end in sight.

For example, pressuring yourself (or succumbing to external pressure) to go faster by cutting corners is almost guaranteed to increase your defect rate and slow down your project. Yet it happens all the time!

But the problem runs deeper than that. Managers are responsible for the worst project disasters. We have people running these projects who, while well-intentioned, have little idea what they are doing. Many of their projects are headed for disaster from the outset (see the "classic" software mistakes below). And by the time they realize that their project is in trouble--usually months after the developers reached the same conclusion--it's often too late to do much about it.

Problem 2: small project development practices don't scale well

The development practices that work relatively well for small projects, don't scale to large, real world projects. Small projects are the only kind of projects most students work on. So they graduate with the false impression that they know how to develop software. But they've been building the equivalent of garden sheds when we are trying to hire them to build the equivalent of skyscrapers. A skyscraper isn't just a really big garden shed--they are completely different things.

And because so few organizations do software development well, many teams employ garden shed-appropriate methods to tackle skyscraper-sized problems. So these poor developers think chaos, confusion, bugs, conflicting requirements, endless testing cycles, missed deadlines, stress, piles of rework, and death marches are all normal parts of software development.

Problem 3: many teams don't have the required skills

You need more than raw technical skills to achieve low defects rates in real-world projects. You need a whole suite of organizational, managerial, and technical level strategies and tactics to pull this off. You'll almost certainly need additional training for almost everyone in your organization and it also requires you to embrace different development practices.

What does it take to achieve that 95% pre-release defect removal rate?

For most organizations, it will take quite an adjustment to achieve that 95% pre-release defect removal rate. But the good news is that even modest improvements in pre-release defect rates will positively impact the economics of your project.

With that in mind, I suggest the following steps:

Accept the truth that high-quality software is faster and cheaper to build than low quality software
Be aware of Steve McConnell's "classic" software mistakes
Use memory-safe languages whenever possible
Start improving your development practices

Let's dive in.

Accept the truth that high-quality software is faster and cheaper to build than low quality software

If you need more evidence than you already have, read The Economics of Software Quality to truly convince yourself and your teammates that this chart is telling the truth. The authors of this book leave very little doubt that:

The best available quality results in 2011 are very good, but they are not well understood nor widely deployed because of, for one reason, the incorrect belief that high quality is expensive. High-quality software is not expensive. High-quality software is faster and cheaper to build and maintain than low quality software, from initial development all the way through total cost of ownership.

Furthermore:

If state-of-the-art combinations of defect prevention, pretest defect removal, and formal testing were utilized on every major software project, delivered defects would go down by perhaps 60% compared to 2011 averages.

Why is that?

Low quality projects spend much more time on testing, debugging, fixing, and rework than high quality projects. In fact, low quality projects contain so many defects that testing often takes longer than construction. And low quality projects frequently stop testing long before they run out of bugs to find.

On waterfall projects they either release the software as is or cancel the project because testing and fixing will go on forever otherwise. On agile projects increments of work are completed quickly at first and then slow to a glacial pace as more and more problems are discovered in existing code. Eventually low quality agile projects reach the same options as waterfall projects: release as is or cancel the project.

High quality projects invest in defect prevention and pretest defect removal activities so that when they do get to testing, there are many fewer defects to find and fix. High quality projects are released sooner and cost less than low quality projects because they have much shorter testing phases and much less rework. And high quality projects also have fewer post-release issues to fix. So when they do need to make changes, the code in high quality projects is easier and cheaper to modify.

Let's look at some key points from The Economics of Software Quality:

Overall quality levels have not changed much between 1973 and 2010. IDEs, new languages, interpreted languages, automated testing tools, static analysis tools, better libraries, frameworks, continuous integration, thousands of books, Agile, Scrum, XP, OOP, TDD, and the whole fricking web haven't moved the needle! That's just depressing. (I know someone is going to argue that this point can't be true. Feel free to look it up on pages 538 and 539 of The Economics of Software Quality).
In low quality software projects nearly 50% of the effort is devoted to finding and repairing defects and rework.
Defect rates rise faster than project size. That means the things you need to do to ensure a 95% pre-release defect removal rate in a 5 KLOC project are completely different than the things you need to do in a 500 KLOC project. Bigger projects not only need more QA activities but they also need different QA activities. Remember, a skyscraper isn't just a really big garden shed.
Testing has been the primary form of defect removal since the software industry began and for many projects, it's the only form used. That's a shame because testing is not that effective. Even if you combine 6 or 8 forms of testing you can't expect to remove more than 80% of the defects in large systems.
Defect prevention methods include reuse, formal inspections, prototyping, PSP/TSP, static analysis, root cause analysis, TDD, and many others. The factors that hurt defect prevention the most are excessive requirements changes, excessive schedule pressure, and no defect or quality measures.
The most effective forms of pretest defect removal are formal inspections and static analysis. But the authors also discuss 23 other methods of pretest defect removal, their range of expected results, and when you might want to use them.
The ROI on the better forms pretest defect removal is more than $10 for every $1 that is spent.
This book discusses 40 kinds of testing, their range of effectiveness, and when you might want to use them.

I think this book's true value comes from the evidence it provides to help you argue against ideas and practices that have an especially negative effect on the cost and schedule of your project. For example, after reading this book it's hard to argue that you don't have time for code reviews.

Be aware of Steve McConnell's classic software mistakes

In chapter 3 of Rapid Development, Steve McConnell lists 36 "classic" software mistakes.

Falling victim to even one of these mistakes can condemn your project to slow, expensive development. You need to avoid all of them if you want to be efficient.

They are:

Undermined motivation
Weak personnel
Uncontrolled problem employees
Heroics
Adding people to a late project
Noisy, crowded offices
Friction between developers and customers
Unrealistic expectations
Lack of effective project sponsorship
Lack of stakeholder buy-in
Lack of user input
Politics placed over substance
Wishful thinking
Overly optimistic schedules
Insufficient risk management
Contractor failure
Insufficient planning
Abandonment of planning under pressure
Wasted time during the fuzzy front end
Shortchanged upstream activities
Inadequate design
Shortchanged quality assurance
Insufficient management controls
Premature or overly frequent convergence
Omitting necessary tasks from estimates
Planning to catch up later
Code-like-hell programming
Requirements gold-plating
Feature creep
Developer gold-plating
Push-me, pull-me negotiation
Research-oriented development
Silver-bullet syndrome
Overestimated savings from new tools or methods
Switching tools in the middle of a project
Lack of automated source-code control

Rapid Development was released over 25 years ago. Yet 35 of those classic mistakes are still extremely common (Subversion and GIT have largely solved #36 "lack of automated source-code control").

Update: 2021-08-17

Check out Steve McConnell's updated and improved list of "classic software mistakes."

Use memory-safe languages whenever possible

This suggestion didn't make it into either book but I think it's an important point in 2019. Around 70 percent of all the vulnerabilities in Microsoft products addressed through a security update each year are memory safety issues.

It's pretty clear at this point that humans just aren't capable of programming large systems in memory-unsafe languages like C and C++ without making an astounding number of mistakes or spending embarrassing amounts of money to find and remove them.

If Microsoft can't keep those errors out of their software, it's unreasonable to think you can do any better. So, if you're starting a new project choose a memory-safe language. There are memory-safe languages for even the most demanding domains so don't let your concerns about the performance implications or compatibility issues stop you from checking them out.

Start improving your development practices

Okay. So, you now believe the chart is correct and you further believe you are to the left of the optimum. What now? Try to get your organization moving down the curve towards the optimal point on the chart, of course.

Your first step is to get buy-in to the fact that low quality is a problem on your project. Since quality is a problem on most projects it shouldn't be hard to come up with evidence to support your case. Chapter 2 of The Economics of Software Quality will help you setup a defect tracking system to track the right things and avoid common measurement pitfalls. Having hard data about the cost of low quality will help you make your case.

Your next step is to convince yourself and your team to stop taking shortcuts because they almost always backfire. Print out the chart and hang out on your walls if you have to.

Next, I suggest you implement a small change across your team, try it for a while, evaluate your results, keep or discard your change, and then choose something else to improve and repeat the cycle.

Not all ideas are equally helpful so I'd start with the suggestions in Software Quality at Top Speed. The three ideas in that blog post are bang on. Start by eliminating shortcuts, replacing error-prone modules, and adopting some kind of code review process. You're welcome to use my code review checklist as a starting point.

Then I'd move on to Rapid Development. This book is all about getting your project under control and delivering working software faster. The 36 classic mistakes to avoid are super important. Go there next, then tackle the topics in the remainder of the book.

The Economics of Software Quality isn't really a how-to book but it will come in handy as a reference. It will help you determine the optimal quality target for your particular project. And it offers you a menu of options to help you choose the right combination of practices to achieve it.

What if nobody's interested in improving quality?

Sadly, this is going to happen to many of you. It's hard to shake the myth that quality is expensive. And it's even harder to convince people to change how they work. Unless you are the top dog on your team you may not have the influence required to spearhead this kind of change.

So you have three options:

Forget about improving quality at the project level and conform to the norms of your team.
Continue advocating for quality improvements and hope that people will eventually agree with you.
Find a new place to work that already cares about quality.

I can't decide for you but let me just say that there are many more developer positions available than qualified developers to fill them. And many employers do care about quality and are actively recruiting people at all levels of experience who can help them improve it.

Wrapping Up

Many of you are just as alarmed and distressed by the quality of the average software project as I am. We know how to do better but we just need to get the ball rolling on more software development teams. And I believe the first step is to show people this chart and the evidence behind it.

I know this problem isn't going to go away overnight. But if you feel the same way I do please do what you can to spread the word, share this post, improve the results in your own projects, and help improve the collective results of our industry.

Agree or disagree? Have a story to share? Let me have it in the comments.

Enjoy this post? Please "like" it below.

How to Crush it at the Start of Your New Developer Job

Blaine Osepchuk — Sat, 15 Jun 2019 15:27:49 +0000

This no B.S. guide will tell you what you need to know to survive and thrive as a software developer at my workplace (or just about any workplace).

Congratulations on being hired. You have a lot to learn. It's not the stuff you think. And you've got to learn it fast

We don't care where you went to school, where you've worked, or about any of the things you've accomplished. That's all in the past. You are starting from zero.

Here's the deal

I assume you:

are a reasonably competent programmer who wants to do a good job and work on interesting projects
want autonomy to choose what you work on, make your own decisions, have your opinion heard, and influence the direction of your projects
want to learn valuable skills that can help you progress in your career
may or may not want leadership or management positions and responsibilities
wouldn't turn down bonuses, raises, and other perks for doing your job well

All of that is on the table in our organization. You set your own limits on the level of autonomy, rewards, training, responsibility, perks, etc. you will receive in this job.

The level of autonomy we grant you is directly proportional to the clarity with which you understand your mission (see below) and the level of competence you exhibit.

High levels of clarity of mission and competence will be met with high rewards. But there's a flip side to that equation. If you are unable to grasp your mission or if you demonstrate low competence despite our best efforts to train you, we'll park you on unimportant tasks and micro-manage you in the best case and fire you in the worst case (I said this is a no B.S. guide).

Your mission

Your mission is to help our company increase revenues and reduce costs. That's it. Every position in our company (and every for-profit company) has the same mission. Your value to our company rests on your ability to deliver on your mission. And we are judging you on that basis. So you need demonstrate that you understand your mission and start delivering on that mission as quickly as you can.

The best way you can help us increase revenues and reduce costs is by working on the most important thing at all times. I can't overstate the importance of this point.

The simplistic formula for calculating the most important thing is "return" divided by "effort."

This is a major problem for you as a new employee because:

you have no way of knowing the amount of "return" (revenue or cost reduction) any given task could be expected to provide
you can only guess at the "effort" required to complete any given task because you don't know how we do things yet

So, it's basically impossible for you to create much value on your own until you can make reasonable assessments of "return" and "effort." And you won't get much autonomy until you prove to us that you can make these assessments reasonably well.

Tips

Let me point out a couple of things:

your most important decision is the first decision (what to work on). If you get that one really wrong nothing you do after that will help you create substantial value. You could write the world's best software but if you solve the wrong problem, it's worthless to us.
it's not enough to find something of value to work on. You are looking for the thing that has the most value.
the most important decisions in your job are rarely 100% technical. Our product owner is an MBA. And he can't tell the difference between an SQL query and a file full of jQuery but he's really good at this stuff. His deep domain knowledge, ability to ask us (his programmers) smart questions, and his strong grasp of how value is created ("return" divided by "effort") are all he needs to make great decisions. Unfortunately for you, this is also a lot harder to learn than the typical things new programming hires ask about such as the mechanics of how we create pull requests or how we name things.
it's a mistake to assume you know the best way to make all these decisions as a new hire. But it's not against the rules to ask a co-worker for help. Hint!

How to do a really good job in your first few days

Focus on:

acquiring domain knowledge - start with the basics (what we sell, why we sell it, how we make money, where our opportunities are, where our challenges are, etc.)
asking your teammates smart questions - what do you want me to learn first? what are our priorities? what do I have to do to be a 10x programmer on this team?
getting into the habit of trying to maximize the value of your efforts. Start doing that math on your decisions. It needs to become a habit.
avoiding silly suggestions - I know you want to sound smart and make valuable contributions but you don't know anything about creating value in this company yet. This is the time for asking questions and learning, not advocating for your favorite technology, methodology, or problem solving approach. Even if you see something that seems absolutely crazy I'd encourage you to assume that there's a good reason for it and just go with it for now (but make a note to circle back to it in a month or two if it still seems crazy).

I want you to succeed but you have to be responsible for yourself

Be prepared. Work hard. Meet your commitments. Don't be late. Don't waste people's time. Ask smart questions. And take notes.

Take responsibility for integrating yourself into the rituals, processes, and culture of your team. Request training and one-on-ones with people who can help you with your mission. If you need resources, ask for them. But know when you are beginning to make too many demands on other people and dial it back.

I'll give you as much of my time as I can but I find that it's better if new hires "pull" information toward themselves than if I "push" it on them by assigning readings and giving tutorials.

To a large extent, you are in control of how much training, orientation, and mentoring you receive above the bare minimum.

Aim to be a "zero" for your first few weeks

In his book An Astronaut's Guide to Life on Earth, Chris Hatfield (a famous Canadian astronaut) said he aimed to be a "zero" at the beginning of a new job.

In your first few weeks in a new position it's very easy to say or do the wrong thing, especially if you're trying to show people how smart and capable you are. And because it can be difficult to overcome first impressions (or get yourself un-fired), you should aim to simply meet expectations and not make any major mistakes--aim to be a "zero."

Your first few weeks are about finding your footing and figuring out what's expected of you.

Never stop learning

Mastering the information in this post will help you make it through your probationary period. But your long term success at our company relies on your continued growth and ability to create value.

You should work through all the information in this post: How to be a wildly successful programmer. There's no secret to being an amazingly valuable member of our team. I wrote it all down for you and I'll help you achieve your goals any way I can. But you have to do the work.

You should also start reading my must read programming books. I can give you customized reading recommendations. Just ask.

Conclusion

So that's my no B.S. (highly opinionated) guide to crushing it at the start of your new developer job. Over the years I've observed many people start all kinds of positions. And hardly anybody does as well as they could during their first few weeks on the job because they are focused on the wrong things. So be prepared for your boss to fall in love with you if you follow this guide.

You can also use this guide to change your fortunes in your current position. It's never too late to change your approach to your job and reap the rewards of being a star employee.

Note: the opinions expressed in this post are mine alone and do not reflect the opinions of my current employer or former employers.

Do you have a good or bad onboarding experience to share? I'd love to hear it in the comments.

Enjoy this post? Please "like" it below.

How to Manage Programmers Without Losing Your Damn Mind

Blaine Osepchuk — Sat, 04 May 2019 19:32:59 +0000

Stop me if this sounds like you. Somewhere along the way someone put you in charge of something at your job. Maybe you're coaching a new hire, maybe you're a technical lead on a small team, or maybe you're running a whole project. Regardless of what you're managing, you need to deal with people and that's not going quite as well as you had hoped because managing programmers is hard. And it can be hard on your sanity.

Maybe you even picked up a book or two about management in the hopes that you'd find something--some trick or skill--to help you manage your people better. I can tell you from experience that most of those books are crap. But there are a couple of exceptions and in this post I want to give you a summary of one of them: The Coaching Habit: Say Less, Ask More & Change the Way You Lead Forever by Michael Bungay Stanier.

The problem

Too many conversations between managers and the people they manage feel like this:

the manager is doing too much of the work
the manager thinks he knows what's wrong and how to fix it
the conversation gets derailed
the manager has too much trouble getting the conversation back into productive territory
the manager is exhausted at the end and hasn't accomplished nearly as much as he hoped

Coaching as a leadership style

Coaching is an effective leadership style but most people don't know how to do it. Managers are programmed to solve problems but solving problems for your staff is not a healthy practice. It makes your staff dependent on you for advice and solutions and you become a bottleneck. Nobody ever grows and performance never improves. When this happens you've fallen victim to the "advice monster" lurking inside of you.

Through the "seven essential questions," The Coaching Habit flips the script. You must resist the urge to dive into advice-giving mode. You need to coach your staff to solve their own problems and do so effectively. This book's approach only takes a few minutes per day. But you have to actually do.

Understanding the different types of coaching

There are two types of coaching:

coaching for performance - how do we solve the problem in front of us as efficiently as possible
coaching for development - goes beyond solving the immediate problem to development of problem solving skills and confidence in the person experiencing the problem

Coaching for development pays greater dividends.

The seven essential questions

What's on your mind?
And what else?
What's the real challenge for you here?
What do you want from me?
How can I help?
If you're saying yes to this, what are you saying no to?
What was most useful to you?

I'll briefly describe each question and how to use it below.

1 - What's on your mind?

Start with this question. You can prefix the question with: "out of curiosity" to take some of the heaviness out of it and make it easier to answer.

If you get a vague answer like "the project," you can put the three "Ps" to use by saying that there are three different facets we could look at:

the project - any challenges around the actual content
the people - any challenges with the people involved
the patterns - if there's a way you're getting in your own way or not showing up in the best way possible

Then you ask "Where would you like to begin?" No matter how the person answers, you are on your way to a deep conversion.

2 - And what else?

It buys you time to think and generates options, which is a good thing. It's also more appropriate than just jumping in with the answer.

Tips for using this question:

stay curious, genuine
ask the question one more time (3-5 times is usually where you end up)
recognize success (success is when the person says there is nothing else)
move on when it's time (if the energy is dropping you can wrap up with a variation: is there anything else?)

Tip: acknowledge the answers you get

Remember to acknowledge the person's answers before moving on to the next "and what else?" to let the person being coached know that you're listening and to encourage them to continue.

Examples:

Fantastic. I like it. Good One
Nice. Yes, that's good.

Another tip: stop offering advice with a question mark attached

When you've got an answer you want to suggest, ask at least one of the seven essential questions first. Then if you still want to offer your suggestion, you can offer it up as an "option" instead of disguising it as a question. People know you're not really asking a question so don't play games. And offering it as an "option" makes it part of a discussion instead of saying "I think we should do X," which is a conversation killer.

3 - What's the real challenge here for you?

When you ask question 1 ("what's on your mind?") you can get three responses that aren't helpful. You can refocus them with "What's the real challenge for you?" You want to get to the real issue and not the first thing that comes to someone's mind.

Three possible unhelpful responses to this question:

proliferation of challenges - the person says everything that's wrong (often a very long list)
coaching the ghost - the person is complaining about a third party or situation (but you can't coach a person who isn't there so this focusing question will bring the person back to what you can do for them)
abstractions and generalizations - if the person is talking about theory and generalizations this focusing question will bring you back to the person at hand and the concrete situation. We're not philosophers, we are just trying to do our jobs.

Tips:

trust that this process is actually useful (the person you're coaching will never learn anything if you always jump straight in with the answer)
remember that there's a place for advice. We're just trying to slow things down so everyone can think and giving people the answer immediately is no longer your default setting. But sometimes it is totally appropriate to give someone the answer immediately
remember the second question: "and what else?" Or "what else is the real challenge for you here?" These questions will help you get past the superficial
the words "for you" are a kind of magic sauce that cuts through all kinds generalities and hypothetical problems

Another tip: stick with questions starting with "what" (and avoid questions starting with "why")

"Why" puts people on the defensive. "Why" asks for the backstory. You don't need the "why" if you aren't trying to solve the problem. Whenever you are tempted to ask a "why" question. Reformulate it into a "what" question. For example, "why did you do that?" becomes "what did you hope to accomplish here?"

4 - What do you want from me?

It's a powerful question that people are rarely asked so they might not have an answer. Nevertheless, it's an important question. You are not a mind reader and you shouldn't assume that you know what the person being coached wants.

You should ask the question without fear you might not be able to grant their request. If you can get to what they actually want, at least you're are having a meaningful conversation.

How to actually help without taking over someone's initiative

Start by assuming that you are playing lessor versions of yourself at least some of the time. You are usually bouncing between one of the roles in the drama triangle:

victim - my life is so hard, my life is so unfair, poor me. Nothing's my fault. I feel stuck because I have no power and no influence. I feel useless.
persecutor - I'm surrounded by idiots and fools. It's not my fault, it's theirs. I feel stuck because I don't trust anyone. I feel alone.
rescuer - don't fight, don't worry. Let me take it on and fix it. It's my fault, not yours. I feel stuck because my rescuing doesn't work. I feel burdened. Rescuers create more victims the more they take over.

Tip: people are hardwired to look for danger all the time

And if they sense danger they start looking for ways to defend themselves instead of being productive and high functioning. As a manger, it is in your interest to have people feeling safe so that they can perform at their best.

Here are four factors that influence whether people feel safe (TERA):

T - is for tribe - are you on the same team or are you opponents?
E - is for expectation - if you know what to expect, you'll feel safer
R - is for rank - if I'm relatively more powerful than you, you'll feel less safe
A - is for autonomy - if you get a say in the outcome, you'll feel safer

It's your job as a manger to increase the TERA quotient for the person you're talking with. It's good for all parties. You do that by asking the seven essential questions and especially the "what do you want" question. (It also helps to not be a jerk the rest of the time.)

Another tip: get used to silence.

Don't immediately fill the void when you ask one of the seven essential questions and don't get an immediate response.

5 - How can I help?

You can't avoid the drama triangle. But you can learn to recognize when you're in it and you can get yourself out by asking: "how can I help?" This question is powerful for two reasons:

it forces the other person to make a specific request, which they might not have thought of before you asked
it keeps you from assuming you know what's wrong and what the right solution is

Tip: if someone directly asks for advice you need to resist the urge to provide an answer directly, which will be extremely difficult

Respond by saying "that's a great question. I have some ideas on that, which I'll share. But before I do that, what are your first thoughts?" And when he answers, which he inevitable will, you nod your head and be engaged and interested, and when he finishes you say, "that's terrific. What else could you do?" More nodding, more being interested. Then you say, "this is all good. Is there anything else you could try here?"

And then, and only then, you could add your own idea to the mix here if you want. And, of course, if the conversation is going well keep asking " and what else?" until he's run out of ideas.

Use this technique every time you get the urge to jump in, help out, and volunteer. And the real insult here--to your time, effort, and good intentions--is that the recipients may not even want or need what you're about to give them.

6 - If you're saying yes to this, what are you saying no to?

It's easy to say "yes" to things but it's hard to get them done unless you take something off your plate. So part of saying "yes" is thinking about what you are not going to do while you're doing this.

Tip: how to say no when you can't

The secret to saying "no" is to shift the focus and learn how to say "yes" more slowly. Saying "yes" more slowly means being willing to stay curious before committing, which means asking more questions:

Why are you asking me?
Whom else have you asked?
When you say this is urgent, what do you mean?
According to what standard does this need to be completed? By when?
If I couldn't do all of this, but could just do a part, what part would you have me do?
What do you want me to take off my plate so I can do this?

Being willing to stay curious will likely provoke one of four responses:

Stop asking annoying questions and get on with it. This means that someone in authority has decided you are going to do this and you are expected to comply.
The person has good answers to your questions, which means he thought this through.
He doesn't have the answers but will find them and get back to you (or not).
He might decide that you're too much work for him and ask someone else.

7 - What was most useful to you?

You can end your coaching session by asking "What was most useful to you?"

This question has four benefits:

you'll get feedback about how the other person perceived the coaching
you might plant the idea in the other person's head that these talks are useful to them
and you create a mini-review to help the person remember what you talked about
it wraps up the coaching session nicely

Tip: coaching over email works

When you get an advice request by email, instead of writing a big, long answer, choose the most appropriate of the seven essential questions and ask it.

Examples:

"Wow, there's a lot going on here. What's the real challenge here for you, do you think?"
"Before I jump into a longer reply, let me ask you: What's the real challenge here for you?"

Additional resources

I've found the following resources about management to be useful:

Managing Humans: Biting and Humorous Tales of a Software Engineering Manager by Michael Lopp (book) Practical advice from someone who's been there and lived to tell the tale (also very funny)
Turn the Ship Around!: A True Story of Turning Followers into Leaders by L. David Marquet (book) This is the amazing and entertaining story of how a first-time submarine commander took over the worst sub in the fleet and made it into the best in 2 years. The Coaching Habit fills in many of the details of how to actually turn the ship around that were a little thin in this book
Google Talk: Turn the Ship Around (video) Great intro to the concepts from Marquet's book

Conclusion:

I wrote this blog post because I really needed this book. I'm a problem solver by nature and I've fallen into pretty much every trap discussed in The Coaching Habit. The parts of my personality that make me a good programmer tend to make me a mediocre leader/manager, which I expect is probably true for most software developers.

I haven't mastered everything in The Coaching Habit yet (not by a long shot) but I'm trying. And just knowing that my tendency to jump into advice-giving mode isn't helping anybody is important all on its own. I tend not to do that so much anymore and I'll continue working on the rest of it.

Have you read this book or do you think you might? Is there another management books that was especially helpful to you? I'd love to hear your thoughts in the comments.

Enjoy this post? Please "like" it below.

29 Must Read Books For Programmers

Blaine Osepchuk — Wed, 14 Nov 2018 00:19:57 +0000

These are my "must read" programming books. They are universally applicable: you'll benefit from reading them regardless of the kind of programming you do. And the knowledge in these books will remain relevant throughout your career.

I hope you'll find a couple of good books to add to your reading list here.

June 16, 2019: I'm no longer updating this post. You'll find my most up-to-date list of must read books for programmers (with working links) on my blog.

Career

The first thing you have to figure out, if you haven't done so already, is what kind of career you want and how to get it. This book will help. Actually, it will help anyone get more satisfaction out of their career.

So Good They Can't Ignore You - Cal Newport (Why follow your passion is bad advice and what you should do instead)

Learn how to learn

Learning new skills is hard. Change is hard. So why not learn how to make new behaviors stick and learning new things easier? These books will help.

The Power of Habit: Why We Do What We Do in Life and Business - Charles Duhigg (Learn how habits actually work so you can make lasting changes by working with your brain instead of against it)
The Spirit of Kaizen: Creating Lasting Excellence One Small Step at a Time: Creating Lasting Excellence One Small Step at a Time - Robert Maurer (Kaizen is the practice of making small, continuous improvements. Toyota used it to improve the quality of their cars and become the biggest car maker in the world. You can use it to improve just about anything as this book demonstrates)
Badass: Making Users Awesome - Kathy Sierra (Most people learn inefficiently. This book shows you a better way) (YouTube summary).

Personal effectiveness

The first three books below are all slightly different takes on the same advice. No matter what your profession or education or task, effective people tend to share a similar mindset and employ similar techniques. Read the book that resonates with you the most.

The 7 Habits of Highly Effective People: Powerful Lessons in Personal Change - Stephen Covey (a classic)
The Effective Executive: The Definitive Guide to Getting the Right Things Done - Peter Drucker (another classic)
Great at Work: How Top Performers Do Less, Work Better, and Achieve More - Morten Hansen (a modern take on the genre)

Checklists can be a game changer. I've got tons of them and you should too.

The Checklist Manifesto: How to Get Things Right - Atul Gawande (Learn the power of simple checklists to improve your performance and reduce errors)

Programming effectiveness

Once you've mastered personal effectiveness, it's time to learn how to be an effective software developer.

The Effective Engineer: How to Leverage Your Efforts In Software Engineering to Make a Disproportionate and Meaningful Impact - Edmond Lau (Gets you focused on the right things)
The Pragmatic Programmer: From Journeyman to Master - Andrew Hunt and David Thomas (Shows beginners how to apply their craft and level up)

Writing code

I still meet programmers who haven't read these classics. What are you waiting for?

Code Complete: A Practical Handbook of Software Construction, Second Edition - Steve McConnell (Evidence-based recommendations on software construction--MY BIBLE)
Clean Code: A Handbook of Agile Software Craftsmanship - Robert C Martin (Every programmer in my workplace is paid to read this book. Learn the importance of readability and maintainability and the cost of owning a mess. I agree with almost all of it (except forcing people to write extremely short methods))
Code Simplicity: The Fundamentals of Software - Max Kanat-Alexander (It's easy to get lost on the theory, opinions, and patterns we are urged to use when creating software. This book puts it all in perspective. It's simply 80 pages of amazing wisdom--MUST READ)

Software engineering

Despite what most people believe, software engineering does have a body of knowledge backed by research. Don't go against the research and expect your project to turn out okay--it probably won't.

Facts and Fallacies of Software Engineering - Robert Glass (Evidence of what works and what doesn't. If you're breaking any of these rules, you better have a good reason to think the research doesn't apply to you)
Rapid Development: Taming Wild Software Schedules - Steve McConnell (Evidence-based recommendations on project/team management--MUST READ)
Making Software: What Really Works, and Why We Believe It - Andy Oram & Greg Wilson (I don't love this book. It's written as a bunch of essays and it's kind of long winded and disjointed. However, if you want to avoid doing something foolish, and you've read the other books in this section, give this book a try after)

Leadership

Leadership is a mindset and a set of skills, not a title or position. Anybody can be a leader. As your career progresses and you gain responsibilities, you need to sharpen your leadership skills.

Measure What Matters: How Google, Bono, and the Gates Foundation Rock the World with OKRs - John Doerr (How to get everybody in your organization pulling as hard as they can in the same direction)
Turn the Ship Around!: A True Story of Turning Followers into Leaders - L. David Marquet (Leader-leader instead of leader-follower. Your organization will perform at the highest levels if everyone is 100% engaged in their job and working towards a common objective. You will learn how to give everyone in your organization the 3 Cs: control, competence, and clarity to make their maximum contribution)

Project management

Just because you're a good coder and you've been around for a while, it doesn't mean you can run a project. Avoid all the beginner mistakes by reading these books.

Rapid Development: Taming Wild Software Schedules - Steve McConnell (Evidence-based recommendations on project/team management--MUST READ)
Essential Scrum: A Practical Guide to the Most Popular Agile Process - Ken Rubin (Excellent advice on scrum and project management in general. Very high signal-to-noise ratio)
The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses - Eric Ries (Learn why building an MVP and using the build-measure-learn cycle is so important)
The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win - Gene Kim & Kevin Behr (Learn how you can apply the Theory of Constraints and Lean to turn around a trouble project (or prevent it from getting into trouble in the first place))

Advanced project management

Once you've got the basics down and your projects are no longer raging garbage fires, consider learning the advanced project management techniques described in the following books.

The Principles of Product Development Flow: Second Generation Lean Product Development - Donald Reinertsen (Corrects the errors people naturally make by trying to apply Lean and Six Sigma techniques from the manufacturing world to the product development world--not suitable for beginners)
Goldratt's Theory of Constraints: A Systems Approach to Continuous Improvement - William H. Dettmer (A systematic approach to finding the constraint in your organization and overcoming it. This is a huge lever--not suitable for beginners)

Unit testing

If you're not unit testing yet, what's stopping you? These books will get you started.

Starting to Unit Test: Not as Hard as You Think - Erik Dietrich (A Beginner's guide to unit testing)
Working Effectively with Unit Tests - Jay Fields (Unit testing best practices for people who know the basics)
Working Effectively with Legacy Code - Michael Feathers (Very famous book on getting existing code covered by automated unit tests--a notoriously difficult task)

Data analysis and statistics

You need data analysis skills to measure your results. Many programmers lack the statistics knowledge and the skills required to correctly apply statistical tests to data and come up with sound conclusions. These books will help.

Data Analysis with Open Source Tools: A Hands-On Guide for Programmers and Data Scientists - Philipp K. Janert (This is the book you need when your boss dumps a ton of data in your lap and says "find the insights." I love this book)

Any introductory book on statistics. The following books are much better than the textbooks I read in university:

Statistics For Dummies - Deborah Rumsey
Statistics II for Dummies - Deborah Rumsey

User interface/user experience

Most programmers don't pay enough attention to user interfaces and user experience. These books are written for programmers and they'll teach you everything you need to know.

Don't Make Me Think, Revisited: A Common Sense Approach to Web Usability - Steve Krug (amazing!)
Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems - Steve Krug (also amazing!)

Wrapping up

So that's my list of "must read" programming books. There's enough material here to keep you busy for a couple of years so please don't think you should read all these books immediately. We already have enough problems with unrealistic expectations in our profession and I don't want to add to them.

Don't just skim through these books so you can check them off some list either. The important point is to take as much time as you need to learn a new skill that's important to you.

I almost certainly missed some really great books. Feel free to suggest more "must read" books in the comments.

How to be fully alert minutes after waking

Blaine Osepchuk — Sun, 28 Oct 2018 14:40:26 +0000

I use a SAD light as soon as I wake up to go from groggy and useless to fully alert in a couple of minutes. Most mornings I'm sitting at my desk and working within 7 minutes of waking up. It's one hell of a productivity hack.

What is sleep inertia?

I used to have a hard time getting up in the morning, especially in the winter. I'd wake up groggy and have to force myself to get out of bed. But getting up, turning on some lights, having a drink of water, and using the bathroom didn't seem to help me get my brain working. Sound familiar?

Everybody experiences a groggy feeling after they wake from a deep sleep. Scientists call it sleep inertia. It can last up to 30 minutes.

Coffee won't help

Most people reach for their favorite caffeinated beverage in the morning to help get their brain firing on all cylinders. But it takes caffeine 30-45 minutes to kick in and sleep inertia usually wears off on its own before that happens. Plus, your brain adapts to caffeine, which leads to reduced effectiveness as a stimulant (and possibly dependence).

What's a SAD light?

It's a special light therapy device designed to treat seasonal affective disorder, which is a kind of depression associated with low light levels people experience in the fall and winter.

Here's the one I have.

It puts out 10,000 lux, blocks UV, and the light strikes my eyes from above--all the important characteristics of an effective light therapy device.

Can a SAD light treat sleep inertia and/or morning grogginess?

I believe it can. I'm not aware of any studies backing this up but, in my experience, if I spend even a couple of minutes sitting in front of my SAD light, my grogginess completely disappears.

My normal procedure is:

to wake up at the same time every day (even on weekends)
get to my desk as fast as I can
turn on my SAD light and start working. (I work with the bright light shining into my eyes from beside my monitor. I become fully alert almost immediately)
after an hour I turn it off, have breakfast, and leave it off for the rest of the day

How I discovered this productivity hack

I used to spend a couple of weeks working outside every summer. I was part of a crew using helicopters to spray herbicide on young forests in northern Alberta, Canada (very fun job). We started our days in the dark so we would be ready to spray as soon as light broke.

I saw stuff like this every morning.

We napped in the afternoons when it was too windy to spray and then worked until after dark.

Those were long days but I noticed that my brain worked surprisingly well as long as I was outside in the daylight. I also noticed that my chronic sleep problems disappeared when I was living and working outdoors. But that they would reappear when I went back to the city and I saw something like this in the mornings.

So that got me thinking about the level of light in my office compared to outside. I learned that even a very overcast day is still much, much brighter than your average office. That lead me to read the research on seasonal affective disorder. I was fascinated and inspired to do an experiment.

My theory is that humans are adapted to living outdoors near the equator. Our ancestors woke with the rising of the sun and went to sleep when the sun set. I believe that living indoors and using artificial lights is messing with our circadian system. I also believe that I can help my body figure out what it's supposed to be doing in the morning by exposing myself to some simulated sunlight (when the real stuff isn't available).

My wife uses a different kind of light to help her wake up

Interestingly enough, my wife also has trouble getting up in the morning. I bought her a Philips Wake-Up Light Alarm Clock that simulates the sunrise by slowly turning on a bunch of LEDs for about 20 minutes before her desired wake-up time. She loves it. It's a total game changer.

Some warnings before you try this yourself

I am not a doctor. And I am not qualified to give medical advice. I'm not telling you what I do. It's up to you and your doctor to figure out what's right and safe for you.

Beware of products claiming to be light therapy boxes. There are dozens of them on Amazon and elsewhere. They are completely unregulated and they might not be safe or effective:

dangerous levels of UV light are my biggest concern (you don't want cataracts and/or skin damage)
they might not work because they aren't bright enough or they emit the wrong frequency of light
this is a more complete list of warnings

The light therapy box I bought was nearly the same one used in a research study that proved that they work for SAD (it's same light from the same company with a different stand). There are much cheaper options out there but I wanted something that was proven to actually work.

Finally, if you think you might have a sleep or mood disorder, don't self-treat with a light therapy box. Being unusually tired or depressed can be a sign of a serious health problem. If in doubt, get that checked out by your doctor just like I did.

Results

I've been using my SAD light for more than 7 years and it helps me work productively right after I wake up. Without the light box, I struggled to get my brain working for the first half hour of my day, especially when the days are shorter. So if you add that up, I'm more productive for an extra 2.5 hours or more per week.

I've also experimented with using my SAD light to help me deal with a poor night's sleep. It's helped a little but it's definitely not a cure.

Additional resources

Here are some resources to help you learn more about light therapy:

Takeaways

I've performed experiments on myself with a SAD light to hack my brain and improve my productivity. And I thought my fellow programmers would find that interesting.

Have you experimented with light to improve your productivity or influence your mood? If so, what did you learn? If not, would you ever try? Tell me all about your experiences in the comments.

Want to write defect-free software? Learn the Personal Software Process

Blaine Osepchuk — Mon, 03 Sep 2018 22:47:44 +0000

I'm on a journey to become a better software developer by reducing the number of defects in my code. The Personal Software Process (PSP) is one of the few proven ways to achieve ultra-low defect rates. I did a deep dive on it over the last few months. And in this post I'm going to tell you everything you need to know about PSP.

The Promise of the Personal Software Process (PSP)

Watts Humphrey, the creator of PSP/TSP, makes extraordinary claims for defects rates and productivity achievable by individuals and teams following his processes. He wrote a couple of books on the topic but the one I'm writing about today is PSP: A Self-Improvement Process for Software Engineers (2005).

This chart shows a strong correlation between the formality with which software is developed and the number of defects in the delivered software. As you can see, PSP/TSP delivered a shocking 0.06 defects/KLOC, which is about 100 times fewer defects than your average organization hovering around CMM level 2 (6.24 defects/KLOC). Impressive, right?

Humphrey further claims:

Forty percent of the TSP teams have reported delivering defect-free products. And these teams had an average productivity improvement of 78% (Davis and Mullaney 2003). Quality isn't free--it actually pays.

Bottom line: defects make software development slower than it needs to be. If you adopt PSP and get your team to adopt TSP, you can dramatically reduce defects and deliver software faster.

It seems counter-intuitive but Steve McConnell explains why reducing defect rates actually allows you to deliver software more quickly in this article: Software Quality at Top Speed

Of course, you can only get those gains if you follow a process like PSP. I'll explain more on that later but I want to take a small detour to tell you a story.

Imagine your ideal job

Imagine you work as a software developer for a truly enlightened company--I'm talking about something far beyond Google or Facebook or Amazon or whomever you think is the best at software development right now.

You are assigned a productivity coach

Now, your employer cares so much about developer productivity that every software developer in your company is assigned a coach. Here's how it works. Your coach sits behind you and watches you work every day (without being disruptive). He keeps detailed records of:

time spent on each task
programming tasks are broken down into categories like requirements, requirements review, design, design review, coding, personal code review, testing, peer code review, etc.
non-programming tasks are also broken down into meaningful categories like meetings, training, administrative, etc.
defects you inject and correct are logged and categorized
anything else you think might be useful

Your coach runs experiments on your behalf

After you go home, your coach runs your work products (requirements, designs, code, documentation, etc.) through several tools to extract even more data such as lines of code added and modified in the case of code. He combines and analyzes additional data from your bug tracker and other data sources into useful reports for your review when the results become statistically significant.

Your coach's job is to help you discover how to be the very best programmer you can be. So, you can propose experiments and your coach will set everything up, collect the data, and present you with the results. Maybe you want to know:

if test-first development works better for you than test-last development?
are design reviews worth the effort?
does pair programming work for you?
do you actually get more done by working more than 40 hours a week?
are your personal code reviews effective?

Those would be the kinds of experiments your coach would be happy to run. Or your coach might suggest you try an experiment that has been run by several of your colleagues with favorable outcomes. The important point is that your coach is doing all the data collection and analysis for you while you focus on programming.

Doesn't that sound great? Imagine how productive you could become.

Unfortunately this is the real world so you have to be your own coach

This is the essence of the Personal Software Process (PSP). So, in addition to developing software, you also have to learn statistics, propose experiments, collect data, analyze it, draw meaningful conclusions, and adjust your behavior based on what your experiments reveal.

What the Personal Software Process gets right

Humphrey gets to the root of a number of the problems in software development:

projects are often doomed by arbitrary and often impossible deadlines
the longer a defect is in the project, the more it will cost to fix it
defects accumulate in large projects and lead to expensive debugging and unplanned rework in system testing
debugging and defect repair times increase exponentially as projects grow in size
a heavy reliance on testing is inefficient, time-consuming, and unpredictable

Here's an overview of his solution:

The objective, therefore, must be to remove defects from the requirements, the designs, and the code as soon as possible. By reviewing and inspecting every work product as soon as you produce it, you can minimize the number of defects at every stage. This also minimizes the volume of rework and the rework costs. It also improves development productivity and predictability as well as accelerating development schedules.

Humphrey convinced me that he's right about all aspects of the problem. He has data and it's persuasive. However, the prescription--what you need to do to fix the problem--is unappealing, as I'll explain in the next section.

Why almost nobody practices the Personal Software Process

PSP is too hard to follow for 99.9% of software developers. The discipline required to be your own productivity coach is beyond what most people can muster, especially in the absence of organizational support and sponsorship. Software development is already demanding work and now you need to add this whole other layer of thinking, logging, data analysis, and behavior changes if you're going to practice the Personal Software Process.

Most software developers just aren't going to do that unless you hold a gun to their heads. And if you do hold a gun to someone's head PSP won't work at all. They'll just sabotage it or quit.

Here's a quote from Humphrey's book just to give you a taste for how detailed and process-driven PSP is:

The PSP's recommended personal quality-management strategy is to use your plans and historical data to guide your work. That is, start by striving to meet the PQI guidelines. Focus on producing a thorough and complete design and then document that design with the four PSP design templates that are covered in Chapters 11 and 12. Then, as you review the design, spend enough time to find the likely defects. If the design work took four hours, plan to spend at least two, and preferably three, hours doing the review. To do this productively, plan the review steps based in the guidelines in the PSP Design Review Script described in Chapter 9....

It goes on from there...for about 150 pages. I can see how you could create 'virtually defect-free' software using this kind of process but it's hard for me to imagine a circumstance where you could get a room full of developers to voluntarily adopt such an involved process and apply it successfully.

But that's not the only barrier to adoption.

Additional barriers to adoption

The Personal Software Process is meant to be taught as a classroom course

The course is prohibitively expensive. So, the book is the only realistic option available to most people.

You'll find that reading the book, applying the process, figuring out how to fill out all the forms properly, and then analyzing your data all without the help of classmates or an instructor is hard.

The book assumes your current process is some version of waterfall

TDD doesn't not play nice with the version of PSP you'll learn in the book. Neither does the process where you write a few lines, compile, run, repeat until you're done. It breaks the stats and the measures you need to compute to get the feedback you require to track your progress. It's not impossible to come up with your own measures to get around these problems but it's definitely another hurdle in your way.

Languages without a compile phase (python, PHP, etc.) also mess up the stats in a similar way.

The chapters on estimating are of little value to agile/scrum practitioners

There's a whole estimating process in PSP that involves collecting data, breaking down tasks, and using historical data to make very detailed estimates about the size of a task that just don't matter that much in the age of agile/scrum. In fact, I didn't see any evidence that the Personal Software Process estimating methods are any better than just getting an experienced group of developers together to play planning poker.

It's very difficult to collect good data on a real-world project

If you're going to count defects and lines of code, you need good definitions for those things. And at first glance, PSP seems to have a reasonably coherent answer. But when I actually tried to collect that data I was flooded by edge cases for which I had no good answers.

For example, a missing requirement caught in maintenance could take one hour to fix or it could take months to fix but you are supposed to record them both as requirement defects and record the fix time. Then you use the arithmetic mean for the fix time in your stats. So that one major missing requirement could completely distort your stats, even if it is unlikely to be repeated.

Garbage in, garbage out. Very concerning.

By the way, there's some free software you can use to help you with the data collection and analysis for PSP. It's hasn't been very useful in my experience but it's probably better than the alternative, which is paper forms.

Your organization needs to be completely on board

Learning the Personal Software Process is a huge investment. I think I read somewhere that it will take the average developer about 200 hours of devoted study to get proficient with the PSP processes and practices. Not many organizations are going to be up for that.

PSP/TSP works because the people using it follow a detailed process. But most software development occurs at CMM Level 1 because the whole business is at CMM Level 1. I think it's pretty unlikely that a chaotic business is going to see the value in sponsoring a super-process-driven software development methodology on the basis that it will make the software developers more productive and increase the quality of their software.

You could argue that the first 'P' in PSP stands for 'personal' and that you don't need any organizational buy-in to do PSP on your own. And that may technically be true but that means you'll have to learn it on your own, practice it on your own, and resist all the organizational pressure to abandon it whenever management decides your project is taking too long.

Is the Personal Software Process (PSP) a waste of time then?

No, I don't think so. Humphrey nails the problems in software development. And PSP is full of great ideas. The reason most people won't be able to adopt the Personal Software Process--or won't even try--is the same reason people people drop their new year's resolutions to lose weight or exercise more by February: our brains resist radical change. There's a bunch of research behind this human quirk and you can read The Spirit of Kaizen or The Power of Habit if you want to learn more.

Humphrey comes at this problem like an engineer trying to make a robot work more efficiently and that's PSP's fatal flaw. Software developers are people, not robots.

So, here are some ideas for how you can get the benefits from the Personal Software Process (PSP) that are more compatible with human psychology.

1. Follow the recommendations without doing any of the tracking

Your goal in following PSP is to remove as many defects as possible as soon as possible. You definitely don't want any errors in your work when you give it to another person for peer review and/or system testing. Humphrey recommends written requirements, personal requirements review, written designs, personal design reviews, careful coding in small batches, personal code reviews, the development and use of checklists, etc.

You can do all that stuff without doing the tracking. He even recommends ratios of effort for different tasks that you could adopt. Will that get you to 0.06 defects/KLOC? No. But you might get 80% of the way there for 20% of the effort.

2. Adopt PSP a little bit at a time

To get around the part of your brain that resists radical change, you could adopt PSP over many months. Maybe you just adopt the recommendations from one chapter every month or two. Instead of investing 200 hours up front, you could start with 5-10 hours and see how that goes.

Or you could just track enough data to prove to yourself that method A is better than method B. And once you're satisfied, you could drop the tracking altogether. For example, if you wanted to know if personal design reviews are helpful for you, you don't need to do all the PSP tracking all the time. You could just setup an experiment where you could choose five tasks as controls and five tasks for design reviews and run that for a week or two, stop tracking, and then decide which design review method to adopt based on what you learn.

3. Read the PSP book and then follow a process with a better chance of succeeding in the long run

Rapid Development by Steve McConnell is all about getting your project under control and delivering working software faster. McConnell, unlike Humphrey, doesn't ignore human psychology. In fact, he embraces it. I believe most teams would follow McConnell's advice long before they'd consider adopting the Personal Software Process (PSP).

Most of the advice in Rapid Development is aimed at the team or organization instead of the individual, which I think it the correct focus. PSP is aimed at the individual but I can't see how you're going to get good results with it unless nearly everyone working on your project uses it. For example, if everyone on your team is producing crappy software as quickly as they can, your efforts to produce defect-free software won't have much effect on the quality or delivery date of the finished software.

What I'm going to do

I develop e-commerce software on a team of two. My colleague and I have adopted a number of processes to ensure only high quality software makes it into production. And we've been successful at that; we've only had 4 critical (but easily fixed) defects and a handful of minor defects make it into production in the last year. Our problem is that we have quite a bit of rework because too many defects are making to the peer code review stage.

I showed my colleague PSP and he wasn't excited to adopt it, especially all the tracking. But he was willing to add design reviews to our process. So we'll start there and improve our processes in little steps at our retrospectives--just like we've been doing.

Rod Chapman recorded a nice talk on PSP and I like his idea of "moving left". If you want go faster and save more money you should move your QA to the left--or closer to the beginning--of your development process. That sounds about right to me.

Additional resources

Here are some resources to help you learn more about PSP:

Nice overview of PSP by Rod Chapman (video)
Watts Humphrey speaking about TSP/PSP (video)
A PSP case study supporting PSP (pdf)
Research paper that casts doubt on the benefits of PSP (pdf)
PSP: A Self-Improvement Process for Software Engineers (book)
Link to the programming exercises for the book (website)
Process Dashboard - a PSP support tool (website)

Takeaways

It's tough to recommend the Personal Software Process (PSP) unless you are building safety-critical software or you've got excellent organizational support and sponsorship for it. Most developers are just going to find the Personal Software Process overwhelming, frustrating, and not very compatible with the demands of their jobs.

On the other hand, PSP is full of good ideas. I know most of you won't adopt it. But that doesn't preclude you from using some of the ideas from PSP to improve the quality of your software. I outlined three alternate paths you could take to get some of the benefits of PSP without doing the full Personal Software Process (PSP). I hope one of those options will appeal to you.

Have you ever tried PSP? Would you ever try PSP? I'd love to hear your thoughts in the comments.

Why I can't recommend Clean Architecture by Robert C Martin

Blaine Osepchuk — Mon, 23 Jul 2018 14:44:20 +0000

Clean Architecture failed to meet my expectations on a number of fronts. Despite Mr. Martin's obvious passion for the topic, Clean Architecture is poorly organized, lacks examples, and is silent on working with existing systems. The author missed a major opportunity to teach us when and how to apply these lessons to our own systems. Let me explain.

Clean Architecture is a poorly organized book

This book takes a long time to get going. The chapters on design paradigms (structured, object oriented, and functional) seem particularly out of place and unnecessary.

The chapters on the SOLID principles are good. I enjoyed seeing the principles broken down and explained well. And I found it interesting to think about their applicability to system architecture. But Uncle Bob presents the SOLID principles like hard rules, which rubbed me the wrong way. In fact, I'm pretty sure a system that never violated the SOLID principles would be a giant mess.

However, this paragraph from page 137 is important:

The primary purpose of architecture is to support the life cycle of the system. Good architecture makes the system easy to understand, easy to develop, easy to maintain, and easy to deploy. The ultimate goal is to minimize the lifetime cost of the system and to maximize programmer productivity.

That's a great observation. Why didn't Uncle Bob put it in the first chapter?

Not enough examples

There are hardly any examples in this book. Chapter 33 contains an example that discusses a video sales e-commerce application. It's okay. But I didn't walk away from it with a firm idea of how I'd apply the concepts to my own systems.

The appendix tells the story of how Uncle Bob came up with the SOLID principles and the rules of clean architecture. It's loaded with examples of past projects. I think it's the most interesting section of the book.

The book is silent on improving the architecture of existing systems

Most developers spend most of their time working on existing systems. So, I expected Clean Architecture to be full of advice on improving existing systems. But the book is conspicuously silent on the subject.

How to create a clean architecture?

In the first half of the book you'll learn that you create a clean architecture by following the SOLID principles to break your system into components along your system boundaries (I'm paraphrasing). If you stopped reading there, you could be forgiven for having the impression that Uncle Bob would not approve of whatever you've been doing for architecture. You could also be forgiven for thinking that the few options he presents are the "right" way to do things. Yet towards the end of the book you'll read this on page 228:

This example is intended to show that architectural boundaries exist everywhere. We, as architects, must be careful to recognize when they are needed. We also have to be aware that such boundaries, when fully implemented, are expensive.

At the same time, we have to recognize that when such boundaries are ignored, they are very expensive to add in later—even in the presence of comprehensive test-suites and refactoring discipline.

So what do we do, we architects? The answer is dissatisfying. On the one hand, some very smart people have told us, over the years, that we should not anticipate the need for abstraction. This is the philosophy of YAGNI: "You aren’t going to need it." There is wisdom in this message, since over-engineering is often much worse than under-engineering. On the other hand, when you discover that you truly do need an architectural boundary where none exists, the costs and risks can be very high to add such a boundary.

So there you have it. O Software Architect, you must see the future. You must guess—intelligently. You must weigh the costs and determine where the architectural boundaries lie, and which should be fully implemented, and which should be partially implemented, and which should be ignored.

So more than half way through the book he says that it is up to us to decide where we should put the boundaries in our systems. And that boundaries are potentially everywhere. Confusing, right?

What this book is missing

So, if the ultimate goal of software architecture is to minimize the lifetime cost of the system, shouldn't a book on architecture help an architect make those decisions?

Unanswered questions

This book left me with a lot of unanswered questions. Where is the discussion of the economics of the various options? How do I find the optimum architecture for my system? Where's the research?

How do I decide if horizontal layering or vertical slicing or something else will minimize the lifetime cost of my system? Or if I have no clearly defined layers, how do I decide which, if any, layering strategy will minimize my lifetime costs?

I have even more questions. Where should you put your effort if you have a limited amount of time to improve the architecture of an existing system? Is separating the database from the business logic is always a good idea? Which advice should you always follow? Which advice depends on the system?

Details that would have made Clean Architecture more valuable

In Code Complete, Steve McConnell talks about the tradeoffs between different non-functional requirements like reliability, dependability, correctness, maintainability, readability, etc. He explains how some requirements move together and others conflict. I would have loved to see something like that for the architecture decisions discussed in this book.

In Clean Architecture, project size, team size, the consequences of project failure, expected code lifetime, and other important factors are under-emphasized as drivers of architecture. For example, Healthcare.gov needs more architecture than the personal to-do list you are developing, even though they are both web apps backed by databases.

What this book is really about

I spent the majority of this book slightly confused. I kind of understood what Uncle Bob was trying to say. But didn't completely get it until I read the appendix. So, let me save you some time.

An example

Imagine you are building a desktop computer for the consumer market (office computers for example). You get to choose the hardware and you are going to write the software for the whole thing (firmware, OS, drivers, applications, everything).

Would you write a monolith?

How would you go about it? Would you write a giant program (a monolith) where the code for the spreadsheet knew about the kind of disk you selected for your computer? Can you imagine updating the spreadsheet code and adding 'if' statements everywhere so that it does one thing if you have a SATA drive and another if you have a SCSI drive? And then doing the same thing for VGA vs DVI vs HDMI video? What about different paths for PS/2 vs USB mouse input? And then repeat the process for the word processor and all the other applications you intend to bundle with your computer?

That would be a nightmare to maintain, right? So what's the solution? Architecture! Your spreadsheet shouldn't know what kind of mouse you're using. Nor what kind of display you have. It should be completely oblivious to those details.

Boundaries in your computer

And that's what you'll find if you look at your computer. There's embedded software in your mouse that communicates with your operating system. Yet the details are hidden from your applications. Your spreadsheet accepts standardized input without knowing or caring what kind of mouse you are using. Then when someone invents a new input device like the touchpad it works with your spreadsheet automatically.

That's just one of the boundaries in your computer. You'll likely come up with hundreds of them. You might assign different teams to work on different parts of the system. You might create or adopt different specs for the various ways the different components interact.

You're probably saying 'duh' at this point. It's obvious that you wouldn't want to change and recompile your spreadsheet every time your hardware changes. Maintenance would be a nightmare. And I agree.

Boundaries are your friend (if you use them correctly)

It's easy to see hardware boundaries. But the same logic that makes hardware boundaries worthwhile also applies to software boundaries. Software boundaries are just harder to see.

So, you might start with the ability to display the spreadsheet on the screen. But you might also want to save it to disk, save it to PDF, save it as a CSV, or print it. So, maybe one of the boundaries in your spreadsheet program is to have an internal data structure that represents each spreadsheet. And then you pass that structure to different code to display, save, or print it in the desired format.

If you keep the data structure completely unaware of how it is displayed, saved, or printed, then you can add a "save to XML" feature down the road without digging through all the code related to the spreadsheet data structure. And if that spreadsheet data structure is several million lines of code, you can imagine how much easier that would be.

That's all Uncle Bob is trying to tell us in this book. You can use the SOLID principles to create boundaries in your systems that hide your implementation details, reduce complexity, and help you reduce the total lifecycle cost of your systems.

A better software architecture book

In many ways, Patterns of Enterprise Application Architecture by Martin Fowler is far superior to Clean Architecture. Fowler describes the patterns he's observed repeatedly in enterprise applications. He gives a simple example if each pattern, describes how it works, and where to use it. I found Patterns of Enterprise Application Architecture to be very readable and applicable to my systems.

Takeaways

There is valuable information in Clean Architecture but you have to work to find it.

I found the chapter on embedded software one of the easiest to understand. It intuitively makes sense to avoid scattering low-level calls throughout your code base. And it also makes sense to make your logic testable in the absence of the hardware (especially since embedded software is often developed before the hardware is available). If you could only read two chapters of this book, I'd recommend you make it this one and the appendix, regardless of what kind of software you write for a living.

Overall, Clean Architecture is a tough read and Uncle Bob left me with more questions than answers. I definitely wouldn't recommend this as your first book on software architecture (check out Patterns of Enterprise Application Architecture by Martin Fowler instead).

Agree or disagree. I'd love to hear your thoughts.

Enjoy this post? Please "like" it below.

How I intend to become a better software developer

Blaine Osepchuk — Tue, 22 May 2018 13:57:09 +0000

My survey of the computer science literature suggests there are only two economical ways to achieve extremely low defect rates (< 1 defect per KLOC). The first way is to follow the Personal Software Process (PSP), which was created by Watts S. Humphrey at CMU. The second way is to use languages and tools that make it difficult to introduce errors into your code in the first place and easier to detect errors if you do manage to get some into your code. In this post I'm going to briefly discuss these two options and how I plan to explore them to become a better software developer.

But first, why do I want to do this? Because my experience building and maintaining software roughly tracks the data coming out of the research: defect-riddled code and all the testing, debugging, correcting, and rework that go along with it are incredibly wasteful. Our industry is overflowing with failure. There's got to be a better way.

I've already written at length on working on the most important thing, avoiding waste, code review checklists, optimal pull request size, and so forth. So, in this post, I'm going to focus solely on what it takes to produce software with very low defect rates.

Personal Software Process

PSP might best be described as the application of lean manufacturing principles to the activities of an individual developer. Programmers tend have poor insight into the various ways they make mistakes and create waste. They make the same mistakes over and over again and think they are awesome (me included). But the truth is that you are the one who is putting all those errors into your code, writing complicated code, writing unmaintainable code, and so on.

PSP teaches programmers how to track their work and their mistakes. And it provides programmers with strategies they can use to improve their performance over time. From what I hear, people who follow this process can improve dramatically.

Better languages and tools

It's very difficult to write correct code in most languages. They're ambiguous and they allow programmers to do things that are almost certainly errors. Things get more interesting when you switch to a safer language like Ada and/or its ultra-safe subset called SPARK. This fits very well with my idea that programming languages need to become significantly safer by default if we're ever going to improve software security.

Ada

Let's talk about Ada first. Strong typing, ranged types, contracts, and a really smart compiler help you avoid the stupid errors that languages like C/C++ and their derivatives ignore. So, if you choose to program in Ada, you're going to have an easier time detecting and localizing defects that you wouldn't even notice with another language.

SPARK

If you want to take things to another level, you can look into using SPARK, which is a subset of Ada that can apply formal methods to your code.

I know it sounds scary but I've been playing around with SPARK and I think it's awesome. First off, you don't need to know or do any serious math to use SPARK. SPARK can prove (or at least attempt to prove) your Ada contacts and type constraints automatically. If you just stopped right there, SPARK basically acts like that best static analyzer you've ever seen.

But you can go further. You can add assertions to your code and SPARK will attempt to statically prove that they will never be violated. How far you go is up to you but you can get your code--or some important parts of it--to the point that SPARK can demonstrate an absence of run-time errors. That means it can't crash from overflows, divide by zero, etc.

Why would you want to do that? Four reasons:

Just like with TDD, writing your code in a way that SPARK can prove results in really well designed, simple, easy to read code.
If SPARK is guaranteeing that whole class of errors can't happen, you don't need to write tests for as many paths through your code.
There's a strong correlation between absence of run-time errors and the absence of logical errors. That means that code proved by SPARK tends to have fewer logical errors than code that hasn't been proved with SPARK.
Once you understand what SPARK can do, you can stop thinking about those things when you are coding. It's similar to the way you don't have to agonize about your spelling when you write an email because the spell checker handles that for you. It's really impressive when you see it in action.

Just a word of warning

There's not much information on the web about Ada/SPARK (compared to more popular languages). And much of what's out there is outdated or just wrong. Both Ada and SPARK went through major changes in their latest releases that addressed some real pain points. Plus, Moore's law has made compiling and proving the correctness of your software orders of magnitude faster than they used to be.

My plan

I'm already experimenting with Ada and SPARK. This kind of programming is a little different than I'm used to but I found some good resources and things are starting to click (see resources below). I plan to build a few toy applications with Ada/SPARK solidify my learning.

For a bunch of reasons I won't go into here, it's unlikely that Ada will ever be a mainstream language. So, I'm not under any delusions that I'll get to write production code in it any time soon. But, I can already see that some of the concepts can be applied to any programming language with very little effort. And I'm looking forward to using some of those ideas to improve my code at my day job.

As for PSP, I ordered the book PSP: A Self Improvement Process For Software Engineers by Watts S. Humphrey and I plan to work through it and the accompanying programming exercises straight away. PSP is language agnostic so if you're interested in upping your game immediately, you might want to start with PSP and explore Ada/SPARK later.

Update: 2018-09-03:
I've been studying PSP for the last few months and I've written a summary of what I've learned so far.

Takeaways

Both PSP and Ada/SPARK are more prominent in the safety-critical, mission-critical, and high security sides of our profession. I've never heard any of the web development gurus mention this stuff so, I have no idea how well these tools will apply to my day job as a back-end ecommerce developer. My guess is that almost anything I can do to prevent defects from getting into our code base will be worth doing.

The research is very clear on the rising costs of fixing defects in the later stages of the development lifecycle. So, I can give up some speed on coding to save time on testing, debugging, and unplanned rework. The trick will be finding the sweet spot.

Would you ever consider learning PSP or Ada/SPARK? I'd love to hear your ideas in the comments.

Resources

How to achieve ultra-low defect rates:
Paper: The Fumble Programmer by Rod Chapman
Video: Murphy vs Satan: Why Programming secure systems is still hard by Rod Chapman
Should we trust computers? (video or word doc) by Martyn Thomas
Making Software Correct by construction (video or word doc) by Martyn Thomas
Safety-Critical Systems (video or word doc) by Martyn Thomas

Learn PSP:
Book: PSP: A Self Improvement Process For Software Engineers by Watts S. Humphrey
Video: An Introduction to PSP by Watts S. Humphrey

Learn Ada/SPARK:
Pluralsight Course: An Introductions to Ada 2012
Contract-Based Programming in Ada 2012 (video or slides) by Jacob Sparre Andersen
Website: Introduction to embedded programming with Ada
Book: Building High Integrity Applications with SPARK by John W. McCormick and Peter C. Chapin
Website: Ada Information Clearinghouse