Forem: Rob Cooper

Technical Debt - The Unseen and Surprising

Rob Cooper — Thu, 18 Mar 2021 07:15:53 +0000

TL;DR;

We accrue technical debt in different, often unseen and innocent ways:
- 'Unexpected' - created by unpredictable or aggressive change.
- 'Unintended' - created by something missed at the design/development phase.
- 'Tools and Tech' - opportunity cost of not leveraging faster, more effective tooling and technologies.
- 'Team changes' - codebase incoherence due to number of contributors (who may no longer be part of the team).
As with all work - get it into the backlog so it's visible.
Weigh up not only the cost of clearing technical debt, but also the value gained from clearing quickly.
If you can't 'sell' the benefits of a refactoring - you probably shouldn't do it.

'Technical Debt' (Tech Debt) is explained nicely by Wikipedia:

Technical debt (also known as design debt or code debt, but can be also related to other technical endeavors) is a concept in software development that reflects the implied cost of additional rework caused by choosing an easy (limited) solution now instead of using a better approach that would take longer.

Many roll with the financial metaphor and discuss 'interest' (time, effort) to clear the debt.

Tech debt is recognition that "we're intentionally cutting corners here". This may sound bad - but there's several great reasons to do take on tech debt - to name a few:

Speed to market - especially if a new product where scale and reliability are less important than early user feedback.
Beta testing - similar to the above - but normally to a closed group. While we may not ship the product to the public - we're willing to 'roll forward' with warts-and-all. This is very typical in Scrum teams where Products are continuously incremented.
Deadlines - the real kind. While generally rare - they do exist. Contractual or legal obligations may force corners to be cut.

I believe these are well understood - and often agreeable. However, I keep seeing 'Technical Debt' getting used for very different types of problems. They need different approaches to solve.

Traditional Tech Debt

Let's get the common and easy one out of the way. When knowingly cutting corners, define up-front when you expect/want to pay that debt off. Similar to how you'd take out a loan.

For example:

"We know this won't scale" - "When we hit a sign up rate of X/Y, we'll get this into the next Sprint"
"We want to get this out ASAP, with some known bugs" - "We'll clean these up next Sprint"
"To meet contratual agreements - we'll do X, but we must spend the next Y Sprints cleaning up"

Generally speaking - the more 'interest' you need to pay, it's best done quickly.

Unexpected Tech Debt

Increasingly common - especially with the rise of more highly distributed, service-orientated systems. Essentially the system becomes a victim of it's own success and/or isn't refactored quickly enough to meet rapidly-changing business demands.

For example:

Explosion of usage (e.g. increased user counts, taking an API public) means that the system is now operating way beyond it's original design. For example: Data stored may not be partitioned effectively given the new usage patterns.
What seemed sensible at the time... Wasn't. While moving fast - things might get implemented in the heat of the moment - and look fine. The holistic view of the system is missed - and a key refactoring should have taken place to accommodate the new demand. For example: Addition of new markets introducing cross-cutting, leaky models into an architecture.

These ones are very tricky and frankly - often painful for the development team. They could start to manifest as performance issues. Or, the 'awesome team that made this business' suddenly starting to look slow and like a bug factory.

A typical phrase I use when working with a team experiencing these is:

"Knowing what you know now - how would you build it?"

If the solution is architecturally very different - you've likely got some unexpected tech debt.

As soon as these are identified - get them in the backlog. Make sure the Product Owners are extremely aware of the problem and how it's only going to get worse. Sitting down and devising some immediate (albeit hacky) mitigations while figuring out a long-term fix is often needed. Give the Product Owner options to work with.

Unintended Tech Debt

If 'unexpected' is the tech debt created in the future, 'unintended' is created in the past. This is essentially "we built with the best of intentions, but we missed something".

Again, this is becoming increasingly common - especially with the glorification of phrases like "move fast and break things".

I like to throw different ones around like:

"Don't break the till" (or 'cash register' for our friends speaking American English 😊). If people can't pay, you don't get paid.
"Don't break the door" - No signups, no users - not to mention reputation damage. Ever seen a busted door during a extremely successful marketing campaign? Not pretty.

There are certain areas where unintended tech debt can be very damaging - so extra care/attention is required. I'm not advocating "Big Design Up Front" (BDUF), but "care and attention to details when and where it matters".

This kind of tech debt can vary in severity based on what's affected. However, normally these come as a nasty shock to the team and can result in high-priority fixes. Not only do these lower the profile of the development team, but also ruin plans in progress. Not fun for anyone involved.

The key here is quality design and testing. Spend enough time in the design/modelling phase. What's your testing plan? Do you have adequate time to perform exploratory testing? What else can be shifted-left?

Tech and Tools

It's worth highlighting these because:

Tools and technologies evolve

What was good yesterday, may be superseded tomorrow by something 100x better. Not leveraging that can introduce debt due to opportunity cost etc.

To help overcome this - I think best thing is a healthy culture of curiosity, learning and sharing among the developers. We can't learn everything so spread the load and get talking about the highlights!

Make sure you and your team has the freedom to learn, play and discover what's new and upcoming so they can be ready to run with it.

Platforms change, as do their costs

I remember having to rent fixed kit with a fixed cost (👴). We now live in incredible times with a veritable feast of PaaS options on the table. These are constantly evolving - potentially offering (significant) cost savings both in usage and (often more importantly) saved developer time and cognitive load.

Similar to the above - a healthy culture creating awareness is key. If you're in an organisation large enough to have Architects around - this should absolutely be one of their core objectives to be on top of this.

Team Changes

We work in teams with multiple Developers. There's churn. A lot of opinions end up in the code. You end up with code that's "kind of OK", but over time has become increasingly awkward or downright dangerous to work with. Yes, we know "it needs a refactor, and better tests". But, few are willing to do it given the risks it might introduce.

Whenever I am working with teams and hear grumbles of "oh no, we need to go near that", I'll start to ask what their understanding of the problem(s) are, how they intend to fix it, and risks.

If it feels like a spike is needed to work out answers, then we'll do that.

Once we're armed with plan and possible solution(s) - I'll then ask:

"What is your estimate if we don't take time to fix this?"
"... and what if had fixed it, per this plan?

If the plan means smaller, more confident estimates - we likely commit to it. Especially if we know that there's a lot of change in that area coming up. Having several 5-pointers turn into a series of 2's by taking the time to 'finally fix that thing' not only feels good from a planning point-of-view, but general pride in the codebase.

'Dislike' Isn't 'Debt'

This is the one that tends to lose me most friends 😉 It's important to call it out though, because it's doing us harm.

When adding tech debt to the backlog - it needs to be based on 'this is causing objective pain' - not "I don't like it" or "this would be cool".

This will often manifest as "The Great Refactoring", only to discover one or more of the following:

It's in a part of the codebase that's barely touched - and no further changes needed on the horizon.
The developer flew solo - often creating more of the 'incoherence' problem. They dip in, changing patterns/conventions in place in just that bit - may look good for them, but the wider team now struggles with it.
It was an opportunity to introduce {coolest pattern/technology of the day} - often adding complexity rather than taking it away.
The Product Owner doesn't really 'get' it, and trusted the developer in that 'we just need it'.

This is tricky to pick up. It can be difficult to judge if a refactor adds enough value. It's important that we try though because these low/no/negative value-add refactorings cost time, energy but more importantly - trust.

As professionals we need to work with our Product Owners and the rest of the team, and be conscious of the investment (our time, opportunity cost, increases in complexity) we're proposing.

A good way of getting a sense of it is to simply do the maths! Take your daily salary - plus some for that of your pair. Guesstimate time you're going to spend on it - what would be required to get that return on investment back? This can be a real eye-opening experience - especially when you consider that some people go down the rabbit hole for entire Sprints or more.

In my experience - there's two common warning signs:

'Tech Debt' Sprints appearing on plans.
Developer X (often a senior) doing a 'thing' for days/weeks that no-one else on the team can really explain. But "they know what they're doing".

Arguably - local leadership could/should pick this up, but we're all human. Everyone has good intentions.

So, be conscious of your investments. If you can't 'sell it' to your Product Owner (as you should), skip it (or spend more time figuring out the benefits).

To close...

Not all tech debt is created equal, much (most?) of it is unintentional. This unintentional debt often has very negative side effects when it becomes apparent. There's several warning signs we can look out for. Once discovered, get it into your backlog to review and resolve.

Understanding the cost of both paying off the debt, and delay in paying of the debt are key for prioritising.

Work with your Product Owner - if you can't 'sell' a refactoring, you probably shouldn't be doing it. Present options/trade-offs. Negotiate.

Now go create.

Happy hacking!

Learn a New Language!

Rob Cooper — Tue, 09 Mar 2021 08:13:14 +0000

In the last post - we talked about why deliberate practice is important, and that I aim to approach my practice with:

Purpose - I'm spending my non-refundable time on something, I want it to count.
Plan - Before I start, I think about how I'm going to structure my practice so that it (hopefully) meets it's purpose.
Progress - Ensuring I take time to reflect on the practice, and adjust as necessary.

To get specific (we're programmers, after all) - this translates to:

Defining Purpose

Creating plans to exercise 'something' that I encounter in my career - for example (and this list is far from exhaustive):

Learning a new programming language.
Working with different IDE's to develop and debug.
Traversing and processing data.
Effective automated testing. From unit to end-to-end.
Creating build and deployment pipelines.
Working with different databases and ORMs.
Performance profiling and testing.
Finding memory leaks.
Rapid prototyping.
Paradigm shifts - e.g. object-oriented to functional.

I will find myself in a situation where either I or a colleague needs some help with a specific area - I then go to my list and see if I either have something defined that works, or start defining what I want from the session based on needs.

Defining a Plan

Given the Purpose I then start to think about a Plan.

Let's say I want to pick up a new language as the client I am working with is on a different stack. I think I generally need to:

1. Get the basics

Can I start to express myself in the new language?

How do I define basic types? (Integers, Floats, Booleans, Strings, Array)
How do I concatenate or interpolate strings?
How do I read from/write to the console?
What are the basic control flow statements? (if/else, foreach, switch etc.)
How do I define functions and methods?
Can I define anonymous functions?
How do I control the visibility of those functions and methods?
How do exceptions work? How do I handle them?

This stage is generally easier if you know how to code already - you'll often ask "How do I do X in Y?" This can lead to easier searches. However, having a checklist makes as a great teaching aid for fledgling programmers.

By the time I'm done with this list - I'm generally able to bumble my way through a problem.

2. Get Productive and Effective

Now that I can write some code - I am starting to think more about how it's really working:

How do I define, run and interpret automated tests?
What data structures do I have available to me? How could/would I implement them if not?
What foundation / base libraries do I have available for common things - for example collections, HTTP, serialisation etc.
How do I use packages/modules that others have created?

At this stage, I'm probably starting to feel a lot more confident. I've got tests backing up my work and I'm using tried-and-tested code from others where possible.

3. Optimise!

I'm writing code - but it's likely not the 'best' code. I'm likely upsetting the veterans in the codebase with every pull request. Now I start to think about:

Going 'native' - how do I write more idiomatic code for that language? Is the method named IsAuthenticated, isAuthenticated, is_authenticated, is_authenticated? or HOW IZ I AUTHENTICATED YR USER?
Thinking about performance - using correct data structures (because not everything belongs in an array, y'all), understanding how they're traversed etc.
Being conscientious with memory usage - how do I get stuff off the heap? Reduce allocations etc.

... Reflect on Progress

I'll open up some notes, and work my way through the above, in order. Getting "the basics" might take an hour or so. From there, I'll try and start solving some simple problems via Katas.

As I start to learn more (e.g. more expressive/cleaner ways of saying things) - I'll continually refine my notes.

I get a real sense of achievement in seeing my notes evolve. Going back to something you've personally written and editing/improving proves progress.

I'll also solidify/verify the growth by tackling random coding problems (e.g. from the likes of CodinGame or CodeWars).
It's amazing how quickly you can get something going!

Rinse and repeat!

Enough Talk, Let's Code!

We've outlined a plan - let's go learn the basics of a new language!

Now - my 'home' language is C#, but I've done some JavaScript, TypeScript, Python, Ruby and Java at varying levels. I'm gonna pick a random just for funsies - Elixir.

🥰 Big thanks to the awesome that is repl.it for enabling me to hack in a browser-based IDE! I highly recommend trying it (or similar) so that you can get to the language without worrying about IDE setup etc.

Run the List

# How do I define basic types? (Integers, Floats, Booleans, Strings, Array)
an_int = 1
an_float = 1.0
an_bool = true
an_str = "Elixir"
an_arr = [1, 2, 3]

# How do I concatenate or interpolate strings?
ans = "Forty-two"
meaning = "The answer to the great question.. Of Life, the Universe and Everything.. Is.. #{ans}."

# How do I read from/write to the console?
user_name = IO.gets("What's your name? ") # Trailing space to look nicer in console
IO.puts("Hello, #{user_name}!")
# Note that you get the trailing newline with gets- so you'll likely want to trim
x = String.trim(IO.gets("?"))
x = IO.gets("?") |> String.trim() # Can also pipe - easier on the eyes?

# What are the basic control flow statements? (`if/else`, `foreach`, `switch` etc.) 
# https://elixir-lang.org/getting-started/case-cond-and-if.html
# https://elixir-lang.org/getting-started/recursion.html
if true do
    # ...
else
    # ...
end

unless true do
    # ...
end

for(i = 0; i < sizeof(array); i++) {
  array[i] = array[i] * 2;
}
# Enum module also has map and reduce functions
# https://elixir-lang.org/getting-started/enumerables-and-streams.html#enumerables
Enum.map([1, 2, 3], fn x -> x * 2 end)
# [2, 4, 6]
IO.puts(Enum.reduce([1, 2, 3], 0, fn x, acc -> acc + x * 2 end))
# 12

beats = case play do
    "scissors" -> "paper"
    "paper" -> "rock"
    "rock" -> "lizard"
    "lizard" -> "spock"
    "spock" -> "scissors"
    "scissors" -> "lizard"
    "lizard" -> "paper"
    "paper" -> "spock"
    "spock" -> "rock"
    "rock" -> "scissors"
end

# How do I define functions and methods?
# (Elixir is functional, and doesn't have concept of Objects, Classes and Methods)
defmodule MyModule
    def sum(a, b)
        a + b
    end
end

# How do I control the visibility of those functions and methods?
defp im_a_private_function
    # ...
end

# Can I define anonymous functions?
sum = fn (a, b) -> a + b end
sum = &(&1 + &2) # Shorthand using &
# Note, these need to be called with .
x = sum.(2, 3) # -> 5

# How do exceptions work? How do I handle them?
# https://elixir-lang.org/getting-started/try-catch-and-rescue.html

raise ArgumentError, message: "Argument out of range"

try do
    # ...Code that might raise
rescue
    ArgumentException -> "Caught the ArgumentException"
end

# Define custom exceptions via a Module with defexception
defmodule MyError
    defexception message: "Default message"
end

Now Go Play...

Define a Purpose - What would you like to learn? Try to keep this in the abstract so it will be reusable for exploring similar scenarios. "Languages" over "C#", "Build Systems" over "TeamCity", "Cloud" over "Azure".
Define a Plan - How will you go about exploring? Make a list - or map - to keep you focused. Iterate and refine this map as you learn more about this "type" of problem. For example - anonymous functions are so useful for cleaning up code, so I added a specific item for in to my list above to make sure I found out if it's possible, and how to do it.
Reflect - How will you solidify your learning? What's next? I really like to have some kind of physical output here - working code FTW, but - it's your practice! Do what you like!

Now go practice.

Happy hacking! 🤩

Programmers Need Deliberate Practice

Rob Cooper — Sun, 06 Dec 2020 21:03:11 +0000

TL;DR;

Consistent, effective practice is a key part of a healthy, enjoyable career.
Most people practice without purpose.
This means they either quit and/or learn less than they could.
Read this for inspiration on how to take a more considered approach to your practice.

Note: I'm English, but opted for the US English spelling of 'Practice' throughout (as opposed to 'practise/practice') to avoid confusion with the difference in spelling within the term 'Deliberate Practice'.

Practice?

I think we can all agree on a couple of things:

People who are experts practice often.
Experts are extremely disciplined with their practice.

This will generally come as no surprise when mentioned, and people will rarely argue those points. They make sense.
Thanks to the growing number of interviews and podcasts - experts are getting more limelight, and we're getting more sneak-peeks into that practice routines, habits and rituals (podcasts like Tim Ferriss' are awesome for this).

However - the one thing that does surprise me when I ask my peers what their practice schedule/routine is - I'm met with blank faces/realisations of "oh, I don't have one".

If they do - it will likely be something that will be pretty loose/unregimented. So if then prodded on the effectiveness of that practice - we're back to blank faces. I don't want that - I want to meet up with people and have them bursting with excitement on how they're growing.

The idea of effective practice has been has been solidified with the concept of deliberate practice. That we shouldn't just "go through the motions" - but need to take a considered, thoughtful approach to our practice - that ensures we get outside our comfort zone and grow.

This is important - because I feel this is something that is sorely missing from the tech community generally. Many of us tend to hack around on things, work on side projects, contribute to open source etc. - which are all AWESOME. But how many of us take a disciplined, thoughtful approach to learning, growing and honing our skillset?

To be clear - I've been hit-and-miss over the years as well - which is why I'm spending real time on it now, and want to help you while I'm at it!

This is important and something I deeply care about - we're all time-poor, and what you spend you aren't getting back. Make it count!

Common Problems with People's Practice

They don't have one! Pretty self-explanatory.
Lack of direction. They do things 'just because' - and work through the motions rather than aim for growth.
Lack of relevancy. The practice doesn't seem relevant/useful to their day-to-day, so they either don't engage well with it - or bin it.
Scope too large. They never get to an "ah-ha" moment - normally ending the session with frustration and lack of desire to come back. This often occurs when we either don't have enough direction/leading questions and/or we've not identified the 20% we need to practice to give 80% of understanding (see 'Pareto Principle').
Additional noise unrelated to the desired growth area. Perhaps you want to practice with a certain design pattern, but spent over an hour battling framework incompatibility issues instead.

But What About Katas?

I know some of you are thinking "Rob! I know, this is why we do Katas!". I hear you, but I've been doing and leading these for years and I have to say - the results vary wildly - sadly leaning on the side of 'poor'.

For the uninitiated - a 'Kata' (the term taken from the usage in martial arts) is essentially repetition of a problem to see how we solve it. Better people than me have done a lot of write-up on this - so check out the links if you want more info. Essentially a kata will play out like:

Bunch of coders get in a room.
They solve a pre-defined problem.
They discuss their code/thought-processes/lessons learned etc.

Now, these are great - and if you've not done one. Do! They're great fun, and you will learn something. I'll link to a few at the end of this post for you.

I think part of the problem is that perhaps 'Kata' isn't an adequate analogy. Katas are about increasing muscle memory or certain movements (getting progressively harder as you gain skills). I think Katas like this have a place, and I will continue to do them (The Gilded Rose is awesome, for example). But I think there's a missing piece - and once found (and done well) you can do away with 'repetitive' Katas all-together.

The goal of this series of blog posts is to help do just that! 🥋 😁

If Not Katas, Then What?

I'm not suggesting you do away Katas. I don't care for reinventing things that are (mostly) working well. Let's just change up how you prep for them. I always think "3 P's":
1. Purpose - Why are you doing this? Is it meaningful to your life right now?
2. Plan - How are you going to effectively meet the purpose of the Kata? (Read: "remove the noise, stay focused, get outside comfort zone, learn").
3. Progress - Reflect. What did you learn? What didn't you learn? (You may discover way after doing the Kata! Go back and update it!) How will you take this forward into current/future projects?

Now each Kata should standardise points 1 and 2 (Purpose and Plan). Most katas lack focus (sound familiar? 'Stories' anyone? Cough 😂) and/or don't get you reaching outside your comfort zone enough. Many people will solve the problem with what they know, and quickly move on. If you take just one thing from this post - take 5-10 mins to think before you practice about WHY you're practicing.

Point 3 (Progress) is often done well (I always enjoy the group chats) - but I rarely (if ever) see Katas refined.

So, let's walk through some quick examples! .⚠️ This series of posts will go into more detailed examples - this is just a primer to get you going/thinking!

Ideas to Get Started/Improve

Gilded Rose

This is one of my favourite Katas - and it will always have a special place in my heart. So let's start with that one.

If you've not done it before - just do it as per the README. 🚧
Now consider the purpose - we're talking about refactoring here - so what common problems come in to play? The goal of refactoring is to make the code easier to understand and (perhaps) easier to extend in the future. Here's some ideas:
- Pair up with someone else! Does the refactoring make sense to you both as you go?
- "Person down the hall test" - fly solo and then have a peer review your code from cold at the end. For an added twist - submit the code anonymously, and only let them view it as a pull request. How quickly can the understand the code and the reason for your changes?
- Define one or more additional requirement - but keep it secret until the end of the Kata and ask participants "What if?" The purpose is then the conversation, not the code.
- Consider the test runner output - if there's a failure when adding new behaviours - do those failure messages make it clear what the expectation is?

Frameworks, Patterns and Paradigms

Last, but not least to plant the seeds of some of what's to come in this series. What about frameworks, [design] patterns and [modelling] paradigms?

Now these are everywhere - likely cross multiple layers (which each has their own purpose. But here's some points to consider for some common ones:

UI - How do we bind to the UI to data? How do we maintain isolation across elements? How do we isolate elements to begin with? How do we cache content? How is that parameterised? How do we bind elements to data? What trade-offs does that design bring? How do we augment content based on device capability?
Data modelling (and implementation) -
Database access (and ORMs) - How do we do basic CRUD operations? How do we do single property/column updates? How do we examine the execution plan and/or examine query performance? How/when do indexes/pages lock? How do we effectively partition data?
Messaging - What options do we have for sending messages across boundaries? What about receiving and processing them? What trade-offs do each entail? What patterns exists for common systems (e.g. eCommerce, Chat, Streaming, Gaming)

Others

Other (but equally important) areas to note:

Algorithms - What algorithms can be used given the problem? What are the pros/cons/trade-offs?
Data Structures - Similar to the above - how can the data structures used by the code be changed/effect the outcome?
Performance - Under what circumstances does your code perform well? When doesn't it? Are those OK? Practice identifying the usage patterns and explaining them to mere mortals ☺️

In Summary

I hope I've given you pause or thought about your practice. Practice is good - great practice is better. Take time to think about the problems you need to solve and create a method to solve them in a way that gets you growing.

I look forward to sharing more about how I do this - code and all - in this series.

Until then - happy hacking! 🤩💻

Forem: Rob Cooper

Technical Debt - The Unseen and Surprising

TL;DR;

Traditional Tech Debt

Unexpected Tech Debt

Unintended Tech Debt

Tech and Tools

Tools and technologies evolve

Platforms change, as do their costs

Team Changes

'Dislike' Isn't 'Debt'

To close...

Learn a New Language!

Defining Purpose

Defining a Plan

1. Get the basics

2. Get Productive and Effective

3. Optimise!

... Reflect on Progress

Enough Talk, Let's Code!

Run the List

Now Go Play...

Programmers Need Deliberate Practice

TL;DR;

Practice?

Common Problems with People's Practice

But What About Katas?

If Not Katas, Then What?

Ideas to Get Started/Improve

Gilded Rose

Frameworks, Patterns and Paradigms

Others

In Summary

Links