Forem: Eric Damtoft

Upgrade your Value Objects in 10 Steps

Eric Damtoft — Tue, 17 Aug 2021 20:40:06 +0000

In object-oriented programming, it's tempting to overuse built-in types. For example, if you think of an object representing a vehicle, that vehicle's model year may be represented as a 32-bit integer, and it's make as a string. For many purposes, this is OK, but defining strong types for those values within your domain can make your code more expressive and allow the compiler become a powerful ally in verifying that your code is being used the way you intended it to.

However, if you look at built-in types, there are a lot of hidden features they implement which can add value. A simple implementation of a domain-specific value-object will likely miss some of these features and eventually lead to frustration for the users of that code. We'll look at some ways to make a true, fully featured value object with all the bells and whistles you've come to expect from .NET types.

As an example, let's use a simple Coordinate with an X and Y value.

class Coordinate
{
    public int X { get; set; }
    public int Y { get; set; }
}

Upgrade 1: Immutability

Not all types should be immutable, but any type which fundamentally represents a value can benefit from immutability. Immutability will generally simplify logic and clarify how your type is supposed to be used.

public class Coordinate
{
    public Coordinate(int x, int y) => (X,Y) = (x,y);

    public int X { get; }
    public int Y { get; }
}

Notice the constructor? This is just for style. Feel free to use standard assignments, but tuple assignments make for a simple one-liner. If you have more than 2 or 3 values to assign, it's probably better to use classic assignments.

Upgrade 2: Make it a struct

Structs have a number of advantages over classes for types which fundamentally represent a single value. They're passed by value and usually only allocated on the stack instead of a reference to the heap so in many cases they'll avoid allocations and reduce garbage collection, thus improving performance. Structs are best avoided for mutable types because of how they're passed, but for a small immutable type like a coordinate point, a struct is the way to go. See more about when to use a struct vs a class here. You can also use the readonly keyword so the compiler will enforce that the type is immutable.

public readonly struct Coordinate

Upgrade 3: Equatability

A good value type should be equatable by it's values. I.E. a coordinate (1,2) should equal another coordinate (1,2). Along with the Equals and GetHashCode methods, we should make sure that you can use the == and != operators and implements IEquatable<Coordinate>.

public override bool Equals(object obj) => obj is Coordinate coordinate && Equals(coordinate);

public bool Equals(Coordinate other) => (X, Y) == (other.X, other.Y);

public override int GetHashCode() => HashCode.Combine(X, Y);

public static bool operator ==(Coordinate left, Coordinate right) => left.Equals(right);
public static bool operator !=(Coordinate left, Coordinate right) => !(left == right);

Notice some interesting syntax in the Equals method. Comparing 2 tuples is an easy way to compare a number of properties. This is again a style choice and feel free to stick with a classic equality check for each property.

Upgrade 4: Formattability

We'It's common to need to specify a way of formatting a value to a string. It's always good practice to add a ToString() method, but there are often a variety of ways to represent any value as a string. In our case, we'll consider the numbers which make up the coordinate and allow them to be formatted like any other number value. To make this work, we'll add the IFormattable interface to our class. You could probably imagine a more sophisticated implementation that allowed for custom delimiter, etc. but for simplicity, we'll just forward the format onto each value.

public string ToString(string format, IFormatProvider formatProvider)
{
    return X.ToString(format, formatProvider) + "," + Y.ToString(format, formatProvider);
}

This way, you can use it in interpolated strings such as $"coordinates: {c:00}" to result in "01,02";

Upgrade 5: Parsability

If you notice, most built-in types have a static Parse and TryParse method. These are useful, and implementing them following the established pattern will make it easy for users.

We'll skip the details of actually parsing a value for brevity

public static Coordinate Parse(string s) => CoordinateParser.Parse(s);

public static bool TryParse(string s, out Coordinate c) => CoordinateParser.TryParse(s, out c);

Upgrade 6: Convertibility

Being able to easily convert between types is crucial to making a user-friendly type.

We can add implicit and explicit conversions to and from other types. This will allow the c# compiler to automatically translate these types either with or without an explicit cast.

For our coordinate class, let's imagine we also had an Vector type with a magnitude and a direction.

public static explicit operator Vector(Coordinate c) => Vector.FromCoordinate(c);

Upgrade 7: Operators

We've already added equality operators, and conversions are also considered by c# to be operators, but depending on the structure, consider adding additional operators specific to your case. In our instance, we'll just add some basic math operations, but you can also consider use cases like DateTime - DateTime = TimeSpan where the types are representative of the actual meaning of the object.

public static Coordinate operator +(Coordinate c) => c; // doesn't do anything, but complements minus
public static Coordinate operator -(Coordinate c) => new Coordinate(-c.X, -c.Y);
public static Coordinate operator ++(Coordinate c) => new Coordinate(c.X+1, c.Y+1);
public static Coordinate operator --(Coordinate c) => new Coordinate(c.X-1, c.Y-1);
public static Coordinate operator +(Coordinate left, Coordinate right) => new Coordinate(left.X + right.Y, left.X + right.Y);
public static Coordinate operator -(Coordinate left, Coordinate right) => new Coordinate(left.X - right.X, left.Y - right.Y);
public static Coordinate operator *(Coordinate left, Coordinate right) => new Coordinate(left.X * right.Y, left.X * right.Y);
public static Coordinate operator /(Coordinate left, Coordinate right) => new Coordinate(left.X / right.Y, left.X / right.Y);
public static Coordinate operator %(Coordinate left, Coordinate right) => new Coordinate(left.X % right.Y, left.X % right.Y);

Upgrade 8: Deconstruction

If we want to easily deconstruct values out of our type, we can add a Deconstruct method.

public void Deconstruct(out int x, out int y) => (x, y) = (X, Y);

This allows us to pull values out using the same syntax as tuple deconstruction.

var (x, y) = new Coordinate(1,2);

Upgrade 9: Debuggability

This one is a bit niche, but can be quite helpful for providing a view of a value geared specifically towards developers. If you inspect a value in the debugger, it'll show you the value from ToString(), but you can customize this by adding a DebuggerDisplayAttribute. In this case, we'll keep it mostly the same as the ToString(), but you could add other useful debugger information or context here as well.

[DebuggerDisplay("({X},{Y})")]
public readonly struct Coordinate : IEquatable<Coordinate>, IFormattable

Upgrade 10: Documentation

This one may go without saying, but once you're done adding features, make sure to take the time to put good XML comments on everything public. Styles and coding guidelines vary, but in general, I tend to go with the principal of only commenting when necessary and to add clarity, but if you're writing a library to be used by others, it's best to take the time to comment everything.

Also, take advantage of more than just the <summary> tag. There are a number of other useful tags for XML comments which are especially valuable if you use tools like DocFX to automatically generate a documentation website with details and examples.

/// <summary>
/// Converts a string into a coordinate object
/// </summary>
/// <param name="s">A string containing a coordinate to convert</param>
/// <returns>A coordinate equivelant to the string contained in <paramref name="s"/></returns>
/// <exception cref="FormatException">A format exception will be thrown if the structure of the string is unexpected</exception>
/// <example>
/// This demonstrates basic usage of the Parse method:
/// <code>
/// var c = Coordinate.Parse("1,2");
/// </code>
/// </example>
public Coordinate Parse(string s)

Putting it all together

We started with a simple class:

public class Coordinate
{
    public int X { get; set; }
    public int Y { get; set; }
}

This might be fine for a simple serialization bucket, but once our upgrades are all done, we've got a class worthy of inclusion in a world-class .NET library. Here's the end result (with comments omitted for brevity):

[DebuggerDisplay("({X},{Y})")]
public readonly struct Coordinate : IEquatable<Coordinate>, IFormattable
{
  public Coordinate(int x, int y) => (X, Y) = (x, y);

  public int X { get; }
  public int Y { get; }

  public override bool Equals(object obj) => obj is Coordinate coordinate && Equals(coordinate);

  public bool Equals(Coordinate other) => (X, Y) == (other.X, other.Y);

  public override int GetHashCode() => HashCode.Combine(X, Y);

  public override string ToString() => ToString(null, null);

  public string ToString(string format, IFormatProvider formatProvider)
  {
    return X.ToString(format, formatProvider) + "," + Y.ToString(format, formatProvider);
  }

  public void Deconstruct(out int x, out int y) => (x, y) = (X, Y);

  public static Coordinate Parse(string s) => CoordinateParser.Parse(s);

  public static bool TryParse(string s, out Coordinate c) => CoordinateParser.TryParse(s, out c);

  public static explicit operator Vector(Coordinate c) => Vector.FromCoordinate(c);

  public static bool operator ==(Coordinate left, Coordinate right) => left.Equals(right);
  public static bool operator !=(Coordinate left, Coordinate right) => !(left == right);

  public static Coordinate operator +(Coordinate c) => c; // doesn't do anything, but complements minus
  public static Coordinate operator -(Coordinate c) => new Coordinate(-c.X, -c.Y);
  public static Coordinate operator ++(Coordinate c) => new Coordinate(c.X+1, c.Y+1);
  public static Coordinate operator --(Coordinate c) => new Coordinate(c.X-1, c.Y-1);
  public static Coordinate operator +(Coordinate left, Coordinate right) => new Coordinate(left.X + right.Y, left.X + right.Y);
  public static Coordinate operator -(Coordinate left, Coordinate right) => new Coordinate(left.X - right.X, left.Y - right.Y);
  public static Coordinate operator *(Coordinate left, Coordinate right) => new Coordinate(left.X * right.Y, left.X * right.Y);
  public static Coordinate operator /(Coordinate left, Coordinate right) => new Coordinate(left.X / right.Y, left.X / right.Y);
  public static Coordinate operator %(Coordinate left, Coordinate right) => new Coordinate(left.X % right.Y, left.X % right.Y);

}

Ultimately, how far down the road you want to go is a judgement call. For some cases, the first version of the type is all that's required and exactly what's needed. But, if you're writing a library or core domain code, it's probably worth breaking out all the stops to make the experience smoother and more predictable for your users.

A Simple Rule for Better Variable Naming

Eric Damtoft — Fri, 13 Mar 2020 21:13:15 +0000

Variable naming is simultaneously one of the most trivial and complex aspects of writing code. Variables names are largely irrelevant to the computer running your code after it's been compiled, but they can make a world of difference for a human who has to read and maintain code.

There's a few general rules of thumb:

Keep names as short and simple as possible while still being descriptive
Follow naming conventions of the system you're coding in

But in addition, here's a more concrete one I've found to be a helpful guideline:

Make booleans affirmative

Think about the following line of code:

var disabled = true;

There's nothing inherently unreadable about that line of code, but when it comes time to use it, we probably are thinking about whether something is enabled and have to invert it. As complexity builds up, this mental flipping can take a toll on comprehensibility of your code.

if (!disabled) {
  doStuff();
}

Instead, we can rename disabled to enabled and reason about it without needing to mentally invert it.

if (enabled) {
  doStuff();
}

This way, we're using true to represent an "on" state and false to represent an "off" state which is considerably more natural to read. Although this seems trivial, it can have a big impact on overall readability of code. If you're not convinced yet, think back to the worst multiple choice test you've ever taken. The question probably looked something like this:

Which of the following is not true?

a) two plus two is not three
b) two minus two is not one
c) two divided by two is not one
d) two times two is not five

If you're like me, your internal dialog just went something like "So if it's not not true then that means... wait..." as you try to flip the logic in your head.

By writing it as an affirmative question, it becomes much easier to comprehend:

Which of the following is true?

a) two plus two is three
b) two minus two is one
c) two divided by two is one
d) two times two is five

Although these two representations are logically equivalent, it's much more human friendly. Just flipping a few logical statements in your head takes a noticeable toll even when the underlying logic is simple. When the underlying logic becomes complex, it can make it completely unintelligible.

Although there's no rule without exceptions, checking to make sure you're not flipping your bits is always a good thing to have in the back of your mind. It's often a simple refactor and can make for much more straightforward code.

Advanced String Templates in C#

Eric Damtoft — Fri, 20 Dec 2019 20:01:30 +0000

c# 6 introduced string interpolation syntax. This feature allows for formatted text to be easily declared in code. For example:

var name = "World";
var message = $"Hello, {name}"; // "Hello, World"

At it's most basic, this is syntactic sugar on top of c#'s string.Format method, but with a much cleaner syntax. However, simply interpolating strings is often a dangerous proposition. Consider the following:

var url = $"https://api.example.com/sample?arg={arg}";

This will work in most cases, but if the parameter includes non-url-safe characters, this is likely to break. This could even expose a serious security vulnerability, expecially if you're doing string interpolation for HTML, javascript, or SQL (don't ever do this!).

Formattable String

Hidden deep in the c# language spec is a minor note about string interpolation. In general, interpolated strings are compiled to a call to string.Format, but you can also cast it to a FormattableString. This type represents the string template and an array of objects which will be interpolated into it.

var make = "Chrysler";
var model = "Town & Country";
var url = (FormattableString)$"https://api.example.com/vehicles?make={make}&model={model}";

Console.WriteLine(url.Format); // "https://api.example.com/vehicles?make={0}&model={1}";
Console.WriteLine(url.GetArgument(0)); // "Chrysler";
Console.WriteLine(url.GetArgument(1)); // "Town & Country";

This provides some interesting opportunities to create much more powerful string templating tools. For example, if we wanted to automatically encode the URL arguments, we can do the following:

public static class Format
{
  public static Uri Uri(FormattableString template)
  {
    var encodedArgs = new object[template.ArgumentCount];

    for (var i = 0; i < template.ArgumentCount; i++)
    {
      var original = template.GetArgument(i);
      encodedArgs[i] = HttpUtility.UrlEncode(original);
    }

    return new Uri(string.Format(template.Format, encodedArgs));
  }
}

The above code creates a new array and populates it with the url-encoded arguments. It then calls string.Format with the original template and new encoded arguments and returns it as a Uri to indicate that it's been safely encoded.

to use it, we can call

var make = "Chrysler";
var model = "Town & Country";
var url = Format.Uri($"https://api.example.com/vehicles?make={make}&model={model}");

Console.WriteLine(url); // https://api.example.com/vehicles?make=Chrysler&model=Town+%26+Country

Custom Formats

Another interesting feature we can make (ab)use of is custom format strings. In a traditional c# string template, you can specify a format for each argument, I.E.

Console.WriteLine($"Today is {DateTime.Now:yyyy-MM-dd}"); // Today is 2019-12-13

This is effectively the equivelant of calling dateTime.ToString("yyyy-MM-dd"). Any object that implements IFormattable can be used with a custom format string, which gives us an opportunity to define a simple syntax when working with string templates. In this example, we'll set up a simple HTML template that will either html encode a value or format it as markdown.

public static HtmlString Html(FormattableString template)
{
  var encodedArgs = new object[template.ArgumentCount];

  for (var i = 0; i < template.ArgumentCount; i++)
  {
    encodedArgs[i] = new HtmlArgument(template.GetArgument(i));
  }

  return new HtmlString(string.Format(template.Format, encodedArgs));
}

class HtmlArgument : IFormattable
{
  public HtmlArgument(object value)
  {
    Value = value;
  }

  public object Value { get; }

  public string ToString(string format, IFormatProvider formatProvider)
  {
    switch (format)
    {
      case "markdown":
        return new Markdown().Transform(Value.ToString());
      case "dangerous-raw-html":
        return Value.ToString();
      default:
        return HttpUtility.HtmlEncode(Value);
    }
  }
}

We can then use this as follows:

var html = Format.Html($"<article><h1>{title}</h1>{content:markdown}</article>");

title will be safely HTML encoded, and content will be rendered as markdown.

Wrapping Up

String interpolation in c# is convenient, but can lead to some traps. If not used carefully, it can break with edge cases or even introduce vulnerabilities. Formattable strings are a little known, but potentially quite useful feature in c# that can be used to make string interpolation smarter and more context-aware.

Hierarchy of Controls for Software Engineering

Eric Damtoft — Fri, 13 Dec 2019 18:43:47 +0000

For workplace safety, the CDC publishes a "Hierarchy of Controls" prioritizing different mechanisms for enhancing worker safety from most effective, to least effective. The principals of reducing physical risk can be mapped to reducing risk in software architecture. Even though the stakes may be lower (depending on your industry), writing reliable software is all about identifying, preventing, and mitigating foreseeable errors.

Elimination

In a workplace safety environment, if there's a ladder that can slip while getting items from a high shelf, putting the items on a lower shelf eliminates the need for the ladder. These solutions are often obvious after the fact but overlooked due to assumptions that were never questioned. The problem may be posed as "how can we make the ladder safer", but the fist thing you should ask is "do I need to use a ladder?". As a software example, a web application I've worked with used extensive caching to reduce the load of expensive SQL calls on an aging SQL server. Caching, while critical to performance, was often a source of bugs. When it came time to re-work that system, we used Elasticsearch and pre-populated it with data in the shape of what we needed to query. Without expensive joins, the need for caching that data in the application layer vanished, taking the bugs (at least for that layer) with it.

Substitution

Continuing with the example of the ladder, it may be that space limitations require items to be stored at height. By replacing the ladder with a staircase with railings, a high-risk action (climbing a ladder) is replaced with a low-risk one (climbing stairs). On the software side of things, we often deal with unreliable systems. In one example, a third party API I had to get data from experienced fairly frequent outages causing long page load times. This caused a chain of service degradation. By switching to an asynchronous importer to load data into an internal database, we were able to substitute our dependency on a unreliable system (the API) to a much more reliable one (our database).

Engineering Controls

This is where most of us start our thinking. Occasionally, a risk cannot be eliminated and substituting it would be unreasonably complex or expensive. In the example of the slipping ladder, an engineer can design a stabilization system for the ladder. Perhaps incorporating high-grip rubber or cables to secure it in place. Engineered solutions can add complexity and additional potential points of failure. Likewise, in code, we can write code to handle exceptions, automatically restart failed processes, or reject invalid operations. When coupled with a CI system, automated tests can be a very effective control. However, the more code we add, the more surface area there is for future bugs.

Administrative Controls

When an engineering solution isn't possible, the next least effective solution is to control the human element. If we can't make it so that ladder is unnecessary, and can't make the ladder safer, then we can put up warning signs and require workers to be trained before using the ladder. For developers, these types of controls are everywhere. Coding guidelines, code reviews, and manual QA testing are all administrative controls. These controls can usually be implemented with the stroke of a pen, so even if less effective, they are efficient. These work well as a short term mitigation or when the risk is low, but always consider if a more effective solution is available.

Protective Equipment and Emergency Preparedness

In workplace safety, the last resort is to try to reduce harm. For our ladder, if a worker falls despite everything, a hard hat might mean a broken ankle instead of a skull fracture and having a first aid kit nearby could save their life. We similarly have last-resort procedures in software. Some examples might be a content-delivery network set to return stale content when a service is unhealthy or performing an emergency hotfix deployment to resolve a critical error. Having these plans in place is critical, but when planning for a disaster recovery, always look at the bigger picture of how to eliminate or mitigate the risk before it happens.

A Simple Formula For When To Use Let, Const, and Var

Eric Damtoft — Fri, 25 Oct 2019 20:40:21 +0000

The new ES6 variable declarations let and const have been available in major browsers for several years, but when to use each still causes a bit of confusion. Although everyone has a different style and preferences, the above flowchart is how I choose which one to use. There are more in-depth guides to the exact differences between each, but this is meant to be a simplified way of how to choose without getting too deep into the nuances of the javascript runtime.

var was the original way to declare a variable. For a while, it had to be used for any non-transpiled code because of compatibility concerns. By now, all major browsers and even IE11 support let and const. The var keyword declares a mutable reference to a value, but it is not block scoped. As someone who mostly writes c#, this means that it will occasionally cause some unexpected behavior. See this stack overflow response for some examples. It also permits bad practices like referencing a variable before it's been declared.

let and const give a more predictable experience and both generally act like you would expect a variable declaration to. The only significant difference is whether you can reassign the value. In my experience, reassignments can often be refactored to be immutable references. This means they never change state which improves clarity. Consider the following:

let value = readInput();

value = value.toUpperCase();

return value;

The variable value is reassigned to hold different values representing different states. At a glance, you can't tell what state the variable holds, which makes complex logic significantly more difficult to comprehend. Instead, by using the const keyword, we can ensure that the variable will only ever hold the state it's initially assigned. You'll also note that our variable names can become much more descriptive when one variable doesn't need to hold multiple states. Reassigned variables often are named generic terms like "output", "value", etc.

const input = readInput();

const upperCasedInput = input.toUpperCase();

return upperCasedInput; // this line is self-explanatory

In general, I find that const is always my preferred way to declare a variable. Sometimes, you can't avoid reassignments, so I fall back to let if there's no way to refactor away from it. I don't even consider using var except when working with legacy code.

Cover Image by Irvan Smith on Unsplash

Food Trucks and Async Programming

Eric Damtoft — Fri, 16 Aug 2019 15:11:22 +0000

A few days ago, we had "Food Truck Day" at DealerOn. Say Cheese, a grilled cheese food truck set up shop out front of our Rockville office and word spread quickly. As a line formed, one person in the back of the truck took orders, and two cooks worked the griddle. When an order was received, they used a queue of "tickets" to track orders, and both cooks worked simultaneously to prepare a variety of grilled cheeses. For such a confined space, the system was efficient and well-orchestrated. It struck me that this was a perfect example of an asynchronous and parallel system architecture. We'll look at some concepts of async, parallel, and distributed programming using a hypothetical food truck as an example.

Synchronous

Imagine a food truck with only a single cook who serves burgers. The line forms and they take your order, put the patty on the grill, sit by the grill waiting for it to cook, add a bun and toppings, and serve it. Then they take the order from the next customer and repeat the same process. While this is the simplest setup, it has obvious inefficiencies. Both the cook and the customer spend a lot of time standing around idle while their food is being cooked and the cook can only cook one patty at a time. Taking an order takes some time, so the grill sits empty for a few minutes between each order. However, this is how we write most code. It executes in a single line and everything must sit and wait for any dependent tasks to complete.

Asynchronous

The cook spends lot of time waiting for the food to cook. It must be on the grill for a few minutes, so after putting a patty on the grill, the cook steps away and takes the next order. A dozen patties can fit on the grill at the same time, so the cook can get started on the next order while the first one is cooking. Although the single cook can only actually flip or serve one patty at a time, multiple can be on the grill concurrently. This is now an asynchronous system.

Waiting for HTTP requests, database queries, or any other kind of external system is a opportunity to gain time back instead of blocking while you wait for the task to complete. In many languages, this is done with async/await syntax to declare that you're letting the worker move on to other tasks until the next step is ready. Javascript operates with this model by having a single-threaded event loop, but requiring that API calls, accepting user input, and other "wait" operations to have "callbacks" to asynchronously continue when they are completed rather than locking up the browser while they stand idle.

Parallel

As an alternate approach, instead of improving the process of a single cook, you could simply hire a second cook. The second cook goes through the same "single-threaded" process as our first example, but both cooks work at the same time. This is the equivalent of introducing multi-threading. Work is still done in a single step-by-step process, but there are more workers operating in parallel. This is handled by running code on multiple threads which can execute at the same time as each-other on a multi-core system.

Thread Pooling

When asynchronous programming joins forces with parallel programming, you can have the best of both worlds. Imagine both cooks can look around and take the next order, work the grill, or serve a customer. Whenever they are idle, they look for the next task that needs to be done. Their efficiency is only limited by the size of the grill. In computing, parallel threads can be "pooled" into a collection of workers. When one task is completed, the next available worker can pick up where it left off. Many languages with asynchronous programming leverage some kind of thread pooling. Different languages have different semantics, but in C#, a Task represents a unit of work that can execute on the runtime-managed thread pool and asynchronously awaited.

Race Conditions

Parallelism is easy when the two processes don't need to share resources, but that's rarely the case. In the example of our food truck, there is a single grill shared by both cooks. Normally, this works fine, but once in a while when the truck is busy, the one cook might put down or flip a patty without the other one seeing it. They loose track of which patties have been on the grill and overcook a few. When this happens in a parallel system, shared memory can become easily corrupted if it's not designed to handle parallel requests. This can lead to particularly difficult-to-diagnose bugs as they may happen only under a high volume of concurrency such as a production environment, and even then only very occasionally. It can be almost impossible to reproduce the exact sequence of events in small controlled development environment.

Distributed Computing

Once the food truck has minimized idle-time, the only remaining bottleneck is the grill itself. As the food truck becomes more popular, the owner buys another food truck. Each new food truck adds a new grill and new cooks. In addition to being able to serve more burgers, if one food truck breaks down, service can still be provided. Likewise, adding additional server instances for a service can improve performance and system resiliency. This works best when each instance can operate independently of each-other, as is the case with the food truck. The complexities of race conditions and concurrency increase drastically as you add network latency and reliability issues to a system.

Although each step can add additional complexity, making a system asynchronous, parallel, and distributed can vastly improve it's performance and efficiency. Any time spent idle can be an opportunity to optimize. From lines for food trucks to complex distributed data processing systems, the core principals tend to stay the same.

What's your Technical Debt's Interest Rate?

Eric Damtoft — Fri, 02 Aug 2019 20:11:25 +0000

Technical debt refers to the work that you leave for your future self when you take a technical shortcut in order to finish a project or feature sooner. This might be deploying code before it's fully documented, tacking a one-off feature onto an existing system that it doesn't cleanly fit into, or simply throwing in a hack in order to fix a bug. Technical debt acts much like monetary debt and is a surprisingly good metaphor.

Technical Debt Isn't Always A Bad Thing

Imagine that you're working on building out the new version of your company's flagship application. It needs to go to market in a few weeks so you decide to skimp on technical documentation for it. After all, it's just your team maintaining it and they all understand how it's put together. You'll write the technical documentation for future developers at some point in the future when you get a chance.

This is a reasonable choice. Getting the app to market on-time will probably generate more value for your company than the extra time which might be lost due to not having good technical documentation. If you pay back the technical debt relatively quickly and catch-up on writing the documentation, then taking on the debt in this case was the right choice.

Technical Debt Accrues Interest

Let's say that once your app goes to market, your team gets bogged down with bug fixes and customer requests. Writing up the documentation takes time which you don't have. Your team hires a few new developers. As the new developers start, it takes them weeks to figure out how the app is put together and the senior developers have to spend hours training each one and explaining how the system works. This time cost will continue until the technical debt is paid off and the system is adequately documented.

Interest on Technical Debt Compounds

Much like how monetary debt charges interest on top of interest, technical debt leads to more technical debt. Continuing with the example of the skipped documentation, some understanding of the architecture will inevitably be missed as new developers begin working on the system without documentation. New features will be built that break the mold and even if they work, now every discussion of application architecture will begin with "so originally, it was supposed to work like…". Over time, you find it takes a full history lesson on the application before anyone can be productive with it. As the architecture frays at the ends, bugs become more common and each quick bug fix has the potential to further worsen the problem.

Not All Technical Debt Has The Same Interest Rate

Anything that's off the perfect happy path of software development best practices introduces some degree of technical debt. Not all technical debt collects "interest" at the same rate, though. Significant architectural shortcomings in an app that's maintained by a large team who interact with the code on a daily basis will cause far more pain than taking a shortcut in an application which you only touch a few times a year. Likewise, skipping some documentation on code which is relatively self-documenting won't cause as much pain as failing to document the underlying requirements and use cases for a system. Going back to the first point that technical debt isn't always bad, but consider the "interest rate" on that technical debt.

Use Technical Debt for Safe Investments

Maxing out your credit cards to buy lottery tickets is a pretty surefire way to find yourself in deep water. Taking out a mortgage to buy a house can be a step towards building a strong financial foundation. In our example of the missing documentation, it was originally a fairly safe investment in a new and improved version of a well established app with a solid user base. The rates on missed documentation were initially low because the team mostly could get by without documentation. It wasn't until the rates started to rise as new developers joined the team that they found themselves in trouble. As long as you can make regular "payments" in the form of refactors and code clean-ups, these technical debts will be paid off over time. Adding in unproven features and taking on technical debt to get them deployed is a much riskier prospect. Sometimes it pays off, but you can also find yourself with a worthless feature and technical debt left over from implementing it.

Paying Off High Interest Technical Debt is a Safe Investment

In general, paying off debt is a safe investment. If you have credit card debt at the US average 19% interest, paying it off effectively guarantees a 19% return on investment, far outpacing stock market returns without the risk of market fluctuations. Although it may not feel like you're making money by paying off debt, it's one of the most powerful investments you can make. The same is true of technical debt. Paying back technical debt on the worst legacy applications can seem insurmountable, but it will make the most difference. Even dedicating a few hours a week to cleanup can be one of the most effective uses of development resources over the long run.

Technical debt can be a useful tool, but when it gets ignored or out of control, trouble will follow. Consider decisions which will introduce technical debt and make them deliberately. Think about the rate at which your technical debt will build up. If you do take on technical debt, come up with a plan for "paying back" the debt, and make sure to make regular refactors and cleanups.

Cover Photo credit Colin Watts on Unsplash

Event Sourcing and the History of Accounting

Eric Damtoft — Fri, 26 Jul 2019 18:02:11 +0000

Event sourcing is a software architecture concept that's based around the idea that instead of focusing on persisting the state of your application, you should persist the stream of events which got it into it's current state. The classic example is a bank ledger. Instead of storing the value of each account at the current moment and updating those values, instead you store each transaction (event), and the value in the account is just a projection of those events.

Auditability is the most obvious benefit of event sourcing, but it also gives you a lot of flexibility. You can go back and "query" the event stream to build up new projections of the original data you never would have thought to. Imagine a banking system where the values of an account were just stored as mutable entries in a database. Even ignoring the lack of an audit trail, you could never go back and ask questions like "what day do most of our transactions occur on". By storing the event stream, you can answer these questions even if you never thought to ask them when you designed the system.

Challenges

Of course, these benefits don't come without downsides. An obvious one is storage. Since each event must be added to an append-only data store, the data will grow forever. Snapshots and retention policies can mitigate this but at the cost of loosing the ability to query back to the beginning of time.

The next challenge is that you must "replay" each event to build up the state of the application. Typically this happens on application startup. As the event store grows, this can become a performance concern. Again, snapshots of the state can easily reduce the pain here by limiting the number of events which must be replayed.

Possibly the biggest concern though is that only the events are stored in the log. The validity of the system is based on the idea that you should always be able to replay the events from the beginning of time and reach the same state your application was in before. But this means that the code which projects those events into their current state must be effectively immutable. If a feature is deprecated, it's code cannot be removed because it is still part of the history of the application and is needed in order to return the application to it's current state. Even a simple bug fix that affects the replay of events can cause a butterfly effect as it runs against every transaction since the beginning of time. This challenge is more fundamental to the concept of event sourcing and is more difficult to overcome.

Learning from the 15th Century

Photo by Henrique Ferreira on Unsplash

Going back to the example of the ledger which is usually the prime example of event souring, these problems aren't new. In the 15th century, Venice found itself the financial capital of the western world. The world was moving beyond simple face-to-face business transactions, but lacked a means of effective accounting. International trade relies on credit, and credit relies on accounting.

In 1494 Luca Pacioli, a friend of Leonardo da Vinci, published Summa de Arithmetica which described the concept of double-entry bookkeeping. The idea was fairly simple. Each transaction would be logged along with the new balance. This was essentially the concept of showing your work while doing a math problem in school. The idea revolutionized accounting and Pacioli is known to this day as the father of modern bookkeeping. Venetian merchants could now track inventory, balances over time, and track credits and debits.

Source: Planet Money: A Mathematician, The Last Supper And The Birth Of Accounting, which is based on "Double Entry: How the Merchants of Venice Created Modern Finance" by Jane Gleeson-White)

Double Entry Accounting and Event Sourcing

Effectively, Pacioli invented the idea of event sourcing. By storing the transactions, Venetians could audit their data and use their transactions to answer questions to help them run their business. This is the same idea behind using event sourcing to power business intelligence in the digital world. Double-entry bookkeeping remains a cornerstone of modern accounting.

But Pacioli's double-entry system had one feature that most event sourcing systems today lack — the double part of double entry. In event sourcing, events are persisted, and state is just a projection of those events. But in double-entry bookkeeping, transactions (events) are stored alongside the new balance (state). From any point, you can check the math and pinpoint any mistakes. If a mistake is found, the record of that mistake persists and a later transaction to reverse the mistake can be added. If the only record of a mistake was that the ending balance was wrong at the end of a long stream of transactions, correcting that mistake accurately would be almost impossible.

One of the most common event sourcing frameworks you use every day takes this approach. A git commit stores a message which describes the intended changes, and a SHA1 hash which acts as a pointer to the new state of the repository. These are essentially parallel to the transaction and balance lines in a transaction ledger. This design makes git fast, resilient, and auditable. You can go back through and audit any commit to pinpoint when a bug was added, and quickly view the repository's state at any point. Looking through and seeing that a line of code doesn't match with the commit message is no different than going through a company's accounting books and checking that a transaction was correctly applied. Like any event sourced system, you can also go through and ask new questions of the original events such as trying to see who contributed the most patches during a given period.

Event sourcing has the power to revolutionize software development much as the advent of modern accounting revolutionized the economy of the 15th century. Event sourcing has challenges, but most of the challenges it faces are not new, or even unique to software. Double-entry bookkeeping can be an important concept in event sourcing. By tracking the new state alongside each transaction, bug fixes can become less dangerous, the need to replay events to restore state on startup vanishes, and you retain the ability to audit your system and glean new information from the event stream. Snapshots in event sourcing systems are usually an afterthought for startup performance, but there can be significant benefits to treating them as a first-class citizen alongside the events.