Forem: Krste Šižgorić

State pattern — no more “if-s”

Krste Šižgorić — Tue, 10 Dec 2024 10:40:49 +0000

State pattern — no more “if-s”

In many cases, the task that is put before us as developers is basically a multi-step process, with each step influencing the subsequent one. In its essence, this is a state machine. We won’t go into the details on how the state machine works as that is beyond the scope of this article.

In this article, we’ll explore the most common way of implementing a solution for this problem in .NET. We will also shed light on another approach that could enhance our code readability and maintainability.

Simple state machine

Let’s say we have a system for writing, reviewing and publishing articles. To keep it simple, our article can be in one of 5 possible states:

Draft — the author is still working on it but wants to save their work.
Submitted — the author thinks that article is done and is awaiting the moderator’s approval to publish it.
Accepted — the article is accepted by the moderator and can be published.
Rejected — the article is rejected and cannot be published.
Published — the article is accessible to the general public.

This is a simple version of this process, but it covers all our needs.

Switch statement version

The easiest way to implement this is via a switch statement. Programming languages and their features have evolved to meet developer’s needs, and so has C# switch statement. Originally, this combination of multiple conditions would require you to have if statements inside switch cases. Now that switch statement supports expressions, in combination with case guards, it can handle all of this on its own.

public class Article 
{
    public int Id { get; set; } 
    public ArticleStatus Status { get; set; } 
    public string Title { get; set; } 
    public string Content { get; set; } 

    public void ChangeStatus(ArticleStatus newStatus) 
    { 
        switch (Status) { 
            case ArticleStatus.Draft when newStatus == ArticleStatus.Submitted: 
                Status = ArticleStatus.Submitted; 
                break; 
            case ArticleStatus.Submitted when newStatus is ArticleStatus.Accepted or ArticleStatus.Rejected: 
                Status = newStatus; 
                break; 
            ... 
            default: 
                throw new NotSupportedTransitionException(); 
        }
    } 
}

This syntax sugar makes it look cleaner and more readable, but it did not resolve the original problem. This is still one big switch statement that handles all business logic for transitioning from one state to another. And that business logic is bound to change.

Naturally, each article needs to have a title and content. If an article lacks either of these, we should not allow it to be submitted. We can treat this as a critical business rule, and as such it should be applied every time.

We should add additional conditions to the switch case, to ensure that there is no data corruption and that the state of the article is always valid. Now our condition is becoming less and less readable.

switch (Status) 
{ 
    case ArticleStatus.Draft when newStatus == ArticleStatus.Submitted 
            && Title is not null && Content is not null: 
        Status = ArticleStatus.Submitted; 
        break; 
    ... 
}

Let’s say we have an additional business rule: articles can be published only by the author. This means that we need to check if the user performing the action is the same user who created the original article.

To check this, we need to forward userId to this method. This requires adding an additional parameter to the method. That parameter is only needed for publishing articles, but not in the case we are accepting or rejecting them. So, it should be an optional parameter.

And that is how a big ball of mud starts to form: a simple idea that turns into something complex and unreadable.

We can mitigate this problem by using the object as an input parameter. However, our conditions are already less readable, and now we have optional properties in the model. Things are getting messy.

The other approach would be to split this one method into multiple smaller methods, each handling a specific transition, to better communicate intent to developer. Calling a method like

Publish(Guid userId)

is easier to comprehend than

ChangeState(ArticleState.Accepted, ArticleState.Published, userId)

But in this case, this will just spread and multiply switch statements. Each of those smaller methods will need to replicate the behavior of the original switch statement to some extent. If something changes, we will need to change that behavior in all switch implementations. This is often a ticking time bomb.

OOP approach

Since C# is an OOP language, let’s take a look at a different, more object-oriented, approach to solve this problem. We can treat the state as an object. We will have an abstract class as base state, and concrete implementation for each required state.

We will break down that one big method in Article class into multiple methods, one for each state transition. These same methods will also be placed in the state classes, and Article will simply forward call to state. The only difference will be that the methods in the state classes will have an additional parameter — the article itself — and return model will be of the ArticleState type.

UML of state pattern by Krste Šižgorić

Just like how we switch one DVD for another in our DVD player, the state returned from the method will replace the existing one in the Status property on Article. Due to this approach, we can mark the setter of the State property as private.

public class Article 
{ 
    public ArticleStatus Status { get; private set; } 
    ... 

    internal void Submit() 
    { 
        Status = Status.Submit(this); 
    } 
    ... 
}

Now that we have delegated the determination of a new state to the existing one, we can put all behavior associated with a particular state into one object. The base class will have default behavior, and concrete implementations will override that behavior with state-specific ones if needed.

public abstract class ArticleState 
{ 
    internal virtual ArticleState Submit(Article article) => 
        throw new NotSupportedTransitionException();

    internal virtual ArticleState Accept(Article article) => 
        throw new NotSupportedTransitionException(); 

    internal virtual ArticleState Reject(Article article) => 
        throw new NotSupportedTransitionException(); 

    internal virtual ArticleState Publish(Article article, Guid userId) => 
        throw new NotSupportedTransitionException(); 

    private sealed class DraftState : ArticleState 
    { 
        internal override ArticleState Submit(Article article) 
        { 
            NotSupportedTransitionException.ThrowIfNull(article.Title); 
            NotSupportedTransitionException.ThrowIfNull(article.Content); 
            return new SubmittedState(); 
        } 
    } 

    private sealed class AcceptedState : ArticleState 
    { 
        internal override ArticleState Publish(Article article, Guid userId) 
        {
            if (article.UserId != userId) 
            { 
                throw new NotSupportedTransitionException("Only the author can publish the article."); 
            } 

            return new PublishedState(); 
        } 
    } 
    ... 
}

The obvious problem is that if we create a new instance of the state object every time an article changes state, it can lead to inefficiencies. Creating new objects takes time and consumes memory. It might not be that dramatic, but it will still slow down our program and add additional work to garbage collector.

However, we can address this by implementing stateless state class, without any properties or fields. If an object does not have an internal state, it can be treated as a set of pure functions and shared across the program. In this case, there would only be one instance of the concrete implementation throughout the entire program.

public abstract class ArticleState 
{ 
    public static readonly ArticleState Draft = new DraftState(); 
    public static readonly ArticleState Submitted = new SubmittedState(); 
    public static readonly ArticleState Accepted = new AcceptedState(); 
    public static readonly ArticleState Rejected = new RejectedState(); 
    public static readonly ArticleState Published = new PublishedState(); 

    ... 

    private sealed class DraftState : ArticleState 
    { 
        internal override ArticleState Submit(Article article) 
        {
            NotSupportedTransitionException.ThrowIfNull(article.Title); 
            NotSupportedTransitionException.ThrowIfNull(article.Content); 

            return Submitted; // return static object 
        } 
    } 
}

Use of smart enums

We still need to persist the article’s state in the database. There will for sure be some queries that will filter out articles based on their status. We may also need business logic that will compare one state to another. Our state objects need to be able to handle that, but they currently cannot.

In C# enums are essentially just fancy numbers, but in some other languages they offer much more functionality. For example, in Java, enums can also contain methods. Having that kind of “smart” enums would simplify the process. This is not a new idea in .NET and it can be implemented using Enumerable objects and some trickery.

But we do not need to go that way because somebody already did this for us and published it as a package. We can use Ardalis.SmartEnum NuGet created by Microsoft MVP Steve Smith (aka Ardalis).

Our implementation is already well-prepared for this, and all we need to do is inherit from the SmartEnum class and add values to the constructor. In the end, our ArticleState class will look like this:

public abstract class ArticleState(string name, int value) 
    : SmartEnum<ArticleState>(name, value) 
{ 
    public static readonly ArticleState Draft = new DraftState(); 
    public static readonly ArticleState Submitted = new SubmittedState(); 
    public static readonly ArticleState Accepted = new AcceptedState(); 
    public static readonly ArticleState Rejected = new RejectedState(); 
    public static readonly ArticleState Published = new PublishedState(); 

    internal virtual ArticleState Submit(Article article) => 
        throw new NotSupportedTransitionException(); 

    internal virtual ArticleState Accept(Article article) => 
        throw new NotSupportedTransitionException(); 

    internal virtual ArticleState Reject(Article article) => 
        throw new NotSupportedTransitionException(); 

    internal virtual ArticleState Publish(Article article, Guid UserId) => 
        throw new NotSupportedTransitionException(); 

    private sealed class DraftState() : ArticleState(nameof(Draft), 1) 
    { 
        internal override ArticleState Submit(Article article) 
        { 
            NotSupportedTransitionException.ThrowIfNull(article.Title); 
            NotSupportedTransitionException.ThrowIfNull(article.Content); 

            return Submitted; 
        } 
    } 

    ... 
}

With some additional configuration for our ORM, we can use this class as a regular enum. Since our methods are marked as internal, we cannot access them outside of our domain project. And by marking the setter of the Status property on Article private, we ensure that status can only be changed by the Article itself which guarantees that the article will always have a valid state.

Adding a new state is nothing more than creating a new implementation of ArticleState class and adding a new property. Yes, we will need to change the implementation of existing concrete classes, but only in the places where transition to a new state is possible. This would happen no matter which implementation we choose, but in our case, we minimize the impact of that change.

Conclusion

Each approach to solving problems has its pros and cons. Same goes for state pattern. It brings a lot of boilerplate code initially, but it greatly enhances readability and maintainability. Most importantly, it establishes solid foundations for future changes and expansions of business logic.

Again, like everything in software development, it depends on the situation. However, for me, this pattern has become the go-to way of solving this kind of problem.

Author: Krste Sizgoric, Senior Software Developer at ELEKS

Originally published at https://nofluffjobs.com on December 10, 2024.

Assumptions in Software Development (with EF Core as example)

Krste Šižgorić — Sat, 11 May 2024 09:10:02 +0000

Assumptions are the mother of all f**k ups. This is a beautiful and unbelievably true statement.

Photo by Bao Menglong on Unsplash

We as software developers are creating, or if you prefer developing, software. And that software is just a set of instructions that execute specific task. Of course, it is a little bit more complex than that, but if we take a look at the pure core of what we are doing, that is basically it.

We write one instruction followed by other instruction, followed by yet another instruction, that eventually leads us to the desired result. With the complexity of our software, the number of instructions will grow. To reduce that number and prevent the need to reinvent the wheel every time we need some functionality, we are delegating some of those instructions to framework, tools, or at least language features.

This is where problems could arise. Our software is relying on us to know what that delegated set of instructions is doing. We could argue that the API we are using should be intuitive enough for us to understand what will happen, and that is true. A lot of learning materials are focused on this topic. Whole principle of “method should not have side-effects” is built upon it.

When we buy a car, we do not read instructions on how to drive it. We sit behind the wheel, turn the car on, and drive it. We might spend some time checking where the reverse is in that specific model, but basically, every car is intuitive enough for us to be able to use it straight away.

But sometimes behavior that is hidden behind a method simply cannot be expressed with a method name. Some tools required us to know what we are doing to be able to operate it correctly. Airplane is more complex than the car because it has more features, and needs to handle more complex scenarios. In software development we often assume that we know how to fly the plane because we know how to drive a car. And then we get angry because we crashed that plane.

EF Core — misunderstood and misused tool

One of those tools that are often used, but rarely truly understood, is Entity Framework Core (EF Core). And with good reason. EF Core is huge, and has a lot of conventions and specific functionalities. Knowing all of them is often not needed because most of default behaviors are the best choice in the majority of cases. So developers do not feel the need to know what is going on under the hood.

Many developers assume how it works based on the API that is exposed. Or based on the produced effect of action that they tested or used before. With those conclusions as base, whole software systems are built. Often that is good enough.

But EF Core is much more than some popular ORM. It has many good features that could improve our code. It could reduce so much manual work or make our software more maintainable and decoupled. It is based on the Unit of work pattern and has a change tracker, which is a great foundation for DDD.

Based on which way we choose to go, it can scaffold entities from database tables (database first), or the other way around, create migrations based on changes in our entities (code first). And all of this without (or with minimal) configuration.

And one of most important features, without which use of EF Core does not make sense, is ability to translate Linq to SQL. If we want, we can write type safe querying that can look almost like SQL.

But all of those could be concluded from the use of EF Core. And that is not the purpose of this article. I would like to cover some small things that are often overlooked but could make a difference.

Configuration — Table names

EF Core uses a lot of conventions. Let’s presume we are using code first, and we generate our migrations from our entities. We probably assume that our table names are generated from entity names. But that is not the case. Or at least not entirely.

If we just create our entities and configuration for them, then the name of our tables will be the same as the name of our entities. For example, if we have an entity Movie, the table in the database will be named Movie. But if we add the DbSet property to our DbContext, the name of our table will be the same as that property.

That is the reason why often our migration generates plural names for table names. Not because there is some mechanism to decide plural names of entities, but because our IDE suggested the plural name for property, and once we create that property, migration will copy it.

If we name property differently, for example Films, the table in the database will be named Films, even though our entity is named Movie.

And of course, we can always explicitly set the name of the table in our configuration, or by setting attribute on entity which will take priority, and “override” convention set by EF Core.

Configuration — Primary key generation

All entities in EF Core need to have a primary key. It could have one or multiple properties (composite key), but it needs to be unique. Once the primary key is set, EF Core convention kicks in. Non-composite primary keys of type short, int, long, or GUID will have new values generated automatically on insert.

We probably assume that this means that it will be generated in the database once the new row is inserted, but that is not necessarily true. How the primary key is generated depends on the provider. For SQL Server, the numeric primary key is automatically set up to be an IDENTITY column as expected. But for GUID, the primary key value is generated in memory by the provider, similar to NEWSEQUENTIALID database function.

We need to be aware that those primary keys are generated in sequence, so behavior stays the same. This is very important to mention, because the primary key is clustered index (if we do not explicitly configure it differently), and clustered indexes sort and store the data rows in the table or view based on their key values. Each new row is “appended” to the end of the table, and there is no fragmentation of data.

Configuration — Global query filters

Global query filters are great way to prevent sensitive data from leaking to users that should not have access to that data. We simply create global filter that reference user specific data (like TenantId) in OnModelCreating method and all queries for user are limited by default. This is “weapon of choice” for many that are implementing simple multitenant application.

We might assume that this filter is built on the fly, when an instance of DbContext is created, but that is not true. Building configuration is expensive, so EF Core builds it based on condition, and default one is the type of DbContext. Since the type of DbContext will never change, it is built once, and that same configuration is used for all instances of that DbContext. In the current version, global query filter values are no longer added to the filter as constant (I believe since v5) and because of that this will work.

So why should we care how configuration is built if this works for us? Well, we might have different requirements. Let’s say we need to build role based limits, and each role has a different way of limiting data access for each table.

We could add a switch statement and for each role add different filters. But it would never work that way. Configuration is built once. That means that OnModelCreating method is run only once.

For all roles the same limits would be applied. To be precise, the global query filter would be set to the role limits of the first user that initialized the DbContext. If a regular user was first one that called Web API, it would be regular user limits. If admin was first, then it would be admin limits.

But there is a way to force DbContext to generate different configuration. This can be achieved by implementing a custom IModelCacheKeyFactory. Once implemented, we can replace existing service with our custom one by calling ReplaceService method on optionBuilder in OnModelCreating method.

This will generate new configuration for each role. But we should be careful. In this case it might not be the problem, because there should not be that many roles. But in essence, we just created a potential memory leak.

If, for example, a user can define its own roles, there is an unlimited number of potential configurations that will be generated. That could fill memory and crash our program, and that is the reason why we should know how configuration is built.

Fetching data — cartesian explosion

When retrieving data, we can include related entities to be returned as well. Those navigation properties could reference one element or list of them, depending how relation between entities are defined.

For example, one director could have multiple movies, but let’s say one movie can have only one director (bear with me). This relationship is one to many. We assume that this would be retrieved by fetching data from the Directors table, then gathering primary keys, and making another request to retrieve Movies for directors’ ids. Many other ORMs work this way, so this is a legitimate assumption. But that is not the case for EF Core. It will join tables and retrieve all data in one round trip to the database.

For example, if we want to return a list of 10 directors, we will receive 10 rows from the database. If we decide to include movies for those directors, the number of rows would grow (in this example it would be 15 rows). And by including movie genres, for each movie we would get additional rows (in this case a total of 42 rows).

Photo by Krste Šižgorić

This case is not that critical. All of this can easily be handled in memory. Entity Framework would decide if a new entity needed to be created, or related entities need to be connected to existing one.

But imagine we include 10 related entities, and each of those entities could have navigation properties that we could include. This is called cartesian explosion and it could easily become a huge performance hit.

There is already a simple solution for this. We could mark the query as AsSplitQuery(), and the query will behave as we are expecting it. It will be split into multiple queries, and each entity will be returned only once.

This could be marked as default behavior, but doing one roundtrip to the database is in most cases faster than doing 10, 15 or whatever number of navigation properties we are including. So now that we know how this works, we could decide for each case what is the best option.

Fetching data — subqueries

Writing queries sometimes requires some little bit more complex logic that expression on IQueriable do not support. If we were writing raw SQL, we would probably write some subquery logic. Often developers assume that this is impossible, but EF Core actually does support it.

As long as we did not call any method that does materialization, like FirstOrDefault, ToList, Count or something similar, we are golden. IQueriable could be used as a condition in other IQueriable. This will generate a subquery, and it is a completely legitimate query to execute by EF Core.

var movieCondition = context.Movies
    .Where(movie => movie.Ranking <= 10)
    .Select(movie => movie.Id);

var directors = await context.Directors
    .Where(director => director.Movies
        .Any(movie => movieCondition.Contains(movie.Id)))
    .ToListAsync();

But our subquery does not need to be limited only to where conditions. We could write subquery in select statements too. This gives us full flexibility of SQL with type safety of strongly typed language. But for safety we should always check what the generated query looks like, just to make sure that EF Core did not generate something that will have poor performance.

var directors = await context.Directors
    .Select(director => new 
    {
        DirectorName = director.Name,
        Keywords = context.Keywords
            .Where(keyword => keyword.Movies
                .Any(movie => movie.DirectorId == director.Id))
            .ToList()
    })
    .ToListAsync();

Fetching data — change tracker

EF Core heavily relies on the change tracker. And this change tracker is a special kind of beast. When an entity is retrieved from the database, by default it is tracked. But the change tracker does not track only changes on entities. It tracks relations between entities too. If we retrieve one entity from the database, and in the next request retrieve some other entity that is somehow related to the original entity, navigation property in the original entity will be filled with newly retrieved entities. This is called navigation fixup.

Let’s explain this through an example. If we retrieve the Director entity from the database, and do not include any navigation property, all navigation properties will be empty for that director. Let’s say it is Peter Jackson. Now, if we retrieve the list of the first 10 movies, the change tracker will do its magic.

In the list of 10 movies that are retrieved there are all three The Lord of the Rings movies. If we now take a look at the Director entity, we will see that the Movies navigation property contains 3 items. Change tracker detected relation between those entities, and connected references.

And if we take a look into the movie entity, even though we did not include the Director property when we were retrieving movies, the navigation property is filled with Peter Jackson entity.

If we decide to return this Peter Jackson entity to the user, without mapping them to DTOs, our operation will fail, because JSON deserialization would fail. This is a circular reference, and after X number of LoTR -> Peter Jackson -> LoTR -> Peter Jackson references it will throw an exception.

Now, if we decide to fetch the next 10 movies, and one of those movies is The Hobbit, that movie will be added to navigation property on Peter Jackson’s Director entity, because Peter Jackson directed that movie.

// director.Movies.Count == 0
var director = await context.Directors
    .FirstOrDefaultAsync(x => x.Name == "Peter Jackson");

// director.Movies.Count == 3
var movies = await context.Movies.Take(10).ToListAsync();

// director.Movies.Count == 4
var movies2 = await context.Movies.Skip(10).Take(10).ToListAsync();

If we do not want our code to behave this way because it is unpredictable to us, or we want to squeeze additional performance, we should disable the change tracker for those queries. This can be achieved by calling AsNoTracking() on DbSet, and then writing the rest of our query. Or we can simply use projection.

Projection is achieved by calling Select() method on our query. This automatically turns off the change tracker. By calling Select eager loading (if there are no additional conditions in Include) are irrelevant too, because we need to explicitly select properties that we want to return.

Modifying data — change tracker

Let’s continue with the change tracker, but this time let’s talk about modifying data. As already mentioned, EF Core is using the change tracker. By default, it creates a snapshot of every entity’s property values when it is first tracked by a DbContext instance.

Those values are later compared against the current values to determine which property values have changed. All those changes will be saved to the database on first call to SaveChanges or SaveChangesAsync. If the database provider supports it, all those changes will be applied in transaction. Developers often are not aware of this.

For example, let’s say we have an endpoint on our Web API, and that endpoint calls SaveChanges only once. In this case SaveChanges guarantees that either all changes are saved, or everything fails. There is no need to explicitly start a transaction. And it doesn’t matter if we are saving one, or we are saving 100 entities.

But let’s go back to the fact that ALL changes on entities are saved on call of SaveChanges method. In combination with fixup of navigation properties, this gives us some nice ability to reduce the number of lines of codes.

Adding a new entity works as we would assume. Once we call Add() method, the entity is tracked by the change tracker, and on SaveChanges it will be stored in the database. If that new entity has some navigation properties filled, those entities will be stored too.

So, if we add a new director, and add new movies to Movies property, both director and movies will be saved. Nice thing about this is that we do not need to explicitly set DirectorId for each movie we are adding. By convention, EF Core presumes that DirectorId needs to be set to Id of related Director.

If this was not the case, we would be forced to save director first, retrieve new id, and only then we would be able to set DirectorId and save those new movies to the database.

And it even works the other way around. If we set the DirectorId property to the id value of the existing tracked Director, and add that new movie entity to the change tracker by calling Add method, that movie will be automatically added to the Director’s Movies property.

// director.Movies.Count == 0
var director = await context.Directors.FirstOrDefaultAsync();

// director.Movies.Count == 0
var newMovie = new Movie { DirectorId = director.Id }; 

// director.Movies.Count == 1
context.Movies.Add(newMovie);

Now all we need to do is to call SaveChanges method, and that entity will be saved to the database. But we do not need to be this explicit to be able to save a related entity. If we simply add a new movie to the Director, we will have the same effect.

var director = await context.Directors.FirstOrDefaultAsync();

director.Movies.Add(new Movie());

await context.SaveChangesAsync();

There is one scenario when this will not work. In case we are setting ID property ourselves EF Core would not be able to recognize that this is a new entity. In that case, if we do not call Add() method explicitly this will cause an exception.

Considering that EF Core is saving entities based on their state in the change tracker, we can conclude that Update method on DbContext is misleading. Many developers assume that it is mandatory to call Update() method to update an entity, but it couldn’t be further from the truth.

It is even recommended that we do not do this. Since the entity is already tracked, EF Core will detect if it needs to be saved. But if we explicitly call Update() method we are marking it as changed, even though nothing changed.

var director = await context.Directors.FirstOrDefaultAsync();

// marked entity as modified
context.Directors.Update(director); 

// update row in database
await context.SaveChangesAsync();

If we are having timestamp columns on our entities (like ModifiedAt and ModifiedBy) that are filled automatically, this will update those columns. Since no data is modified, the value of these columns is greatly reduced, because we are losing true info about the last change.

By not using Update method explicitly we can utilize one more nice feature of EF Core. When we are mapping our command or DTO to our entity, the change tracker compares new and old values, and if they are the same, properties won’t be marked as modified. Meaning, once we call SaveChanges, the change tracker will look at the state of entities, see that there is nothing to save, and there will be no roundtrips to the database. Simple.

When an entity has been modified, EF Core will detect which properties are modified, and update only those properties. If we explicitly use Update method, that will not be the case. All properties will be marked as modified, and the SQL update statement will contain all columns in the table (except primary key).

In some cases, for deleting items, we can avoid explicitly calling the Remove method too. If we remove one item from the navigation property, that item will be marked as deleted, and SaveChanges will execute deletion of that row in the database.

var director = await context.Directors.FirstOrDefaultAsync();

var movieToDelete = director.Movies.First();
director.Movies.Remove(movieToDelete);

await context.SaveChangesAsync();

By now we should understand some of the concepts of how the change tracker works. By understanding that EF Core is applying changes to the database based on state in the change tracker, we can do things like deleting an entity without retrieving it from the database. Something like this:

context.Movies.Remove(new Movie() { Id = 1 });

await context.SaveChangesAsync();

If there is no row with provided ID, an exception will be thrown (The database operation was expected to affect 1 row(s), but actually affected 0 row(s)), so we should be careful when and how we use this. Same goes for updates. We can create an entity, set the primary key of that entity, and attach it by calling Update method. And this is finally a valid case for using Update method on DbContext.

Finally, if we utilize all these functionalities, we could change the way we write our code. We could load the main entity (root aggregate), do changes on it (add new, update existing and remove not needed items in navigation properties) and call SaveChanges. All this will be saved in the database in transaction, so there is no need to explicitly start it, or handle rollback logic.

Base for DDD

If we now stop and take a look at what EF Core provides us (especially the change tracker), we can see a couple of ways to decouple our business logic. We could use in-process message dispatching to delegate side effects of our business logic to handlers.

If we dispatch messages right before we call SaveChanges, we could load additional entities in those handlers and make changes on them. We do not save those changes to the database, but simply leave them like that. Change tracker will keep references to those entities, so the garbage collector cannot remove them from memory.

Once all handlers are done, we can call SaveChanges, and all data will be saved. Both our business logic, and side effect logic from handlers. If any of those fails, everything will be rolled back. Simple, extendible, and decoupled. And with minimal lines of code.

Now, if we want to go a step further, we can encapsulate our navigation properties, create domain events inside the entity itself if something is changed, add those events to a private list of domain entities, and dispatch them inside our repository just before we call SaveChanges. This is deferred approach that is a little bit different than the original DDD’s idea of domain events. But it is clean, testable, and maintainable.

Conclusion

This article is overwhelming on purpose, to demonstrate how something that we take for granted could be more complex than we expected. It is not important how we misuse EF Core. We could do the same thing with AutoMapper, RabbitMQ, AWS SQS or whatever tool or service we are using. We could do the same thing with patterns or architecture. Or even language features.

Fact is that we should dedicate some time to understand what we are using, and how it works. We should not assume that we have a car, just because it has wheels. Or we will end up getting all confused because we keep scratching everything and causing damage and jams, but ignoring the fact that we are driving the plane on the highway.

Making custom tools

Krste Šižgorić — Mon, 22 Apr 2024 11:13:50 +0000

When a new feature, part of the system or system as whole needs to be implemented, there is always debate which way to go.

Photo by Salvador Godoy on Unsplash

Do we go for an existing solution and adjust it, or do we make it ourselves. And to what extent do we choose one or the other option.

There is great benefit of making things from the ground up, where they are fully adapted to our needs, but it is a longer way. It is time consuming, and we need to put in much more effort. And in the end our problem might not even be that different to require this level of customization.

Existing solutions give us the benefit of being able to use it right away. It could give us a nice base to build upon, but it forces us to adjust to its principals. Perhaps the way we do things is that advantage that makes us stand out, and those principals could invalidate it. Or simply, we do not need complexity that existing solution force upon us, and simpler solution would better meet our needs.

Ownership

To make things easier, let’s focus on libraries. When we need functionality, and we decide to go with a custom solution, we are controlling the direction in which that code will evolve. If we need a feature, we simply implement it, and we implement it the way we want.

If there is breaking change, it is there because our needs are requiring it. We can control the extent of that braking change and mitigate impact on existing codebase. In case there is a need, we can go for micro-optimizations. We can do what we want because we have ownership over it.

By using the library, we are given up all control over that part of code. In exchange we often get the benefit of not worrying about it. It just works. And if it does not work, there is often someone that will fix it for us. If it needs to be optimized, someone will probably optimize it. Or it was already considered when it was originally implemented.

With the rise of popularity of open-source things are shifting a little bit, so we do have some control over code. We can contribute to the open-source project, but there is still the fact that maintainers are deciding if our contribution and use cases are in line with their vision of how their library should work.

This is always the risk for our project. Good example of this is RestSharp. It is a simple HTTP API client library. At some point in 2022. in version 107 maintainers have decided to do some refactoring, and in that process it was decided to remove all interfaces except IRestClient. Is it a good or bad decision is not the point, but the impact that had on consumers of the library.

Consumers could have either updated their implementation and unit tests to reflect new changes or decide not to update library and lock themselves into using the last compatible version.

Code updates require an unknown amount of time, and in business time is money. And there is always opportunity cost. Effort of adjusting code due to library change could have been directed into implementing new features.

Not updating and using a not maintained version of a library exposes them to potential vulnerabilities and bugs, and deprives them of new functionalities and potential bug fixes.

Neither option seems appealing. After heated discussion there was compromise, and interfaces were returned in version 109. And this is a simple HTTP API client library.

Maintenance

Our code is our responsibility. It is up to us to test it properly, discover bugs and fix them. Documentation of behaviors and functionalities is mandatory if we want to avoid unnecessary bugs due to presumption of how things work. This is a lot of additional effort.

Libraries often provide us with those. Instead of investing time and money in creating something that already exists, we could simply reuse existing tools. Libraries often provide additional configuration with which we can adjust the behavior to our needs. And there is always the possibility to abstract it and add additional behavior.

In a custom tool, if there is a bug or vulnerability, we might not even know that it exists. Existing tools often have pretty good coverage. And if they do not, we can always choose a different tool.

Each bug in a custom tool stretches our resources more and more, and we might get ourselves in a situation where we spend more time on maintaining the tool we are using, instead developing a system that is made for. For specific topics it might even require us to have specialized knowledge to make it work.

This is an additional load on developers. Alongside knowing the domain, there is an additional need of knowing technical details that are not directly related to that domain, but are tool specific. We no longer work on one project, but two. We might not see it that way, but it is.

Using well established tools gives us certainty that if something is not working it is probably due to a mistake in our code. It depends on time invested into creating it, but we often can not say the same for a custom tool. And we find ourselves in situations where we are examining and verifying a custom tool’s code because something does not work while we are using it.

I don’t know why, but I stumble upon a lot custom ORMs. In dotnet ecosystem there is a lot of well establish ORMs like Entity Framework Core, NHibernate, Dapper or Linq2Db. Pretty much all functionality possible are covered and each of them are properly tested. And there is truly no need to create yet another ORM just for one specific project.

Yes, it is fun to make them, but for a specific project there is no direct benefit of making them. What benefits systems are mostly functionalities, and rarely how they are implemented under the hood.

Only when our implementation requires too many compromises to use a certain tool there is justification of ditching it and doing it from scratch. Otherwise, we are doing custom implementation just because it is fun.

And it stops being fun when a critical bug is discovered in our custom tool, and we need to stop everything and try to fix it as fast as possible. Not to mention that it could block other developers from doing their job until we fix it. Whole development is on standby, and damage to the business (even not directly visible) is massive.

Or when our custom tool finally goes to production and ends up having horrible performance. Something needs to be optimized, but each “optimization” breaks some other feature. Never-ending spiral of making it work could eventually degrade code quality to the extent it is unmaintainable.

Idea of creating a custom logging tool is great until you have thousands of log requests, and you start to deal with concurrent writes to the destination. By using a library for logging, we do expose ourselves to vulnerabilities like one with Log4J, but on the other hand, in pursuit of functionalities we could have implemented the same vulnerability without even knowing about it and not being able to locate it or knowing how to fix it.

Importance of abstraction

It is hard to determine what external dependency is. For example, if we have a custom implementation of mediator pattern to communicate between different parts of our system, is it that external dependency or not? If we decide to extract it in a different project, and use it as a library, does the nature of that code suddenly change? Is it now an external dependency? We still have full control of the code we are using.

Can we look at some other libraries the same way? MediatR is a simple implementation of mediator pattern. Should we treat it differently from our custom implementation of mediator pattern? If it is providing us with some fundamental building block for our solution, is it dependency or just that — building block? How does it differ from components from the framework we are using?

There is no one-fits-all solution. In the case of RestSharp, simple abstraction would have saved a lot of trouble for anyone using it. But the same thing could happen to us with a custom tool. Just because we are controlling the code it does not mean that we did not lock ourselves to specific implementation.

We might want to replace our custom tool with an existing library because it is hard to maintain it, but if there is no clear abstraction between implementation and consumption, we might not be able to do so. And then we simply keep using something that nobody wants, and it is a source of a lot of frustration.

We should differentiate between what is the fundamental building block of our system, something that we choose to tighten ourselves to, and what is a tool that does not affect our business logic but is just mean to the end.

First one does not require abstraction. Like numbers and formulas in math, it is the base we build everything else in our system. In the other case, we should have some level of abstraction for something that is in its essence just a tool. Simple thing layer that protects us from something that we cannot control or could jeopardize our project.

Conclusion

Custom tools are a double-edged sword. On the one hand, it is fun making them, and it is a great way of learning technology and design practices. On the other hand, it is additional work and potential threat to the system. Even more so than some external tools, because we do not look at it as dependency and it can creep up to us. And unmaintained dependency is the worst.

So, when choosing which way to go, let’s just use common sense. If the purpose of custom implementation is a fun side project, go for it. If it is work-related or supporting functionality… think carefully about what it brings to the table and is it worth it. We should not settle for OK, just because it is easy. But a good solution is better than a perfect unachievable one.

Software Architecture Should Not Be the Enemy

Krste Šižgorić — Fri, 05 Apr 2024 16:10:22 +0000

Software architecture is more than folder structure.

Photo by Hermes Rivera on Unsplash

Wikipedia says it is a set of structures needed to reason about a software system and the discipline of creating such structures and systems. So, it is a pattern that we follow to make it easier for us to comprehend how the system works.

Often it is not just one single pattern, but a set of architectural patterns that work well together. Each of those patterns brings some benefits and some restrictions. Each pattern is also a solution for a particular problem. Once we are aware of that, we can utilize the full potential that it brings.

If we do not have a problem that some pattern solves, then we definitely should not use it. Purpose of the solution is not to demonstrate that we know all the fancy patterns, but to solve real world problems. On the other hand, the purpose of the pattern is to solve a recurring problem in a safe and maintainable way. Additional benefit of a well established pattern is that it can be recognized by other developers, which makes it easier to understand the code and its purpose.

And best of all, if we understand the pattern that is used, we should be aware that breaking it when it is needed is OK. We should be the masters of our architecture, not the other way around. But in that case, when we break consistency, we need to have a really good reason for it. That reason should be commented on, and well visible in our code. Because the first question that comes to mind is “Why? Why are you doing this?”, and we should have an answer ready, written in the comments of the code.

Separating reads from writes

There are two types of actions that we can do with data. Either we are reading it (query), or we are altering it (command/mutation). We can mix them together, but there is a great benefit of keeping them separated.

By ensuring that retrieving data will never alter it, we can have confidence of fetching data wherever and whenever we want, and as many times as we want. There will never be unpleasant surprises of unconsciously making change on it.

When writes do not depend on reads, we can handle them differently. If needed, we can alter one side without forcing change on the other.

In case of update action on our REST API, if we are following RESTful guidelines, we should return a new state of resource. But in practice we often do not need it. Common practice on frontend after update is to display a success message and redirect to the index page. In this case our response is not needed. All we did is increase traffic between API and client.

The client could use cache, and by returning a new state it could update that value in the cache. But if response is not needed right away, we could just invalidate cache and new state could be fetched when and if needed, by simply doing that additional API request.

In that case we can do some magic on the backend. If our system ever grows big and we start having problems with performance due to a lot of data, we can use a different structure or source of data for the read side.

We can use a materialized view pattern and prefill a different table with data that is needed for querying and avoid the need for joining multiple tables. This makes the query execute much faster. We can split our read and write side in two different services, and scale them independently. We can do whatever we want, as long as we decide not to return a new state of resource on data mutation.

Yes, this can be done even if we do not split read and writes, but in that case every change on the read side will ultimately change the write side. If a model that needs to be returned changes, we need to change it in all places, including commands/mutations.

If we use different tables for data that are served for UI, we will need to generate new value directly from the application, immediately after create/update, so we can return it. This makes our application even slower, and it defeats the whole purpose of optimizing the read side.

Now that we are aware of this we can decide if this additional complexity is necessary. Do we optimize on frontend and return a new state right away? Or do we sacrifice that optimization on frontend to have benefits of separate reads from writes, and count that eventual consistency will be fast enough to meet our needs? It is up to us to decide what fits for each specific case.

Distinguish business logic by its purpose

Shortest way from point A to point B is a straight line. Equivalent of this in programming is procedural code. It is easy and straightforward. But that does not mean that it is the smartest choice. Crossing the crossroad diagonally is not something that we should practice on regular bases, even though it is the shortest way. It might be OK for small towns, where traffic is one car per hour, but in big cities you would be run over on the first try.

Same goes for procedural programming. It seems like it is the best solution until functionalities start to change and grow. In that case procedural methods tend to grow too. First solution is to split them into multiple methods, but code is still procedural.

Since we are already talking about update action on REST API, let’s continue with that. It does not matter if we are writing our logic in controller, service or handler, the end result is often the same. Code is written in a single method. Once additional functionality is needed, we simply extend that method. In case when it grows too big, we might split it into multiple methods that are executed one after another. But it is still pretty much procedural.

We could look at the logic of the method from a different perspective. Validation probably is not part of this method. It is done in middleware and, if the request is not valid, our method won’t even start to execute. Code responsible for fetching data is abstracted in a repository or some similar pattern.

What is left is business logic, which is something that is the main purpose of the system we are building. But not all code written in our method is the same. If we take a look of its purpose, we can see three different kinds of logic:

Main business logic — purpose of method
Critical business rules — rules that define our process and need to be applied every time when action is performed
Side effects — additional logic that needs to be executed in case particular event happened during method execution

We tend to mash them all together and keep them in the same place, but this does not need to be the case. Our method could only contain its main purpose, something that can be expressed with the name of the method. In case we have a blog, this would be publishing the article.

Critical business rules are something that distinguish our system from others, and represent our process. For blogs, a critical business rule would be that attachments of published articles need to be automatically set to public. This kind of logic is something that needs to be executed every time without exception. And it does not depend on anything except data inside entities, and their interaction. So, it is best kept within the entity itself.

Instead of setting the value of IsPublic field on our article from false to true, we could implement the method Publish() that will switch state to true, go through post’s attachments and make them all public. This way it is not possible to forget to do this critical business rule. To ensure that the IsPublic field is changed only through this method we could restrict it so it can be changed only from within the entity itself. We are now forced to call the method Publish() if we want to publish a post. Therefore, we are forced to apply the business rule every time. If the business rule changes, we can change it without affecting the main method.

But there might be some additional logic that needs to be executed on post publishing. For example, an author entity might have a counter of published posts. This should be incremented once a post is published. If we have subscribers, we want them to receive mail that a new post is published, too.

We can implement this in the same method, just after we publish a post. But we can decouple this logic by emitting a domain event that post is published. Then we can create handlers that will process that logic: one for author update, and other for sending email.

We can have as many handlers as needed. Additional functionality on post publishing is now just a new implementation of event handler. Main method is untouched. Other side effects are untouched. We cannot introduce bugs in existing code because we did not change it. The only things that need to be changed are integration tests, to check for new functionality.

Being aware that this approach exists, we can now determine if we need it. Do benefits outweigh complexity that is introduced? Is this the right fit for the system we are building?

Business logic in a controller

In today’s development placing business logic inside controllers is a big no-no. It has basically become a default way of implementing APIs in languages like Java and C#. We are structuring our projects this way without even thinking about it.

And with a good reason. That additional layer of abstraction greatly increases reusability, compatibility, scalability, and testability of our software. By abstracting the way our business logic is consumed, we are removing dependencies on the framework we are using. That makes it easier to migrate our code to a different/new technology if needed.

But if we do not need these kinds of benefits, placing business logic inside controllers is a completely viable option. It even makes our code simpler. Less abstraction means less code. Less code, less chance to have bugs.

We do not need some specific way to handle stop of execution. We can simply return a response on any line of code. Business logic placed in service/handlers need a way to handle this scenario. We could return http response directly from service/handler, but then we simply invalidate the purpose of an additional layer of abstraction.

So, viable options are throwing exceptions or returning the result model (result pattern). Both approaches have benefits and pitfalls.

Throwing exceptions keeps code clean, but is expensive. It greatly affects our performance and resource consumption. However, it should not be a concern for most applications. If we are having validation, exceptions should be very rare, and our code could cover only the happy path. When exceptions do happen, in most cases it would indicate we have a bug because the user managed to do something that the system should not support. Or our user has malicious intentions and purposely tries to do something that should not be possible. In both cases that is an exception that needs to be addressed.

Another option is the result model. It tries to fix performance problems with exceptions by simply not throwing them, and instead returning a model with a flag that indicates if the operation succeeded or not. Basically, it is a wrapper around a real model that has field IsSuccess.

This approach forces us to handle not only a happy path, but also failures. First, we need to check if the operation succeeded, and then we can access our true response by accessing the Data field (or whatever we choose to call it). Our code gains additional complexity.

But if we place our logic in the controller, we can simply return NotFound or Forbidden response. We avoid complexity, and do not have any performance penalties. But we now mix business and presentation/infrastructure logic. And we are heavily dependent on the framework we are using. In some cases that is completely OK.

Let’s say we have a Rest API that has only GET endpoints. Data is fetched/processed in background jobs, and we only need to access it somehow. Our endpoint can call the database directly, check if an entity exists, and return appropriate results. What are we gaining by adding that additional abstraction layer here? Do we really need it? Are we ready to heavily depend on the framework we are using? It is up to us to decide.

Conclusion

Users do not care about architecture. It is here for developers, as a last line of defense against complexity we are going to face. It should be a sum of answers to the questions we are asking ourselves while implementing the project. Adjusted to fit our needs, and to make things easier.

If things are getting complicated because we are following some architecture, then we are basically fighting against the rules that we are forcing upon ourselves. And that does not make sense. We should understand the problem we are facing, and solve it. Nothing more, nothing less. And our architecture should reflect that.

Local development made easy

Krste Šižgorić — Tue, 19 Mar 2024 18:04:18 +0000

Local development should be easy. For developing a new feature, we should focus only on code, and forget all about external dependencies.

Photo by Brevitē on Unsplash

Each external dependency is an additional, often unnecessary, effort in the development process. The more time it takes to set up the development environment, the less effective we are. And if we, on top of that, need to take care of our in-progress changes affecting others, we definitely will not be as productive as we could be by just focusing on our code.

Local environment ideally would simply work after we clone the repository and run it. Of course, this is not always possible, but we should strive towards it. We can come close to this “ideal” working conditions by mocking as much as possible.

Database

First thing that comes to mind is the database. Using a shared database for the whole development team complicates things a lot. Adding new features that do not touch existing tables should not be the problem. But if we are altering existing tables, we are effectively breaking the development environment for all other members of the team.

Testing different solutions for the same problem becomes more about how we can rollback changes if we choose not to go that way, than implementing that feature. Or even worse, we choose the first thing that comes to our mind and just go with that, to avoid the effort of rolling back database changes while keeping the solution stable.

If we choose to use a local database, all these problems go away. Now we will need to implement a seeder to fill our database with meaningful data. Arranging and maintaining this will take some effort. But it gives us an opportunity to analyze data and better understand our system. And we do not need to do it all manually. There are tools that can generate fake data that make sense.

This can be done directly in the database, but I like to keep it in my solution, and use libraries like Bogus, Faker.js or FakerPHP. This way we can determine when and how data will be seeded.

Email

We all worked with a system that needs to send emails. This is often done by having a set of predetermined emails that you can use for testing. In most cases this is an OK approach. Although it is a little bit annoying that we need to switch between different accounts to check if an email was received. Or we need to configure the server to redirect all emails to one account.

Once set up, it is not that much of the effort, but it still requires extra attention. We can avoid that and simply use a dummy SMTP server like smtp4dev. Messages received in smtp4dev can be viewed and inspected locally, instead of login on a remote system.

File upload/download

In modern development we are focused on developing solutions for the cloud. This has changed how we handle files. We used to upload files to the server, and our API would save it to local or network disk. Now it is more common that our solution uses some services for storing files, like AWS S3 bucket.

So, each environment in that case will have its own bucket. That often includes the local development too. But there is a better solution. LocalStack enables us to emulate popular AWS services locally.

This is great for AWS, but not everybody is using AWS. No matter what service we are using, with little “trickery” and well-established abstraction we can solve this problem too. We can have two different implementations: one for environments, and one for local development.

The one for local development can simply store and read files from our local disk. We should not care about the source of our files, only how we can use it to implement new functionality. And a lot of our problems regarding cloud storage, access and/or rights are avoided this way.

We do not need to have access to infrastructure to be able to implement things. Our development process should focus on programming only, but a lot of companies do not see it this way. They do not see a clear separation of responsibilities between developers and devops, and expect developers to use and tweak infrastructure on a daily basis. Don’t get me wrong, we should know how it works and how to use it, but not necessarily explicitly use it.

External APIs

These days there are really small amounts of software that do not integrate with some external API. System we are trying to integrate with could provide us with a test instance. But that is not always the case. And even if we do have a test instance, we may choose to use it only for our environment, because there is a rate limit, or single account available.

In this case we can reproduce behavior or that external API by creating a mock server. Postman has the ability to create a mock server from collection, and that collection can be saved alongside our solution in our repository or on a team account.

We can also use mock libraries like WireMock.Net in which case we reuse those same mocks in integration tests.

No matter which way we decide to go, by eliminating external dependency we are no longer dependent on it and can focus on our development, regardless of limits that API can have. By controlling response, we are able to test all possible scenarios locally, even those hard to reproduce, and implement adequate logic for it.

Message queue

If we are working with event driven architecture, or simply want to delegate some work to out-of-process tasks we are probably using some version of a message queue. If we are using RabbitMQ or Kafka we can easily run docker images locally. If we are using AWS SQS we can use LocalStack as mentioned above. If we are condemned to use Azure Service Bus, we are probably using real subscriptions or topics, and adding the suffix “local” or developer name.

But we should really look at what we need for local development in this case. For receiving messages, we are using some library, or we are using our custom implementation. In both cases this should already be covered with tests either by us or by the creator of the library. So, message publishing and receiving should be outside of our scope.

What is in our scope, and what we should focus on, is the processing of the received message. We do not care how we received it and what the source of that message is. And since we do not care about the source, that source does not necessarily need to be the same as in production.

With good abstraction we can replace real consumers with a simpler implementation like database or mock endpoint. Postman can create this kind of mock server. Or if we use a library for this, and it supports multiple different implementations, we could simply switch to RabbitMQ or some other implementation that can be run locally.

We can have configuration parameters that will determine which implementation will be used. For the local environment we can set it to mock implementation by default.

This way there is no need for extra effort. No need to create subscriptions or topics only for local development. There is no possibility that we forget about those subscriptions or topics, and accidentally fill subscription storage. There is also no possibility that we trigger something that should not be triggered because we forgot to change our consumer/publisher to use a topic for our local development. And all of this can work offline, without needing to have access to a given resource.

Conclusion

I worked on many different projects, but when I have an opportunity to work on a greenfield project, I tend to make development on that project as autonomous as possible. One of my fondest memories in my career is the moment when a junior came to the project I set up. The same day when he arrived, at a daily meeting, the project manager asked him if he needed any help setting up the project. To this the junior replied that everything is already set up and he can start working immediately.

The reason for this was explained in this article. Project was configured to set up an environment if it was run with dev configuration. If the database did not exist, it would create a new one locally and execute all migrations.

After that seeder would check if there was data in the database, and if tables were empty it would generate some random, but credible, data.

For messaging, we used AWS SQS, but if run locally it would be replaced with an in-memory queue. This was a smaller project, so it was implemented as a monolith. So, in this case in-memory solution was good enough.

For emails there was an option for “demo” mode intended for local development. Regular implementation was wrapped with a decorator that would replace the real email address with a template that looked like test-mail+{original-mail}@project-domain.com. Email providers would ignore everything after plus sign and all emails were sent to the same email address. We could distinguish who was the intended receiver because the display name and part of the email address after + was filled with real data (for example John Doe test-mail+john-doe@project-domain.com).

Again, for local development instead of the AWS S3 bucket we used an implementation that stored and read data from a local disk.

These simple “tricks” enabled a junior developer to get onboard and start working on new functionalities within hours. Everything just worked on the first try. There was no unnecessary time wasted on introducing 100 dependencies and setting up accounts and rights. He did not need any of that to be productive. And that is a reason enough why we should care about the local development environment more.

A Journey Through Anti-Patterns and Code Smells

Krste Šižgorić — Fri, 15 Sep 2023 13:48:47 +0000

To understand today’s best practices, we should understand the journey that led to them. And why they could change in the future.

Photo by Dino Reichmuth on Unsplash

I must admit that I was really lucky to be able to work on a variety of projects that gave me somewhat unique insights into the software development process. From doing greenfield projects, to maintaining and expanding two decades old ones. Doing short 3-month projects or working on the same project multiple years. Working backend, frontend, and desktop development. Sometimes I worked on products, and sometimes it was tools or libraries.

And I did not use only one technology. My language of choice is C# in which I did(do) most of my work, but I did use some other languages like JavaScript, PHP, Ruby, or Java. Back then the changes within the languages themselves and frameworks were not so drastic, so it was easier to be good in multiple ones. Today this is much more difficult because languages are changing rapidly, including what is considered best practices. But some fundamental knowledge stays the same. And I would like to share some of my conclusions that could be placed in that category.

The Beginning

Most web-based projects are based on MVC pattern. In this context everything related to data is planned to be part of the Model (M), representation part of the View (V), and business logic part of the Controller (C). Or at least this was the way it was implemented in many projects that I worked on. I followed this pattern and loved it.

If new action was needed on the system, I would create a new endpoint and implement it. If functionality was big, the controller method was also big. Each method would be one unit that represented everything that action was doing. If that action needed to be changed, well… everything was in one place.

It did not take me long to realize that this is not the best way to do things. Most actions in the system are not independent. One action would do the same thing as the other, just a little bit differently. If something needed to be changed in business logic, it needed to be changed in multiple places.

The problem would go even further than that. Sometimes I would implement already existing logic because I did not know it existed in some other method. This led to the fact that one thing, which should be consistent system wide, would be implemented and used in several different ways. This was not good.

I started to extract that duplicated logic into separate methods. Logically, inside the same controller. This way I could reuse existing logic inside different method. But only inside that controller. Eventually, I did find myself in a situation where I needed to use one controller inside another controller. Solution to this was to push the method further down, to the base controller class. Now every controller had access to all that business logic, and I could choose what to use and where.

The base controller started to get big and it was doing a lot of things. Naming methods became difficult because the names would collide with existing ones. This was no longer sustainable. The final straw was a situation where I put the wrong access modifier on a method. Instead of making the method protected I set it to public. The consequence of this was that each controller got an additional endpoint, and the Access Control List used on the system generated rights for each of those endpoints. It made me think if the base controller is the best place for those methods. It was time for a change.

I Need Help(er)

So how to fix this problem? I started to create a type of classes that helped me reuse the same logic in multiple places — helpers. It was a great solution, because I could reuse logic without fear that some helper would grow big. Splitting a helper class was easy. If something needs to be done in multiple places… put it into a helper. So if you are working with dates… use DateHelper. Done.

There was a lot of logic that was delegated to the helpers. Controller methods now looked much leaner and more comprehensible. Changes were easier and the system was more stable and consistent.

Now everything looked like it could be placed in helper classes. If something was used twice, I would move it there. There was not only DateHelper kind of stuff. There was AccountHelper, MatchHelper or BetHelper, depending on what I was working on. And that’s where the problems started again.

I was not working alone. If someone else was working on a new feature, he or she might not know that some logic is already implemented and could be found inside the helper. Sometimes I was not aware that I already implemented some logic, ending up re-implementing that same small feature yet again inside the controller.

And all the same problems as with the controller start resurfacing again. I needed a method that was already implemented, but only a little bit different. I needed one helper inside another. And then some more. In an attempt to reuse logic I ended up using an existing method that directly called resources (database, disc, or external API). This resulted in bad performance, because external resources in some cases were called in for/foreach loop. Those calls could have been done much more efficiently if the same methods had not been reused.

I ended up dividing the helper into two categories: true helpers and business logic helpers. True helpers were kind of helpers that were composed of what today is called pure functions. Method that receives input, does some logic, and returns the result. Those helpers were OK, and I never had a problem with them. I kept them in the same form as they were. But business helpers needed to change. I once again started looking for a different place to put that business logic.

Modeling solution

Business logic placed inside business helpers was always related to M part of MVC — the model. And models are accessible within the whole system. Then why use helpers (that you need to know about) when you can deliver data and business logic together, within the model? I started to push logic from business helpers to models. Although there was some improvement, all the same problems started appearing once again. As well as logical solutions for them.

If you are not careful, you could still change the model directly instead using methods that have some common logic for that part. This was fixed with encapsulation. If you prevent model from being changed from outside, you can force developers to use intended methods.

Another thing was that I still had problems with performance. By pushing dependencies inside the model, I could still end up calling external resources inside the for/foreach loop. But if I call an external resource inside the controller, and pass the result to the model I have much more flexibility in utilizing external resources. Calling external resources multiple times is expensive compared to calling it once and doing the work inside memory.

Without dependencies, the model was now containing only logic for changing itself and interacting with other models. This fixed the problem with needing the same, but slightly different method. Business logic was placed inside the controller where it belongs, and behavior for instance of the model was placed inside the model itself.

I could have still pushed dependency inside the model, as a parameter of a method, when some complex behavior needed to be executed on that instance. But I choose not to do that. There was no strict reason for this, but I just had a feeling that eventually this will lead toward pushing inappropriate logic within the model.

A Step Further

But there were still some things that could be improved. When some action is executed, there is often a set of additional commands that need to be executed as well. Doing this manually after every action, inevitably leads to long methods and hard-to-maintain code. This could be replaced with an event that will trigger the parallel execution of those commands. This way if additional behavior needs to be added it can simply be done by adding another event listener/handler.

This could be difficult to imagine without an example, so let’s try with this one. One of the features of some system I was working on was that users can be part of the group. That was not the main functionality, but it was important. Each user could have been only in one group, and each group could have one, and only one, leader. Based on this, user could have access to some resources and permissions to do some actions.

When a user was added to a group, this logic should have been updated to reflect those rules. When the leader was changed inside the group, again, this logic should have been updated to reflect these rules. The same was for the case that the user is deleted/deactivated.

But that was not the only thing that needed to be done. Each group had some expenses and was treated as a cost center. Each user had an additional cost. For each user there should be calculated direct (user cost) and indirect cost (group cost divided by number of users). So, if a user was added, demoted, promoted, or removed from the group it should have been recorded.

Those two features were not directly related, but the cost center was dependent on group administration in one simple way. It needed information that something happened, an event, so it can do its own thing. There is no need for the administration of the group to know anything about costs of the group, or even of their existence.

With time there was a need to add history of changes for group administration. Again, the administration of the group should not care about this. New event handler was added, and an audit was created.

And last, but not least, new feature was a list of notifications on user’s dashboard. Again, in case of group changes all users inside the group should get their notification about this event. Once again, there was no need that process of administrating group needed to know about this. New event handler was processing new business logic that was independent. It did not touch (or potentially break) group administration, cost centers logic, nor audit.

All of this made this amount of business logic more comprehensive and easier to maintain and extend. To ensure that all of this is done atomically (everything succeeded or nothing succeeded) I wrapped the whole process in a transaction. This ensures that data saved in the database is valid and consistent.

Conclusion

A good part of what I know about programming I learned the hard way. That helped me understand why some things are done the way they are done. Over the years I started to learn from other people’s mistakes, but this transition from Anemic Model toward Rich Domain Model is not one of them.

There are always exceptions to the rule, but these are my ideas for developing software. I picked them up over time, and I use them as a guidance:

Write code as you are writing an email, descriptive and straightforward. Do not mix multiple topics inside one “action”. Separate reads from writes to simplify things. Eventually you will read your own code and will need to understand it quickly. Do yourself a favor and write it that way.
Never call the same type of classes from that type (controller inside controller, service inside service, handler inside handler, events inside events). Business logic that needs to be shared between services should be pushed into entities, placed in manager/processor/helper classes (to mark the difference from services), or delegated to in-process/out-of-the-process handlers. This prevents circular dependency and keeps logic lean.
Keep business logic as lean as possible. If you need to dig deep to find out what one “action” on the system is doing then something went horribly wrong. You will end up with bad performance and unwanted behavior/side effects.
Be consistent. If you do things a certain way, do them that way everywhere. Sooner or later, someone will need to make changes to the system, and it will be much easier if they (or you) don’t have to learn how things are done for each feature.
Minimize who can do what. Encapsulate as much as possible and have as little as possible access points. This minimized breaking points. And if something is not working as expected it is easier to test and locate it this way.
Split your system into subsystems if possible. Smaller parts are much easier to maintain than one big whole.
If there is an exception to the rule, always comment on it well and make it as visible and as obvious as possible. This will prevent a lot of confusion later.
Think for yourself, question authority. It does not matter who said something. If for a particular situation it doesn’t pass a sanity check or test of time, it should not be used.

P.S. To prevent this article from becoming a small book I threw out some (a lot of) parts. Some of them are moving from MVC toward Layers architecture, getting circular dependencies with services, and ending up with some version of Layered architecture that is similar to Onion architecture. Point of this article is to present how I was moving the placement of logic in an attempt to increase maintainability of software I was working on. And all the most important points that led me to today’s way of thinking about software development are listed within this article.

Neglecting Legacy Code? It’s a Potential Gold Mine of Learning

Krste Šižgorić — Thu, 18 Aug 2022 14:49:01 +0000

There are so many places and ways to learn to program, we only need to know where to look. Yet, we often neglect legacy code.

Photo by Matthew Feeney on Unsplash

You’re about to enter another dimension. A dimension not only of sight and sound, but of mind. A journey into a wondrous land of imagination. Next stop: The Legacy Code.

For those who didn’t recognize it, or didn’t watch the show, this is (altered) intro to The Twilight Zone episode. It is a cult SF/horror TV show. Each episode tells a stand-alone story of characters that find themselves dealing with often disturbing or unusual events leading to the surprising ending.

Our journey through the lines of legacy code could feel quite similar sometimes. But just like in episodes of The Twilight Zone, in every legacy code mystery there is a lesson to be learned.

Borrowed knowledge

Programming can be learned in many ways. But no matter how we did it, we didn’t do it on our own. In our journey, there were a lot of books, online courses, and documentation. While going through these materials we stumbled upon a bunch of advice on how things could be done and what should be avoided. All those pieces of advice are an accumulation of experience from the people who wrote those materials, and many others that shared their experiences before them.

This is what I like to call borrowed knowledge. It is not our knowledge, we are “borrowing” it. We are doing things in a certain way, but that doesn’t guarantee that we understand what we are doing.

_- We are doing microservice architecture.

Why? What problem are we trying to solve? What are the benefits? What are disadvantages? Are we really doing microservice architecture or are we doing distributed monolith? Do we need microservices?_

This is probably the easiest example. Microservices are lately used like mantra. It is presented as the ultimate solution to all of our problems. Don’t get me wrong, microservice architecture is a beautiful pattern, but it has its own place.

Until we know what that place is, microservices are not our knowledge. It is something that someone else told us to do. And recommending it would be like recommending a movie that we did not watch. We put our trust in someone else’s taste in movies. We are “borrowing” it.

This is not necessarily a bad thing. Take into consideration that we work to live, not live to work. So knowing everything is not the ultimate goal that we should strive for. That is why we have opinionated technologies and coding standards. To show us the way and to eliminate the need to learn things the hard way. We do not need to get hit by a car to learn to look both ways when crossing the street.

Having a structure of a project, a tool or a pattern that we can simply “plug” into our solution without much fuss is great. But what definitely should be our goal is to understand what we are doing, why, and when we should not do it that way.

Legacy code

How does legacy code fit in this story? What is a better way to see why we should or should not do something than seeing it in action? Sometimes code is just a victim of time or circumstances, but in most cases legacy code is a result of bad decisions.

It is harder to notice those bad decisions in the beginning. When we are starting new projects, we are mostly focused on what needs to be done. As time passes and software is used, bad decisions begin to surface. And there is no better teacher than a bad decision.

In software development we should avoid tightly coupling as much as possible. But do we really understand why? And how can we achieve this? Sometimes it is OK to tightly couple things if we know what we are doing.

For example, if we use a certain ORM it is fine to directly use it in business logic… if we know for certain that we are not going to change it. If we are not that certain then we should put a level of abstraction between our business logic and data access layer by using repositories, data access objects or something similar.

If we have ever been in a position where we use an ORM that is out of support, a custom solution that no longer meets our needs, or simply majority of employees do not know how to use it (and there is no good documentation), then we definitely understand why we need that level of abstraction. Changing ORM would be a nightmare, and staying on existing ORM limits us in what we can do. This is a lose-lose situation.

Another example, so characteristic for legacy code, would be code duplication. We all (should) agree that code duplication is a bad thing. Yet there are so many developers that still duplicate code. Sometimes even on a massive scale. Any of us that have ever fixed some bug, and there is still faulty behavior in production because there are two, three, or more copies of the same code containing the bug, know why we shouldn’t duplicate code.

In a new codebase, we might get away with it because there is still not that much code, and we still remember every place that we need to change, but that doesn’t make it less wrong.

Lessons learned

All of this is well said, but it is too abstract to show what I am really talking about. So let me describe one of my experiences with legacy code.

Early in my career I came to work on a certain project. It was version 2 that was already 4 years into development at the time. It was a MVC project covering a variety of functionalities, and there was a substantial amount of code. It had over 100 tables, and more than half of those entities needed to be able to have attachments. And that feature (attachments) is a feature I would like to cover in this section.

To speed up development, the original team used generic controllers and code generation. Among other things, they generated views for CRUD operations for most entities. These views would be generated and then altered according to requirements. Most entities needed to have attachments. To address this they have put code for handling display, upload and modifying attachments inside the view template.

Lesson: Duplicated code is hard to maintain and should be avoided

Since over 70 entities needed to have attachments, logic related to that feature was generated in over 70 places. There was a list of formats that were supported by the system for upload purposes. When a new format was required I needed to go through all those 70+ places to update business logic.

Once I did that, the new format was supported. But after some time the user complained that it does not work. It took me some time to figure out I forgot to update the view template. Next view that was generated after the change did not contain the newly supported format. It was a dumb mistake, but in my defense, I was pretty green at the time.

Lesson: If you generate code do not generate duplicated code. Or, if you do, make sure that you can later regenerate it to reflect new logic.

To be honest, it wasn’t all my fault. The view template should not have contained the view part for attachments. That code was not a candidate for code generation. It should have been a separate component that is only referenced in the view. Although it was generated code, duplicated code is still duplicated code.

If views could have been overridden by generating a new version of it, there would not be any problem. Newly generated versions would contain changes and everything would work as expected. But views were altered with manually written logic, so that was not the option.

Lesson: Generics are great. Use it when you can, but do not force it.

I decided to create components for attachments anyway. Replacing all that duplicated code might not be possible right away, but I could use it in the new code. This way I’m not creating more work for myself once I start switching things later.

Up until then, I worked on projects that used generics, but I personally have never written a line of generic code. Even though I knew what generics are, I didn’t understood their full potential.

The original idea behind the component was to have a unified view part. But as I started to use it, I recognized that every time I reference it in view I’m duplicating code in the controller too. It might be a good idea to push that fetching logic inside the component then.

This was an eye-opening realization for me. With help of reflection and generics, I managed to create an encapsulated component that handled fetching, displaying, and authorizing action on attachments.

But some pages needed to have a specific list of attachments that are not related just to a specific entity. I tried to push this into the component, and it became messy fast. So I decided to split the view and logic part of the component, and use only the view part. The business part was manually written every time because it did not fit into generic logic.

Lesson: Know the tools you are using and what your code is doing under the hood

At some point, users started to complain about the performance of a certain page. Again, to speed up initial development, the original development team decided to fetch all data on page load. Paging, filtering, and ordering was done on the client side.

With time the number of rows grew by over 5000. By itself, it was not the problem. What really killed performance was attachments.

It looked like someone concluded (just like I did when I was extracting the component) that there is a lot of code duplication when fetching attachments.

To prevent this duplication, eager loading was turned on. For each entity, ORM would load all of its attachments by default. Relations between entity and attachments were one-to-many, so ORM could not fetch those attachments within the same query. This meant that for each entry (row) ORM would make an additional request to the database.

Since the original request needed to fetch two entities to display the data in the view, the result for 5000 rows was over 10 000 requests to the database on page load. And attachments were not even used on this page.

I couldn’t simply turn off eager loading because it was used in too many places, and I couldn’t be 100% sure that I covered them all. At the end, “the solution” was to go through all pages that display lists, and manually turn off eager loading for attachments.

Conclusion

Maintaining legacy code is often depressing. Keeping things running while dealing with a bunch of other people’s (or yours) bad decisions get frustrating fast. Every mail, call, or message we receive means that something is not working and we need to fix it. It is hard not to get affected by that.

But working with legacy code does not need to be a nightmare. We can look at it as an opportunity to sharpen our skills. Every code is the final product of a chain of decisions made to solve some problem. By knowing when, why and how things went the wrong way, we can learn a great deal about software development. And just maybe the code that we left behind will be better than the code that we inherited.

The Three-body Problem in Software Development

Krste Šižgorić — Mon, 16 May 2022 18:21:05 +0000

Tight coupling in code is something that builds up with time because we treat different parts as one big whole. We should avoid doing that.

Photo by Heramb kamble on Unsplash

It’s been a rough couple of months. A lot of work had to be done and in the end I just needed a break. Reading relaxes me so I picked up The Three-Body Problem by Liu Cixin.

I did not know anything about the book or the problem before I started reading it and I can say that I was positively surprised.

It is a science fiction novel, the first book in the Remembrance of Earth’s Past trilogy. The book takes a three-body problem, one of the most difficult problems in classical mechanics, and builds a story around it.

So let me, without spoiling the original story for you, do the same thing in my own way.

The three-body problem

To be able to explain the three-body problem and how it could relate to software development, let me start by explaining the one-body problem. It is better known as the central-force problem. Central-force problem tries to determine the motion of a particle when there is a central unmovable source of a force acting on it.

To put it bluntly, it could be used to well enough describe movement of light planet orbiting a heavy star, where the star can be treated as stationary. This movement can be represented with trigonometric functions.

To complicate things, now let’s imagine we have two massive objects affecting each other’s movements. This is known as the two-body problem. It could be used to describe movements of Earth and Moon orbiting around each other. Or even better, Pluto and Charon, just like it is represented in the animation below.

An oblique view of the Pluto–Charon system showing that Pluto orbits a point outside itself (Wikipedia)

For many forces, including gravitational ones, the general version of the two-body problem can be reduced to a pair of one-body problems, allowing it to be solved completely. Again, we have a formula.

But if we add another massive object, and make the whole thing the three-body problem, things become unpredictable and chaotic. The three-body problem, for most initial conditions, does not have a general closed-form solution like one-body problem or two-body problem.

Building up n-body problem in software development

How does the three-body problem relate to software development? Well, it doesn’t. But if we look at both processes we could find similarities in behavior. Tight coupled features affect one another, and very loose coupled features can peacefully coexist within the same system without forcing change on one another. Let’s compare software development to the n-body problem.

In the beginning everything is easy and understandable. We have one feature, one central thing, and everything revolves around it. There are a small number of functionalities which do not collide with each other.

For example, we are building an inventory app. All we need is the possibility to insert new items, decrease or increase quantity, and have insight into the state of stocks. So we implement that.

With time we eventually want to add new things. It would be really nice to have a webshop to be able to sell our products online. So we start adding things in our inventory.

First, a web page. We fetch the state of inventory with available quantities. Now the web page needs to describe the state of available inventory. But this doesn’t mean that those items are no longer in inventory, only that they are in different state. So we need to introduce states in inventory. We need to have quantity for “on stock” state, and for “ready for shipping”.

But now fetching of available quantities for the web shop needs to be changed to reflect this change. It doesn’t matter how many items are really in the warehouse if we can’t sell them. We only want to display “on stock” quantity on our web shop. We need to change the web shop again.

The gravity of one part of the system pulls other part of that system into changing. There is a “power struggle” between features until a stable relationship is created between those two. Once we finally polish functionalities everything falls back into predictable motion. Everything works just as we planned and we are happy. We can still relatively easily predict that this will happen, and how change in one part reflects on the other.

Two bodies with a “slight” difference in mass orbiting a common barycenter (Wikipedia)

But things can be better. We could provide our customers with a delivery service. So we go through the existing system and make changes on every step of the way. Delivery service changes inventory, inventory changes web page, web page changes delivery service, inventory changes delivery service…

Delivery service has helped our business grow. One warehouse is no longer enough to meet our needs. We want to expand our business to more locations and enable the system to support this way of working. But how will this affect the existing system? Inventory needs to be changed to enable multiple locations. Since the web page is decreasing the quantity of items in inventory, it needs to be changed to support multiple locations. But how to do this? This will force changes on inventory and delivery service… Chaos.

Approximate trajectories of three identical bodies located at the vertices of a scalene triangle and having zero initial velocities (Wikipedia)

Reduce to one body problem

So how can we avoid this problem? How can we avoid that one feature dramatically influences the other?

Solar system is full of objects that are big enough to affect movement of other objects. Yet, if we try to predict Earth’s orbit around the Sun, we can safely ignore all of them and focus only on Sun and Earth. This will give us a good initial approximation of a true motion. Same goes for Jupiter, or any other planet.

If we decouple features from one another we can treat them like planets in the Solar system. Gravity of one planet should not affect the orbit of another planet. It actually does, but we can ignore it in most cases. Same thing will happen with features. They will still influence one another, but those changes will be less intense and in some cases even non-existing.

Imagine if we are trying to calculate dates for Easter for the next 10 years. Easter is Christian holiday celebrated on the first Sunday after the first full moon after the spring equinox. Do we really care about Jupiter’s 79 moons when we are trying to calculate these dates? Do we want to care about them? Certainly not. And we do not have to.

We break our solution into small pieces and let them circle around “the Sun”. “The Sun” in our case can be a message broker, service bus, or even just well established contracts (interfaces). We decide how decoupled our Solar system needs to be. It doesn’t matter if those small pieces that orbit around the Sun are modules, domains or micro-services, the important thing is that they are independent as much as possible. That makes them easier to comprehend. This way calculation of Easter dates could be done without even knowing that Jupiter has 79 moons.

Earth’s Moon affects its orbit greatly, that is the fact. If we look at the relations of the Sun and the Earth we do not talk about Earth and Moon’s orbit around the Sun. We talk only about Earth’s orbit. No matter how complex a feature is (Jupiter with 79 moons) in the whole (Solar) system we should treat it like one whole.

This way we do not have (approximately) 1 200 000 objects in the Solar system. Nor 700 planets, minor planets or satellites. We have 8 planets. Because, generally speaking, when we talk about the Solar system we only care about these 8 planets. Now we can determine their movements much easier. Even though the result is not perfect it is good enough for us to be able to work with it without problems.

Simplifying things

When we think of an inventory, what do we think about? Probably about some big warehouse and a lot of items stored in it. What is the job of an inventory? To store things and keep it safe until someone comes to take it away. Our software solution should then only cover those functionalities.

Web shop should only be about displaying items and creating purchases. But purchases on web shop need to change the state of inventory. So, how to do this? Now, this could be done in many different ways, but bare with me.

Purchase is complex enough functionality. It gets orders, checks inventory to see if items are available, does the billing part and creates shipping. This might seem like just another feature, but due to its size it can easily be separated into an individual part.

We create strict contract for the inventory where we can get a list of all items, all active items, check if items are available, and decrease their quantity. Web shop only needs to know about active items. If we decide at some point to support soft delete, or multiple states like we did in the example above, the web shop does not need to know about those changes. The only thing that will be changed is the data it receives, and that is going to happen without web shop knowing about it.

For the purchase feature we need to do the same thing. Contract needs to have an action to make a purchase. Web shop calls that action and its job is done, purchase feature takes over. It checks if an item is available and, if everything is OK, it completes the purchase and decreases quantity in the inventory.

If we look at our system we have a clear separation of features that can stand on its own (some more than the other). We started with an inventory, so it definitely can be a system of its own. Then we added a web shop and purchase feature that could be implemented as independent.

We have a delivery service, but so far the whole process did not need to know of its existence. So we should treat it as a system of its own too. We should not forcefully push it into existing feature just to make it fit.

And there we go. Now we do not have a system that has a list of features with complex dependencies. We have a set of subsystems, each with specific complexity, that together make one comprehensible and maintainable solution.

Conclusion

Building, maintaining and extending software is hard. It might seem easy in the beginning. “Just add this feature”. But the more things we add, the more complicated the equation becomes. If we try to push in too many things eventually we will find ourselves with a problem that can’t be solved. And there is a thin line between those two.

We could, on the other hand, try to simplify it and break it into many small pieces. There is a larger number of people who can work with those smaller pieces. Solving one body problem is done by teenagers in school. On the other hand, the three-body problem seems to be an unsolvable problem. Yet, the difference between those two looks so small at first glance.

Unit tests — Playing chess against yourself

Krste Šižgorić — Mon, 17 Jan 2022 19:42:58 +0000

Unit tests — Playing chess against yourself

Recently I read an article about how many incredible developers don’t use debuggers at all. It is a really good read. I did write a lot of code in PHP so I was familiar with the concept. When I was writing PHP it did not have a debugger in the full sense of the word, so most of the debugging was print_r/die combination. I can confirm that not having capability to just put breakpoints on some line and simply go through the code do make the difference.

It forces you to think about what code really does and why something broke, and not only how to find it and fix it. That eventually leads to better understanding of the product you are working on, and how to avoid the same mistakes later.

Not having a debugger did make me a better programmer, but writing unit tests has more effect on my progress as a software developer. It helped me to better analyze the problem I was working on. And as a result I would locate problems faster, if not preventing them.

Playing both sides

Chess is a strategy game where both sides have perfect information. This means, unlike when playing poker, each player is perfectly informed of all the events that have previously occurred. Your goal is to capture your opponent’s king. So you plan ahead, and try to set a trap to do so. It might sound simple, but traps need to be sophisticated because both players see everything that is going on the board.

Did you ever tried to play chess against yourself? You make moves for both sides, White and Black. This is fascinating experience. Regularly, when you are playing against other person, you analyze the position as best as you can in your head, and try to determine what is best course of action for both sides. This helps you to predict opponent’s next moves and how to react to them. In the way you are already playing some portion of the game against yourself.

But when you are truly the only one moving pieces on the board you already know the other person’s (yours) intentions. This gives you an opportunity to study your gameplay, find weaknesses in it, and determine what could have been done better. Not only that it helps you avoid repeating mistakes next time, but it trains you to do an analytic thought process that is necessary to play chess.

Writing unit tests

Unit tests are a type of test where you are testing a small portion of code called unit. Unit can be a method or an object. I never really saw the purpose of it before I started writing them. I thought that you were trying to find a bug in your code. But that is not true. Purpose of unit tests is to validate behavior.

Unit tests belong in a group of so-called white box tests. This means that the person who writes them is aware of how things are implemented. Just like in chess, you know “opponent’s” intentions. And now you can study their gameplay and try to find weaknesses in it.

You want to know exactly how code will behave under different circumstances. So you write every possible scenario and see how things will play out. Unlike chess, the goal is not to win, but to be certain that you know what will happen. Code may throw an exception on invalid input, and that is ok, but you need to know that that is going to happen.

This was an eye-opening for me. First time when I was writing unit tests I thought that I knew how my code behaved. I was wrong. There were a couple of edge cases that I did not take into consideration. My code would throw unwanted exceptions. But that was a minor problem. Big problem was the use case where code would return the wrong value. Everything after that would work fine. Invalid data would be saved in the database and it would cause data corruption. Since there was no immediate effect this case would be harder to detect.

Unit tests remove assumptions about code. They are documenting behavior. Let’s say you are using some internal library. You are not sure how something should behave. There is no documentation or it is obsolete. Unit tests got you covered. Find (or write) unit tests for your case and you are golden. Run it and you are 100% sure what to expect. Even better, integrate tests in your deployment process. This will ensure that deployment fails if code does not behave as expected in unit tests, preventing unwanted behavior in production.

Good naming convention

I have tendency to name tests by combining 3 things:

name of unit that need to be tested
conditions under which you are testing that unit
expected result

For example:

CalculateDiscount_ByPercentageForValue1000_Return100

From name there is pretty clear what is going on. You are testing method CalculateDiscount. That method receive input 1000 and type of discount (ByPercentage). Result need to be 100. If this test fails you know just by looking the name of unit test that something is wrong with CalculateDiscount method if type of discount is ByPercentage. This is a great start for finding bugs.

These are my personal preferences. In .Net ecosystem there is convention to name tests like:

CalculationDiscount_should_return_100_for_ByPercentage_when_price_is_1000

Use whatever you like, just keep it consistent. In both cases the name is pretty descriptive and that is all that matters.

What not to cover with unit tests

Writing unit tests is time consuming. You need to write a test for every possible use case. So be sure that you know what you are testing. Never test dependencies, only business logic of the unit you intend to test. Just presume that dependencies work perfectly. These dependencies should have their own unit tests to validate behavior.

Do not test external libraries. For example, if you are using AutoMapper then do not write tests for it. Do not write tests for cases when AutoMapper returns null as result of mapping because that will never happen. If there is a mapping profile between two objects then AutoMapper will always return a valid object. If not, it will throw an exception. But that case should be covered with configuration validation. So presume that AutoMapper is working perfectly, and just test your own business logic.

This is also true for internal dependencies. Let’s say you have some utility class. You ensure behavior of that class with unit tests, and when you use it somewhere in your code you only need to cover expected behavior.

On one project I had a service that was used as an additional layer of authorization. Based on role, current status and new status it would return true or false as indicator if user is able to change issue from current to new status. Since there were 4 roles and 16 statuses, there were around 800 unit tests for this method. Since it returns only 2 possible outcomes any unit that used this service had only two use cases for this dependency, one when service would return true, and one when it would return false.

If dependency is internal, and there are no IO actions (Database, reading files from disc…), you do not even need to mock it. If you use some utility class that is POCO you can simply use it as dependency. You are breaking “rules” of unit tests because you are not fully controlling the state of the test, but let’s be honest, you are writing tests to validate behavior, not to demonstrate that you are implementing a certain paradigm.

Failed unit tests are friends, not the enemy

True power of unit tests comes to light once you start changing things. Let’s say the client wants you to extend some functionality. You need to add new things, but old use cases need to remain the same. You implement new business logic, but now 5 unit tests fail. You broke something, and because of unit tests you know it right away.

Conclusion

It is fun to look at code from different perspectives. On one hand you are just trying to make a feature, and on the other you are trying to break it. Experiencing your code through unit tests will make you a better developer. Just like in chess, you will see weaknesses in your code and strengthen up your gameplay for the next match. With time you will cover edge cases by default because you will know where an error may occur. And one day, when new requirements come, you will recognize it just like chess masters recognize chess positions, as a face of someone you already know, and unit tests will prepare you to know what to expect and what to look out for.

Writing code like an email

Krste Šižgorić — Fri, 05 Nov 2021 15:47:56 +0000

Communication is a process by which information is exchanged between individuals through a common system of symbols, signs, or behavior. Writing code can easily fit in this definition. Some solutions might be written by one person, but if the product is used and extended it will eventually outgrow that one developer. Best example is Facebook. It was written by Mark Zuckerberg, but now there are many more developers working on it.

That being said, when we write code we are not writing it only for machines to understand, but for other developers too. Code should be as readable as possible because it is a channel of communication between people. And what makes code readable? Well, if the other person understands you then you have succeeded. The shorter it takes that person to understand what you were trying to say, the more readable your code is. So we should strive to keep that time as short as possible.

Break it into small pieces

Did you ever receive an email that is one big chunk of text? If not, you are a lucky one. Big blocks of text are repulsive and intimidating. To make things easy for readers we should break text into short paragraphs that represent a logical whole.

This way readers can quickly go through it without reading everything. If some part of text does not concern them they can easily skip it and see what is in the next part. Even better, if we break it into different chapters, and put descriptive titles, the intention of an email will be as transparent as it can be.

So break your code into smaller classes and methods, break statements into smaller ones, and separate statements with different intentions with empty line. Making an enormously long one-liner is not clever. You may be proud to have done it all in one line, but you just undermined the readability and maintainability of your code. You have just put your preferences in front of your project’s well-being.

Reading a long sentence is a nightmare. Imagine reading “Dancing Lessons for the Advanced in Age” by Bohumil Hrabal and try to comprehend it in one go. It is a 128 page long novel written in one sentence. How about doing that under pressure of deadlines?

Same thing goes for the code. Long statements, long methods, big classes, multiple classes in one file… all that makes code harder to read. We are writing a set of instructions, so it should be as simple as possible. Short methods are easier to understand for other programmers and we should use them instead of long “smart” ones.

Be as straightforward as possible

“The Story” of one class or method should be as straightforward as possible. If you want to say something you should say only that. And the name of the class/method should be descriptive enough to indicate the purpose of it. This may sound silly when put on paper, but in practice it is much harder to stick to it.

For example, writing an email to the client. Let’s say our client is outsourcing some project to us, and we need to arrange code and documentations transfer, and arrange a call for developers to do knowledge transfer. In this case how we format email can be a crucial step determining speed of project transfer. If we write a long text explaining every part in smallest details, this creates a lot of noise. There is a high probability that our client will not read it and we will not get all the necessary information.

Everything will end up in the video call where we will need to repeat all the information from the mail. If we structure it into an email as a list of things there is a better chance that we will get all we need and that call will be unnecessary and everything will be done quicker.

Dear client,

Development of project [name of project] should start next month. To make this possible we need to finish all necessary preparations. We need access to:

- source control
- documentation
- development and test environment

Best regards

And if someone needs additional explanation, they can always ask for it.

What preparations?

Why do you need access to source control? Who needs it?

What kind of documentation do you need?

What kind of access to our environment do you need?

In code this is the equivalent of digging deeper into implementation. If we are analyzing business logic we do not need to know implementation details for fetching data. Is it SQL, NoSQL or something else? It is not important for understanding what the method is doing.

If we put a lot of information in one method it will create a lot of noise. Opening SQL connection, writing SQL query, mapping response to entity class and similar things will distract us from the real purpose of the method. Delegating this to the repository hides all of that from us and helps us focus on important parts.

Do one thing at a time

Let’s do the follow up on the previous section and say we should contact that same client about the developer position that needs to be filled and for this a job interview needs to be arranged. We didn’t finish the previous mail. Do we add an additional paragraph in that email, or should we send it as a new one? It is the same client and most of the people from the first mail should receive this one too.

What subject should we put for mail that contains both topics? “Project transfer and job interview”? We can see that we are doing two things at once. At first it doesn’t seem like a big problem. But imagine when a client starts replying to your mail, adding new recipients. Some of them start answering the first part, others the second part. Now you need to reference that mail, but only part about the job interview. You can’t do that, you are including them in the whole thread. Your dev-ops need the part related to environments. Again, you are including them in the same thread. Everybody receives every email. This is quickly becoming a mess.

Translated into code, this means one thing: classes and methods should have only one responsibility. Keeping it this way we are going to save ourselves a lot of time once the code base becomes big and features start to pile up.

Be descriptive when naming things

I have already touched on this topic in the previous section. When we send an email we need to set a subject. Imagine you receive an email “Important — read”. Or maybe “It is not working”. What is the purpose of this email? We need to open email and read what is inside. Later, when we need to find it between a bunch of emails that we have received, how will we ever know what to search for?

We should have a tendency to set the subject of email short, but as descriptive as possible. This way the recipient can see the intentions of an email just by reading the subject (“Project transfer”, “Job interview”).

Also, when we name things in our code we should follow the same logic. For example, we have a property that determines if a customer can have some feature enabled. I have a tendency to name boolean values like questions (IsDeleted, IsPublished, HasFeatureEnabled…). This way when I write code it would look something like this:

if(customer.HasFeatureEnabled) {  
 // do something  
}

When we read it later we are basically reading plain old English (If customer has feature enabled do something).

If the subject of our email is becoming too big, that is a sign that we are doing too many things at once and that we should probably split it into two or more separate emails. For example, “Project transfer and job interview” mentioned before. This is what I am talking about. You can see that this is something that doesn’t belong together and that will probably lead to unreadable and messy mail thread.

Same goes for code. If the name of a class or method is too big, that is one of the signs that we are doing too many things at once. Changing one thing from that class/method could break other behavior in that same class/method. And good naming convention warned us about it.

Be consistent when naming things

Other important thing that we need to take care of when writing email is consistence. We should use expressions and naming that recipient understand, and stick to it. We shouldn’t juggle with multiple names for same thing, nor name different thing with same name. For example, let’s say we are creating multi tenant software. It is some form of CRM.

Now, our client is a company that has other companies for clients that have people as their clients. In this sentence word client is used to describe three different things, and word company is used to describe two different things. In two cases client and company are the same thing.

If we write an email and talk about a client, who are we referring to? And if we once say client, and other time company we are just complicating things for whoever is reading our email. We should agree how to call something and stick to it.

If something is called Tenant it should be called Tenant in all emails. If we sometimes call it Account, Client or Company, even though it is correct, we didn’t agree on that and it could cause confusion.

Avoid slang

I have a friend that for some particular reason refuses to write full words in a sentence. I always felt like I had a stroke reading his messages. This became so irritating to the point that sometimes I simply would not read his message and would just call him.

Many people write code this way and this eventually leads to two things: 1) you either need to do everything by yourself or 2) you will need to explain your code to the other developers every time someone else is doing changes on it.

When we are debugging code we are scanning through it, not really reading it, but trying to interpret logic behind it. We don’t want to read it all and why should we? We only need a general picture of what is going one and to find the place where things need to be changed.

Let’s say we need to do something by specific instructions. These instructions were sent to us in an email. We find the email thread and start to go through it. We do not read every email in it, we only scan through it to find things that we need. Imagine now that my friend sent us these instructions using made up abbreviations and acronyms. Now again, imagine reading that under pressure of deadlines.

I found a lot of things like this in the code I was working on. Namespace called GTK that has nothing to do with UI, AccService that has nothing to do with Accounts, 8+ letters long names that do not form any meaningful word and look like random generated string… Some of those were mine, some were other programmers’ code. But in the end, in all these cases, it was harder to work on the code than it should be.

Break it early

If some condition will break execution, it should be put first. This way a person does not need to spend the whole day analyzing something that doesn’t refer to their case.

On multiple occasions I received mail that goes something like this:

We are informing you regarding [some actions]. If you already did [that action], or it does not apply to your case, please ignore this email.

[content of mail]

This saved me so much time reading unnecessary mails, and not to mention double guessing myself if I already did requested action.

Conclusion

Writing code is easy. Reading it later, not so much. At the moment when we were writing code we had all the facts in our head and we were fully devoted to the task. But with time that knowledge fades away and we find ourselves in a situation where we need to relearn it. If someone else needs to do the task on our old code and it is not readable, there is a good chance that that task will eventually find its way back to us. This will waste the time of two developers instead of speeding things up. Time is the expensive resource and we should invest some of it now so we can spare it later.

Benefits of stepping into the unknown

Krste Šižgorić — Sat, 07 Aug 2021 10:48:59 +0000

Programmer’s job is to solve problems. It applies to everyone in the industry no matter which programming languages or framework we are using. Our job starts with an idea that needs to be transformed into something that can be used by others. And that process of transforming ideas into implementation can be done in so many different ways. That is why there are so many technologies out there, so many different approaches. People tend to find different solutions for the same problems.

However, once we do solve it, if we run into the same or similar problem again, we will probably implement it the same way. Fact is, we do not do things in the most efficient way but in the way that originally led us to the satisfying result. We are not looking for the best, we are looking for something to get things done and let us move on. If we find something that works, we stick to it.

Leaving the comfort zone

It is hard to look at the same problem in a different light. Even if we try to do things differently we tend to fall into the same tracks and in the end we create very similar solutions. If the only tool you have is a hammer, you tend to see every problem as a nail. But a hammer is not the only tool in the bag. And we should try to have as many tools in the bag as we can.

One way of achieving this, that I would definitely recommend, is by experimenting with different technologies. Every technology has its way of doing things. If some technology has a big community, it means that a lot of people like to do things that way. So there should be something good that we can learn from it.

Experimenting with different languages

I am a C# back-end developer, but I did experiment with different programming languages. I even worked as a PHP developer for a couple of years. While working on one of these PHP projects, I stumbled upon an interesting feature inside ORM. This system had 19 roles. To make business logic more reusable, on each model there were access limits per role. This meant that additional conditions would be appended on each query for each table inside that query. That additional condition would vary depending on the user’s role.

This is something, as far as I know, that was not possible in Entity Framework nor NHibernate at the time. You could append additional conditions, but only on query level (that is now changed with Global Query Filters in Entity Framework Core). And this way of thinking was definitely not the way any .Net programmer I know was thinking.

Besides PHP I also had a little “field trip” to Ruby. Even though I didn’t work a lot with Ruby on Rails, I did however had the chance to witness the power of “Rails Magic”. Rails is based on convention over configuration, and with use of Active Admin you could get really fast development. If you name things a certain way you can leverage certain behavior and things would work out of the box. This seemed so nice to me. You do things once, and with the right naming convention, everything will just work.

Doing things the different way

I was given a task of setting up an architecture for a particular system. Programming language would be C#, and for storage we would use SQL server. Pretty standard. I knew that this system will have a lot of business logic, but equally so a good portion of the functionality will be standard CRUD operations. System will have multiple roles and will be used by multiple internally developed client apps. Also, development was supposed to be fast due to short deadlines.

Since application development speed was one of the factors, and a lot of actions would be simple CRUD operations, generics was a logical choice. Reusing as many things as possible would be great. And if something like this could be done with configuration, or even convention over configuration, that would be even better.

So this was my proposal: CRUD operations would be covered with generic implementations. If generic implementation doesn’t fit some use case, specific implementation would be created.

To limit the need for different actions for different roles, Entity Framework Core would be used in combination with Global Query Filters. For each entity we would configure access limits based on the role. This would increase reusability of generic implementation. You could use the same method for admin, manager or a basic user. That business logic will produce different results based on the user role. Same goes with update and delete methods. Method will only work for entries users have access to. If a new role is added to the system, most of the functionalities would already work just by adding global query filters for that role.

Another thing that I thought would be nice was to delegate select logic to models and use auto mapper projection to retrieve only needed data. I combined this with generic implementations which would receive generic request and response models. Remember, these are simple CRUD operations. If you want to update an entity, you would receive a generic model, retrieve entity from database, remap request to entity and save it. Since you can remap different kinds of models to the same entity, this opens a whole new level of reusability of generic implementation.

Result

Whole architecture was broken to small pieces which were building blocks for a solution. We use what we could and only if we need we do specific implementation. This way of doing things do have a big learning curve and might seem a bit weird, but it was good choice for this case. Four team members (starting from empty solution) implemented almost 400 endpoints within two months. Everything we could, we delegated to generic implementation, which gave us more time to devote to complex business logic.

If we needed to expand an existing endpoint to return an additional field, all we needed to do is to add that property to the model. By following the naming convention, auto-mapper would do its magic and remap the entity’s property to a model and additional fields would be added to select statements.

In case there is a need for two different sets of data based on the same entity (index page with list, dropdown list or specific formatting) you could reuse the same method by forwarding a new model into it and getting different results. This could be done with OData, but we had multiple clients, so I decided that it is better to do this on the back-end side and prevent reinventing the wheel on each client app.

End solution was a hybrid of ideas from different technologies compacted into something specific for the problem we faced. I believe this is a pretty unique approach in the dotnet ecosystem. And it would never be done this way if I didn’t work on a PHP project with ORM filtering, or if I never worked with Ruby on Rails.

Conclusion

By experimenting with different technologies we expand our skill set. Of course, there is no need to learn all technologies out there, specialization is still the best way of perfecting yourself, but healthy amount of exposure to different technologies do give us new insights.

This makes us more flexible and resilient to changes. And we do need to be resilient. Software development is a very dynamic profession. Paradigms change all the time. Before React.js there was consensus that business logic should be separated from presentation. Then came React.js, change all of that, and became most popular SPA framework out there.

But trends are not reason why we should learn new things. We should learn new things so we can use them in situations where they are the best solution for the given problem. There is no silver bullet, one approach to rule them all. We have a bag of tools and we choose which one to use. The more tools we have, the easier it gets to adjust to given situations.

Code as a source of inspiration

Krste Šižgorić — Sat, 03 Jul 2021 11:19:47 +0000

The most important thing in software development is an idea. Idea is a spark that ignites and drives process of creating something useful, helpful or simply fun. As developers, we prefer good quality code and architectural design, but to users that is completely irrelevant. If bad code is making their life easier then in their eyes that is a great product. If that is not the case, you can write the cleanest code ever, but they will not care or ever use it.

Every program is a tool created to solve some specific problem. More problems it solves, more valuable it is to a user. Guided by that logic, we try to make make our solution more valuable. Our product grows and evolves so idea become something bigger, something that we originally didn’t plan.

But how does our idea grow? Where does inspiration come from? Often, users are the source of changes and improvements. They share their problems with us and we try to solve them. We can try to anticipate what user wants and suggest/implement changes by thinking how we can make things better. And sometimes, if done right, our solution can show us a full potential of the original idea.

Structure as foundation for a great idea

For code to be the source for improvements, it needs to be well structured. This is best explained with example. I was a lead developer on a project with a product that was integrated with three external systems from three different providers. All three integration were separate projects and developed as separated API-s that shared Data layer. This was something that we inherited and we didn’t question it. It seemed like a good idea at the time because we could scale any API the way we wanted, depending on traffic.

All three integrations are doing the same thing, but in different way. Each partner had something unique and specific that would differ one implementation from the other. Once I moved myself away from specific implementation, and looked at the project from higher perspective, I noticed that all three projects had almost the same steps in the process. The process could be unified.

I suggested that we merge business logic of all three projects into one. That way we would maintain only one project. So less work for everyone. To speed things up we would choose best and most generic of three implementations, and change it into something that would suit the flow of all three integrations. Existing integrations would be changed into adapters and would reference unified business logic. Each adapter would transform the request from external system into something that new implementation would understood and could process. Difference in logic could be specifically implemented if needed.

At the time I didn’t understood full potential of this idea. For me, this was only to make things easier for me and the development team. That was probably the reason why I didn’t get a green light to go with this changes. There was too many things that needed to be developed first, and this was not a priority. More like “nice to have”. Other arguments were that if we have a bug, we will have it in all three projects now. This meant that fix needed to be deployed for all three projects. When we were deploying something, we would always have downtime and product would not work for a short period of time. And in heavily traffic solutions like this one, that is something that we wanted to avoid. So this idea was postponed.

True benefits

I never had a chance to implement this idea. Not long after my suggestion I was needed on another project and I was transferred to the different team. But colleague of my did implement it. And this idea has grown far beyond what I intended to.

He implemented unified business logic and transformed existing integrations into adapters. As direct consequence of this change, integration with new (fourth) partner was implemented blazingly fast. And a reason for this was one simply fact: set of actions from unified business logic arose and contract was created. All adapter needed to do was transfer input from external system into this contract. It is not as simple as it sounds from this sentence (there was a lot of work to be done), but still better than implementing whole thing.

Once contract was established there was clear set of actions for new API that could be created. Until that point, all integrations needed to be done within solution. Now, with this API, work of integration could be done in external system. This brings benefit of working with new partner without cost of developing adapter for their system. With original implementation, this was not that easy to see. And implementing API without unified business logic would definitely require more time.

But why stop there? This product was used directly by end-users. What if, just like with external systems, contract could be created in communication with users? Now anyone could communicate with external systems. All they need to do is respect given contract (interface) provided by the system. Now you are in provisioning business.

You could split system into two different products: one providing service to the end-user, and other providing integration service to external systems you already integrated with. Whoever wants to start or expand its business could do it by integrations with each external system on its own or to just integrate with you and get integration with all of them.

Should this be done is up to the management to decide, but the idea was born from well formed product. Original implementation never shown this potential. And the fulfillment of these ideas would definitely require much more time and effort.

Conclusion

Idea is a great starter for any story. But without form, idea is just that — an idea. It needs to be explored, refined and tested. For idea to become a story, it needs structure: introduction, rising action, climax, falling action and resolution. That is a template for every book, movie or TV show.

Same goes for software development. We need template that we will follow. That template gives us a chance to explore and see what else could be done. This way “the story” (software) itself can show us where we can go. It is up to us to choose the path, but now we have a map to see what else is out there.