Forem: Alexander Demin

Data Validation: Scratching Surface of Code Responsibility

Alexander Demin — Sun, 11 Feb 2024 18:16:42 +0000

Introduction

As responsible developers, our goal is to write code that is not just functional but also clear, maintainable, and adaptable. In this process, we often face a key but somewhat elusive question: the responsibility problem in clean code. Hold on, don’t close the page just yet. I’m not planning to describe the well-known Single Responsibility Principle (SRP). Instead, the idea is to take a look at something broader and yet sometimes more tricky — where should specific functionalities be placed within a system’s architecture?

To bring this topic into focus, let’s take a look at various elements that exist in almost any modern software system — data validation, error handling, database transaction management, caching logic, logging, metrics, and more. Each of these components has an important role, and yet, their optimal placement within a system’s architecture is a subject of ongoing debate (or it should be).

The idea of this blog post is to showcase my current view on some parts of this question and, hopefully, to help some of you architect your services in a cleaner way. If it raises more questions than answers, even better. In discussions, we can reveal the full picture and make an informed decision.

But before we dive into the main topic, let’s first go over some basic concepts. These are important for understanding the ideas we’ll discuss and how they affect decisions in software design.

Just One More Look To The Layered Service Architecture

Let’s think about layered architecture in software not as a clear abstraction, but as a company with different departments, each doing its part to provide value. Understanding this helps us figure out where to put different parts of our code later.

Imagine a software system as a company. In this company, we have different departments working together, each with a specific role:

Incoming Communication Layer (Transport Department). This layer acts as the bridge between external inputs (like user requests, and messages from other services) and the internal business logic of the software. It is responsible for receiving data, transforming it into a format understood by the domain layer, and forwarding it appropriately. This layer ensures that the data communication to and from the business logic is streamlined and efficient.
Business Logic Layer (Production Department). Think of this as the production department of the company. It takes the requests from the transport department and works on them. This is where the main action happens — the rules and processes of the software are applied here.
Persistence Layer (Storehouse). This is like the storehouse of the company. Once the production department has processed the request, the results (data) need to be stored. This layer handles saving data to databases or files and retrieving it when needed.

Though this analogy might seem like an oversimplification, it isn’t. This definition helps to make an informed decision later when deciding where to put what functionality. And it is important to understand not only what each layer does but, equally crucial, what it DOES NOT do.

Of course, there are usually more components than just those listed here, but overall, they typically fall in the same general logic (and it really does not matter what approach you follow — onion, hexagonal, or anything else):

Driving infra layer with synchronous HTTP/gRPC/console handlers, async events/commands consumers.
Domain layer with all the business logic.
Driven infra layer with database repositories, caches, async events/commands publishers, clients to other services.

Thinking about layered architecture like a company with different departments helps us see how each part of our software has its own job. Each layer focuses on its role without stepping into others’ tasks. This organization makes our software easier to work with, fix, and change.

If you, by any chance, are not familiar with these concepts, I’d suggest checking out some dedicated articles, there is an enormous amount of them on the internet.

Validation Place Is Obvious. Well, Not Quite.

Now that we’ve refreshed our understanding of the layers concept in software, let’s dive into a specific aspect of functionality placement. To keep things concise, we’ll focus on one example, exploring which part of the validation should live in which layer.

Typical Framework Approach to Validation

You might ask — why is it even a question? Don’t we do it for years and this functionality is supported by the majority of existing frameworks, right out of the box? Well, yes, and not really. Most frameworks offer a straightforward approach to validation. Here are just a few examples:

In Java Spring, you might use the @Valid annotation in the controller parameter, along with specific annotations for each important field in the DTO (Data Transfer Object).

@Entity
public class User {
    @NotNull
    @Size(min = 5, max = 255)
    private String name;

    @NotNull
    @Min(18)
    @Max(150)
    private Integer age;
}

@RestController
public class UserController {
    @PostMapping("/users")
    ResponseEntity<String> addUser(@Valid @RequestBody User user) {
        // ...
    }
}

PHP Laravel framework typically involves explicit request validation right in the controller.

public function addUser(Request $request) {
   $validated = $request->validate([
      'name' => 'required|min:5|max:255',
      'age' => 'numeric|min:18|max:150',
   ]);
}

In Golang, despite the real absence of frameworks, many libraries follow a similar approach for validation. Let’s take a look at an example with github.com/go-playground/validator/v10 library:

type User struct {
    Name string `json:"name" validate:"required,min=5,max=255"`
    Age  int    `json:"age" validate:"gte=18,lte=150"`
}

func (a *API) createUserHandler(w http.ResponseWriter, r *http.Request) {
    var user User

    err := json.NewDecoder(r.Body).Decode(&user)
    if err != nil {
        // write error to the response
        return
    }

    validate := validator.New()

    err = validate.Struct(user)
    if err != nil {
        errors := err.(validator.ValidationErrors)
        // write errors to the response, 
        // most likely mapping specific validator errors to something
        // that can be understood by a client
        return
    }

    user, err = a.userService.CreateUser(user.Name, user.Age)
    if err != nil {
        // write error to the response
        return
    }

    w.WriteHeader(http.StatusOK)
    // write created user to the response
}

In these examples, the validation is done right in the Incoming Communication Layer, before any data is sent to the Domain Layer.

A Closer Look at Layer Responsibilities

When we examine the roles of the Incoming Communication Layer and the Domain Layer a bit deeper, a different approach to validation emerges:

Incoming Communication Layer is where data first enters our system. This layer is primarily concerned with transporting data, such as transforming received data into a format the Domain Layer can understand. Usually, it’s not aware of the deeper business logic or requirements.
Domain Layer is where the main action of our software occurs. This layer knows all the requirements and business invariants.

Considering this separation of responsibilities, does it still look like a good idea to put detailed validation to the infra layer? Just imagine if your transport department receives accounting reports, opens all the boxes, and examines received documents on validity without asking the responsible department. They simply can’t do it even if they want to — there is not enough knowledge in place to decide what is right or wrong.

What the Incoming Communication Layer (read “HTTP handler/controller”) can really do is only perform basic checks like ensuring the data is in a generally correct format (e.g., valid JSON), check the payload size, and maybe do other checks related to its area of responsibility.

On the other hand, the Domain Layer has all the information and really can perform thorough validation, ensuring the data is meaningful and matches with the domain requirements. This even includes checking for mandatory fields, the length of strings, and the correctness of field formats.

This approach differs from the simple usage of a validation library in an HTTP Controller/Handler. It creates a clear separation of responsibilities between the Infrastructure (Incoming Communication Layer) and the Domain Layer for the task of data validation:

Incoming Communication Layer has limited knowledge of payload internals, focuses on basic payload format and size checks.
Domain Layer is fully informed about the data, checks data against business rules, ensuring it meets all necessary criteria.

Adopting this idea prevents business logic from leaking outside the Domain Layer. Each layer does its part efficiently, supporting each other without overstepping boundaries. This method ensures that our software remains well-organized, with clear responsibilities, leading to better maintainability and scalability.

Let’s take a look at the revised example in Golang with the discussed separation of responsibilities:

package api

type User struct {
    Name string `json:"name"`
    Age  int    `json:"age"`
}

func (a *API) createUserHandler(w http.ResponseWriter, r *http.Request) {
    var dto User

    err := json.NewDecoder(r.Body).Decode(&dto)
    if err != nil {
        // write error to the response
        return
    }

    user, err = a.userService.CreateUser(dto.Name, dto.Age)
    if err != nil {
        // proper domain error handling
        return
    }

    w.WriteHeader(http.StatusOK)
    // marshal user entity and write to the response
}

package domain

type User struct {
    Name string
    Age  int
}

var ErrUserTooYoung := NewValidationError("user_too_young")
var ErrUserTooOld   := NewValidationError("user_too_old")
// define other needed errors

func NewUser(name string, age int) (*User, error) {
    if age < 15 {
        return nil, ErrUserTooYoung
    }

    if age > 150 {
        return nil, ErrUserTooOld
    }

    // ... other validation checks

    return User{Name:name, Age:age}, nil
}

type UserService struct {
    // ...
}

func (s *UserService) CreateUser(name string, age int) (*User, error) {
    user, err := NewUser(name, age)
    if err != nil {
        return nil, err
    }

    // store user in the database

    return user, nil
}

If you feel that the NewUser function becomes too complex, it might be a signal that it is time to use the Value Object approach. Some would even say that it is better to use Value Object from the very beginning.

Let’s say with Age as a Value Object the example could look like this:

package domain

type Age int

func NewAge(age int) (Age, error) {
    if age < 15 {
        return 0, ErrUserTooYoung
    }

    if age > 150 {
        return 0, ErrUserTooOld
    }

    return Age(age), nil
}

type User struct {
    Name string
    Age  Age
}

// NewUser already accepts the Age value object instead of a generic int.
func NewUser(name string, age Age) (*User, error) {
    // ... other validation checks

    return User{Name:name, Age:age}, nil
}

func (s *UserService) CreateUser(name string, age int) (*User, error) {
    // here we create age as a value object which validates itself
    userAge, err := NewAge(age)
    if err != nil {
        return nil, err
    }

    user, err := NewUser(name, userAge)
    if err != nil {
        return nil, err
    }
}

And so on. From the first look, it might seem like more complex code, but in reality, it just becomes explicit — it is always clear what errors the code can return and in what format. It does not hide business rules but instead showcases them directly, right in the domain layer where they belong. With this approach, it becomes nearly impossible to forget something and create a wrongly structured domain entity. Validation becomes a first-class citizen of the domain.

Benefits

By segregating validation responsibilities according to the layers, the code becomes more organized. The Incoming Communication Layer handles basic data integrity, while the Domain Layer manages complex business rule validation.
With each layer performing its distinct validation checks, the system becomes more robust against invalid or malicious data inputs, reducing security vulnerabilities. Especially when other types of transport are added to a service, e.g. you have HTTP and add gRPC or async handlers.
There is no coupling to a specific validation library, everything is explicit and clear for other developers.

Potential Pitfalls

If responsibilities are not clearly defined, they may overlap, leading to redundant validations and increased complexity.
Coordinating error responses and providing meaningful feedback across different layers and to clients can be challenging.

For a deeper dive into effective error-handling strategies, particularly in a domain-centric approach using Go, refer to my dedicated article here. Domain Centric Approach to Error Handling Using Go

Conclusion

The way we handle responsibilities in software architecture has serious implications on the design and maintainability of our systems.

The discussed principle doesn’t just apply to the Incoming Communication and Domain Layers, it covers the driven infrastructure layers as well, such as repositories and event publishers. Their primary task is just to store and retrieve domain entities without concerning with the internals of these entities. It is the domain entities responsibility to determine the correctness of their internals.

I hope this article has been a bit thought-provoking and insightful for you. And I’ll be happy to hear your feedback!

Do you agree or disagree with the points raised? Why?
Or maybe the topic is obvious to you and you’re wondering why it was even brought to the table?
Would you find it useful to explore other examples of functionality responsibility issues in further posts?

Your views are really important, and I’m excited to keep talking about this. Thank you for reading, and stay tuned for more articles! ❤️

Find me on Linkedin: https://www.linkedin.com/in/alex-demin-dev/

Thoughts on Beyond Coding: With Business in Mind

Alexander Demin — Wed, 07 Feb 2024 13:40:17 +0000

Recently, I came across this question on Threads: “As a developer, if you want to impress me, you should _____”. And it triggered me to think, what would I write there being a software engineer myself? What would impress me?

Is it being good at coding? Like writing code that is easy to understand and maintain?
Or it is to be able to work on big projects by themselves?
Maybe having insanely high code coverage of all the code they write?
Knowing dozens of programming languages and ways to build software?

Saying that these skills are impressive is like saying that just doing the job is enough to stand out, which is typically not.

But there’s something that In my opinion makes a developer stand out, and it’s not about coding or purely engineering skills. It’s about understanding the business.

It is not a secret that many developers just focus on their coding tasks. They get a job to do, they do it, and move on to the next one. They may rarely ask why they’re doing it or how it helps the business. This happens a lot, even in companies that make their own software products. The result is work that only does what someone else thought was needed, without really seeing the big picture and taking it into technical consideration.

What can make a developer truly impressive is when they understand not just what and how to do, but why it needs to be done. This knowledge gives them a special ability to help the business. Often, the tasks that developers are given already have a planned solution, which might not always be the best or the only way to do things. Since developers know the technical side very well, they can suggest better solutions to existing business problems that can save a lot of time and effort.

Developers who know about the business side of things can do more than those who don’t. They can offer new ideas, find smarter ways to solve problems, and make sure their work really helps the company’s goals. They become much more valuable because they not only do their job but also contribute to the company’s success in a bigger way.

So, my answer to this question is definitely:

As a developer, if you want to impress me, you should know the business.

While being good at software development is important, it’s not everything. To stand out as a developer, we, as developers, need to understand the business we’re working in. This is what makes a developer not just a deeply technical person, but a valuable part of the company’s success.

So, if you’re a developer and find this idea insightful or at least meaningful, start by learning more about the business side of your projects. Perhaps, a simple question “why” on the next sprint planning session could be a good start.

Stay tuned for more discussions on Go programming, software architecture, and beyond. Happy coding!

Find me on Linkedin: https://www.linkedin.com/in/alex-demin-dev/

Context Matters: Advanced Error Handling Techniques in Go

Alexander Demin — Tue, 30 Jan 2024 18:33:52 +0000

TL;DR: This article explores advanced error handling in Go, focusing on contextual errors that provide rich details for debugging by embedding key-value metadata into errors. It demonstrates implementing these errors using a dedicated package, offering insights into better error resolution and organized logging. The approach enhances error understanding in Go applications, balancing detail and simplicity.

Before reading this post, consider checking the first article about error handling in Golang: Domain-Centric Approach to Error Handling using Go

Introduction

In the previous blog post, we explored the domain errors approach to error handling in Go, ensuring that error messages are both clear and user-centric. However, while this approach effectively addresses one aspect of error handling, it falls short in another critical area - providing sufficient contextual information. The absence of context can create significant challenges for developers and QA specialists as they attempt to investigate and understand the underlying problems in an application.

In this post, we will try to tackle this issue. We'll start by taking a brief look at the standard approach to error handling in Go, then go through a typical example that highlights the limitations of this method. Finally, we will introduce an approach that could help solve this problem, or at least make it more manageable and future-proof.

Limitations of the Standard Way

When using traditional Go error handling, providing useful context is pretty difficult. Errors are just simple text messages without any structured information. An error is typically created in a deeply nested service (e.g. a database repository) and bubbles up through several layers before finally being handled. By the time it reaches the surface, the original context - the 'why' and 'how' of the error - is often lost.

To address this lack of context, a common practice is to embed context directly into the error message using formatted errors. Typically, this is done by using statements that include additional information with the error, such as:

fmt.Errorf("failed to store entity with ID=%s: %w",
    entity.ID, err)

fmt.Errorf("entity %s is empty", entity.ID)

fmt.Errorf("failed to process entity with id=%s, 
    userID=%s: %v", entity.ID, user.ID, err)

However, this approach leads to its own set of problems. As an error passes through various layers, accumulating additional context, the final error string becomes increasingly complex and convoluted:

failed to process request, requestID=491002, userID=4881:
doWork called, failed to do work, entityID=482003, 
userID=4881: failed to fetch entity with ID=482003 from 
the database: database connection lost

Such error strings, although rich in information, are not great:

Long, nested error messages become increasingly difficult to read and understand. The deeper the error is nested, the harder it is to quickly determine the root cause or the primary message.
These verbose error strings are hard to parse by logging analysis tools. Extracting specific pieces of information from these long strings can be problematic and error-prone.
As errors bubble up through the application layers, the same information often gets repeated. For example, user IDs or entity IDs might be appended at multiple levels, leading to redundancy in the error message.

Exploring a Practical Example

To illustrate the practicality and necessity of contextual errors in Go, let's consider a common scenario in a typical Go application with a simple layered architecture: an HTTP handler, a Domain Service, and a Database Repository.

HTTP Handler (package api)

This layer handles incoming HTTP requests, interacts with domain services, and responds to the client. It's the entry point for most operations initiated by the user. For simplicity, in the exampleHandleSomeRequest does not operate with http.ResponseWriter and http.Request directly, leaving it to a more generic caller.

package api

type DomainService interface {
    DoWork(ctx context.Context, userID string) error
}

type API struct {
    domainService DomainService
}

func (a *API) HandleSomeRequest(ctx context.Context, userID string) *Response {
    err := a.domainService.DoWork(ctx, userID)
    if err != nil {
        slog.Error("error doing work", "error", err)
        return NewErrorResponse(err)
    }

    return nil
}

Domain Service (package domain)

This layer contains the business logic. It's responsible for executing domain-specific operations, often interacting with the repository using an interface.

package domain

type Repository interface {
    LocateEntity(ctx context.Context, userID string) (*Entity, error)
    SaveEntity(ctx context.Context, entity *Entity) error
}

type Service struct {
    repo Repository
}

func (s *Service) DoWork(ctx context.Context, userID string) error {
    // find specific entity which might be different depending 
    // on the user internal state on the moment of the request
    entity, err := s.repo.LocateEntity(ctx, userID)
    if err != nil {
        return err
    }

    // perform some work on the entity
    if err := entity.DoImportantEntityWork(); err != nil {
        return err
    }

    // save the entity back in the database
    if err := s.repo.SaveEntity(ctx, entity); err != nil {
        return err
    }

    return nil
}

As you may see, the process of error propagation to higher layers is quite straightforward and doesn't provide much extra information. For example, in the absence of any context, an HTTP handler would not be able to log any specific details of the entity that caused the error. This information is only available in the domain layer and is not transferred to higher layers.

Let's take a look at how to address this problem and improve this code without adding this information right in the error message.

Introducing Structured Contextual Errors

Structured contextual errors differ from traditional error wrapping by including additional metadata in a structured format, typically key-value pairs. This metadata can be represented by various data relevant to the error, such as entity IDs, user details, input parameters, or internal state information. By embedding this information directly into the error, we create a self-contained error object that carries its context along with it.

Let's refer back to our previous example and improve our error handling by implementing structured contextual errors. Instead of returning a generic error message, we will wrap it with additional context when an error occurs with a simple contract likeWithMetadata(err error, pairs …any) which adds provided metadata to the error and returns an enriched version.

package domain

// … [Other code] …

func (s *Service) DoWork(ctx context.Context, userID string) error {
    entity, err := s.repo.LocateEntity(ctx, userID)
    if err != nil {
        return metaerr.WithMetadata(err, "operation", "locate_entity")
    }

    if err := entity.DoWork(); err != nil {
        return metaerr.WithMetadata(err, "operation", "do_work", 
            "entity_id", entity.ID)
    }

    if err := s.repo.SaveEntity(ctx, entity); err != nil {
        return metaerr.WithMetadata(err, "operation", "save_entity", 
            "entity_id", entity.ID)
    }

    return nil
}

In the HTTP handler layer, we can now include structured metadata when logging errors, making the logs more informative. It can be done with a handy GetMetadata(err) []any function that extracts all metadata gathered throughout the call chain and returns it as a slice of key-value pairs suitable for a standard Go structured logging library.

package api

// … [Other code] …

func (a *API) HandleSomeRequest(ctx context.Context, userID string) *Response {
    err := a.domainService.DoWork(ctx, userID)
    if err != nil {
        slog.With(metaerr.GetMetadata(err)...).
            With("user_id", userID, "error", err).
            Error("error doing work")
        return NewResponse(err)
    }

    return nil
}

MetaErrors Implementation

Having our desired contract in place, let's take a look at a potential implementation of the metaerr package.

First thing we need a structure type that will represent an error with attached metadata and will support all required error-related interfaces. Let's name it errMetadata and make it private to the package, as there should not be any need to reference it from outside. This struct is the cornerstone of the metaerr package and will enable us to store additional error context.

// errMetadata represents an error with attached structured metadata
type errMetadata struct {
    // err is the original error
    err error
    // metadata is the container for structured error context
    metadata []any
}

// Error returns the original error message, 
// ensuring compatibility with the standard error interface.
func (e *errMetadata) Error() string {
    return e.err.Error()
}

// Unwrap allows errors wrapped by errMetadata to be compatible with
// standard error unwrapping mechanism
func (e *errMetadata) Unwrap() error {
    return e.err
}

The WithMetadata function is responsible for wrapping an existing error with additional context. It takes an error and a series of key-value pairs and then wraps this error in the errMetadata struct along with the provided metadata.

package metaerr

// WithMetadata creates a new error with metadata
func WithMetadata(err error, pairs ...any) error {
    return &errMetadata{
        err:      err,
        metadata: pairs,
    }
}

The GetMetadata function is designed to extract metadata from errors in the form of a slice, suitable for the Go standard structured logger (slog). In the future, more specific methods may be introduced, such as returning key-value pairs as a map or merging values for duplicated keys. However, for now, let's limit the example to logging purposes and keep things simple.

// GetMetadata returns metadata from the error chain
// if there is no metadata in the chain, it will return an empty slice
func GetMetadata(err error) []any {
    data := make([]any, 0)

    // we will iterate over all errors in the chain
    // and merge metadata from all of them
    for err != nil {
        // if current error is a metadata error
        // we will add its metadata to the existing
        // already collected metadata
        if e, ok := err.(*errMetadata); ok {
            data = append(data, e.metadata...)
        }

        err = errors.Unwrap(err)
    }

    return data
}

With the metaerr package, error handling in our example application becomes more informative. When an error is created or passed through the DoWork function in the domain service, WithMetadata is used to attach relevant context. At the API layer, logging these errors with GetMetadata helps to capture a complete picture of the error scenario, including all important contextual details.

Benefits of Contextual Error Handling

The primary benefit of contextual errors is the rich detail they provide, which is invaluable for debugging:

Unlike traditional error messages that often require developers to dig through logs or replicate issues, contextual errors can provide immediate insights into what went wrong and under what conditions.
Logging systems can leverage the structured nature of these errors to create more organized and searchable logs, making it easier to filter and analyze them.
Logs that include detailed error metadata offer a more comprehensive view of issues, aiding in post-mortem analysis and continuous improvement processes.

Some Rules Of Thumb

To effectively utilize contextual errors in Go applications, consider the following rules of thumb:

Only include metadata that is useful for understanding the error. Avoid cluttering errors with irrelevant or redundant information.
Add information at the point where it is first obtained. For example, a parameter passed to a function should be added to the error in the function where it was created, not necessarily where the error is first encountered. It will help to reduce the risk of adding the same parts of metadata in multiple places.
Use consistent key names for metadata across different parts of the application. This consistency is crucial for effective log analysis.
Be mindful of how errors are wrapped. Preserve the original error using Go's wrapping mechanism (%w with fmt.Errorf) to maintain the error chain and gather metadata.
Contextual errors should complement, not replace, existing error-handling strategies. They should fit seamlessly into the current architecture.
Be cautious about including sensitive information in error metadata, especially when logs might be stored or processed in less secure environments.

Conclusion

By integrating structured contextual errors, particularly through the implementation of a library similar to the discussed one, developers can achieve a deeper understanding and more efficient resolution of issues that arise in their applications.

Key Takeaways

Contextual errors provide information that goes beyond the traditional error message, offering insights into the application's state at the time of the error.
With detailed metadata attached to errors, the debugging process becomes more straightforward, leading to quicker resolutions and enhanced application reliability.
Structured error metadata leads to more organized and actionable logs, aiding in both real-time issue resolution and long-term analysis.
Implementing contextual errors effectively involves a balance of including relevant metadata, maintaining consistency, and ensuring performance and simplicity are not compromised.

Potential pitfalls

Keep in mind that metadata is not a replacement for typed errors, it is just a context. If you need to provide some specific information to a client, do it explicitly with typed domain errors, generic metadata is not a good fit for domain-related information.
Adding too much metadata can lead to performance overhead. Ensure that the process of attaching metadata is efficient and doesn't significantly impact application performance.
The error-handling logic can become more complex with the addition of metadata. Strive for a balance between detail and simplicity.

Final Thoughts

Complexities of error management in Go demonstrate that the decision to implement contextual and structured error handling is both a practical and strategic choice. It's about enhancing the code we write and the experiences of those who interact with our applications - developers, support teams, or quality assurance specialists.

Embracing contextual structured error handling and logging is a step in that direction, offering a method to handle errors as sophisticated and nuanced as the applications we build.

I hope you found this exploration insightful and that it inspires you to implement these practices in your Go projects. Your feedback and experiences are invaluable, so feel free to share your thoughts and how you have applied these concepts in your work.

Stay tuned for more discussions on Go programming, software architecture, and beyond. Happy coding!

Find me on Linkedin: https://www.linkedin.com/in/alex-demin-dev/