Forem: Jonathan Hall

Testify is making your Go tests worse

Jonathan Hall — Tue, 21 Apr 2026 14:33:25 +0000

Pop quiz.

Does this testify assertion pass or fail?

    var x []int
    y := []int{}
    require.Equal(t, x, y)

If you're like me, you have no idea. Arguments for both passing and failing seem reasonable. Let's jump to the docs:

func Equal
func Equal(t TestingT, expected, actual interface{}, msgAndArgs ...interface{}) bool
Equal asserts that two objects are equal.

Well that's no help. I still don't know if an empty slice is considered equal to a nil slice! To answer the question, I have to dig through the source code, or just try it.

Is the learning curve worth it?

Assertion libraries are popular in Go, and for a solid reason: they make tests shorter. But in exchange, you're signing up for some hefty costs, which are often overlooked. A few, from the official Go wiki:

They require developers to learn a whole new sub-language just to read or write tests
Most assertions abort a test early, omitting potentially useful information from the output
They hurt expressiveness, both in communicating what is being asserted, and in how you present failures
They make it easy to write imprecise tests
They duplicate features already found in the language, such as evaluation, comparison, and often more

And a few more reasons of my own:

Many assertion libraries are internally inconsistent (testify is one such example)
They require a combinatoric explosion of functions to express slight variants on certain checks, which are much easier expressed using first-class language features such as ! or &&

Then why is `testify` so popular?

If any of these downsides are true, why do assertion libraries remain popular in Go?

In a word: Habit. In many languages, there is no testing framework provided by the language. This means that to do unit testing, you must use a third-party tool, and historical momentum has made xUnit-style testing libraries common. So when folks start learning Go, they naturally carry those habits with them, because it's comfortable.

But Go treats testing as a first-class citizen, shipping with its own testing framework, as well as the testing package, and related libraries, included in the standard library. So the explicit need for a third-party testing framework isn't there. Is there sufficient reason to still prefer a third-party assertion library? Below are the most common arguments in favor, and why they don't really hold up.

"It's easier to read." — While "readability" is mostly subjective, what's readable is mostly what's familiar. If all else is equal, any convention can be readable if you're familiar with it. As we'll discuss later, though, assertion libraries, in general are not equal.
"It's shorter." — While technically true, this is mostly irrelevant. Yes, it makes many assertions shorter, but is that a virtue? Not if it makes them harder to understand.
"It makes complex comparisons easier." — If you're comparing assert.Equal to a long list of individual struct field comparisons, yes, the assert library is the clear winner on all counts: easier to write, easier to read, easier to reason about. But there's an alternative that's better than either of these.

The Go philosophy

Why does Go do tests differently from so many other languages?

It's not a dig at other languages. It's a matter of Go's philosophy. Every language prioritizes certain things over others. And one thing Go prioritizes is making things obvious. Go's notoriously verbose error handling is a good example of this principle in action. You've no doubt written if err != nil more times in the last year than you've written your own postal code in your entire lifetime. And while this is an endless source of annoyance for some, it exists for a reason: It makes it obvious when an error might occur, and how it's being handled. The common alternative of exceptions, by comparison, makes problems opaque.

Assertion libraries are to Go's testing philosophy what exceptions are to Go's error handling.

An idiomatic Go test

Let's pause to look at what an idiomatic Go test would actually look like, as a point of reference for the rest of this discussion. Here's what the test from the quiz above might look like, written using only the testing library:

    var x []int
    y := []int{}
    if !slices.Equal(x, y) {
        t.Error("expected x and y to be equal")
    }

And, of course, you'll be forgiven if you think this looks less "readable" than the assert version above. It certainly is more verbose. But it gives us several improvements over the testify-based version. Let's talk about them.

"Readability" FTW

It's time for the big reveal: the assert.Equal(x, y) call from the quiz will fail. Testify treats empty and nil slices as unequal.

In contrast, slices.Equal(x, y) returns true, so that version of the test will pass. That's different! Why?

Well, while assert.Equal's precise behavior isn't explicitly documented, slices.Equal's is (emphasis added):

func Equal
func Equal[S ~[]E, E comparable](s1, s2 S) bool
Equal reports whether two slices are equal: the same length and all elements equal. If the lengths are different, Equal returns false. Otherwise, the elements are compared in increasing index order, and the comparison stops at the first unequal pair. Empty and nil slices are considered equal. Floating point NaNs are not considered equal.

Now as I said at the top of the post, I can imagine arguments both for and against nil and empty slices being considered equal. And I don't know exactly why the Go team decided that slices.Equal should treat them as equal—though I suppose that in the context of slice comparison, for practical purposes, they are generally equal. But I do know that I use the slices package all the time in my code, and knowing how slices.Equal works is valuable information regardless of how I write my tests. If I use assert.Equal in my tests, I have to learn a second set of rules just for my tests.

Comparing assert.Equal to slices.Equal might seem like a nitpick. And if that were the only issue with using an assertion library, it probably would be. Here are a few other quiz questions for you:

Does calling Contains on a map check against keys, values, or both? (answer: keys)
What's the difference between Equal and EqualValues? (answer: EqualValues coerces values for comparison)
How does Empty treat bool values? (answer: zero values are considered empty, so false is "empty")

`testify` is a whole new language

Of course, testify doesn't expose just one equality assertion function. Let's take a quick look at the other equality functions included in the package:

Equal
EqualError
EqualExportedValues
EqualValues

And most of these have Not* and *f variants, for a total of 12 equality assertion functions.

Oh, but I forgot these:

Exactly
False
Nil
Same
True
Zero

These are also equality assertions by a different name. That makes a total of 30 functions that all handle subtly different senses of exact equality—effectively wrapping either == or !=. And that's to say nothing of the over 100 other assert functions in the library, many of which directly mirror functionality built into the standard library, with slightly different calling conventions; Contains, DirExists, and LessOrEqual to name a few.

And the language doesn't even agree with itself. Equal takes (expected, actual). EqualError takes (actual, expected). Both are asserting that two things are equal. There's no design reason for the flip. You just have to memorize it, function by function.

That's a lot of assertion functions to either memorize, or be constantly looking up, every time you write or read a test. Is it really easier to read something that requires looking up vocabulary in the dictionary all the time?

Okay, but what about complex data structures?

Nobody wants to read or write this test:

    if person.Name != "Alice" {
        t.Errorf("Expected Name = Alice, got %s", person.Name)
    }
    if person.Age != 30 {
        t.Errorf("Expected Age = 30, got %d", person.Age)
    }
    if person.Email != "alice@example.com" {
        t.Errorf("Expected Email = alice@example.com, got %s", person.Email)
    }
    if !person.CreatedAt.Equal(time.Date(2020, time.January, 1, 0, 0, 0, 0, time.UTC)) {
        t.Errorf("Expected CreatedAt = 2020-01-01, got %s", person.CreatedAt)
    }

This is one of the most common objections I hear when presenting about avoiding assertion libraries. "So the Go wiki says to avoid assert libraries, but I don't want to write 50 lines of boilerplate code to test my structs!" Fair criticism! But the Go wiki also says something else:

If your function returns a struct, don’t write test code that performs an individual comparison for each field of the struct. Instead, construct the struct that you’re expecting your function to return, and compare in one shot using diffs or deep comparisons. The same rule applies to arrays and maps.

That sounds like a valuable thing that assertion libraries actually provide! But can we get that value without the weight of an assertion library? Sure! cmp.Diff is the tool that the Go wiki suggests for just this problem. And not only does it replace the one-shot, deep-equality checks of most assertion libraries, it does it with a lot more flexibility!

cmp.Diff is part of the github.com/google/go-cmp/cmp package, from Google. It's not part of the standard library (at least not yet, though it has been considered for inclusion), but it is from Google, and is officially recommended by the Go team.

So how does it work? Let's illustrate by revisiting our original quiz example one more time:

    var x []int
    y := []int{}
    if d := cmp.Diff(x, y); d != "" {
        t.Errorf("expected x and y to be equal, diff: %s", d)
    }

In this version of the test, cmp.Diff returns a textual diff representation of the difference between the two arguments. In this case, it returns:

  []int(
-   nil,
+   {},
  )

The - line is the first argument (x, our nil slice), and the + line is the second (y, the empty slice). One line, one type label, one difference called out. So now we're back to testify's semantics—treating nil and empty as distinct. But remember when I said cmp.Diff is flexible? We can instruct it to treat empty and nil slices as equal, if that's the correct behavior for our application. Just one line changes:

    if d := cmp.Diff(x, y, cmpopts.EquateEmpty()); d != "" {

And beyond that, the cmpopts package provides the ability to ignore specific fields, do approximate time-matching, and a ton of other useful options. It even lets you write your own custom comparison functions, when you really need to get into the details.

The exception to the rule

Having made the case against using testify, I have a confession to make: I was writing tests with testify just this morning.

That's because there's one specific case where I see using testify, or other assertion libraries, as justifiable: You're writing code for a team whose established convention dictates it.

One of my clients uses testify for their tests. Now, I could use the idiomatic Go approach when adding tests to their codebase. And maybe the individual tests would be better as a result. But the project as a whole would suffer. Rather than a single, coherent testing style, there would be two conflicting styles. If my goal is to reduce the burden of reading tests, consistently using testify is clearly better than using an alternative half of the time. That's what I was doing this morning.

Unless the team has decided to migrate from one test style to another, I'd rather adhere to established convention.

Going deeper

Writing idiomatic tests in Go is a lot more than just avoiding testify. How do you avoid the overuse of mocks? How do you test code that calls a live, third-party service? How can you avoid hard-coded sleeps when testing concurrency? Is there any way to avoid flaky tests that depend on the clock? If these are topics you'd like me to help you explore, I'd like to invite you to join my live 6-week course, Idiomatic Testing in Go, starting May 5. It's a small group: 15 seats, weekly 90-minute sessions, and I review your actual test code each week. Early bird enrollment ends April 28.

See the syllabus and reserve your seat.

Three things that can go before a package clause in Go

Jonathan Hall — Mon, 14 Oct 2024 23:09:12 +0000

This post is an excerpt from my email list, Boldy Go: Daily. Join for more content like this every day.

If you’ve done any Go coding at all, you’ve seen at least one package clause:

package main

And you probably know that it comes at the top of each Go source file.

But did you know there are three other things that may come first? Let's look at all three:

Comments, including GoDoc. It's good practice to include a descriptive GoDoc paragraph immediately before the package clause (i.e. with no blank lines between). You need only do this in a single file (in case of a multi-file package).
```
// Package foo Foos all the Bars.
package foo
```
Build constraints and other directives (these are treated as comments, but are worth mentioning separately).
```
//go:build !js

package foo
```
In this example, we tell the compiler to ignore this file when building for a JS target (i.e. WebAssembly or GopherJS).
A shebang line (only as the very first line of the file).
```
#!/usr/local/bin/go run
package main
```
What?!

Didn't know that was possible?

This actually isn't a Go thing at all, it's more a shell thing. But all common Unix/Linux shells will recognize such a line, and interpret it as the name of an interpretor to pass the file to. You really should not ever need this in Go, but some folks have fun writing Go for shell-script types of tasks. If that describes you, this will apply. I included this here mostly for completeness sake, not as an endorsement of this approach. 😉

What if we put all three together, just for fun?

#!/usr/local/bin/go run

// this comment does nothing special. It's not GoDoc, becuase
// there's no package clause or declaration on the next line,
// and it's not a build constraint.

//go:build !js

//go:generate /path/to/utility

// Package main is a utility that does something interesting.
package main

Don't panic!

Jonathan Hall — Thu, 28 Mar 2024 08:56:18 +0000

This post comes from my daily email list: Boldly Go! Daily. Sign up to learn a bit more about Go every day!

Don’t panic.

Don’t panic is such ubiquitous Go advice that it’s one of the famous Go proverbs.

So why do we have a built-in panic function if we’re not supposed to use it?

Well, it’s not so much that it’s never appropriate to use, as much as it’s often heavily overused. Especially by programmers who are accustomed to using exceptions and try/catch blocks to manage control flow.

Don’t do that!

At the moment I’m helping one client with a massive refactor of their application to remove the problematic, non-idiomatic pattern of using panic rather than proper Go error handling.

I’m sure we’ll talk a lot more about this in the future, but probably the most important reason not to panic is that it’s very non-obvious. This problem exists in languages that uses exceptions for error handling, too (and is why Go’s design doesn’t encourage this pattern). Consider this code (taken from the above mentioned client’s code base, with names edited to protect the innocent):

func CreateRecordWithIDAndType(ctx context.Context, id model.ID, typ model.ResourceType) {
    userID := appctx.AuthenticatedUserIdOptional(ctx)
    if userID.Valid() {
        recordReq := &RecordRequest{
            ActionRequest: ActionRequest{
                Type: learnType,
                Id:   id,
            },
        }

        CreateRecordWithCDP(ctx, recordReq)
    }
}

Here’s the question: Can the call to CreateRecordWithCDP result in an error condition?

It’s impossible to be sure. Although we have one strong clue: It accepts a context.Context value, which is often, though not always, an indication that the function call may block, and be terminated early if the context is cancelled. This would suggest that an error state is possible. Although the context could just be used to read a value, in which case it may legitimately not be possible for CreateRecordWithCDP to result in an error condition. It also doesn’t return anything, so whatever it’s “creating”, presumably is stored somewhere–probably in a database. And database operations, as a rule, can err.

In any case, since we don’t know, we’re forced to either investigate CreateRecordWithCDP (and any functions it calls) to determine if it might panic, or just play it safe, and add a deferred recover before we call it.

Wouldn’t it be much easier if we had some obvious indication? Perhaps like this new version?

func CreateRecordWithIDAndType(ctx context.Context, id model.ID, typ model.ResourceType) error {
    userID := appctx.AuthenticatedUserIdOptional(ctx)
    if userID.Valid() {
        recordReq := &RecordRequest{
            ActionRequest: ActionRequest{
                Type: learnType,
                Id:   id,
            },
        }

        return CreateRecordWithCDP(ctx, recordReq)
    }
  return nil
}

This make the fact that calling CreateRecordWithCDP (and in turn CreateRecordWithIDAndType) can result in an error condition. And it makes programming to use these functions much simpler, and eliminates a lot of guesswork.

Of course, this does lead to the situation many complain about, of cluttering your code with if err != nil { ... }. But IMO, while this is a bit annoying to type, typing is a very small price to pay for the clarity it provides.

Error handling in Go web apps shouldn't be so awkward

Jonathan Hall — Wed, 10 Jan 2024 15:18:21 +0000

In this post I’m going to describe an error-handling pattern I’ve found to be fairly elegant when writing REST, gRPC, or other services in Go. I have three goals in writing this post:

To explain the pattern I’ve implemented for a few clients, so that others developing on the same codebase will understand it.
To give others a pattern they may wish to implement in their own applications.
To solicit feedback. Is there a better pattern out there I haven’t seen yet? Are there tweaks I can make to this pattern to make it better?

The problem

Before I explain the pattern I use, let me explain what it replaces, so that we might understand the problems it's meant to solve.

Let's look at a simple HTTP handler, using the standard library's HandlerFunc pattern, which simply fetches a widget record from the database, and serves it to the client as JSON

func (s *Service) GetWidget(w http.ResponseWriter, r *http.Request) {
    if err := r.ParseForm(); err != nil {
        http.Error(w, err.Error(), http.StatusBadRequest)
        return
    }
    id, err := strconv.Atoi(r.Form.Get("widget_id"))
    if err != nil {
        http.Error(w, err.Error(), http.StatusBadRequest)
        return
    }
    widget, err := s.db.GetWidget(id)
    if err != nil {
        if errors.Is(err, sql.ErrNoRows) {
            http.Error(w, err.Error(), http.StatusNotFound)
            return
        }
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    widgetJSON, err := json.Marshal(widget)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    w.Header.Set("Content-Type", "application/json")
    w.Write(widgetJSON)
}

While this should be a more or less realistic example, it's intentionally over-simplified from what I typically find in production services. In particular, I've never seen http.Error used in a real service. Much more likely, you'll have a custom error format you want to send back. Possibly utilizing a JSON error response, with some additional error context or internal error codes, etc. Or maybe you want to render the error as HTML. In any case, I'll assume your app replaces the http.Error() call with something more sophisticated. Which likely means your code is even more annoying and repetitive than what I've shown above.

That aside, let me call out a few specific problems I see with the above code:

The error handling is repetitive, and non-idiomatic. Go is (in)famous for its if err != nil { return err } idiom. Yet we can't even use that here, because the HandlerFunc signature doesn't return an error. Instead, for every error, we must (a) serve the error, and separately (b) return.
We must explicitly handle the HTTP status for every error case. If you have dozens or hundreds of handlers (and you probably do), this quickly becomes repetitive, and error-prone. There's no DRY here. In a single handler like this, maybe it's not a big deal. But it would be nice if we had some sort of default HTTP status code for an error—probably 500 / Internal Server Error.
This handler has to concern itself with database internals. In particular, it checks whether we received a sql.ErrNoRows error. The HTTP handler should be completely database agnostic, so this detail should not need to be exposed here. This is some ugly tight-coupling we can get rid of.

What if, instead...

What if, instead:

for every error, we could simply return err, and the right thing would happen? The error would be rendered to the proper format, and sent to the user?
the magic that renders the error would know the proper HTTP status to set, too? 400 for invalid input, 404 for not found, 401 for unauthorized access, etc?
the data store, whether an SQL database, or MongoDB, or the filesystem, would just tell us "this error means not found", and that could be automatically converted to a 404 instead of the handler knowing the implementation details?

The pattern I'm about to describe gives us all of these things. Not only that, but it enables a number of other emergant patterns which are quite powerful as well. I'll mention some of them at the end, and may write more extensively about some of them later on (let me know if that would interest you).

Idiomatic error handling

The three behaviors I've described that we want all depend on two things, the first of which is "idiomatic error handling". We need to be able to simply return err in our handlers. Unfortunately, the standard libray doesn't give us this. But some third-party frameworks do. The most popular one I'm familiar with is labstack echo, whose HandlerFunc looks like this:

type HandlerFunc func(c Context) error

But I don't believe you should need to adopt a heavy framework like Echo just to get handy error-handling primitives. So you can do this yourself. Here's a simple adapter function pattern you can use:

// customHandler converts an error-returning handler to a standard http.HandlerFunc.
func customHandler(f func(http.ResponseWriter, *http.Request) error) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        err := f(w, r)
        if err != nil {
            http.Error(w, err.Error(), http.StatusInternalServerError) // Wait, alwyas 500? More on that later
        }
    }
}

With such an adapter function, our earlier handler gets simplified to:

func (s *Service) GetWidget(w http.ResponseWriter, r *http.Request) error {
    if err := r.ParseForm(); err != nil {
        return err
    }
    id, err := strconv.Atoi(r.Form.Get("widget_id"))
    if err != nil {
        return err
    }
    widget, err := s.db.GetWidget(id)
    if err != nil {
        return err
    }
    widgetJSON, err := json.Marshal(widget)
    if err != nil {
        return err
    }
    w.Header.Set("Content-Type", "application/json")
    w.Write(widgetJSON)
}

Of course this requires that we actually use the adapter function when setting up our routes:

mux.Handle("/widget", customHandler(s.GetWidget))

Of course we've also actually broken this endpoint. It now treats all errors as internal server errors. So we'll address that next.

But first, an experiment I'm working on

But before we do, I want to call out an experimental library I'm working on, with the hope it may eventually become an official proposal to the standard library (although I think it's a long shot it would be accepted) to extend the definition of the http.HandlerFunc type to include an optional error return value. The library is gitlab.com/flimzy/httpe, and it adds -WithError variants to http.Handler, http.HandlerFunc, ServeHTTP, and related middlewares. It's based on work I've been using for years with clients, but now living in its own stand-alone library for easy inclusion, if you wish.

If you choose to use this library, the new version of the handler remains unchanged, but in places of calling customHandler, you could do:

import "gitlab.com/flimzy/httpe"

mux.Handle("/widget", httpe.ToHandler(s.GetWidget))

The main advantage to the httpe library over your own custom handler is that it provides support for middleware adapters, and inter-mixing standard and error-enabled handlers, with some behind-the-scenes error propagation. But that's beyond the scope of this post.

How to handle different HTTP statuses

The second thing these improvements depends on is some way to specify an HTTP status. We've observed that while this new handler pattern makes error handling simpler, it also breaks it, by treating all errors as 500 / Internal Server Error (or whatever arbitrary status you set in your customHandler function). Let's address that now.

Errors are interfaces

Recall that in Go, the error type is an interface type, define as:

type error interface {
    Error() string
}

This is powerful, because it means we can create our own custom error types. And what's more, for our purposes, we can extend the error type to include other methods.

We want to take advantage of both of these capabilities to create a custom error type that includes an HTTP status, and add a method to expose that status. Here's the simple custom type we'll be using:

type statusError struct {
    error
    status int
}

Now already this is a "complete" error type. It already satisfies the error interface by virtue of embedding the error type (so its methods are promoted to our type's methods). And it includes a status code. But we need a couple more pieces to make this complete. First, let's add an Unwrap method, to allow errors.Unwrap and the related errors.Is and errors.As, etc, to work properly:

func (e statusError) Unwrap() error {
    return e.error
}

And now we also want to add a method to expose the included status. Strictly speaking, this isn't necessary. You can get at the status code by type-converting an error back to the statusError type with a type assertion, or with the use of errors.Is or errors.As. But it's a bit cubersome, and requires exporting the field (unless your entire application is in a single package--I sure hope that's not the case!) Further, by exposing the status via an interface method, we have the freedom to use multiple implementations of our custom error type, which is something I virtually always do. So let's add that detail:

func (e statusError) HTTPStatus() int {
    return e.status
}

Now, you could name your method whatever you want. I've settled on HTTPStatus, after initially using simply Status(), because it's less ambiguous, but still short enough not to be annoying. You can just as eaisly use any other method (or multiple methods). For example, maybe you want JSONRPCStatus() if you're building a JSON-RPC service. Or if you're building a gRPC service, there's already an interface defined for you: GRPCStatus() *status.Status.

Using our custom error type

Now that we have our statusError type, let's incorporate it into our handler, to un-break our status code handling:

func (s *Service) GetWidget(w http.ResponseWriter, r *http.Request) error {
    if err := r.ParseForm(); err != nil {
        return statusError{error: err, status: http.StatusBadRequest}
    }
    id, err := strconv.Atoi(r.Form.Get("widget_id"))
    if err != nil {
        return statusError{error: err, status: http.StatusBadRequest}
    }
    widget, err := s.db.GetWidget(id)
    if err != nil {
        return statusError{error: err, status: http.StatusInternalServerError}
    }
    widgetJSON, err := json.Marshal(widget)
    if err != nil {
        return statusError{error: err, status: http.StatusInternalServerError}
    }
    w.Header.Set("Content-Type", "application/json")
    w.Write(widgetJSON)
}

So now we've (mostly) unbroken our status codes. The one exception is the database call. We treat all errors as status 500, when we should treat a missing widget as 404. The solution here is to make our data access layer aware of these new error types:

func (db *DB) GetWidget(id int) (*Widget, error) {
    widget, err := db.Get(/* ... */)
    if errors.Is(err, sql.ErrNoRows) {
        return nil, statusError{error: err, status: http.StatusNotFound}
    }
    if err != nil {
        return nil, statusError{error: err, status: http.StatusInternalServerError}
    }
    return widget, nil
}

One last thing: We need to update our customHandler to understand this new error type:

// customHandler converts an error-returning handler to a standard http.HandlerFunc.
func customHandler(f func(http.ResponseWriter, *http.Request) error) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        err := f(w, r)
        if err != nil {
            var status int
            var statusErr interface {
                error
                HTTPStatus() int
            }
            if errors.As(err, &statusErr) {
                status = statusErr.HTTPStatus()
            }
            http.Error(w, err.Error(), status)
        }
    }
}

Okay... so now we're back to a fully-functional widget handler. And we've also decoupled our databaes logic from our HTTP handler. So that's a win.

Further improvments

But our handler is still kind of ugly, with a bunch of repeated customErrror{} structs. We also have both our handler and our data access layer depending on a concrete statusError type. Which isn't even exported, which implies that our data access layer and handler are in the same package. Ick. We really don't want that. So let's move our custom error type to its own package. And we'll add a convenient and descriptive constructor function, as well.

package apperr // Use a descriptive name

type statusError struct {
    error
    status int
}

func (e statusError) Unwrap() error   { return e.error }
func (e statusError) HTTPStatus() int { return e.status }

func WithHTTPStatus(err error, status int) error {
    return statusError{
        error: err,
        status: int,
    }
}

Now our handler can be updated to the slightly more readable:

func (s *Service) GetWidget(w http.ResponseWriter, r *http.Request) error {
    if err := r.ParseForm(); err != nil {
        return apperr.WithStatus(err, http.StatusBadRequest)
    }
    id, err := strconv.Atoi(r.Form.Get("widget_id"))
    if err != nil {
        return apperr.WithStatus(err, http.StatusBadRequest)
    }
    widget, err := s.db.GetWidget(id)
    if err != nil {
        return err // No call to apperr.WithStatus here, as we trust the db has already set the appropriate status code for us
    }
    widgetJSON, err := json.Marshal(widget)
    if err != nil {
        return apperr.WithStatus(err, http.StatusBadRequest)
    }
    w.Header.Set("Content-Type", "application/json")
    w.Write(widgetJSON)
}

Setting a default status

We can do one other big improvement to this setup: Setting a default status.

In fact, you may have noticed that our improved customHandler func has no default status. This means that if we pass it an error that doesn't include an HTTP status, it will try to serve an HTTP response with status of 0. Probably not ideal.

Let's solve this problem by adding a helper function to our apperr package, which can also be used from other places:

package apperr

// HTTPStatus returns the HTTP status included in err. If err is nil, this
// function returns 0. If err is non-nil, and does not include an HTTP status,
// a default value of [net/http.StatusInternalServerError] is returned.
func HTTPStatus(err error) int {
    if err == nil {
    return 0
    }
    var statusErr interface {
        error
        HTTPStatus() int
    }
    if errors.As(err, &statusErr) {
        return statusErr.HTTPStatus()
    }
    return http.StatusInternalServerError
}

With this new function in our pocket, our custom handler can be simplified and improved:

// customHandler converts an error-returning handler to a standard http.HandlerFunc.
func customHandler(f func(http.ResponseWriter, *http.Request) error) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        err := f(w, r)
        if err != nil {
            http.Error(w, err.Error(), apperr.HTTPStatus(err))
        }
    }
}

And our handler can also be improved by omitting the call to apierr.WithStatus when we want the default status of 500 / Internal Server Error.

Further improvements

This is really only the beginning of what using an app-wide standard error extension and matching handlers can provide.

The next areas I usually improve on apps where I implement this pattern is to add a couple standard middlewares:

Logging middleware

A middleware to log, with an associated error, if relevant, is a great addition. And it eliminates the "need" to log an error every time it happens--just pass it to your caller, and let the middleware log it for you. Here's a simplified example, using the function signatures in gitlab.com/flimzy/httpe:

func loggingMiddleware(logger *slog.Logger) func(next httpe.HandlerWithError) httpe.HandlerWithError {
    return httpe.HandlerWithErrorFunc(func (w http.ResponseWriter, r *http.Request) error {
        err := next.ServeHTTPWithError(w, r)
        status := http.StatusOK
        if err != nil {
            status = apperr.HTTPStatus(err)
            logger.Error("request served with error", "status", status, "request", r, "error", err)
        } else {
            logger.Info("request served", "status", status, "request", r)
        }
    })
}

Keep in mind that a handler can still call w.WriteHeader with a status distinct from that contained in err (or even in the absence of an error). So a robust implementation will check for that as well.

Error-serving middleware

An improvement over the customHandler function shown above, is to move the error serving into a middleware. This does require the httpe package, or a similar implemenatation, that can work with middlewares.

func serveErrors() func(next httpe.HandlerWithError) httpe.HandlerWithError {
    return httpe.HandlerWithErrorFunc(func (w http.ResponseWriter, r *http.Request) error {
        err := next.ServeHTTPWithError(w, r)
        if err != nil {
            http.Error(w, err.Error(), apperr.Status(err))
        }
    })
}

Panc-recovery middleware

Most web apps have this (or they should!), but a version that works with error-returning handlers can be nice, as it just has to convert panics to errors, rather than serving them directly:

func serveErrors() func(next httpe.HandlerWithError) httpe.HandlerWithError {
    return httpe.HandlerWithErrorFunc(func (w http.ResponseWriter, r *http.Request) (err error) {
        defer func() {
            if r := recover(); r != nil {
                switch t := r.(type) {
                    case error:
                        err = t
                    default:
                        err = fmt.Errorf("%v")
                }
            }
        }()
        return next.ServeHTTPWithError(w, r)
    })
}

Domain-specific error codes

What's better than having your entire application set HTTP statuses on errors all over the place, is to define your own domain-specific error codes. For a small, web-only app, maybe HTTP status codes are sufficient, but in most real-world apps, they aren't. Everything else about this pattern can still be used with your own domain-specific error codes. Simply make your custom error types also return the appropriate HTTP (or JSON-RPC or gRPC or whatever...) codes as well. After such a change, our above database method might look more like:

func (db *DB) GetWidget(id int) (*Widget, error) {
    widget, err := db.Get(/* ... */)
    if errors.Is(err, sql.ErrNoRows) {
        return nil, apperror.ErrWidgetNotFound
    }
    if err != nil {
        return nil, err
    }
    return widget, nil
}

Then the various callers can do their own error inspection, as appropriate:

internalCode := apperror.Code(err) // The internal error code

httpStatus := apperror.HTTPStatus(err) // The HTTP status

I leave the exact implementation of apperror.ErrWidgetNotFound and the associated functions as an exercise for the reader.

Caveats

This approach doesn't come without some drawbacks. It's worth calling a few of them out.

It's non-standard. Obviously. There's a cost in verbosity to be paid to convert from an error-returning handler to a standard handler. Although it seems many people are happy to pay this cost, in the form of adopting a heavy framework that offers this benefit (along with others, of course).
There are now two ways to send responses to the client. This annoys me. But I have yet to find any way around it. And in practice, it doesn't seem to be a big problem, but it does require keeping it in mind. You can set the HTTP status of a resonse either by calling w.WriteHeader(), or by returning an error. Each response can have only a single status, obviously. And if you call w.WriteHeader(), that one generally takes precident (unless you've implemented your own http.ResponseWriter with different behavior).
It makes certain behaviors implicit. Or at least it can. As an example, the shown apperr.HTTPStatus() function returns a default status for errors that don't contain a status. While I believe this makes good sense, and is a benefit, it is a bit "magical", and may surprise someone who's not familiar with the pattern at play. It can also be confusing to see apperr.WithStatus(err, http.StatusNotFound) the first time. While it should be apparent upon plain reading that it's including an HTTP status with an error, it's not apparent what other code consumes that status, or how its used. Of course, the purpose of this post is to help solve this drawback.

Other limitations

This is by no means a one-size-fits-all solution. A couple of obvious limitations I have run into on various applications:

It doesn't provide any ergonomic way to specify a non-200, non-error status (such as 201). For this, you still must fall back on w.WriteHeader().
Some applications would be better served with a function signature such as func(*http.Request) (any, error), such that a response (likely to be renderd as JSON) is the first return argument.

What's next?

I mentioned at the outset that this pattern tends to lend itself to additional improvements. Let me just mention a few, without going into detail here. If you'd like a longer explanation on any of these, let me know.

Include additional metadata in errors. Stack traces are an obvious one, which is nicely provided by github.com/pkg/errors, for example. Expand your logging middleware to extract the stack trace and include it in logs.
Hide error messages for certain HTTP statuses. I typically write my error-serving middleware to return simply Internal Server Error to the client any time there's a 500 status, to avoid the risk of potentially reporting sensitive information, which can happen in some unsanitized errors. HTTP statuses 401 and 403 are also good candidates for this.
Similar to obscuring certain errors, maybe you want to expose a user-friendly version of an error message to the users of your app, while logging the nitty gritty details that the error originally included. Add a Public() string method to such errors, and send the Public() version to your users, and the Error() version to your logs.
HTTP statuses not detailed enough for you? Maybe you need to distinguish between widget not found and user not found? You can create your own internal error status/codes, for use internally, which resolve to a common HTTP status.
Look for ways to DRY up your code. For example, from the handler example, consider moving the call to r.ParseForm and strconv.Atoi to a common function—or use a validation library such as github.com/go-playground/validator/v10 in place of strconv calls—which returns an error with 400 status. Then your handler can just pass that error through.

What other use cases have you seen, or can you think of? I'd love to hear from you.

Is there a better way?

I don't know.

I'm always on the lookout for a better way to do things.

If you know of a better pattern, or even small ways I can improve on this pattern, please, let me know! I'd love to learn from you!

Banner image by ian munroe, CC-BY-2.0

I'm going back to school, Live

Jonathan Hall — Mon, 16 Oct 2023 12:31:50 +0000

I’m going back to school. And you’re invited along.

Specifically, I’m preparing to go through boot.dev’s online course, “Learn Web Servers”.

Boot.dev is building the smartest way to master backend development by tackling the hardest problem with e-learning: boredom. The platform uses tactics learned from modern game design to push ambitious students toward their goal: a career in backend development. Through hands-on lessons that balance theory with practice, and an active Discord community, students are prepared to tackle technical interviews and forge a successful programming career.

It offers courses in Python, a wee bit of JavaScript, and most important, obviously… Go!

If you’re new to Go or web servers, this will be an easy introduction to the concepts, and a chance to ask questions as you watch me make mistakes.

Save the link, and be sure to subscribe to the Boldly Go youtube channel. I'll be going live in just a half an hour!

Or if you're reading this in the future, of course you can still watch the replay.

I hope to see you there!

Resolving type parameter ambiguities

Jonathan Hall — Thu, 03 Aug 2023 07:41:38 +0000

I stumbled upon a bit of a WTF in the Go Specification, while writing my daily Go email and had to dig in a bit to make sense of it:

Type parameter declarations

…

A parsing ambiguity arises when the type parameter list for a generic type declares a single type parameter P with a constraint C such that the text P C forms a valid expression:
type T[P *C] …
type T[P (C)] …
type T[P *C|Q] …
…

If you’re up for it, I’ll try to break this thing down. If you’re not up for it, no worries. See you later! 👋😆

Let’s consider the first example provided: type T[P *C] …. How can this be ambiguous?

Well, it looks like an array type declaration, where the array length is derrived from a constant expression, such as in this example:

const P = 3
const C = 4

type T [P * C]int // T is of type [12]int

Other expressions are possible, too, which is what leads to this possible ambiguity.

So, we need a way to solve this WTF scenario, and Go gives us two of them…

In these rare cases, the type parameter list is indistinguishable from an expression and the type declaration is parsed as an array type declaration. To resolve the ambiguity, embed the constraint in an interface or use a trailing comma:
type T[P interface{*C}] …
type T[P *C,] …

So either wrap your constraint in interface{…}, or add a magical comma at the end. Poof! problem solved.

But still. WTF?!

Building a Go linter from scratch

Jonathan Hall — Mon, 31 Jul 2023 08:47:50 +0000

🤔 Ever wonder how linters work in Go? ❓

In my latest video, I go through Denis Isaev's tutorial "Writing Useful go/analysis Linter", to learn how to build a linter myself to learn what's involved. Follow along and see if you can learn something, too...

The 10 best Go versions, ranked

Jonathan Hall — Thu, 27 Jul 2023 07:29:48 +0000

Go 1.21 is just around the corner, which will be the 22nd major Go release to date. So I thought it would be a good time to look back on the best Go releases we've had to date.

Go Lexical elements: Rune literals pt 3

Jonathan Hall — Sun, 09 Jul 2023 06:00:00 +0000

Let’s continue our disection of rune literals. If you missed the parts, check them out from Friday when we discussed Unicode, and yesterday when we discussed quoting single characters.

Today we’re looking at the various escape sequences supported by the rune literal syntax.

Rune literals

Several backslash escapes allow arbitrary values to be encoded as ASCII text. There are four ways to represent the integer value as a numeric constant: \x followed by exactly two hexadecimal digits; \u followed by exactly four hexadecimal digits; \U followed by exactly eight hexadecimal digits, and a plain backslash \ followed by exactly three octal digits. In each case the value of the literal is the value represented by the digits in the corresponding base.

Although these representations all result in an integer, they have different valid ranges. Octal escapes must represent a value between 0 and 255 inclusive. Hexadecimal escapes satisfy this condition by construction. The escapes \u and \U represent Unicode code points so within them some values are illegal, in particular those above 0x10FFFF and surrogate halves.

Let’s take these one at a time.

A single octal byte — \OOO

You'll probably never use this, so let's get it out of the way first. But it's allowed. You can specify a byte using octal notation. However, note that as described, this is limited to values 0-255 inclusive, which means you can create an invalid rune representation this way:

  var x = rune('\400') // # 400 octal == 256 decimal

Produces the following error:

  octal escape value 256 > 255

One, two, or four hexidecimal bytes — \xXX, \uXXXX, \UXXXXXXXX

This allows you to a single byte with two hexidecimal digits, (\xXX), two bytes with four digits (\uXXXX), or the full 4 bytes of a rune with eight hexidecimal digits (\UXXXXXXXX).

And finally, there are some special escape sequences supported for rune literals:

After a backslash, certain single-character escapes represent special values:

\a   U+0007 alert or bell
\b   U+0008 backspace
\f   U+000C form feed
\n   U+000A line feed or newline
\r   U+000D carriage return
\t   U+0009 horizontal tab
\v   U+000B vertical tab
\\   U+005C backslash
\'   U+0027 single quote  (valid escape only within rune literals)
\"   U+0022 double quote  (valid escape only within string literals)

(That last one arguably doesn't belong here, as it's not valid in a rune literal, but it's nice to know that it's explicitly excluded here.)

Let's round out today's email with the rest of the rune literal section, which is just the boring EBNF syntax, and some examples, which we don't need to discuss in any detail.

An unrecognized character following a backslash in a rune literal is illegal.

rune_lit         = "'" ( unicode_value | byte_value ) "'" .
unicode_value    = unicode_char | little_u_value | big_u_value | escaped_char .
byte_value       = octal_byte_value | hex_byte_value .
octal_byte_value = `\` octal_digit octal_digit octal_digit .
hex_byte_value   = `\` "x" hex_digit hex_digit .
little_u_value   = `\` "u" hex_digit hex_digit hex_digit hex_digit .
big_u_value      = `\` "U" hex_digit hex_digit hex_digit hex_digit
                           hex_digit hex_digit hex_digit hex_digit .
escaped_char     = `\` ( "a" | "b" | "f" | "n" | "r" | "t" | "v" | `\` | "'" | `"` ) .

'a'
'ä'
'本'
'\t'
'\000'
'\007'
'\377'
'\x07'
'\xff'
'\u12e4'
'\U00101234'
'\''         // rune literal containing single quote character
'aa'         // illegal: too many characters
'\k'         // illegal: k is not recognized after a backslash
'\xa'        // illegal: too few hexadecimal digits
'\0'         // illegal: too few octal digits
'\400'       // illegal: octal value over 255
'\uDFFF'     // illegal: surrogate half
'\U00110000' // illegal: invalid Unicode code point

Go Lexical elements: Rune literals pt 2

Jonathan Hall — Sat, 08 Jul 2023 06:00:00 +0000

I write a daily email about Go, and this is a repost from my series Exploring the Go Spec. You're invited to sign up and follow along.

Let’s continue our exploration of rune literals, which began on Monday. In summary from yesterday, a rune in Go represents a Unicode code point. Continuing from there…

Rune literals

A rune literal is expressed as one or more characters enclosed in single quotes, as in 'x' or '\n'. Within the quotes, any character may appear except newline and unescaped single quote. A single quoted character represents the Unicode value of the character itself, while multi-character sequences beginning with a backslash encode values in various formats.

The simplest form represents the single character within the quotes; since Go source text is Unicode characters encoded in UTF-8, multiple UTF-8-encoded bytes may represent a single integer value. For instance, the literal 'a' holds a single byte representing a literal a, Unicode U+0061, value 0x61, while 'ä' holds two bytes (0xc3 0xa4) representing a literal a-dieresis, U+00E4, value 0xe4.

There are a few things I want to call out about this section of the spec, that aren’t always obvious, or are easily forgotten. Especially if you’re not already very familiar with Unicode.

A rune represents a single “character” (technically: unicode code point, see yesterday's discussion).
A rune is not a single byte. (In fact, rune is an alias for int32, so it’s actually 4 bytes)
A rune is not necissarily a single visible character, as many visible characters are built by combining multiple codepoints.

As pointed out in the spec, both 'a' and 'ä' are valid rune literals. The first also corresponds to a single ASCII (or Unicode) byte: 0x61. The second corresponds to two UTF-8 bytes: 0xc3, 0xa4. So it’s immediately clear that a rune may contain multiple bytes.

But recall the example from yesterday as well: 'ў' is a valid rune literal, and represents two bytes: 0xd1, 0x9e. But in contrast, the visually identical 'ў' is not a valid rune literal, because it contains two Unicode code points, each of two bytes: у (0xd1, 0x83) followed by the breve mark, ˘, (0xcc, 0x86).

As you might expect, this can be an easy place to get tripped up. What you see on the screen is quite frequently not the whole story. I know of no fool-proof way to solve this confusion. The best I know is to be aware that the confusion exists, so when you see an error along the lines of “more than one character in rune literal”, you know where to begin your search.

Go Lexical elements: Rune literals pt 1, Intro to Unicode

Jonathan Hall — Fri, 07 Jul 2023 14:12:36 +0000

I write a daily email about Go, and this is a repost from my series Exploring the Go Spec. You're invited to sign up and follow along.

Runes… Oh boy! This is one of bits of Go that shines for its elegant simplicity, but constantly trips up everyone (myself included). As such, I think this may be a 2-, or maybe even a 3-parter.

Let’s get started.

Rune literals

A rune literal represents a rune constant, an integer value identifying a Unicode code point.

If you’re already familiar with Unicode, and have a strong understanding of what a “code point” is, you can probably skip this one. See you later! 👋

If you’re not completely sure what a Unicode code point is, stick around… I’ll do my best to untangle this seemingly simple phrase.

First off, what is Unicode? It’s emojis, right? Ehh. If that’s your understanding of Unicode, which would be completely understandable, if your career in software is relatively young, then you need a bit more context. I encourage you to read a bit about the history of why Unicode was invented, and the problems it was meant to solve. This PDF, Introduction to Unicode: History of Character Codes is a good starting point.

But here are the highlights:

Before Unocide, we had many different coding systems. The most popular was ASCII, the American Standard Code for Information Interchange. But since not all of the world is American, this had obvious drawbacks. Different countries or languages would often have their own coding schemes, but this made it very difficult to share documents between regions.

If I wrote a document using the Cyrillic alphabet, then sent it to my colleague in France, he would likely see a jumble of French letters in seemingly random order.

So Unicode came along to Unify all the codes.

Great. So now instead of 127 possible characters in ASCII, we have a virtually unlimited number of characters, right?

Not so fast.

While there’s room in Unicode for more than 1 million individual code points, most are not (yet) defined. But what’s more, Unicode is smarter than ASCII in a number of ways. It is possible to combine Unicode code points to form a single physical character.

For example, if you want to display the Cyrillic letter ў, this can be done by combining the Cyrillic Y (у) with the breve mark (˘), to give you ў. But while this looks like a single character, and in print terms it is, it’s actually two Unicode codepoints. As such, this code won’t compile:

x := rune('ў')

Because while it visually looks like we’re quoting a single character, that character is composed of two Unicode codepoints, and (as we’ll see in the next section), a rune literal must be a single Unicode codepoint.

Now if that’s not confusing enough, this code will compile:

x := rune('ў')

What’s the difference?

Well, Unicode includes a number of precomposed characters. This is a nice convenience for languages that commonly use a large number of these types of diacritics. But it’s an incovenience for us programmers. Not only does it mean that of these two character representations, only one is a valid rune, it also introduces certain headaches when trying to compare Unicode strings for equality, or when sorting, etc.

A last note for today, especially for anyone very new to Unicode. This concept of combining characters to add diacritics and other markings to an existing letter is the same way that Unicode emojis are modified to change skin tone, gender, or other attributes.

Unicode is pretty powerful. And confusing at times.

Guess-Driven Development

Jonathan Hall — Tue, 04 Jul 2023 00:00:00 +0000

Have you ever seen a pull request with half a dozen, seemingly unrelated changes, all in the name of fixing a specific bug?

The last time this happened to me, I requested clarification on the purpose of a number of the changes, suggested a couple small refactors, pointed out that a couple of the changes were unnecessary or even counter-productive…

After a few back-and-forths, the PR went from about 40 lines changed to 2 lines changed. All the unrelated, unnecessary bits had been removed.

This is the result of what I have come to call Guess-Driven Development.

It happens when someone doesn’t really know what’s causing a bug, so they just start changing things, until the bug goes away. Then they commit their entire change set, without much thought.

If you enjoyed this message, subscribe to The Daily Commit to get future messages to your inbox.

Forem: Jonathan Hall

Testify is making your Go tests worse

Is the learning curve worth it?

Then why is testify so popular?

The Go philosophy

An idiomatic Go test

"Readability" FTW

testify is a whole new language

Okay, but what about complex data structures?

The exception to the rule

Going deeper

Three things that can go before a package clause in Go

Don't panic!

Error handling in Go web apps shouldn't be so awkward

The problem

What if, instead...

Idiomatic error handling

But first, an experiment I'm working on

How to handle different HTTP statuses

Errors are interfaces

Using our custom error type

Further improvments

Setting a default status

Further improvements

Logging middleware

Error-serving middleware

Panc-recovery middleware

Domain-specific error codes

Caveats

Other limitations

What's next?

Is there a better way?

I'm going back to school, Live

Resolving type parameter ambiguities

Type parameter declarations

Building a Go linter from scratch

The 10 best Go versions, ranked

Go Lexical elements: Rune literals pt 3

Rune literals

Go Lexical elements: Rune literals pt 2

Rune literals

Go Lexical elements: Rune literals pt 1, Intro to Unicode

Rune literals

Guess-Driven Development

Then why is `testify` so popular?

`testify` is a whole new language