Forem: Jonathan Hall

Three things that can go before a package clause in Go

Jonathan Hall — Mon, 14 Oct 2024 23:09:12 +0000

This post is an excerpt from my email list, Boldy Go: Daily. Join for more content like this every day.

If you’ve done any Go coding at all, you’ve seen at least one package clause:

package main

And you probably know that it comes at the top of each Go source file.

But did you know there are three other things that may come first? Let's look at all three:

Comments, including GoDoc. It's good practice to include a descriptive GoDoc paragraph immediately before the package clause (i.e. with no blank lines between). You need only do this in a single file (in case of a multi-file package).
```
// Package foo Foos all the Bars.
package foo
```
Build constraints and other directives (these are treated as comments, but are worth mentioning separately).
```
//go:build !js

package foo
```
In this example, we tell the compiler to ignore this file when building for a JS target (i.e. WebAssembly or GopherJS).
A shebang line (only as the very first line of the file).
```
#!/usr/local/bin/go run
package main
```
What?!

Didn't know that was possible?

This actually isn't a Go thing at all, it's more a shell thing. But all common Unix/Linux shells will recognize such a line, and interpret it as the name of an interpretor to pass the file to. You really should not ever need this in Go, but some folks have fun writing Go for shell-script types of tasks. If that describes you, this will apply. I included this here mostly for completeness sake, not as an endorsement of this approach. 😉

What if we put all three together, just for fun?

#!/usr/local/bin/go run

// this comment does nothing special. It's not GoDoc, becuase
// there's no package clause or declaration on the next line,
// and it's not a build constraint.

//go:build !js

//go:generate /path/to/utility

// Package main is a utility that does something interesting.
package main

Don't panic!

Jonathan Hall — Thu, 28 Mar 2024 08:56:18 +0000

This post comes from my daily email list: Boldly Go! Daily. Sign up to learn a bit more about Go every day!

Don’t panic.

Don’t panic is such ubiquitous Go advice that it’s one of the famous Go proverbs.

So why do we have a built-in panic function if we’re not supposed to use it?

Well, it’s not so much that it’s never appropriate to use, as much as it’s often heavily overused. Especially by programmers who are accustomed to using exceptions and try/catch blocks to manage control flow.

Don’t do that!

At the moment I’m helping one client with a massive refactor of their application to remove the problematic, non-idiomatic pattern of using panic rather than proper Go error handling.

I’m sure we’ll talk a lot more about this in the future, but probably the most important reason not to panic is that it’s very non-obvious. This problem exists in languages that uses exceptions for error handling, too (and is why Go’s design doesn’t encourage this pattern). Consider this code (taken from the above mentioned client’s code base, with names edited to protect the innocent):

func CreateRecordWithIDAndType(ctx context.Context, id model.ID, typ model.ResourceType) {
    userID := appctx.AuthenticatedUserIdOptional(ctx)
    if userID.Valid() {
        recordReq := &RecordRequest{
            ActionRequest: ActionRequest{
                Type: learnType,
                Id:   id,
            },
        }

        CreateRecordWithCDP(ctx, recordReq)
    }
}

Here’s the question: Can the call to CreateRecordWithCDP result in an error condition?

It’s impossible to be sure. Although we have one strong clue: It accepts a context.Context value, which is often, though not always, an indication that the function call may block, and be terminated early if the context is cancelled. This would suggest that an error state is possible. Although the context could just be used to read a value, in which case it may legitimately not be possible for CreateRecordWithCDP to result in an error condition. It also doesn’t return anything, so whatever it’s “creating”, presumably is stored somewhere–probably in a database. And database operations, as a rule, can err.

In any case, since we don’t know, we’re forced to either investigate CreateRecordWithCDP (and any functions it calls) to determine if it might panic, or just play it safe, and add a deferred recover before we call it.

Wouldn’t it be much easier if we had some obvious indication? Perhaps like this new version?

func CreateRecordWithIDAndType(ctx context.Context, id model.ID, typ model.ResourceType) error {
    userID := appctx.AuthenticatedUserIdOptional(ctx)
    if userID.Valid() {
        recordReq := &RecordRequest{
            ActionRequest: ActionRequest{
                Type: learnType,
                Id:   id,
            },
        }

        return CreateRecordWithCDP(ctx, recordReq)
    }
  return nil
}

This make the fact that calling CreateRecordWithCDP (and in turn CreateRecordWithIDAndType) can result in an error condition. And it makes programming to use these functions much simpler, and eliminates a lot of guesswork.

Of course, this does lead to the situation many complain about, of cluttering your code with if err != nil { ... }. But IMO, while this is a bit annoying to type, typing is a very small price to pay for the clarity it provides.

Error handling in Go web apps shouldn't be so awkward

Jonathan Hall — Wed, 10 Jan 2024 15:18:21 +0000

In this post I’m going to describe an error-handling pattern I’ve found to be fairly elegant when writing REST, gRPC, or other services in Go. I have three goals in writing this post:

To explain the pattern I’ve implemented for a few clients, so that others developing on the same codebase will understand it.
To give others a pattern they may wish to implement in their own applications.
To solicit feedback. Is there a better pattern out there I haven’t seen yet? Are there tweaks I can make to this pattern to make it better?

The problem

Before I explain the pattern I use, let me explain what it replaces, so that we might understand the problems it's meant to solve.

Let's look at a simple HTTP handler, using the standard library's HandlerFunc pattern, which simply fetches a widget record from the database, and serves it to the client as JSON

func (s *Service) GetWidget(w http.ResponseWriter, r *http.Request) {
    if err := r.ParseForm(); err != nil {
        http.Error(w, err.Error(), http.StatusBadRequest)
        return
    }
    id, err := strconv.Atoi(r.Form.Get("widget_id"))
    if err != nil {
        http.Error(w, err.Error(), http.StatusBadRequest)
        return
    }
    widget, err := s.db.GetWidget(id)
    if err != nil {
        if errors.Is(err, sql.ErrNoRows) {
            http.Error(w, err.Error(), http.StatusNotFound)
            return
        }
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    widgetJSON, err := json.Marshal(widget)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    w.Header.Set("Content-Type", "application/json")
    w.Write(widgetJSON)
}

While this should be a more or less realistic example, it's intentionally over-simplified from what I typically find in production services. In particular, I've never seen http.Error used in a real service. Much more likely, you'll have a custom error format you want to send back. Possibly utilizing a JSON error response, with some additional error context or internal error codes, etc. Or maybe you want to render the error as HTML. In any case, I'll assume your app replaces the http.Error() call with something more sophisticated. Which likely means your code is even more annoying and repetitive than what I've shown above.

That aside, let me call out a few specific problems I see with the above code:

The error handling is repetitive, and non-idiomatic. Go is (in)famous for its if err != nil { return err } idiom. Yet we can't even use that here, because the HandlerFunc signature doesn't return an error. Instead, for every error, we must (a) serve the error, and separately (b) return.
We must explicitly handle the HTTP status for every error case. If you have dozens or hundreds of handlers (and you probably do), this quickly becomes repetitive, and error-prone. There's no DRY here. In a single handler like this, maybe it's not a big deal. But it would be nice if we had some sort of default HTTP status code for an error—probably 500 / Internal Server Error.
This handler has to concern itself with database internals. In particular, it checks whether we received a sql.ErrNoRows error. The HTTP handler should be completely database agnostic, so this detail should not need to be exposed here. This is some ugly tight-coupling we can get rid of.

What if, instead...

What if, instead:

for every error, we could simply return err, and the right thing would happen? The error would be rendered to the proper format, and sent to the user?
the magic that renders the error would know the proper HTTP status to set, too? 400 for invalid input, 404 for not found, 401 for unauthorized access, etc?
the data store, whether an SQL database, or MongoDB, or the filesystem, would just tell us "this error means not found", and that could be automatically converted to a 404 instead of the handler knowing the implementation details?

The pattern I'm about to describe gives us all of these things. Not only that, but it enables a number of other emergant patterns which are quite powerful as well. I'll mention some of them at the end, and may write more extensively about some of them later on (let me know if that would interest you).

Idiomatic error handling

The three behaviors I've described that we want all depend on two things, the first of which is "idiomatic error handling". We need to be able to simply return err in our handlers. Unfortunately, the standard libray doesn't give us this. But some third-party frameworks do. The most popular one I'm familiar with is labstack echo, whose HandlerFunc looks like this:

type HandlerFunc func(c Context) error

But I don't believe you should need to adopt a heavy framework like Echo just to get handy error-handling primitives. So you can do this yourself. Here's a simple adapter function pattern you can use:

// customHandler converts an error-returning handler to a standard http.HandlerFunc.
func customHandler(f func(http.ResponseWriter, *http.Request) error) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        err := f(w, r)
        if err != nil {
            http.Error(w, err.Error(), http.StatusInternalServerError) // Wait, alwyas 500? More on that later
        }
    }
}

With such an adapter function, our earlier handler gets simplified to:

func (s *Service) GetWidget(w http.ResponseWriter, r *http.Request) error {
    if err := r.ParseForm(); err != nil {
        return err
    }
    id, err := strconv.Atoi(r.Form.Get("widget_id"))
    if err != nil {
        return err
    }
    widget, err := s.db.GetWidget(id)
    if err != nil {
        return err
    }
    widgetJSON, err := json.Marshal(widget)
    if err != nil {
        return err
    }
    w.Header.Set("Content-Type", "application/json")
    w.Write(widgetJSON)
}

Of course this requires that we actually use the adapter function when setting up our routes:

mux.Handle("/widget", customHandler(s.GetWidget))

Of course we've also actually broken this endpoint. It now treats all errors as internal server errors. So we'll address that next.

But first, an experiment I'm working on

But before we do, I want to call out an experimental library I'm working on, with the hope it may eventually become an official proposal to the standard library (although I think it's a long shot it would be accepted) to extend the definition of the http.HandlerFunc type to include an optional error return value. The library is gitlab.com/flimzy/httpe, and it adds -WithError variants to http.Handler, http.HandlerFunc, ServeHTTP, and related middlewares. It's based on work I've been using for years with clients, but now living in its own stand-alone library for easy inclusion, if you wish.

If you choose to use this library, the new version of the handler remains unchanged, but in places of calling customHandler, you could do:

import "gitlab.com/flimzy/httpe"

mux.Handle("/widget", httpe.ToHandler(s.GetWidget))

The main advantage to the httpe library over your own custom handler is that it provides support for middleware adapters, and inter-mixing standard and error-enabled handlers, with some behind-the-scenes error propagation. But that's beyond the scope of this post.

How to handle different HTTP statuses

The second thing these improvements depends on is some way to specify an HTTP status. We've observed that while this new handler pattern makes error handling simpler, it also breaks it, by treating all errors as 500 / Internal Server Error (or whatever arbitrary status you set in your customHandler function). Let's address that now.

Errors are interfaces

Recall that in Go, the error type is an interface type, define as:

type error interface {
    Error() string
}

This is powerful, because it means we can create our own custom error types. And what's more, for our purposes, we can extend the error type to include other methods.

We want to take advantage of both of these capabilities to create a custom error type that includes an HTTP status, and add a method to expose that status. Here's the simple custom type we'll be using:

type statusError struct {
    error
    status int
}

Now already this is a "complete" error type. It already satisfies the error interface by virtue of embedding the error type (so its methods are promoted to our type's methods). And it includes a status code. But we need a couple more pieces to make this complete. First, let's add an Unwrap method, to allow errors.Unwrap and the related errors.Is and errors.As, etc, to work properly:

func (e statusError) Unwrap() error {
    return e.error
}

And now we also want to add a method to expose the included status. Strictly speaking, this isn't necessary. You can get at the status code by type-converting an error back to the statusError type with a type assertion, or with the use of errors.Is or errors.As. But it's a bit cubersome, and requires exporting the field (unless your entire application is in a single package--I sure hope that's not the case!) Further, by exposing the status via an interface method, we have the freedom to use multiple implementations of our custom error type, which is something I virtually always do. So let's add that detail:

func (e statusError) HTTPStatus() int {
    return e.status
}

Now, you could name your method whatever you want. I've settled on HTTPStatus, after initially using simply Status(), because it's less ambiguous, but still short enough not to be annoying. You can just as eaisly use any other method (or multiple methods). For example, maybe you want JSONRPCStatus() if you're building a JSON-RPC service. Or if you're building a gRPC service, there's already an interface defined for you: GRPCStatus() *status.Status.

Using our custom error type

Now that we have our statusError type, let's incorporate it into our handler, to un-break our status code handling:

func (s *Service) GetWidget(w http.ResponseWriter, r *http.Request) error {
    if err := r.ParseForm(); err != nil {
        return statusError{error: err, status: http.StatusBadRequest}
    }
    id, err := strconv.Atoi(r.Form.Get("widget_id"))
    if err != nil {
        return statusError{error: err, status: http.StatusBadRequest}
    }
    widget, err := s.db.GetWidget(id)
    if err != nil {
        return statusError{error: err, status: http.StatusInternalServerError}
    }
    widgetJSON, err := json.Marshal(widget)
    if err != nil {
        return statusError{error: err, status: http.StatusInternalServerError}
    }
    w.Header.Set("Content-Type", "application/json")
    w.Write(widgetJSON)
}

So now we've (mostly) unbroken our status codes. The one exception is the database call. We treat all errors as status 500, when we should treat a missing widget as 404. The solution here is to make our data access layer aware of these new error types:

func (db *DB) GetWidget(id int) (*Widget, error) {
    widget, err := db.Get(/* ... */)
    if errors.Is(err, sql.ErrNoRows) {
        return nil, statusError{error: err, status: http.StatusNotFound}
    }
    if err != nil {
        return nil, statusError{error: err, status: http.StatusInternalServerError}
    }
    return widget, nil
}

One last thing: We need to update our customHandler to understand this new error type:

// customHandler converts an error-returning handler to a standard http.HandlerFunc.
func customHandler(f func(http.ResponseWriter, *http.Request) error) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        err := f(w, r)
        if err != nil {
            var status int
            var statusErr interface {
                error
                HTTPStatus() int
            }
            if errors.As(err, &statusErr) {
                status = statusErr.HTTPStatus()
            }
            http.Error(w, err.Error(), status)
        }
    }
}

Okay... so now we're back to a fully-functional widget handler. And we've also decoupled our databaes logic from our HTTP handler. So that's a win.

Further improvments

But our handler is still kind of ugly, with a bunch of repeated customErrror{} structs. We also have both our handler and our data access layer depending on a concrete statusError type. Which isn't even exported, which implies that our data access layer and handler are in the same package. Ick. We really don't want that. So let's move our custom error type to its own package. And we'll add a convenient and descriptive constructor function, as well.

package apperr // Use a descriptive name

type statusError struct {
    error
    status int
}

func (e statusError) Unwrap() error   { return e.error }
func (e statusError) HTTPStatus() int { return e.status }

func WithHTTPStatus(err error, status int) error {
    return statusError{
        error: err,
        status: int,
    }
}

Now our handler can be updated to the slightly more readable:

func (s *Service) GetWidget(w http.ResponseWriter, r *http.Request) error {
    if err := r.ParseForm(); err != nil {
        return apperr.WithStatus(err, http.StatusBadRequest)
    }
    id, err := strconv.Atoi(r.Form.Get("widget_id"))
    if err != nil {
        return apperr.WithStatus(err, http.StatusBadRequest)
    }
    widget, err := s.db.GetWidget(id)
    if err != nil {
        return err // No call to apperr.WithStatus here, as we trust the db has already set the appropriate status code for us
    }
    widgetJSON, err := json.Marshal(widget)
    if err != nil {
        return apperr.WithStatus(err, http.StatusBadRequest)
    }
    w.Header.Set("Content-Type", "application/json")
    w.Write(widgetJSON)
}

Setting a default status

We can do one other big improvement to this setup: Setting a default status.

In fact, you may have noticed that our improved customHandler func has no default status. This means that if we pass it an error that doesn't include an HTTP status, it will try to serve an HTTP response with status of 0. Probably not ideal.

Let's solve this problem by adding a helper function to our apperr package, which can also be used from other places:

package apperr

// HTTPStatus returns the HTTP status included in err. If err is nil, this
// function returns 0. If err is non-nil, and does not include an HTTP status,
// a default value of [net/http.StatusInternalServerError] is returned.
func HTTPStatus(err error) int {
    if err == nil {
    return 0
    }
    var statusErr interface {
        error
        HTTPStatus() int
    }
    if errors.As(err, &statusErr) {
        return statusErr.HTTPStatus()
    }
    return http.StatusInternalServerError
}

With this new function in our pocket, our custom handler can be simplified and improved:

// customHandler converts an error-returning handler to a standard http.HandlerFunc.
func customHandler(f func(http.ResponseWriter, *http.Request) error) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        err := f(w, r)
        if err != nil {
            http.Error(w, err.Error(), apperr.HTTPStatus(err))
        }
    }
}

And our handler can also be improved by omitting the call to apierr.WithStatus when we want the default status of 500 / Internal Server Error.

Further improvements

This is really only the beginning of what using an app-wide standard error extension and matching handlers can provide.

The next areas I usually improve on apps where I implement this pattern is to add a couple standard middlewares:

Logging middleware

A middleware to log, with an associated error, if relevant, is a great addition. And it eliminates the "need" to log an error every time it happens--just pass it to your caller, and let the middleware log it for you. Here's a simplified example, using the function signatures in gitlab.com/flimzy/httpe:

func loggingMiddleware(logger *slog.Logger) func(next httpe.HandlerWithError) httpe.HandlerWithError {
    return httpe.HandlerWithErrorFunc(func (w http.ResponseWriter, r *http.Request) error {
        err := next.ServeHTTPWithError(w, r)
        status := http.StatusOK
        if err != nil {
            status = apperr.HTTPStatus(err)
            logger.Error("request served with error", "status", status, "request", r, "error", err)
        } else {
            logger.Info("request served", "status", status, "request", r)
        }
    })
}

Keep in mind that a handler can still call w.WriteHeader with a status distinct from that contained in err (or even in the absence of an error). So a robust implementation will check for that as well.

Error-serving middleware

An improvement over the customHandler function shown above, is to move the error serving into a middleware. This does require the httpe package, or a similar implemenatation, that can work with middlewares.

func serveErrors() func(next httpe.HandlerWithError) httpe.HandlerWithError {
    return httpe.HandlerWithErrorFunc(func (w http.ResponseWriter, r *http.Request) error {
        err := next.ServeHTTPWithError(w, r)
        if err != nil {
            http.Error(w, err.Error(), apperr.Status(err))
        }
    })
}

Panc-recovery middleware

Most web apps have this (or they should!), but a version that works with error-returning handlers can be nice, as it just has to convert panics to errors, rather than serving them directly:

func serveErrors() func(next httpe.HandlerWithError) httpe.HandlerWithError {
    return httpe.HandlerWithErrorFunc(func (w http.ResponseWriter, r *http.Request) (err error) {
        defer func() {
            if r := recover(); r != nil {
                switch t := r.(type) {
                    case error:
                        err = t
                    default:
                        err = fmt.Errorf("%v")
                }
            }
        }()
        return next.ServeHTTPWithError(w, r)
    })
}

Domain-specific error codes

What's better than having your entire application set HTTP statuses on errors all over the place, is to define your own domain-specific error codes. For a small, web-only app, maybe HTTP status codes are sufficient, but in most real-world apps, they aren't. Everything else about this pattern can still be used with your own domain-specific error codes. Simply make your custom error types also return the appropriate HTTP (or JSON-RPC or gRPC or whatever...) codes as well. After such a change, our above database method might look more like:

func (db *DB) GetWidget(id int) (*Widget, error) {
    widget, err := db.Get(/* ... */)
    if errors.Is(err, sql.ErrNoRows) {
        return nil, apperror.ErrWidgetNotFound
    }
    if err != nil {
        return nil, err
    }
    return widget, nil
}

Then the various callers can do their own error inspection, as appropriate:

internalCode := apperror.Code(err) // The internal error code

httpStatus := apperror.HTTPStatus(err) // The HTTP status

I leave the exact implementation of apperror.ErrWidgetNotFound and the associated functions as an exercise for the reader.

Caveats

This approach doesn't come without some drawbacks. It's worth calling a few of them out.

It's non-standard. Obviously. There's a cost in verbosity to be paid to convert from an error-returning handler to a standard handler. Although it seems many people are happy to pay this cost, in the form of adopting a heavy framework that offers this benefit (along with others, of course).
There are now two ways to send responses to the client. This annoys me. But I have yet to find any way around it. And in practice, it doesn't seem to be a big problem, but it does require keeping it in mind. You can set the HTTP status of a resonse either by calling w.WriteHeader(), or by returning an error. Each response can have only a single status, obviously. And if you call w.WriteHeader(), that one generally takes precident (unless you've implemented your own http.ResponseWriter with different behavior).
It makes certain behaviors implicit. Or at least it can. As an example, the shown apperr.HTTPStatus() function returns a default status for errors that don't contain a status. While I believe this makes good sense, and is a benefit, it is a bit "magical", and may surprise someone who's not familiar with the pattern at play. It can also be confusing to see apperr.WithStatus(err, http.StatusNotFound) the first time. While it should be apparent upon plain reading that it's including an HTTP status with an error, it's not apparent what other code consumes that status, or how its used. Of course, the purpose of this post is to help solve this drawback.

Other limitations

This is by no means a one-size-fits-all solution. A couple of obvious limitations I have run into on various applications:

It doesn't provide any ergonomic way to specify a non-200, non-error status (such as 201). For this, you still must fall back on w.WriteHeader().
Some applications would be better served with a function signature such as func(*http.Request) (any, error), such that a response (likely to be renderd as JSON) is the first return argument.

What's next?

I mentioned at the outset that this pattern tends to lend itself to additional improvements. Let me just mention a few, without going into detail here. If you'd like a longer explanation on any of these, let me know.

Include additional metadata in errors. Stack traces are an obvious one, which is nicely provided by github.com/pkg/errors, for example. Expand your logging middleware to extract the stack trace and include it in logs.
Hide error messages for certain HTTP statuses. I typically write my error-serving middleware to return simply Internal Server Error to the client any time there's a 500 status, to avoid the risk of potentially reporting sensitive information, which can happen in some unsanitized errors. HTTP statuses 401 and 403 are also good candidates for this.
Similar to obscuring certain errors, maybe you want to expose a user-friendly version of an error message to the users of your app, while logging the nitty gritty details that the error originally included. Add a Public() string method to such errors, and send the Public() version to your users, and the Error() version to your logs.
HTTP statuses not detailed enough for you? Maybe you need to distinguish between widget not found and user not found? You can create your own internal error status/codes, for use internally, which resolve to a common HTTP status.
Look for ways to DRY up your code. For example, from the handler example, consider moving the call to r.ParseForm and strconv.Atoi to a common function—or use a validation library such as github.com/go-playground/validator/v10 in place of strconv calls—which returns an error with 400 status. Then your handler can just pass that error through.

What other use cases have you seen, or can you think of? I'd love to hear from you.

Is there a better way?

I don't know.

I'm always on the lookout for a better way to do things.

If you know of a better pattern, or even small ways I can improve on this pattern, please, let me know! I'd love to learn from you!

Banner image by ian munroe, CC-BY-2.0

I'm going back to school, Live

Jonathan Hall — Mon, 16 Oct 2023 12:31:50 +0000

I’m going back to school. And you’re invited along.

Specifically, I’m preparing to go through boot.dev’s online course, “Learn Web Servers”.

Boot.dev is building the smartest way to master backend development by tackling the hardest problem with e-learning: boredom. The platform uses tactics learned from modern game design to push ambitious students toward their goal: a career in backend development. Through hands-on lessons that balance theory with practice, and an active Discord community, students are prepared to tackle technical interviews and forge a successful programming career.

It offers courses in Python, a wee bit of JavaScript, and most important, obviously… Go!

If you’re new to Go or web servers, this will be an easy introduction to the concepts, and a chance to ask questions as you watch me make mistakes.

Save the link, and be sure to subscribe to the Boldly Go youtube channel. I'll be going live in just a half an hour!

Or if you're reading this in the future, of course you can still watch the replay.

I hope to see you there!

Resolving type parameter ambiguities

Jonathan Hall — Thu, 03 Aug 2023 07:41:38 +0000

I stumbled upon a bit of a WTF in the Go Specification, while writing my daily Go email and had to dig in a bit to make sense of it:

Type parameter declarations

…

A parsing ambiguity arises when the type parameter list for a generic type declares a single type parameter P with a constraint C such that the text P C forms a valid expression:
type T[P *C] …
type T[P (C)] …
type T[P *C|Q] …
…

If you’re up for it, I’ll try to break this thing down. If you’re not up for it, no worries. See you later! 👋😆

Let’s consider the first example provided: type T[P *C] …. How can this be ambiguous?

Well, it looks like an array type declaration, where the array length is derrived from a constant expression, such as in this example:

const P = 3
const C = 4

type T [P * C]int // T is of type [12]int

Other expressions are possible, too, which is what leads to this possible ambiguity.

So, we need a way to solve this WTF scenario, and Go gives us two of them…

In these rare cases, the type parameter list is indistinguishable from an expression and the type declaration is parsed as an array type declaration. To resolve the ambiguity, embed the constraint in an interface or use a trailing comma:
type T[P interface{*C}] …
type T[P *C,] …

So either wrap your constraint in interface{…}, or add a magical comma at the end. Poof! problem solved.

But still. WTF?!

Building a Go linter from scratch

Jonathan Hall — Mon, 31 Jul 2023 08:47:50 +0000

🤔 Ever wonder how linters work in Go? ❓

In my latest video, I go through Denis Isaev's tutorial "Writing Useful go/analysis Linter", to learn how to build a linter myself to learn what's involved. Follow along and see if you can learn something, too...

The 10 best Go versions, ranked

Jonathan Hall — Thu, 27 Jul 2023 07:29:48 +0000

Go 1.21 is just around the corner, which will be the 22nd major Go release to date. So I thought it would be a good time to look back on the best Go releases we've had to date.

Go Lexical elements: Rune literals pt 3

Jonathan Hall — Sun, 09 Jul 2023 06:00:00 +0000

Let’s continue our disection of rune literals. If you missed the parts, check them out from Friday when we discussed Unicode, and yesterday when we discussed quoting single characters.

Today we’re looking at the various escape sequences supported by the rune literal syntax.

Rune literals

Several backslash escapes allow arbitrary values to be encoded as ASCII text. There are four ways to represent the integer value as a numeric constant: \x followed by exactly two hexadecimal digits; \u followed by exactly four hexadecimal digits; \U followed by exactly eight hexadecimal digits, and a plain backslash \ followed by exactly three octal digits. In each case the value of the literal is the value represented by the digits in the corresponding base.

Although these representations all result in an integer, they have different valid ranges. Octal escapes must represent a value between 0 and 255 inclusive. Hexadecimal escapes satisfy this condition by construction. The escapes \u and \U represent Unicode code points so within them some values are illegal, in particular those above 0x10FFFF and surrogate halves.

Let’s take these one at a time.

A single octal byte — \OOO

You'll probably never use this, so let's get it out of the way first. But it's allowed. You can specify a byte using octal notation. However, note that as described, this is limited to values 0-255 inclusive, which means you can create an invalid rune representation this way:

  var x = rune('\400') // # 400 octal == 256 decimal

Produces the following error:

  octal escape value 256 > 255

One, two, or four hexidecimal bytes — \xXX, \uXXXX, \UXXXXXXXX

This allows you to a single byte with two hexidecimal digits, (\xXX), two bytes with four digits (\uXXXX), or the full 4 bytes of a rune with eight hexidecimal digits (\UXXXXXXXX).

And finally, there are some special escape sequences supported for rune literals:

After a backslash, certain single-character escapes represent special values:

\a   U+0007 alert or bell
\b   U+0008 backspace
\f   U+000C form feed
\n   U+000A line feed or newline
\r   U+000D carriage return
\t   U+0009 horizontal tab
\v   U+000B vertical tab
\\   U+005C backslash
\'   U+0027 single quote  (valid escape only within rune literals)
\"   U+0022 double quote  (valid escape only within string literals)

(That last one arguably doesn't belong here, as it's not valid in a rune literal, but it's nice to know that it's explicitly excluded here.)

Let's round out today's email with the rest of the rune literal section, which is just the boring EBNF syntax, and some examples, which we don't need to discuss in any detail.

An unrecognized character following a backslash in a rune literal is illegal.

rune_lit         = "'" ( unicode_value | byte_value ) "'" .
unicode_value    = unicode_char | little_u_value | big_u_value | escaped_char .
byte_value       = octal_byte_value | hex_byte_value .
octal_byte_value = `\` octal_digit octal_digit octal_digit .
hex_byte_value   = `\` "x" hex_digit hex_digit .
little_u_value   = `\` "u" hex_digit hex_digit hex_digit hex_digit .
big_u_value      = `\` "U" hex_digit hex_digit hex_digit hex_digit
                           hex_digit hex_digit hex_digit hex_digit .
escaped_char     = `\` ( "a" | "b" | "f" | "n" | "r" | "t" | "v" | `\` | "'" | `"` ) .

'a'
'ä'
'本'
'\t'
'\000'
'\007'
'\377'
'\x07'
'\xff'
'\u12e4'
'\U00101234'
'\''         // rune literal containing single quote character
'aa'         // illegal: too many characters
'\k'         // illegal: k is not recognized after a backslash
'\xa'        // illegal: too few hexadecimal digits
'\0'         // illegal: too few octal digits
'\400'       // illegal: octal value over 255
'\uDFFF'     // illegal: surrogate half
'\U00110000' // illegal: invalid Unicode code point

Go Lexical elements: Rune literals pt 2

Jonathan Hall — Sat, 08 Jul 2023 06:00:00 +0000

I write a daily email about Go, and this is a repost from my series Exploring the Go Spec. You're invited to sign up and follow along.

Let’s continue our exploration of rune literals, which began on Monday. In summary from yesterday, a rune in Go represents a Unicode code point. Continuing from there…

Rune literals

A rune literal is expressed as one or more characters enclosed in single quotes, as in 'x' or '\n'. Within the quotes, any character may appear except newline and unescaped single quote. A single quoted character represents the Unicode value of the character itself, while multi-character sequences beginning with a backslash encode values in various formats.

The simplest form represents the single character within the quotes; since Go source text is Unicode characters encoded in UTF-8, multiple UTF-8-encoded bytes may represent a single integer value. For instance, the literal 'a' holds a single byte representing a literal a, Unicode U+0061, value 0x61, while 'ä' holds two bytes (0xc3 0xa4) representing a literal a-dieresis, U+00E4, value 0xe4.

There are a few things I want to call out about this section of the spec, that aren’t always obvious, or are easily forgotten. Especially if you’re not already very familiar with Unicode.

A rune represents a single “character” (technically: unicode code point, see yesterday's discussion).
A rune is not a single byte. (In fact, rune is an alias for int32, so it’s actually 4 bytes)
A rune is not necissarily a single visible character, as many visible characters are built by combining multiple codepoints.

As pointed out in the spec, both 'a' and 'ä' are valid rune literals. The first also corresponds to a single ASCII (or Unicode) byte: 0x61. The second corresponds to two UTF-8 bytes: 0xc3, 0xa4. So it’s immediately clear that a rune may contain multiple bytes.

But recall the example from yesterday as well: 'ў' is a valid rune literal, and represents two bytes: 0xd1, 0x9e. But in contrast, the visually identical 'ў' is not a valid rune literal, because it contains two Unicode code points, each of two bytes: у (0xd1, 0x83) followed by the breve mark, ˘, (0xcc, 0x86).

As you might expect, this can be an easy place to get tripped up. What you see on the screen is quite frequently not the whole story. I know of no fool-proof way to solve this confusion. The best I know is to be aware that the confusion exists, so when you see an error along the lines of “more than one character in rune literal”, you know where to begin your search.

Go Lexical elements: Rune literals pt 1, Intro to Unicode

Jonathan Hall — Fri, 07 Jul 2023 14:12:36 +0000

I write a daily email about Go, and this is a repost from my series Exploring the Go Spec. You're invited to sign up and follow along.

Runes… Oh boy! This is one of bits of Go that shines for its elegant simplicity, but constantly trips up everyone (myself included). As such, I think this may be a 2-, or maybe even a 3-parter.

Let’s get started.

Rune literals

A rune literal represents a rune constant, an integer value identifying a Unicode code point.

If you’re already familiar with Unicode, and have a strong understanding of what a “code point” is, you can probably skip this one. See you later! 👋

If you’re not completely sure what a Unicode code point is, stick around… I’ll do my best to untangle this seemingly simple phrase.

First off, what is Unicode? It’s emojis, right? Ehh. If that’s your understanding of Unicode, which would be completely understandable, if your career in software is relatively young, then you need a bit more context. I encourage you to read a bit about the history of why Unicode was invented, and the problems it was meant to solve. This PDF, Introduction to Unicode: History of Character Codes is a good starting point.

But here are the highlights:

Before Unocide, we had many different coding systems. The most popular was ASCII, the American Standard Code for Information Interchange. But since not all of the world is American, this had obvious drawbacks. Different countries or languages would often have their own coding schemes, but this made it very difficult to share documents between regions.

If I wrote a document using the Cyrillic alphabet, then sent it to my colleague in France, he would likely see a jumble of French letters in seemingly random order.

So Unicode came along to Unify all the codes.

Great. So now instead of 127 possible characters in ASCII, we have a virtually unlimited number of characters, right?

Not so fast.

While there’s room in Unicode for more than 1 million individual code points, most are not (yet) defined. But what’s more, Unicode is smarter than ASCII in a number of ways. It is possible to combine Unicode code points to form a single physical character.

For example, if you want to display the Cyrillic letter ў, this can be done by combining the Cyrillic Y (у) with the breve mark (˘), to give you ў. But while this looks like a single character, and in print terms it is, it’s actually two Unicode codepoints. As such, this code won’t compile:

x := rune('ў')

Because while it visually looks like we’re quoting a single character, that character is composed of two Unicode codepoints, and (as we’ll see in the next section), a rune literal must be a single Unicode codepoint.

Now if that’s not confusing enough, this code will compile:

x := rune('ў')

What’s the difference?

Well, Unicode includes a number of precomposed characters. This is a nice convenience for languages that commonly use a large number of these types of diacritics. But it’s an incovenience for us programmers. Not only does it mean that of these two character representations, only one is a valid rune, it also introduces certain headaches when trying to compare Unicode strings for equality, or when sorting, etc.

A last note for today, especially for anyone very new to Unicode. This concept of combining characters to add diacritics and other markings to an existing letter is the same way that Unicode emojis are modified to change skin tone, gender, or other attributes.

Unicode is pretty powerful. And confusing at times.

Guess-Driven Development

Jonathan Hall — Tue, 04 Jul 2023 00:00:00 +0000

Have you ever seen a pull request with half a dozen, seemingly unrelated changes, all in the name of fixing a specific bug?

The last time this happened to me, I requested clarification on the purpose of a number of the changes, suggested a couple small refactors, pointed out that a couple of the changes were unnecessary or even counter-productive…

After a few back-and-forths, the PR went from about 40 lines changed to 2 lines changed. All the unrelated, unnecessary bits had been removed.

This is the result of what I have come to call Guess-Driven Development.

It happens when someone doesn’t really know what’s causing a bug, so they just start changing things, until the bug goes away. Then they commit their entire change set, without much thought.

If you enjoyed this message, subscribe to The Daily Commit to get future messages to your inbox.

Go Constants can't overflow

Jonathan Hall — Fri, 23 Jun 2023 12:50:04 +0000

Did you know that constants in Go can't overflow?

Quoting the Go spec:

Numeric constants represent exact values of arbitrary precision and do not overflow. Consequently, there are no constants denoting the IEEE-754 negative zero, infinity, and not-a-number values.

Now this is a very interesting thing about constants in Go, and it’s something that confuses a lot of people. So let’s play around with this one a bit.

If you’re experienced with compiled languages, you’re probably already familiar with numeric overflows. Each language handles this differently, but in general, you can’t store more than 8 bits of data in an 8-bit variable.

var x int8
x += 256

will predictably complain:

256 (untyped int constant) overflows int8

But notice the parenthetical note in that error message: “untyped int constant”

We’ll talk more about untyped constants soon, but for now the interesting thing to point out is that it’s not the constant expression (i.e. 256) that overflowed. It’s the variable assignment (i.e. x +=) that overflowed.

This is because, as quoted from the spec above, constants do not overflow.

You can create constant expressions in Go of arbitrary precision, and they’re valid. Check this out:

var x int8
x += 2560000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

It still produces essentially the same error:

2560000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 (untyped int constant) overflows int8

That value is effectively a 312-bit integer (if my math isn’t off). There’s no data type in Go that can store a 312-bit integer. So you might expect that constant to overflow. But it doesn’t. Because constants do not overflow.

But as soon as you try to assign a constant to a variable, that assignment may cause an overflow.

Jumping ahead a bit to the end of this section in the spec, we see a note on implementation requirements:

Implementation restriction: Although numeric constants have arbitrary precision in the language, a compiler may implement them using an internal representation with limited precision. That said, every implementation must:

Represent integer constants with at least 256 bits. -Represent floating-point constants, including the parts of a complex constant, with a mantissa of at least 256 bits and a signed binary exponent of at least 16 bits.

Give an error if unable to represent an integer constant precisely.

Give an error if unable to represent a floating-point or complex constant due to overflow.

Round to the nearest representable constant if unable to represent a floating-point or complex constant due to limits on precision.

These requirements apply both to literal constants and to the result of evaluating constant expressions.

So there’s a bit of a caveat to this “constants do not overflow” rule. An implementation may choose to round a constant representations to 256 bits, for example. So my 312-bit example above may be rounded to 2559999999999999972095373170245681839337968901451475214262959301931782193648482834514033246208 in such an implementation (again: if my math isn’t off).

So if you want to be super-paranoid, limit your constant literals to 256 bits of precision. 😉

If you learned something reading this, you'll learn something every day by subscribing to my daily Go mailing list! Sign up for Boldly Go: Daily.