Forem: Momchil Atanasov

Implementing a Dynamic Compressor

Momchil Atanasov — Sun, 15 Mar 2026 16:21:55 +0000

Background

As part of my hobby game engine "lacking", I am implementing an audio mixer in Go for native builds. It is based on miniaudio but uses its own node-based mixing logic. In many ways, I am trying to keep the API and behavior as close as possible to the Web Audio API, since my engine supports WebAssembly builds where Web Audio API is already internally used.

While most Node types have proven intuitive to implement and/or there are sufficient resources online, the Dynamic Compressor node turned out elusive. Unfortunately, the Web Audio API specification provides limited information on the matter: https://webaudio.github.io/web-audio-api/#DynamicsCompressorOptions-processing

The goal of this article is to share what I have learned in the process and hopefully help someone else that might be stumbling with the same stuff. That said, keep in mind that I’m a complete beginner in the audio processing space, so take this article with a grain of salt.

Side note: Using AI can get you 90% along the way, or maybe even 100% if you know the subject and are able to guide it along the way. In fact, I did use the help of AI to figure out a lot of the stuff here. Regardless, I prefer to understand what I am implementing as much as possible (except for Quaternions - still can't create a visual understanding in my head), hence this article.

Algorithm

The main goal of the Dynamic Compressor is to reduce the output volume if the input signal consistently passes a certain threshold.

It appears that there are a few ways to implement a compressor, but the general flow is as follows:

Consume audio frames (PCM frames)
Determine input power (in dB units)
Determine target compression amount (in dB units)
Calculate desired audio gain (in ratio units)
Gradually move current gain towards desired gain
Apply gain to input frame and pass to output

(Step 1) Consume audio frames

In my particular case, there is nothing fancy about this. Input PCM frames arrive in chunks, and there is a for loop in the handler function that processes each frame in turn.

func (n *CompressorNode) Process(ctx ProcessContext, inputFrames, outputFrames FrameList) {
  for i, frame := range inputFrames {
    // Steps 2 to 6 happen here.
  }
}

(Step 2) Determine input power

Here we need to determine how loud the input is. Some implementations use algorithms like RMS (root mean square) to determine that. From what I understand, this requires tracking multiple samples in a row to calculate that.

For this initial implementation, I am just basing it on the value of the current input frame.

peak := max(sprec.Abs(frame.Left), sprec.Abs(frame.Right))
peakDB := audio.GainToDB(max(1.0e-8, peak)) // avoid log of zero

The equation for converting audio amplitude to dB is as follows:

db(x) = 20\log_{10}(x)

This is exactly what happens in the audio.GainToDB function.
Since log is not defined for 0, a max function with a small lower boundary is used to prevent -Inf values.

(Step 3) Determine target compression amount

This was the most problematic step for me. I could not figure out what compression function should be used and how this function relates to gain.

The reduction function

A compression function is often depicted like in the following image (it shows a hard compressor).

The X-axis represents the input dB, and the Y-axis represents the output dB that should ideally be produced in return. Hence, what we actually care about is how much dB the input signal loses due to compression. This is achieved by subtracting the function value from what would otherwise have been the original output value.

The following picture should make that clearer.

We care about the vertical stripes. To put that into mathematical terms, having our compression function f(x), we actually care about the reduction r(x) it produces.

r (x) = f (x) - i (x)

Here i(x) stands for the identity function. This is just the diagonal line that represents no compression (output equals input; Y equals X) and has the following form.

i (x) = x

All of this is very important. Once we figure out what the f(x) function is, we will actually need the r(x) function to use it in code, since it gives us the compression dB amount. And in many cases, the r(x) function actually turns out to be more simple and pleasant to read.

The compression function

For now, we will stick with the hard compressor function. Once we get a grasp on things and cover all steps, we will rework it into a soft-compressor (at the end of the article).

As a reminder, here is what the function f(x) looks for a hard compressor.

We can clearly see that initially the output signal equals the input signal. This is basically the identity function. After a certain input signal strength, the output signal is compressed.

The point at which this transition happens is called the threshold and is a configurable parameter on a Dynamic Compressor. Input frames that cross that threshold are expected to be compressed.

Since this function cannot be represented by a single math expression, we will need to split it into two parts (regions), determined by input ranges.

Non-compression region

This is the region before the threshold (denoted as t) is reached and can be expressed as:

x <= t

As mentioned above and as can be seen in the diagram, this region does not perform any compression and the f(x) function is just the identity function.

f (x) = i (x) = x

However, we actually care about the reduction in dB.

r (x) = f (x) - i (x) = i (x) - i (x) = 0

This makes sense: we expect 0 dB in reduction (in audio systems, 0 dB means full volume, or 1.0 gain).

Hard-compression region

This is the region after the threshold (denoted as t) and can be expressed as:

t < x

Here we need to perform compression, which is done through a scaling factor (q). This can be described with the following equation:

\frac{x - t}{f(x)-i(t)}=q

This might seem intimidating at first, but it actually represents the rate of ascent of the compressed output line:

The q constant determines how much the input moves along the x axis for a unit increase in the y axis. This ratio is normally configurable in a Dynamic Compressor.

If we consider that $i (t) = t$ , we have all we need to derive f(x) by transforming the equation above.

q (f (x) - t) = x - t

q f (x) = x - t + qt

\frac{x}{q} - \frac{t}{q} + t

\frac{x}{q} - (\frac{t}{q} - t)

\frac{x}{q} - (\frac{1}{q} - 1) t

We now have the compression function for the hard compression region. Just like with the non-compression region, we actually need to calculate the reduction function, since this is what we care about and what will be used in the code.

\frac{x}{q} - (\frac{1}{q} - 1) t - x

\frac{x}{q} - x - (\frac{1}{q} - 1) t

(\frac{1}{q} - 1) x - (\frac{1}{q} - 1) t

(\frac{1}{q} - 1) (x - t)

Using the reduction functions in code

Now that we have the reduction functions for both compression regions, we can put that into code.

var reductionDB float32
if peakDB <= threshold {
    reductionDB = 0.0
} else {
    reductionDB = ((1.0 / ratio) - 1.0) * (peakDB - threshold)
}

(Step 4) Calculate desired audio gain

This is fairly trivial. We need to convert the reductionDB value to an amplitude fraction value.

Keep in mind that our r(x) functions always produce values in the range 0.0 to -inf, which is ideal since this maps to the gain value range of 1.0 to 0.0, respectively.

The required transformation is the opposite to the one we used previously.

10^{\frac{x}{20}}

Which can be put into code.

targetGain := audio.DBToGain(reductionDB)

(where DBToGain holds the implementation for the gain-from-dB function)

(Step 5) Gradually move current gain towards desired gain

So far, we have determined the input power, we have calculated the desired db reduction and have transformed that into a fractional value (i.e. gain).

Should the input signal pass the threshold, the gain value would start oscillating up and down at the same rate as the input signal. We don't want that. Instead, we want the gain value to smoothly transition towards the desired value.

To be even more precise, we'd like the actual gain value to ignore high frequency changes to the desired gain value and instead follow the low frequency changes - in essence a low-pass filter.

We achieve that by using the equation for a first-order low-pass filter.

g_{new} = (1 - \beta) g_{target} + \beta g

More information can be found on the Wikipedia page.

However, we don't calculate $β\beta$ in terms of a cut-off frequency, as would normally be done. Instead, we explore the low-pass filter as an exponential smoothing function and configure it in such a way as to ensure ~63% transition to the target value after $τ\tau$ duration.

As such, the equation for $β\beta$ is:

\beta = e^{-\frac{1}{\tau f_s}}

Here $f_s$ is the discrete sampling frequency. Usually this is 44100 though other standards are available as well. Regardless, this is information that is internally available to the processor node.

How exponential smoothing works is explained well in the Exponential Smoothing article.

I would be lying if I said that I fully understand why this is used in real life and it seems that not all analog Dynamic Compressors follow that approach, as is explained in (https://www.audiotechnology.com/tutorials/understanding-compression-2).

That said, this is the approach we are taking. In code, there is a very elegant way to implement the first equation through the usage of the mix function (popular with graphics shader developers).

n.currentGain = sprec.Mix(targetGain, n.currentGain, coeff)

Here, coeff represents $β\beta$ in the above equations and controls the rate of convergence.

We should still keep in mind that this filter isn't being applied to the input signal here. It is being applied to the gain value, which controls the amount of compression. It is common to want the compressor to be quicker to act when the signal requires compression, but more relaxed when backing off.

These are the so called attack and release parameters. And this is also why, depending on whether the gain is increasing or decreasing, the coeff argument is calculated differently.

In code, this looks as follows:

const ( // normally these would be configurable
    attack  = 0.005
    release = 0.25
    sampleRate = 44100.0
)

var coeff float32
if targetGain < n.currentGain {
    coeff = float32(math.Exp(-1.0 / (sampleRate * attack)))
} else {
    coeff = float32(math.Exp(-1.0 / (sampleRate * release)))
}

n.currentGain = sprec.Mix(targetGain, n.currentGain, coeff)

(Step 6) Apply gain to input frame and pass to output

This is the most trivial step. We take the input values and multiply them by the gain value. Then we pass them as output.

outputFrames[i] = Frame{
    Left:  frame.Left * n.currentGain,
    Right: frame.Right * n.currentGain,
}

If we piece together all the code, we get the following implementation:

type CompressorNode struct {
    currentGain float32 // this should be initialized to 1.0
}

func (n *CompressorNode) Process(ctx ProcessContext, inputFrames, outputFrames FrameList) {
    const ( // these should be configurable
        sampleRate = 44100.0
        attack  = 0.005
        release = 0.25
    )

    for i, frame := range inputFrames {
        peak := max(sprec.Abs(frame.Left), sprec.Abs(frame.Right))
        peakDB := audio.GainToDB(max(1.0e-8, peak)) // avoid log of zero

        var reductionDB float32
        if peakDB <= threshold {
            reductionDB = 0.0
        } else {
            reductionDB = ((1.0 / ratio) - 1.0) * (peakDB - threshold)
        }
        targetGain := audio.DBToGain(reductionDB)

        var coeff float32
        if targetGain < n.currentGain {
            coeff = float32(math.Exp(-1.0 / (sampleRate * attack)))
        } else {
            coeff = float32(math.Exp(-1.0 / (sampleRate * release)))
        }
        n.currentGain = sprec.Mix(targetGain, n.currentGain, coeff)

        outputFrames[i] = Frame{
            Left:  frame.Left * n.currentGain,
            Right: frame.Right * n.currentGain,
        }
    }
}

Soft Dynamic Compressor

While the hard compressor is easy to implement, it has a sharp edge when transitioning from the non-compression region into the hard-compression region. This can become noticeable.

To mitigate that, a soft-compression region is introduced, so that it smooths the transition between the two regions.

The compression function looks as follows:

The region between the blue vertical lines is called the knee and it spans $k2\frac{k}{2}$ units to the left of the threshold and $k2\frac{k}{2}$ units to the right of the threshold.

What is important to note is that the non-compression and hard-compression regions still have the same functions, it is just that their ranges have changed, as depicted in the following image.

In this case, the whole compression function should be split into three parts - non-compression region, a soft-compression region and a hard-compression region. The tricky part is figuring out the soft-compression region, since it needs to bridge the two neighbouring regions in a smooth fashion.

Non-compression region

The range for this region has now changed to:

\frac{k}{2}

As stated, the compression function remains the same:

f (x) = x

The reduction function is preserved as well:

r (x) = 0

Hard-compression region

The range for this region has now changed to:

\frac{k}{2} <= x

As stated, the function remains the same:

\frac{x}{q} - (\frac{1}{q} - 1) t

And the reduction is preserved as well:

(\frac{1}{q} - 1) (x - t)

Soft-compression region

The range for this region is as follows:

\frac{k}{2} < x < t + \frac{k}{2}

However, we don't have the equation for the function. We need a function that is smooth, monotonically increasing and that connects to the non-compression and hard-compression regions seamlessly (i.e. has the same value and first-order derivative at the boundary).

It is clear that using a linear function would not work here, since it will create two sharp edges at the boundary. Instead, it turns out that a quadratic polynomial fits perfectly.

First things first, let's simplify our problem a bit. If we translate our function a bit so that the soft-compression region starts at the origin, this will make the math much simpler.

So let us instead explore a function g(x) that represents this translated version as depicted here:

This way, $g (0) = 0$ , which makes things much simpler.

To get back to the f function from this new g function (that we are yet to derive), we can use the following equation.

\frac{k}{2})) + i(t - \frac{k}{2})

It might look complicated at first, but this adjustment just takes into account the amount by which the input and output of g are shifted. We can input the lower boundary value to verify that it holds true.

\frac{k}{2}) = g((t - \frac{k}{2}) - (t - \frac{k}{2})) + i(t - \frac{k}{2})

\frac{k}{2}) = g(0) + i(t - \frac{k}{2})

\frac{k}{2}) = i(t - \frac{k}{2})

This is exactly what we expect, so our translation works.

Lower boundary

We have translated g to start at the origin, hence we know the following:

g (0) = 0

Furthermore, the derivative must match that of the identity function, which is always 1 (easy to derive, not shown here).

g^{'} (0) = 1

Upper boundary

In addition, the end of the region must match the start of the hard-compression region.

f_{hard}(t + \frac{k}{2}) - f_{non}(t - \frac{k}{2})

This just checks that the elevation at the upper boundary of the g function matches the height difference between the start of the hard-compression region and the end of the non-compression region.

The above thus leads to the following.

\frac{t + \frac{k}{2}}{q} - (\frac{1}{q} - 1) t - (t - \frac{k}{2})

\frac{t}{q} + \frac{\frac{k}{2}}{q} - \frac{t}{q} + t - t + \frac{k}{2}

\frac{\frac{k}{2}}{q} + \frac{k}{2}

(\frac{1}{q} + 1) \frac{k}{2}

Lastly, we want the derivative at the end of g to match the derivative at the start of $f_{hard}$ . But the derivative of $f_{hard}$ is also always a constant equal to $1q\frac{1}{q}$ (easy to derive, not shown here).

Hence, we have:

\frac{1}{q}

The quadratic polynomial

Now we try to find the parameters of a quadratic polynomial that satisfy the above constraints.

g(x) = a x^2 + b x + c

We need to find out what are the values of a, b and c. Let's use the constraints to figure that out.

We know that $g (0) = 0$ , hence we have:

g(0) = a 0^2 + b 0 + c = c

Thus, for this to work, c needs to be 0 and we can ignore it from now on. So we are looking for the parameters a and b.

g(x) = a x^2 + b x

The derivative of g is as follows:

g^{'} (x) = 2 a x + b

Knowing that $g^{'} (0) = 1$ , we have the following:

g^{'} (0) = 2 a 0 + b = b = 1

So we can determine that $b = 1$ . All we are left to calculate is a. We will use the upper-range constraint to do that.

(\frac{1}{q} + 1) \frac{k}{2}

k^2 + k = (\frac{1}{q} + 1) \frac{k}{2}

(\frac{1}{q} + 1) \frac{1}{2}

\frac{1}{q} + 1

\frac{1}{q}

\frac{1}{q} - 1

\frac{\frac{1}{q} - 1}{2 k}

Excellent, we now have the value for a. The last thing we need to do is verify that the derivative matches. Recall that:

g^{'} (x) = 2 a x + 1

But we know what a is, so it becomes:

(\frac{\frac{1}{q} - 1}{k})x + 1

Now we plug the value k and see if that matches the derivative for the hard-compression range.

(\frac{\frac{1}{q} - 1}{k})k + 1

\frac{1}{q} - 1 + 1

\frac{1}{q}

Which is exactly what our derivative constraint for the upper bound required. We now have a complete solution for the g function.

(\frac{\frac{1}{q} - 1}{2k})x^2 + x

Having the g function, we need to transform it in order to get the compression function f.

\frac{k}{2})) + (t - \frac{k}{2})

(\frac{\frac{1}{q} - 1}{2k})[x - (t - \frac{k}{2})]^2 + [x - (t - \frac{k}{2})] + (t - \frac{k}{2})

(\frac{\frac{1}{q} - 1}{2k})[x - (t - \frac{k}{2})]^2 + x

The reduction function becomes:

(\frac{\frac{1}{q} - 1}{2k})[x - (t - \frac{k}{2})]^2 + x - x

(\frac{\frac{1}{q} - 1}{2k})[x - (t - \frac{k}{2})]^2

All of this results in the following code:

var reductionDB float32
switch {
    case peakDB <= (threshold - knee / 2.0): // non-compression
        reductionDB = 0.0
    case peakDB >= (threshold + knee / 2.0): // hard-compression
        reductionDB = ((1.0 / ratio) - 1.0) * (peakDB - threshold)
    default: // soft-compression
        z := peakDB - (threshold - knee / 2.0)
        reductionDB = ((1.0 / ratio) - 1.0) * z * z / (2.0 * knee)
}

Threshold translation

Before we can call it a day, we need to take into account one last thing. As I mentioned at the beginning of the article, I need my implementation to match what is described in the WebAudio API specification.

Notice that unlike our design so far, the soft-compression region starts at threshold and not at threshold - knee/2. Furthermore, the hard compression starts at threshold+knee and not threshold+knee/2. So the code needs to be adjusted to shift the threshold by knee/2 to the right, in order to match the specification.

Non-compression region

The range is adjusted to be as follows:

x <= t

The function is preserved:

f (x) = x

And the reduction is preserved as well:

r (x) = 0

Hard-compression region

The range is adjusted to be as follows:

t + k <= x

The function is adjusted by shifting the threshold:

\frac{x}{{q}} - (\frac{1}{{q}} - 1)(t + \frac{k}{2})

And the reduction is adjusted as well:

(\frac{1}{{q}} - 1)[x - (t + \frac{k}{2})]

Soft-compression region

The range is adjusted to be as follows:

t < x < t + k

The function is adjusted by shifting the threshold:

(\frac{\frac{1}{q} - 1}{2k})(x - t)^2 + x

The reduction function is also adjusted and becomes:

(\frac{\frac{1}{q} - 1}{2k})(x - t)^2

Code change

The code becomes as follows:

var reductionDB float32
switch {
    case peakDB <= threshold: // non-compression
        reductionDB = 0.0
    case peakDB >= threshold + knee: // hard-compression
        reductionDB = ((1.0 / ratio) - 1.0) * (peakDB - (threshold + knee / 2.0))
    default: // soft-compression
        z := peakDB - threshold
        reductionDB = ((1.0 / ratio) - 1.0) * z * z / (2.0 * knee)
}

Final Code

The following is the final state of the source code.

type CompressorNode struct {
    currentGain float32 // this should be initialized to 1.0
}

func (n *CompressorNode) Process(ctx ProcessContext, inputFrames, outputFrames FrameList) {
    const ( // these should be configurable
        sampleRate = 44100.0
        attack  = 0.005
        release = 0.25
    )

    for i, frame := range inputFrames {
        peak := max(sprec.Abs(frame.Left), sprec.Abs(frame.Right))
        peakDB := audio.GainToDB(max(1.0e-8, peak)) // avoid log of zero

        var reductionDB float32
        switch {
            case peakDB <= threshold: // non-compression
                reductionDB = 0.0
            case peakDB >= threshold + knee: // hard-compression
                reductionDB = ((1.0 / ratio) - 1.0) * (peakDB - (threshold + knee / 2.0))
            default: // soft-compression
                z := peakDB - threshold
                reductionDB = ((1.0 / ratio) - 1.0) * z * z / (2.0 * knee)
        }
        targetGain := audio.DBToGain(reductionDB)

        var coeff float32
        if targetGain < n.currentGain {
            coeff = float32(math.Exp(-1.0 / (sampleRate * attack)))
        } else {
            coeff = float32(math.Exp(-1.0 / (sampleRate * release)))
        }
        n.currentGain = sprec.Mix(targetGain, n.currentGain, coeff)

        outputFrames[i] = Frame{
            Left:  frame.Left * n.currentGain,
            Right: frame.Right * n.currentGain,
        }
    }
}

There you have it. This is how to implement a basic Dynamic Compressor with a soft transition that matches the threshold requirements of the WebAudio API specification.

What can be improved is the calculation of the input power and most constants in the code should be parameterized.

While I have tried to derive as much as possible myself and present it above (with the help of the Desmos tool), it is possible that I might have missed something or miscalculated. Feel free to comment below with tips and tricks on the matter.

Go main-run pattern

Momchil Atanasov — Tue, 12 Sep 2023 21:24:47 +0000

I'll try to keep this short. It's just something I have seen way too many times and that has always resulted in poor code down the road.

Most example code online will have you write your main function as follows:

func main() {
    client := somelib.NewClient("https://example.com")
    err := client.DoSomething()
    if err != nil {
        log.Fatalf("Error: %v", err)
    }
}

For a simple hello world example, this is fine. However, it then often leads to the following code:

func main() {
    cfg, err := loadConfig()
    if err != nil {
        log.Fatalf("Error: %v", err)
    }

    db, err := openDB(cfg.DB)
    if err != nil {
        log.Fatalf("Error: %v", err)
    }
    defer db.Close()

    client := somelib.NewClient(cfg.ClientURL)
    err := client.DoSomething()
    if err != nil {
        log.Fatalf("Error: %v", err)
    }
}

It gets even worse if you are using a custom logging package that doesn't have Fatalf.

func main() {
    cfg, err := loadConfig()
    if err != nil {
        log.Printf("Error: %v", err)
        os.Exit(1)
    }

    db, err := openDB(cfg.DB)
    if err != nil {
        log.Printf("Error: %v", err)
        os.Exit(1)
    }
    defer db.Close()

    client := somelib.NewClient(cfg.ClientURL)
    err := client.DoSomething()
    if err != nil {
        log.Printf("Error: %v", err)
        os.Exit(1)
    }
}

There are a number of problems with this code.

If you happen to forget os.Exit in some error branch, you will likely visit panic-land.
The code gets bloated. I have seen code that has 10+ such checks in its main function.
Any decent linter will complain that defer db.Close() might not get called since os.Exit halts the program immediately and no cleanup is performed.

The main-run pattern

The main-run pattern (not an official name, just something I have used to reference it throughout the years in internal discussions) is as trivial as it gets.

func main() {
    if err := runApp(); err != nil {
        log.Printf("Error: %v", err)
        os.Exit(1)
    }
}

func runApp() error {
    cfg, err := loadConfig()
    if err != nil {
        return fmt.Errorf("error loading config: %w", err)
    }

    db, err := openDB(cfg.DB)
    if err != nil {
        return fmt.Errorf("error connecting to db: %w", err)
    }
    defer db.Close()

    client := somelib.NewClient(cfg.ClientURL)
    err := client.DoSomething()
    if err != nil {
        return fmt.Errorf("error calling client: %w", err)
    }
    return nil
}

With larger main functions, this is clearly better:

There is only one place that has os.Exit so its hard to forget it.
The code is much more compact and straight to the point.
The defer statements are run even on failure, ensuring linters are happy and proper cleanup is performed.

Signal handling

With this new design, there is also an elegant way to introduce a lifecycle context to the function.

func main() {
    ctx, ctxDone := signal.NotifyContext(context.Background(), os.Interrupt)
    defer ctxDone()
    if err := runApp(ctx); err != nil {
        log.Printf("Error: %v", err)
        os.Exit(1)
    }
}

func runApp(ctx context.Context) error {
    ...
    return nil
}

Custom error codes

Ok, you might object, saying that the original design allows for custom exit codes.

To be honest, I have rarely seen an app that really needs that. But even if you do want custom exit codes, you can always use custom errors and a mapping function.

func main() {
    ctx, ctxDone := signal.NotifyContext(context.Background(), os.Interrupt)
    defer ctxDone()
    if err := runApp(ctx); err != nil {
        log.Printf("Error: %v", err)
        os.Exit(exitCode(err))
    }
}

var ErrConfigNotFound = errors.New("config not found")
var ErrDatabaseOffline = errors.New("database offline")

func exitCode(err error) int {
    switch {
    case errors.Is(err, ErrConfigNotFound):
        return 2
    case errors.Is(err, ErrDatabaseOffline):
        return 3
    default:
        return 1
    }
}

Summary

Well, that's pretty much it. Just something to keep in mind. And even if you don't like the proposed approach, use whatever option best suits you, just make sure to avoid the pitfalls that the os.Exit approach has.

Beware of Go interfaces

Momchil Atanasov — Wed, 30 Aug 2023 19:03:50 +0000

Interfaces play a huge part when working with the Go programming language. The implicit implementation approach that Go takes makes them very flexible and easy to write. But interfaces in Go have a dark side when dealing with nil comparisons and this article aims to shine some light on it.

No time to waste, let's directly dive into it. Since error is probably the most important and most commonly used interface in Go, it will be used in the examples that follow. However, the problem is applicable to any other interface type.

What just happened

Let's have a look at the following code.

package main

import (
    "fmt"
    "log"
)

type DivisionByZeroError struct {
    Numerator   int
    Denominator int
}

func (e *DivisionByZeroError) Error() string {
    return fmt.Sprintf("division by zero: %d / %d", e.Numerator, e.Denominator)
}

func divide(numerator, denominator int) (int, error) {
    var (
        result int
        err    *DivisionByZeroError
    )
    if denominator != 0 {
        result = numerator / denominator
    } else {
        err = &DivisionByZeroError{
            Numerator:   numerator,
            Denominator: denominator,
        }
    }
    return result, err
}

func main() {
    result, err := divide(10, 5)
    if err != nil {
        log.Fatalf("Division by zero: %v", err)
    }
    log.Printf("Answer: %d", result)
}

It might be a bit overengineered for a simple divide function but it's all for demonstration purposes.

The important thing to note is that we have defined a custom type DivisionByZeroError that implements the error interface and is returned by divide whenever one tries to divide by zero.

We run the program and since we divide 10 by 5 we expect to get Answer: 2 but instead we get the following error:

2009/11/10 23:00:00 Division by zero: <nil>

NOTE: You can run the program in the Go Playground.

This makes absolutely no sense. Not only did Go decide that err != nil was true, when it should have been false, but it also printed <nil> to the console, contradicting itself.

What is going on? Is this a compiler error? Was the custom error type incorrectly defined?

Simplified example

Well - no. The problem from above has to do with how Go converts from concrete types to interfaces and how it compares interfaces.

Here is a simplified example of the same problem.

package main

import (
    "log"
)

type CustomError struct{}

func (e *CustomError) Error() string {
    return "custom error"
}

func main() {
    var typed *CustomError = nil
    var err error = typed
    if err != nil {
        log.Fatalf("Error: %v", err)
    }
}

NOTE: You can run the program in the Go Playground.

The problem occurs on the following line:

var err error = typed

This converts the nil value of type *CustomError to an interface of type error. The problem is that the interface is not nil, despite what one may think.

It prints as <nil> to the console but this is only because it references a nil value of *CustomError but the interface itself is not nil.

Inner workings

We have to look at how interfaces are represented internally to understand what is going on.

An interface in Go can be thought of as a struct that has two fields - a reference to the value and a reference to the type it represents.

type interface struct {
  value   *internal.Pointer
  type    *internal.Type
}

NOTE: This is an over-simplified Go-syntax representation, which acts only for demonstration purposes and does not accurately reflect the actual design.

When you assign a variable to an interface, the value field holds a pointer to your variable and the type field holds a reference to the Type definition of your type. That way, Go can know what methods the value has and how to cast it, if needed.

When you compare an interface in Go with nil, Go checks that value and type are both nil. If value is nil but type is not, then the check will fail.

In our case, when we assign var err error = typed, in essence we do var err error = (*CustomError)(nil).

This can be thought of as resulting in the following:

var err error = interface {
  Value:  nil,
  Type:   CustomError.(type),
}

NOTE: Again, this is total nonsense Go code but it should help you get a feel for how it works internally.

As explained above, since type is not nil, the check err == nil returns false, or vice versa, the check err != nil returns true, which happens to be our case.

But why

I have known of this "problem" for a long time now and I have long been asking myself why did the Go developers choose to implement interface comparison this way.

Why didn't they just check that the interface value isn't nil and ignore the type?

Only recently did I put two and two together and figured out why this was the case. Up to that point, I always thought it was for legacy / backward compatibility reasons.

The thing is, nil values in Go are actually usable. And since it will be hard to explain it in words, here is an example with our CustomError from above.

package main

import (
    "log"
)

type CustomError struct{}

func (e *CustomError) Error() string {
    return "custom error"
}

func main() {
    var err *CustomError = nil
    log.Println(err.Error())
}

NOTE: You can run the program in the Go Playground.

You might expect the program to panic, since we are calling the Error method on the err value, which is nil. However, instead, the program runs successfully and outputs the following to the console.

2009/11/10 23:00:00 custom error

Go allows you to call a method on a nil value of a type. And as long as the implementation of the method does not dereference the receiver, nothing will panic. In fact, in our particular case we can define the Error method of CustomError in this simplified form - without a receiver variable name since we are not using the receiver in any way.

func (*CustomError) Error() string {
    return "custom error"
}

What this means is that in some cases passing a nil value of a type might actually be e valid / working implementation of some interface.

Hence, it would be incorrect from Go to treat the interface as nil (unusable), since in fact it is perfectly valid, even if there is no actual value behind it and only a type.

Now what

Ok, so now that we know why Go compares interfaces with nil in such a way, what can we do to prevent the problem we had from the first example?

An easy solution is to never implicitly pass nil to an interface and instead always be explicit. That is, the original "divide" example would instead look as follows:

package main

import (
    "fmt"
    "log"
)

type DivisionByZeroError struct {
    Numerator   int
    Denominator int
}

func (e *DivisionByZeroError) Error() string {
    return fmt.Sprintf("division by zero: %d / %d", e.Numerator, e.Denominator)
}

func divide(numerator, denominator int) (int, error) {
    if denominator == 0 {
        return 0, &DivisionByZeroError{ // explicit
            Numerator:   numerator,
            Denominator: denominator,
        }
    }
    return numerator / denominator, nil // explicit
}

func main() {
    result, err := divide(10, 5)
    if err != nil {
        log.Fatalf("Division by zero: %v", err)
    }
    log.Printf("Answer: %d", result)
}

NOTE: You can run the program in the Go Playground.

The code became much cleaner. Since this is probably how one would have written it to begin with, one might assume that it would be hard to get into the problem from the initial example. But I have seen it in practice a number of times. Often it is not so trivial and has to do with more complex functions being chained.

Most often it starts with a function that returns a concrete error type instead of the error interface...

func doSomething() (int, *CustomError) {
  ...
}

...and a caller that propagates the error implicitly.

func caller() error {
  result, err := doSomething()
  if err == nil {
    log.Println(result)
  }
  return err // implicit
}

Instead, the caller function should perform explicit error returns as follows.

result, err := doSomething()
if err != nil {
  return err // explicit
}
log.Println(result)
return nil // explicit

And the doSomething function should not have returned *CustomError as a result parameter. Instead, if possible, it should have used the error interface instead.

func doSomething() (int, error) {
  ...
}

Lastly, avoid named result parameters.

func caller() (err error) {
  var result int
  result, err = doSomething() // implicit
  if err == nil {
    log.Println(result)
  }
  return // implicit
}

They tend to get people in trouble when not careful.

Fin

If you got so far, thank you for following along. Hopefully, this article was helpful.

It does not aim to bash Go in any way. As has been shown, there is a very good reason for Go to compare interfaces in such a way. Instead, my goal is for this to be helpful to new Go developers that are still learning the language and spare them a painful debugging session.

Feel free to add details in the comments if you think there is something I got wrong or if there is something important I forgot to mention.

Happy Go coding!

Go’s true can be false

Momchil Atanasov — Tue, 15 Aug 2023 19:17:17 +0000

This is something that not all Go developers are probably aware of. Go allows you to do things like the following.

var true = false

Which also means that the following code compiles and runs.

package main

import "fmt"

var true = false

func main() {
  var status bool = true // This will actually be false.
  if status {
    fmt.Println("True")
  } else {
    fmt.Println("False")
  }
}

And it outputs False.

Even code like this is possible.

package main

import "fmt"

var int = false // Wait, what?!?

func main() {
  var status bool = int // Really?!?
  if status {
    fmt.Println("True")
  } else {
    fmt.Println("False")
  }
}

So it seems that some keywords are actually not keywords in Go. I filed an issue a while back and got the reply that this was by design. Not sure what purpose it serves. If you know, please share in the comments.

I wonder if this will be changed in v2, where backward compatibility could be broken.

In the meantime, this is not something to really worry about. Since all these keywords are lowercase, in practice they are private to the package so it's not possible for a dependency to mess your code up.

NOTE: This post is a more condensed version of an old post of mine.

Proper HTTP shutdown in Go

Momchil Atanasov — Tue, 15 Aug 2023 19:00:18 +0000

Up to this day, I continue to come across code that has problems with shutting down an HTTP server gracefully in Go. This is why I decided to write a post about this.

Background

We should first talk about graceful HTTP shutdown and when this might be important. If you already have experience with horizontally scaled microservices and rolling updates then you can skip this section.

When you have multiple instances of your HTTP application deployed and you would like to update these instances with a newer version, in general, you want to do it in a way that avoids downtime or failed HTTP requests.

The most common practice is to do a rolling update. Simply put, you start a new instance with the new version of the application and you configure your ingress to include this new instance when routing HTTP requests. Next, you shutdown one of the old instances. But before you can shut it down, you first need to configure your ingress to stop routing HTTP requests to that particular instance. You repeat this process until all old instances are replaced by new ones.

Side note: The actual algorithms for rolling update might differ a bit from what I explained above, depending on the platform you are using, but the general idea is the same.

If you are using some type of PaaS or IaaS infrastructure (e.g. Cloud Foundry or Kubernetes) that whole process is somewhat automated or at least made easy to implement.

However, there is one important aspect to consider. While you may have configured new requests to no longer be routed to a particular instance you plan to shut down, you may have active connections still in progress. If you were to shut the instance down, connected clients would get connection reset errors or similar. If those calls originate from a browser (e.g. the endpoints are meant to serve front-end requests and not microservice-to-microservice requests) and there isn’t some retry mechanism in place, it will result in poor user experience for the customers whenever you perform updates.

Usually, when a platform shutdowns your instance, it sends a SIGTERM or SIGINT signal to inform your application that it is time to shutdown and it is up to your application to ensure that all connections have completed processing before exiting. And this is where gracefully shutting down your HTTP server comes to play. It ensures that connections are properly drained.

Problematic Implementations

Now that we understand why graceful shutdown is important, let us explore some of the most common Go HTTP server implementations and how they fail at proper HTTP shutdown.

The Hello World approach

This is probably the most common code you would come across when getting started with Go and HTTP.

package main

import "net/http"

func main() {
    http.Handle("/", http.FileServer(http.Dir("./public")))
    http.ListenAndServe(":8080", nil)
}

To be honest, for a Hello World it is probably fine. The problem is that it creates a lot of wrong assumptions for new developers that can be hard to unlearn.

The first problem is that http.ListenAndServe returns an error. So one should probably handle it. We get to the following code.

package main

import (
    "log"
    "net/http"
)

func main() {
    http.Handle("/", http.FileServer(http.Dir("./public")))
    if err := http.ListenAndServe(":8080", nil); err != nil {
        log.Fatalf("HTTP server error: %v", err)
    }
}

Better, right? Well, wrong. It turns out that when http.ListenAndServe returns normally (note that it is a blocking call), it actually returns an http.ErrServerClosed error.

Side note: My personal opinion is that this was a mistake on the Go team’s side. I see no reason why returning nil wouldn’t have been better and more properly aligned with Go convetions. If anyone knows the answer to this, please write a comment.

So we make another iteration and fix the problem from above.

package main

import (
    "errors"
    "log"
    "net/http"
)

func main() {
    http.Handle("/", http.FileServer(http.Dir("./public")))
    if err := http.ListenAndServe(":8080", nil); !errors.Is(err, http.ErrServerClosed) {
        log.Fatalf("HTTP server error: %v", err)
    }
}

We have even decided to be fancy and have used the errors.Is function to compare the error. Surely this should be enough.

Unfortunately, it isn’t.

See, as I mentioned in the previous section, when the application is being stopped it gets sent a SIGINT (CTRL+C on some platforms) or SIGTERM signal.

If you check the documentation of the signal package, you will see that the default behavior of a Go program when it receives one of these two signals is to exit. What this means is that the program is abruptly stopped. It never gets to return from the http.ListenAndServe call and to perform the error check. It is as though os.Exit was called.

Let’s extend the code above with the following log statements to see what happens.

package main

import (
    "errors"
    "log"
    "net/http"
)

func main() {
    log.Println("Starting...")
    http.Handle("/", http.FileServer(http.Dir("./public")))
    if err := http.ListenAndServe(":8080", nil); !errors.Is(err, http.ErrServerClosed) {
        log.Fatalf("HTTP server error: %v", err)
    }
    log.Println("Stopped.")
}

If we run this code, followed by CTRL+C in the terminal, we get the following output.

$ go build -o experiment .; ./experiment
2022/01/14 00:19:51 Starting...
^C
$ echo $?
130

Side note: I am using go build instead of go run since go run always returns an exit code equal to 1, even when the app is properly written.

As you can see, we never got the Stopped. log statement. Instead, signal: interrupt was printed and checking the exit code of the program shows that it exited with 130 (non-zero) exit code.

The signal handling approach

After some digging around, we realize that in order for our Go application not to exit so abruptly, we need to handle incoming signals. We soon end up using the signal package. We also find out that we need to create a dedicated http.Server instance, since there is no way to tell http.ListenAndServe to unblock. We end up with the following code.

package main

import (
    "errors"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
)

func main() {
    server := &http.Server{
        Addr: ":8080",
    }

    go func() {
        sigChan := make(chan os.Signal, 1)
        signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
        <-sigChan

        if err := server.Close(); err != nil {
            log.Fatalf("HTTP close error: %v", err)
        }
    }()

    http.Handle("/", http.FileServer(http.Dir("./public")))
    if err := server.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
        log.Fatalf("HTTP server error: %v", err)
    }
}

What we have done is to spawn a goroutine that starts listening for signals and whenever a SIGINT or SIGTERM is received we close the server.

While one step closer to the truth, this code still does not achieve a graceful shutdown, since Close immediatelly terminates all active connections without waiting for them to be processed.

We do some more reading and we change our implementation to use Shutdown with some timeout (through the usage of a timeout context).

package main

import (
    "context"
    "errors"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"
)

func main() {
    server := &http.Server{
        Addr: ":8080",
    }

    go func() {
        sigChan := make(chan os.Signal, 1)
        signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
        <-sigChan

        shutdownCtx, shutdownRelease := context.WithTimeout(context.Background(), 10*time.Second)
        defer shutdownRelease()

        if err := server.Shutdown(shutdownCtx); err != nil {
            log.Fatalf("HTTP shutdown error: %v", err)
        }
    }()

    http.Handle("/", http.FileServer(http.Dir("./public")))
    if err := server.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
        log.Fatalf("HTTP server error: %v", err)
    }
}

We run the program and we see that we no longer get any signal exit errors and we are using Shutdown so we should be ok to go. We deploy the application and feel happy with a work well done.

Except that, after some time has passed, and once we have applied this pattern to many of our microservice applications, we start to notice connection reset errors in our logging dashboard. We start troubleshooting what is going on and eventually we realize that our applications are not shutting down properly after all.

So what went wrong, we are using Shutdown after all, aren't we?

The problem is that a lot of developers don’t actually read the documentation thoroughly and end up in this pitfall. For those of you that know what I am talking about, the code above may seem silly and unlikely to occur, but I have seen it a number of times in practice. Sometimes it is not so trivial to spot, since there is some custom framework in place or the code has been split into multiple functions (maybe across files; often having channels and contexts involved).

If we read the documentation for Shutdown carefully, we will notice the following warning:

When Shutdown is called, Serve, ListenAndServe, and ListenAndServeTLS immediately return ErrServerClosed. Make sure the program doesn’t exit and waits instead for Shutdown to return.

In our code above, we call Shutdown from within a goroutine. This immediately unblocks the server.ListenAndServe call in the main function, running on the main goroutine.

And the second problem is that Go has one very specific rule about the main function that is often forgotten or is overlooked by even more senior Go developers — If the main function returns, then the program is immediatelly terminated. All other goroutines are killed, without even running any defer statements.

So, the moment the server.ListenAndServe unblocks, the program exists and the server.Shutdown call never gets a chance to drain the connections and to properly release resources. We can verify this easily by adding some logging statements.

package main

import (
    "context"
    "errors"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"
)

func main() {
    server := &http.Server{
        Addr: ":8080",
    }

    go func() {
        sigChan := make(chan os.Signal, 1)
        signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
        <-sigChan

        shutdownCtx, shutdownRelease := context.WithTimeout(context.Background(), 10*time.Second)
        defer shutdownRelease()

        if err := server.Shutdown(shutdownCtx); err != nil {
            log.Fatalf("HTTP shutdown error: %v", err)
        }
        log.Println("Graceful shutdown complete.")
    }()

    http.Handle("/", http.FileServer(http.Dir("./public")))
    if err := server.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
        log.Fatalf("HTTP server error: %v", err)
    }
    log.Println("Stopped serving new connections.")
}

We get the following output:

$ go run main.go
^C
2022/01/13 23:44:54 Stopped serving new connections.

We never get to see the Graceful shutdown complete. message.

A working graceful shutdown

There is a very simple solution to the above problem. You can just swap the locations of the Shutdown and ListenAndServe calls, where the former can be called from the main function and the latter from the goroutine.

package main

import (
    "context"
    "errors"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"
)

func main() {
    server := &http.Server{
        Addr: ":8080",
    }

    http.Handle("/", http.FileServer(http.Dir("./public")))

    go func() {
        if err := server.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
            log.Fatalf("HTTP server error: %v", err)
        }
        log.Println("Stopped serving new connections.")
    }()

    sigChan := make(chan os.Signal, 1)
    signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
    <-sigChan

    shutdownCtx, shutdownRelease := context.WithTimeout(context.Background(), 10*time.Second)
    defer shutdownRelease()

    if err := server.Shutdown(shutdownCtx); err != nil {
        log.Fatalf("HTTP shutdown error: %v", err)
    }
    log.Println("Graceful shutdown complete.")
}

If we run this program and then stop it, we can see the following output.

$ go run main.go
^C
2022/01/14 20:49:25 Stopped serving new connections.
2022/01/14 20:49:29 Graceful shutdown complete.

Finally, the outcome we wanted to achieve.

The example above can be further improved. You may decide to add a server.Close call inside the error branch of the server.Shutdown call. That way if the graceful shutdown is unable to complete within the specified timeout, you can still force the server to close.

Summary

The main key points to take away from this post are as follows:

Always read the Go documentation for methods you are using. There are often hints to important corner cases or unexpected outcomes.
The httpServer.ListenAndServe method unblocks immediatelly when httpServer.Shutdown is called.
Make sure you never return from the main function until you are actually ready to quit. Either structure your code in such a way (as in the example above) or use synchronization primitives (e.g. wait groups, channels).

Final remarks

I hope that the examples above were useful to you and that this helps you avoid a common pitfall. I am sorry about the long post but I wanted this to be useful to both beginner and advanced Go developers.

I should mention that in a more complicated application it might not be possible to have the Shutdown call block on the main goroutine, since you may have multiple HTTP Servers that you want to run and stop concurrently. Most often you would use some type of framework for starting and stopping concurrent sub-processes (for a lack of a better term). Even so, make sure that all Shutdown calls have unblocked before you exit the main function. Use WaitGroups or other synchronization mechanisms to your advantage.

Something to keep in mind when using Kubernetes is that even if your code is well written you may still end up with connection problems during rollout. The problem is that it takes time for Kubernetes to adjust its ingress routing and prevent new connections from reaching your instance (pod/container in this case) that is being stopped. In such cases, instead of connection reset errors you may observe bad gateway errors or similar, depending on your ingress implementation.

To solve this, you could use a preStop hook on your container to ensure that it is given a number of seconds between being detached (or rather, requested for detachment) from the ingress and receiving a SIGTERM signal.

I should also point out that in the microservice and cloud world there is a common understanding that cloud applications should be resilient to failure and as such they should be implemented in a way that they can handle abrupt application shutdowns.

While I agree with that and I strongly advice that you employ tactics like retrying failed requests, circuit breakers and the like, keep in mind that all of these mechanisms result in error logs, connections that might not be immediatelly cleaned up, extra processing overhead, message queues that need time to figure out what is going on and start resending messages to working instances, etc.

As such, my personal opinion is that applications should try and stop as gracefully as possible (properly releasing all resources and disconnecting from services) but should also have mechanisms in place to handle failures. After all, it happens that applications crash, VMs on which they are running disappear, the network starts acting out, and so on. If you really want to test your environment and ensure that it is resilient, you may consider using something like Chaos Monkey.

Note: This article has been ported from another platform as I gradually migrate here. Sorry if you have seen this before.

Understanding the Color type in Go

Momchil Atanasov — Sun, 13 Aug 2023 13:57:03 +0000

While working on my personal projects, I have had to deal with Go's image package a number of times. Transitively, I have also had to deal with Go's color package as well.

In the past, I used to struggle with alpha-premultiplied colors, which are the default ones in Go, and their representation. In this post I am hoping to shed some light on the matter as well as to show some micro-optimization techniques when implementing your own Color type.

Quick Theory

Before diving into the details, we should first start with the basics. Colors are often represented by four components - R, G, B and A - each representing the amount of Red, Green, Blue and Alpha (transparency/translusency) that a color is made up of.

In practice, the A is sometimes omitted when one is dealing with fully opaque colors. Also, this is not the only format out there (e.g. there is CMYK) but I will stick to RGBA in this blog post.

The most common machine representation of RGBA is probably R8G8B8A8 (RGBA8 for short). This means that there are 8 bits dedicated to each component. In essence, each component can range from 0 to 255. The benefit is that this is capable of representing a sufficient number of unique colors while also using little memory. In fact, a single color can be nicely stored in a single uint32 variable.

Other common options are R16G16B16A16 (RGBA16 for short), where each component gets 16 bits (values range from 0 to 65535), and R32FG32FB32FA32F (RGBA32F for short), where each component is a floating point number (values range from 0.0 to 1.0, though there are exceptions), allowing for easy transformations, without losing much precision, at the cost of a higher memory footprint.

Alpha-premultiplied colors

In my personal experience, I have always needed my colors to be in non-alpha-premultiplied form so I cannot pretend to fully understand the benefits of alpha-premultiplied colors. I know that there are certain algorithms which work better/faster with alpha-premultiplied colors and I believe that Go's draw package uses such algorithms internally.

Regardless, by default Go returns colors in alpha-premultiplied form so one needs to know how such colors are constructed, if one is to use Go's color or image packages correctly.

So what are alpha-premultiplied colors. These are colors, where the r, g, and b components have been multiplied by a.

...and as much as this explanation is correct, it is also very wrong. It makes certain assumptions that might not be obvious to the reader, unless they have deep knowledge on colors. This same assumption is made by the Go documentation as well.

If we were to take an r component from an RGBA8 color that has a value 128 (half intensity) and were to multiply it with an a component that has a value 128 (half transparency), we would get a new r value of 0 (no intensity), where in fact we would have expected 64 (quarter intensity). The problem is that the number overflows the 8 bits we have available.

In fact, what is usually understood is that the multiplication occurs in floating point space. In our example above, it would mean that we convert the 128 values to 0.5 (half intensity), then we multiply them with each other (i.e. 0.5 x 0.5), resulting in 0.25. Lastly, we convert the float back to uint8, resulting in 64.

Go's built-in color types

We should have a look at some of the most popular Go color types and see how they match to the formats we discussed previously.

The most popular color type is RGBA. It represents an alpha-premultiplied RGBA8 color.

Another popular color type is NRGBA. It represents a non-alpha-premultiplied RGBA8 color.

Derived from these type names, we also have the 16 bit component types.

The first one is RGBA64, which represents an alpha-premultiplied RGBA16 color.

Note: Go takes a different approach to color naming when compared with OpenGL, for example. In Go's case, the 64 in the type name indicates that the whole struct uses 64 bits (4 x 16 bits) and not that each component is 64 bits in size.

And the second one is NRGBA64, which represents a non-alpha-premultiplied RGBA16 color.

The RGBA function

Understanding the built-in color types is good but what unifies all colors in Go is the Color interface. Each color is expected to implement that interface. As such, if you were to ever implement your own color representation, you would need to implement that interface.

The only method that the Color interface requires is the RGBA one.

RGBA() (r, g, b, a uint32)

And even if you were to never implement your own Color, you are more than likely to use the RGBA method. As such, it is important to understand how it relates to the colors.

This particular function, for me personally, has been the source of a lot of confusion in the past. Let's have a look at the official documentation.

RGBA returns the alpha-premultiplied red, green, blue and alpha values for the color. Each value ranges within [0, 0xffff], but is represented by a uint32 so that multiplying by a blend factor up to 0xffff will not overflow. An alpha-premultiplied color component c has been scaled by alpha (a), so has valid values 0 <= c <= a.

There are two important things here that are easy to get wrong.

The first one is that colors returned from the RGBA function are in the alpha-premultiplied R16G16B16A16 format. However, the components are returned inside 32 bit variables. The idea is that if one were to multiply two R16G16B16A16 colors with each other, the values would overflow the 16 bit representation, whereas 32 bits per component would handle such an operation.

Still, the important point to remember here is that each component is 16 bit, even if placed inside a 32 bit variable, and that the color components are alpha-premultiplied. It is sufficient to check the official source code for the RGBA64 type (recall that it represents an alpha-premultiplied R16G16B16A16 color).

type RGBA64 struct {
    R, G, B, A uint16
}

func (c RGBA64) RGBA() (r, g, b, a uint32) {
    return uint32(c.R), uint32(c.G), uint32(c.B), uint32(c.A)
}

As can be seen, the 16 bit components are just cast to uint32 but their values remain in the [0, 65535] range. Also, there is no transformation done, since the color is already alpha-premultiplied.

The second important thing to note when reading the documentation is the part about the color component c has been scaled by alpha (a). What is meant here by scaled by alpha is that the alpha is converted to a float32 and only then multiplied with each color component. Recall the explanation in the Alpha-premultiplied colors section.

Implementing the Color interface

So, now that we have covered the fundamental aspects of colors in Go, let's try and implement our own color type. We will try to implement the non-alpha-premultiplied RGBA8 type.

Initial implementation

Following the principles from what's been discussed so far, we end up with the following implementation.

type CustomColor struct {
    R uint8
    G uint8
    B uint8
    A uint8
}

func (c CustomColor) RGBA() (uint32, uint32, uint32, uint32) {
    // convert components to float32 in the range [0.0, 1.0]
    r32f := float32(c.R) / float32(255.0)
    g32f := float32(c.G) / float32(255.0)
    b32f := float32(c.B) / float32(255.0)
    a32f := float32(c.A) / float32(255.0)

    // perform alpha-premultiplication
    r32f = r32f * a32f
    g32f = g32f * a32f
    b32f = b32f * a32f

    // convert back to 16 bit
    r16 := uint16(r32f * 65535.0)
    g16 := uint16(g32f * 65535.0)
    b16 := uint16(b32f * 65535.0)
    a16 := uint16(a32f * 65535.0)

    // return as 32 bit (without upscaling)
    return uint32(r16), uint32(g16), uint32(b16), uint32(a16)
}

If we test this implementation we will see that it works correctly. The question now is how well does it compare to an official implementation.

Official implementation

As mentioned already, the color.NRGBA64 type in Go implements a non-alpha-premultiplied RGBA8 color, exactly what we implemented above. Let us have a look at the official source code and compare how well we did.

func (c NRGBA) RGBA() (r, g, b, a uint32) {
    r = uint32(c.R)
    r |= r << 8
    r *= uint32(c.A)
    r /= 0xff
    g = uint32(c.G)
    g |= g << 8
    g *= uint32(c.A)
    g /= 0xff
    b = uint32(c.B)
    b |= b << 8
    b *= uint32(c.A)
    b /= 0xff
    a = uint32(c.A)
    a |= a << 8
    return
}

There is clearly a major difference between the two implementations. How is it that the official implementation avoids the usage of float32 values and what are all the bit-wise operations doing?

Iterative optimization

Let us try to optimize our implementation one step at a time until we get to something that resembles the official implementation. Going on that journey should expand our knowledge in micro-optimizations and give us pointers on how to optimize the Color implementations we might do in the future.

First, we should realize that we don't need to go through the uint16 type but can instead directly cast to uint32, as long as we ensure that the final values don't exceed 65535. And since our float representations are in the [0.0, 1.0] range and we multiply them by 65535.0, this should be ensured. We end up with the following code.

func (c CustomColor) RGBA() (uint32, uint32, uint32, uint32) {
    // convert components to float32 in the range [0.0, 1.0]
    r32f := float32(c.R) / float32(255.0)
    g32f := float32(c.G) / float32(255.0)
    b32f := float32(c.B) / float32(255.0)
    a32f := float32(c.A) / float32(255.0)

    // perform alpha-premultiplication
    r32f = r32f * a32f
    g32f = g32f * a32f
    b32f = b32f * a32f

    // convert back to 16 bit, stored inside 32 bit variables
    r16 := uint32(r32f * 65535.0)
    g16 := uint32(g32f * 65535.0)
    b16 := uint32(b32f * 65535.0)
    a16 := uint32(a32f * 65535.0)
    return r16, g16, b16, a16
}

Ok, this is cleaner, but is not much of an actual performance improvement.

Looking more at the code, one thing we may ask ourselves is whether we really need to convert the color components to the [0.0, 1.0] range. In fact, as long as it is of type float32 and the alpha component is in the [0.0, 1.0] range, we could do the alpha-premultiplication within the [0.0, 255.0] range, and then just scale it up to the [0.0, 65535.0] range. This will remove one division per component, meaning a total of three float divisions. We end up with the following code.

func (c CustomColor) RGBA() (uint32, uint32, uint32, uint32) {
    // convert alpha to the range [0.0, 1.0]
    a32f := float32(c.A) / float32(255.0)

    // perform alpha-premultiplication in the range [0.0, 255.0]
    r32f := float32(c.R) * a32f
    g32f := float32(c.G) * a32f
    b32f := float32(c.B) * a32f

    // convert to the [0.0, 65535.0] range
    r16 := uint32(r32f * 257.0)
    g16 := uint32(g32f * 257.0)
    b16 := uint32(b32f * 257.0)
    a16 := uint32(a32f * 65535.0)
    return r16, g16, b16, a16
}

Note: We multiply the r, g and b components by 257 since 255 x 257 = 65535, where 255 is the maximum value in the [0.0, 255.0] range. In essence, we ensure a correct mapping from the [0.0, 255.0] range to the [0.0, 65535.0] range.

Since we are multiplying each color component by 257 and it is an integer constant, we could perform that multiplication while the color component is still in its integer representation, converting three float multiplications to three integer ones.

func (c CustomColor) RGBA() (uint32, uint32, uint32, uint32) {
    // convert alpha to the range [0.0, 1.0]
    a32f := float32(c.A) / float32(255.0)

    // perform alpha-premultiplication in the range [0.0, 255.0]
    r32f := float32(uint32(c.R)*257) * a32f
    g32f := float32(uint32(c.G)*257) * a32f
    b32f := float32(uint32(c.B)*257) * a32f

    // convert to the [0.0, 65535.0] range
    r16 := uint32(r32f)
    g16 := uint32(g32f)
    b16 := uint32(b32f)
    a16 := uint32(a32f * 65535.0)
    return r16, g16, b16, a16
}

Note: We cast the color components to uint32 before performing the multiplications, as otherwise the multiplication would overflow the uint8 size that each color component has.

Let's rework the code a bit further by moving the color-component-to-alpha multiplication to the end of the code.

func (c CustomColor) RGBA() (uint32, uint32, uint32, uint32) {
    // convert alpha to the range [0.0, 1.0]
    a32f := float32(c.A) / float32(255.0)

    // perform integer upscaling of the color components
    upscaledR := uint32(c.R) * 257
    upscaledG := uint32(c.G) * 257
    upscaledB := uint32(c.B) * 257

    // convert to the [0.0, 65535.0] range
    r16 := uint32(float32(upscaledR) * a32f)
    g16 := uint32(float32(upscaledG) * a32f)
    b16 := uint32(float32(upscaledB) * a32f)
    a16 := uint32(a32f * 65535.0)
    return r16, g16, b16, a16
}

Something else we should realize is that we can split the color component multiplication from uint32(c) * 257 to uint32(c) * 256 + uint32(c). Why would we want to do this? Instead of a single multiplication, don't we now have both a multiplication and an addition?

We have to know a bit base-2 theory to spot the potential performance improvement here. Multiplication by 256 can also be represented through an 8-wise bit-shift to the left. A single bit-shift to the left is equal to a multiplication by 2. Doing 8 bit-shifts to the left is equal to doing eight multiplications by 2, or 2^8, which is 256. The critical thing here is that in general bit-shifts are much faster than arbitrary multiplications or divisions.

Side note: Doing a bit-shift to the right is equal to the reverse - a division by 2.

This allows us to rewrite the code as follows.

func (c CustomColor) RGBA() (uint32, uint32, uint32, uint32) {
    // convert alpha to the range [0.0, 1.0]
    a32f := float32(c.A) / float32(255.0)

    // store the 8 bit values inside 32 bit variables so that we can
    // perform multiplications without overflowing
    upscaledR := uint32(c.R)
    upscaledG := uint32(c.G)
    upscaledB := uint32(c.B)

    // perform integer upscaling of the color components
    upscaledR = upscaledR + (upscaledR << 8)
    upscaledG = upscaledG + (upscaledG << 8)
    upscaledB = upscaledB + (upscaledB << 8)

    // convert to the [0.0, 65535.0] range
    r16 := uint32(float32(upscaledR) * a32f)
    g16 := uint32(float32(upscaledG) * a32f)
    b16 := uint32(float32(upscaledB) * a32f)
    a16 := uint32(a32f * 65535.0)
    return r16, g16, b16, a16
}

By now you should start to see a resemblance to the official source code.

Another thing to realize is that adding an 8 bit number to any number whose first 8 bits are zeroes is the same as just replacing the first 8 bits.

For example, if we were to have a number A that has ????????00000000 as its 16 bit representation (where each ? can by either a 0 or 1) and were to add a number B that has 10110110 as its 8 bit representation, then the sum of the two would be ????????10110110. This is basically a bit-wise | (or) operation.

In our code above, we multiply each component by 256 by doing 8 bit-shifts to the left. This means, that we have 8 lower bits that are zero. This means that we can use bit-wise | to perform the addition. In general, bit-wise | is much quicker than arbitrary addition.

We get the following code.

func (c CustomColor) RGBA() (uint32, uint32, uint32, uint32) {
    // convert alpha to the range [0.0, 1.0]
    a32f := float32(c.A) / float32(255.0)

    // store the 8 bit values inside 32 bit variables so that we can
    // perform multiplications without overflowing
    r := uint32(c.R)
    g := uint32(c.G)
    b := uint32(c.B)

    // perform integer upscaling of the color components
    r = r | (r << 8)
    g = g | (g << 8)
    b = b | (b << 8)

    // convert to the [0.0, 65535.0] range
    r16 := uint32(float32(r) * a32f)
    g16 := uint32(float32(g) * a32f)
    b16 := uint32(float32(b) * a32f)
    a16 := uint32(a32f * 65535.0)
    return r16, g16, b16, a16
}

Let's look at the alpha component for a bit. We do the following steps to evaluate it.

// convert alpha to the range [0.0, 1.0]
a32f := float32(c.A) / float32(255.0)
// convert to the [0.0, 65535.0] range
a16 := uint32(a32f * 65535.0)

We convert the alpha down to the [0.0, 1.0] range, only so that we can scale it back up to [0.0, 65535.0]. In essence, we divide it by 255.0 and afterwards multiply it by 65535.0. Well, (v / 255.0) * 65535.0 equals (v * 65535.0) / 255.0, which equals v * 257.0. We see the well familiar 257-multiplication paradigm. Let's apply the same logic as we did with the color components.

func (c CustomColor) RGBA() (uint32, uint32, uint32, uint32) {
    // convert alpha to the range [0.0, 1.0]
    a32f := float32(c.A) / float32(255.0)

    // store the 8 bit values inside 32 bit variables so that we can
    // perform multiplications without overflowing
    r := uint32(c.R)
    g := uint32(c.G)
    b := uint32(c.B)
    a := uint32(c.A)

    // perform integer upscaling of the components
    r = r | (r << 8)
    g = g | (g << 8)
    b = b | (b << 8)
    a = a | (a << 8)

    // convert to the [0.0, 65535.0] range
    r16 := uint32(float32(r) * a32f)
    g16 := uint32(float32(g) * a32f)
    b16 := uint32(float32(b) * a32f)
    return r16, g16, b16, a
}

Note: We still need the a32f variable that holds the alpha in the [0.0, 1.0] range, since we are still using it for the alpha-premultiplication of the color components. Is there something we could do about this as well?

In the code above, we are using the following equation for color alpha-premultiplication.

c_alpha_premultiplied = uint32(float32(c) * (float32(a) / 255.0))

If we move the brackets a bit, we can get to the following, which yields the same output.

c_alpha_premultiplied = uint32((float32(c) * float32(a)) / 255.0)

In the above equation, we can multply c and a before casting them to float32, since they are integer values.

c_alpha_premultiplied = uint32(float32(c * a) / 255.0)

Now that we have rearranged the equation in this simplified form, we can notice that we don't need to use a floating point division. Since we are casting the end result to an integer (uint32 in this case), we are losing any digits after the decimal, hence we might as well use integer division which has that same behavior instead.

c_alpha_premultiplied = uint32(c * a) / 255

Note: This is only possible and valid if c * a fits inside uint32. In our case c is an upscaled value that occupies 16 bits and a (as used here; prior to upscaling) occupies 8 bits. This means that in total a maximum of 24 bits would be needed and since we have 32 to work with, we should be fine.

Let's apply the above reasoning to our code.

func (c CustomColor) RGBA() (uint32, uint32, uint32, uint32) {
    // store the 8 bit values inside 32 bit variables so that we can
    // perform multiplications without overflowing
    r := uint32(c.R)
    g := uint32(c.G)
    b := uint32(c.B)
    a := uint32(c.A)

    // perform color alpha-premultiplication of the color components
    r = r | (r << 8)
    r = (r * a) / 255
    g = g | (g << 8)
    g = (g * a) / 255
    b = b | (b << 8)
    b = (b * a) / 255

    // do the alpha integer upscaling last
    a = a | (a << 8)

    return r, g, b, a
}

Lastly, we need to recall the following Go short-hand expressions.

c = c | (c << 8)

// is the same as

c |= (c << 8)

c = (c * a) / 255

// is the same as

c *= a
c /= 255

// which is the same as

c *= a
c /= 0xff

Using these and a bit of rearranging, we get to the following code.

func (c CustomColor) RGBA() (uint32, uint32, uint32, uint32) {
    r := uint32(c.R)
    g := uint32(c.G)
    b := uint32(c.B)
    a := uint32(c.A)

    r |= (r << 8)
    r *= a
    r /= 0xff

    g |= (g << 8)
    g *= a
    g /= 0xff

    b |= (b << 8)
    b *= a
    b /= 0xff

    a |= (a << 8)

    return r, g, b, a
}

Except for a few syntactic differences, we have done it - we have simplified and optimized our Color implementation of a non-alpha-premultiplied RGBA8 color to match the official one.

Summary

I realize that not everyone has to deal with Go's image or color packages. Even fewer people would ever need to implement their own Color type.

Still, I am hoping that this managed to shed some light on alpha-premultiplied colors and the most popular Go color formats to those readers that will need to use them.

Also, while micro-optimizations are usually the last line of defense when dealing with performance problems, in situations where a function is expected to be called a million times within a short duration of time (as is the case with RGBA), micro-optimizations and knowing how to apply them can be a critical.