Forem: Matt Dale

The Go Playground: Using Modules

Matt Dale — Mon, 06 Nov 2023 07:22:10 +0000

This is part of an upcoming series about useful and surprising features of the Go Playground. Follow me to read more about the Go Playground!

The Go Playground is extremely useful for sharing runnable Go code and testing out standard library functions, but what if you want to test out code that's in an external library? It turns out that the Go Playground can automatically fetch Go modules when you import packages that aren't in the standard library.

Let's check out an example of generating a random UUIDv4 using the github.com/google/uuid package:

package main

import (
    "fmt"

    "github.com/google/uuid"
)

func main() {
    u, err := uuid.NewRandom()
    if err != nil {
        panic(err)
    }

    fmt.Println(u)
}

Run

When we run that code, the Go playground fetches the github.com/google/uuid module, then compiles and runs our code. By default, the Go Playground fetches the latest available module version for the specified package. But what if we want to use a different version of the module?

Major Versions

To fetch a different major version, add a major version suffix to the import path and the Go Playground will fetch that major version instead! For example, to fetch package mypackage from version v2.x.x of the github.com/name/example module, import package

github.com/name/example/v2/mypackage

Let's check out an example of writing a simple HTTP handler using v5 of the github.com/go-chi/chi module:

package main

import (
    "fmt"
    "io"
    "net"
    "net/http"
    "os"

    "github.com/go-chi/chi/v5"
    "github.com/go-chi/chi/v5/middleware"
)

func main() {
    r := chi.NewRouter()
    r.Use(middleware.Logger)
    r.Get("/", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("hi"))
    })

    listener, err := net.Listen("tcp", ":8080")
    if err != nil {
        panic(err)
    }
    go http.Serve(listener, r)

    fmt.Println("Sending request...")
    res, err := http.Get("http://localhost:8080/")
    if err != nil {
        panic(err)
    }

    fmt.Println("Reading response...")
    if _, err := io.Copy(os.Stdout, res.Body); err != nil {
        panic(err)
    }
}

Run

That example highlights another feature of the Go Playground: you can run and test local network services, like HTTP servers! Your code can't access the Internet, but you can run as many local services as the Go Playground sandbox can handle.

Any Version

So far we've let the Go Playground fetch the latest version and a specific major version, but we can also specify the exact version of a module if we want. To do that, we'll use another hidden feature of the Go Playground: you can add multiple source code files!

In this example, we'll add a go.mod file that specifically requests v1.11.7 of the go.mongodb.org/mongo-driver module:

package main

import (
    "fmt"

    "go.mongodb.org/mongo-driver/version"
)

func main() {
    fmt.Println(version.Driver)
}
-- go.mod --
module mymodule

require go.mongodb.org/mongo-driver v1.11.6

Run

That's all for now! In the next Go Playground post, we'll look at displaying images and GIFs in the Go Playground. Subscribe for more!

Sometimes "math/rand" Uses Weighted Dice

Matt Dale — Mon, 19 Jun 2023 03:39:28 +0000

Go's math/rand package provides pseudo-random number generators, or PRNGs, that are used in a ton of Go software. I've personally used the math/rand package in the vast majority of Go projects I've worked on. It's great for generating IDs, shuffling lists, introducing jitter in timing, and lots of other things.

⚠️ math/rand should not be used for security-sensitive purposes. For that, use crypto/rand instead.

As with any PRNG, the number sequence appears random but is actually deterministic and depends on the seed value. As a result, it's possible to get math/rand to generate the exact same number sequence by giving it the same seed value.

Let's look at an example:

rand.Seed(1)
fmt.Println(rand.Int63(), rand.Int63())
// Output: 5577006791947779410 8674665223082153551

rand.Seed(1)
fmt.Println(rand.Int63(), rand.Int63())
// Output: 5577006791947779410 8674665223082153551

Go Playground

We used the same seed and got the same random numbers each time, just like we expected! Now let's try using different seed values:

rand.Seed(1)
fmt.Println(rand.Int63(), rand.Int63())
// Output: 5577006791947779410 8674665223082153551

rand.Seed(2)
fmt.Println(rand.Int63(), rand.Int63())
// Output: 1543039099823358511 2444694468985893231

Go Playground

Great, we got different numbers! Let's try two more seed values:

rand.Seed(243697394791045426)
fmt.Println(rand.Int63(), rand.Int63())
// Output: 5463449640670714849 6224151428804650039

rand.Seed(-6474076765971414729)
fmt.Println(rand.Int63(), rand.Int63())
// Output: 5463449640670714849 6224151428804650039

Go Playground

Oops, that's not right. We gave it totally different seed values, but the sequences are the same. What happened?

It turns out that math/rand will sometimes generate the same number sequence for different seed values. The rand.Seed function docs describe the unexpected behavior:

Seed values that have the same remainder when divided by 2³¹-1 generate the same pseudo-random sequence.

That's not great, that means math/rand might generate the same number sequence even if our seed values don't repeat.

Why'd you do that math/rand?! You're supposed to be (pseudo) random!

Automatic Seeds in Go 1.20

If you've looked at the rand.Seed documentation recently, you may have noticed that the function is deprecated. In fact, starting with Go 1.20, the package-global math/rand PRNG is automatically initialized with a random seed value. Great, so we can ignore this seed nonsense and just live our lives, right?

Mostly yes. However, it's still possible to run into problems with the new math/rand automatic seed values. I found that if you start about 100,000 processes that generate numbers with math/rand using the automatic seed values, two of those processes will probably generate the same sequence.

Why 100,000? (click for details)
This is an example of the birthday paradox, which is the counterintuitive fact that you're likely to get a repeating value in a series of random numbers much sooner than you might expect. In this case, rand.Seed basically truncates the int64 input into 31 bits of entropy. Based on that, we can calculate that we have about a 90% chance of getting a repeating value after generating 100,000 random 31-bit numbers.

Granted, most use cases don't require starting 100,000 processes. However, it could cause problems if, for example, you deploy an agent to every container in a very large server fleet and you need those agents to generate different number sequences.

Some Possible Fixes

If that is a problem for us, is there anything we can do about it? Yes!

Use a different PRNG library.

One solution is to use a different PRNG that doesn't suffer from the same unexpected behavior as math/rand. Go issue #36133 suggests a possible alternative is the golang.org/x/exp/rand module, which implements a different type of PRNG. Importantly, the x/exp/rand module does not truncate seed values, so is less likely to cause collisions between seeds. However, it is not automatically seeded like math/rand, so you must provide a seed value yourself!

Use `crypto/rand` instead.

Another option is to use the crypto/rand package instead, which uses the operating system's or runtime's cryptographically-secure randomness source. Cryptographically-secure RNGs are typically slower than PRNGs, but if you don't need to generating a ton of values, then using crypto/rand can be a simple option.

Partition the PRNG values.

A more complex solution is to "partition" the math/rand output with a value that is different for each process. For example, consider the following code that generates hexadecimal strings to use as request IDs. It reads a random value from crypto/rand at startup and appends that to all values read from math/rand, reducing the probability of collisions between processes:

var partition uint32

func init() {
    err := binary.Read(cryptorand.Reader, binary.LittleEndian, &partition)
    if err != nil {
        panic(err)
    }
}

func newRequestID() string {
    return fmt.Sprintf("%016x%08x", rand.Uint64(), partition)
}

Go Playground

⚠️ Adding partitions to pseudo-random values can change the distribution of values, so it's extremely important to understand how it affects your use case. For example, a new distribution of values might introduce "hotspots" in a database index that wasn't a problem without the partition.

Additional Reading

Go issue #54880 - math/rand: seed global generator randomly
Go issue #21835 - proposal: use PCG Source in math/rand for Go 2

^{Cover photo is Closeup Photo of Two Red Dices Showing 4 and 5 by Jonathan Petersson

Body photo is Yelling formal man watching news on laptop

by Andrea Piacquadio}

Early Perspectives on Go Generics

Matt Dale — Tue, 01 Mar 2022 07:26:31 +0000

Go 1.18 is scheduled for release sometime soon (originally Feb 2022) and will be the first version of Go to support generic programming, commonly called "generics". The lack of generics in Go has been a contentious topic for a long time, as Ian Lance Taylor pointed out in his GopherCon presentation in 2019:

Go was released November 10, 2009. Less than 24 hours later, we saw the first comment about generics.
— Ian Lance Taylor, GopherCon 2019: Generics in Go

There are already lots of good articles, tutorials, and talks about generics in Go, so this article will just focus on the experience of writing and using generic code in Go.

Disclaimer: Go 1.18 is not released yet! These early perspectives on Go generics are using the latest gotip version as of Feb 21, 2022. Some of the behaviors described in this article may change in the final release.

What's the Big Deal with Generics?

Before we get into what using generics in Go is like, let's take a step back and ask, "Why do developers use generics and what improvements do generics offer?" Let's turn to a quote by Alexander Stepanov, original designer of the C++ Standard Template Library, for help:

Generic programming is about abstracting and classifying algorithms and data structures ... Its goal is the incremental construction of systematic catalogs of useful, efficient and abstract algorithms and data structures.
— Alexander Stepanov, Short History of STL

So generic programming is primarily about being able to focus on writing code once and using it many times with many types. How does it work in Go?

Go Generics: The Really Really Short Version

Go generics adds a "type constraint" syntax that allows functions and types to specify multiple types that they can receive and return. Let's make that more concrete with a simple Min function that accepts two numeric values and returns the minimum value of the two. Our Min function can accept two parameters, both either an int64 or float64, and returns a value the same type as the inputs.

func Min[T int64 | float64](x, y T) T {
    if x < y {
        return x
    }
    return y
}

To call our generic Min function, we call it with values that satisfy the type constraint, int64 in this case. The Go compiler infers appropriate types, resulting in code that looks the same as non-generic Go code.

var x int64 = 5
var y int64 = 12
Min(x, y) // returns 5

For a more in-depth look at generics in Go, check out the (beta) Go generics tutorial.

Early Perspectives

Here's what you came for, some selected early perspectives on using Go generics!

Generic Type Inference

Type inference is a concept already familiar to most Go developers. When you assign a value to a variable using the := assignment operator, you're asking the Go compiler to infer the correct variable type. Go generics use a similar concept to try to determine the correct types to use when calling a generic function.

The good news is type inference works great with function parameters! Calling functions that use generic input parameter types is very intuitive in most cases, and is often syntactically identical to calling non-generic functions.

However, calling functions with a generic return type often requires you to specify the type in the call. For example, let's consider the following generic function that returns a positive infinity floating point value as a float32 or float64:

func Inf[T float32 | float64](t) T {
    return T(math.Inf(1))
}

Now let's try to call the Inf function to assign a +Inf value to a float64 variable:

var x float64
x = Inf()

Oops, we got a compile error!

./main.go:20:15: cannot infer T (./main.go:10:10)

Instead, we have to specify the function type explicitly:

var x float64
x = Inf[float64]()

As a result, calling generic functions where the Go compiler can't use the input parameters to infer the return type can be somewhat awkward. There's an open discussion about whether or not Go should support generic function type inference when assigning to a variable here, but for now you have to declare the return type explicitly.

APIs and Type Switches

Generics also gives us new options when we want to add functionality to APIs while maintaining backward compatibility. Consider a function Do that takes an int value and performs some action.

func Do(x int) {
    // Perform some action with int x.
}

Now let's say we want Do to support either an int or a bool, and perform a different action for with each type. Before generics, we might have just added new functions DoInt and DoBool, and kept the Do function to maintain API backward compatibility.

func DoInt(x int) {
    // Perform some action with int x.
}

func DoBool(x bool) {
    // Perform some action with bool x.
}

func Do(x int) {
    DoInt(x)
}

With generics, we can add a type constraint to our Do function to accept either an int or a bool and use a type switch to do the appropriate action for each type.

func Do[T int | bool](x T) {
    switch any(x).(type) {
    case int:
        // Perform some action with int x.
    case bool:
        // Perform some action with bool x.
    }
}

Note: any is a new shorthand for interface{}.

We've reduced our API surface area by only using a single Do function, and maintained API backward compatibility in most cases (calls to Do using reflection may still be impacted), but the resulting implementation code is less clear than two separate functions. Additionally, there's no compile-time check to ensure that the type constraint and switch cases match, so they may get out-of-sync and lead to bugs.

Whether or not it's a good idea to update APIs to accept more types using type constraints is still unclear. There is an open discussion about amending the type switch to work more gracefully with type parameters, so follow that if you're interested in the outcome!

Testing

With more types come more tests! If your functions only support a single type, then you only need to test with values of a single type. Once your functions support many types, you need to test with values of every supported type. That can balloon into lots of test cases quickly!

To complicate things, some common testing practices won't work when using generics. For example, anonymous functions and closures cannot use type parameters (check out the discussion here).

As a result, you may end up with quite large tests with lots of subtests and lots of test cases, like the ones here. We'll need to develop new testing best practices to handle the new challenges that come with testing generic code in Go.

Wrapping Up

Go generics will be available soon and may significantly change how we write Go. There is a lot to learn about the best ways to use Go generics and we will undoubtedly make plenty of mistakes along the way. Considering that, Rob Pike's suggestion to keep generics out of most of the standard library for the Go 1.18 release seems very wise.

Finally, here's a plug for the gmath library, my attempt to write generic versions of commonly used functions from the Go math package. Check it out if you need math functions for numeric types other than float64!

Sending 😀 in Go

Matt Dale — Wed, 14 Jul 2021 03:16:16 +0000

So you're a Go developer and you're building your latest and greatest web app. You decide to add some extra flare to your JSON API by adding a 😀 to the end of your success message. You define a success message and marshal it as JSON.

jsonMsg, _ := json.Marshal(map[string]interface{}{
    "ok":      true,
    "message": "Success 😀",
})
fmt.Println(string(jsonMsg))
// -> {"message":"Success 😀","ok":true}

We got the JSON message we expected, including our 😀, so that worked. But how does the smiley face in our code editor end up in a JSON message without breaking something or requiring some kind of Base64-encoded image in the message? It turns out there are actually numerous standards and systems that must work together to make handling Unicode text, including Emoji, seamless.

The first reason it works is because 😀 is not just any image, but an image that's part of a font (called a glyph) that your code editor and most other software use to display text. The image data is already present on your computer and was likely included in your operating system or downloaded with your code editor.

The second reason it works is because your code editor, the Go compiler, the Go runtime, the JSON standard, and the Go JSON library all use Unicode text encoding, specifically UTF-8 encoding. Unicode text is encoded as a series of "code points": numbers that tell your software which glyphs to display. UTF-8 is a Unicode encoding that stores text as a series of 8-bit values and represents code points as one to four 8-bit values. For a helpful summary of text encoding terminology, check out this Stack Overflow post.

Why is any of this important? For me, this exploration started when I was attempting to improve handling of Unicode surrogate pair values in the MongoDB Go driver's Extended JSON unmarshaler. Don't worry if that sounds like word salad now, we'll explore those concepts more later.

Unicode in Go

Building on what we just learned about Unicode text encoding, let's quickly review how Go text types work. A Go string is a wrapper around a byte array that typically holds UTF-8-encoded text (although it can technically contain any arbitrary bytes). A Go rune represents a single Unicode code point and is an alias of int32. A string can be directly converted to and from a []byte or []rune. For more information about text representation in Go, read Rob Pike's 2013 blog post Strings, bytes, runes, and characters in Go.

Let's see that encoding in action by printing the []byte representation of our JSON message. Note that json.Marshal returns the encoded JSON as a []byte, so we just need to print the returned variable instead of converting it to a string.

fmt.Println(jsonMsg)
// -> [123 34 109 101 115 115 97 103 101 34 58 34 83 117 99 99 101 115 115 33 32 240 159 152 128 34 44 34 115 117 99 99 101 115 115 34 58 116 114 117 101 125]

Well, we just got a bunch of numbers. Where's the 😀? Let's use the IndexRune function in the "bytes" package to find the index of the 😀 glyph in the byte slice.

idx := bytes.IndexRune(jsonMsg, '😀')
fmt.Println(idx)
// -> 21

Great, now we know where it is! Let's make sure reading the byte at index idx gives us the 😀 we expected.

fmt.Println(string(jsonMsg[idx]))
// -> ð

Something's not right, we got ð, not a smiley face. Remember earlier when we talked about how some Unicode code points need more than one byte when encoded as UTF-8? Let's try reading a few different size byte slices and see what we get.

fmt.Println(string(jsonMsg[idx : idx+1]))
// -> �

Hmm, that's not a smiley face. We got �, the Unicode replacement character, which means Go couldn't figure out how to decode the bytes as a valid UTF-8 string. Let's try reading more bytes.

fmt.Println(string(jsonMsg[idx : idx+2]))
// -> �

Nope.

fmt.Println(string(jsonMsg[idx : idx+3]))
// -> �

Not yet.

fmt.Println(string(jsonMsg[idx : idx+4]))
// -> 😀

There, we got a smiley face by decoding 4 bytes! What happens if we try to decode 5 bytes?

fmt.Println(string(jsonMsg[idx : idx+5]))
// -> 😀"

OK, now we're getting extra code points. That makes sense because the maximum size a UTF-8 code point can be is 4 bytes. Going back to the original question, let's look at the UTF-8 bytes that represent a 😀.

fmt.Println(jsonMsg[idx : idx+4])
// -> [240 159 152 128]

We can confirm that the value we found is indeed a 😀 by building the byte slice literal and decoding it as a UTF-8 string.

fmt.Println(string([]byte{240, 159, 152, 128}))
// -> 😀

We just successfully built a valid UTF-8 string from individual byte values!

In the Go runtime, strings are UTF-8 encoded byte arrays, but what about the actual ".go" file? Let's try reading the "main.go" file we're writing and see how the 😀 glyph is encoded.

main, _ := ioutil.ReadFile("main.go")
idx := bytes.IndexRune(main, '😀')
fmt.Println(string(main[idx : idx+4]))
// -> 😀
fmt.Println(main[idx : idx+4])
// -> [240 159 152 128]

It's the same as the Go string encoding! In fact, the Go compiler expects ".go" files to be UTF-8-encoded. The consistency of UTF-8 text encoding across different software definitely makes handling 😀 in Go easier.

Unicode in JSON

Great, we understand how to encode and decode our 😀 glyph in Go source code and Go strings! But weren't we talking about JSON messages? That's right, we marshalled a JSON message with a 😀 and were trying to figure out how it worked. The short answer is that the JSON Character Encoding specification says that JSON can contain any UTF-8-encoded text, including 😀. If that's true, we should be able to unmarshal a JSON message declared as a Go string literal and converted to a []byte. Let's try.

var msg map[string]interface{}
json.Unmarshal([]byte(`{"message":"Success 😀","ok":true}`), &msg)
fmt.Println(msg)
// -> map[message:Success 😀 ok:true]

Because a Go string is UTF-8-encoded text, converting one to a []byte and using the "encoding/json" package to unmarshal it works as expected.

Weird Unicode in JSON

We've mostly answered our original question, but let's consider a world where some legacy software doesn't handle JSON messages with multi-byte UTF-8 code points correctly. What if we really, really need to limit our JSON messages to only ASCII values, but we still want to send the 😀?

JSON actually supports encoding Unicode code points as "escape sequences" in strings, like "\u2603", which is the Unicode escape sequence for the snowman emoji (☃). How do we get the Unicode escape sequence for 😀?

The Go "encoding/json" library doesn't support marshaling JSON with only ASCII values (there's an interesting discussion about that here), so let's try using the QuoteToASCII function from the "strconv" package to get a Unicode escape sequence.

fmt.Println(strconv.QuoteToASCII("😀"))
// -> "\U0001f600"

Unicode escape sequences in JSON must start with "\u" (lowercase), so let's lowercase that "\U" and try to unmarshal our message, replacing the literal 😀 with the Unicode escape sequence.

var msg map[string]interface{}
json.Unmarshal([]byte(`{"message":"Success \u0001f600","ok":true}`), &msg)
fmt.Println(msg)
// -> map[message:Success f600 ok:true]

Spoiler
Unicode escape sequences output by QuoteToASCII are always valid in Go string literals, but are not always valid in JSON strings. Go string literals support UTF-32 escape sequences that start with an uppercase "\U" (e.g. "\U0001f600"), but JSON does not.

Oops, that didn't work. What happened to our Unicode escape sequence and our 😀? It turns out that JSON Unicode escape sequences must start with "\u" followed by exactly 4 hexadecimal digits. The JSON unmarshaler reads the first 4 hex digits ("0001"), decodes it as the Unicode "start of heading" code point (a non-printing control character), then reads the following values "f600" as a literal string. In fact, we can't encode 😀 as a single JSON Unicode escape sequence because its code point value requires 17 bits, which is more than the 16 bits we can write as 4 hex digits. The Unicode spec helpfully includes a quirky feature called surrogate pairs that lets us encode Unicode code points larger than 16 bits as a pair of UTF-16 values. We can use the EncodeRune function from the "unicode/utf16" package to get those two UTF-16 runes, then use strconv.QuoteToASCII again to get the corresponding Unicode escape sequence.

high, low := utf16.EncodeRune('😀')
pairStr := string([]rune{high, low})
fmt.Println(strconv.QuoteToASCII(pairStr))
// -> "\ufffd\ufffd"

That seems like it worked, but if we take a closer look we see that both Unicode escape sequences are "\ufffd", which is the escape sequence for the Unicode replacement character �. We've run into one of the quirks of Unicode surrogate pairs, which is that each "surrogate half" is not individually a valid Unicode code point. As a result, when we convert our []rune into a string, Go interprets each surrogate half as two separate invalid Unicode code points and replaces each with the Unicode replacement character. If we try to run just the []rune to string conversion code, we see the same result.

fmt.Println(string([]rune{high, low}))
// -> ��

Alright, no string this time. Let's use the Unicode format verb from the "fmt" package to get the raw Unicode code point for each surrogate half.

fmt.Printf("%U %U\n", r1, r2)
// -> U+D83D U+DE00

Great, let's write those as JSON Unicode escape sequences and try it again.

var msg map[string]interface{}
json.Unmarshal([]byte(`{"message":"Success \uD83D\uDE00","ok":true}`), &msg)
fmt.Println(msg)
// -> map[message:Success 😀 ok:true]

It worked! To be clear, Unicode surrogate pairs are confusing and not necessary in the vast majority of cases, but they can help us understand how different Unicode encodings work.

Wrapping Up

If you write Go code, you use Unicode and UTF-8 all the time. Go's strong UTF-8 support probably isn't random, as Rob Pike and Ken Thompson, two of the original authors of Go, wrote the original implementation of UTF-8 in the Plan 9 operating system in 1992. Today, most common software handles UTF-8-encoded text properly, but when something doesn't work, understanding the layers of text encoding can be incredibly helpful to pinpoint the problem.

Additional Reading

^{Cover photo background by Nick Fewings on Unsplash}

Forem: Matt Dale

The Go Playground: Using Modules

Major Versions

Any Version

Sometimes "math/rand" Uses Weighted Dice

Automatic Seeds in Go 1.20

Some Possible Fixes

Use a different PRNG library.

Use crypto/rand instead.

Partition the PRNG values.

Additional Reading

Early Perspectives on Go Generics

What's the Big Deal with Generics?

Go Generics: The Really Really Short Version

Early Perspectives

Generic Type Inference

APIs and Type Switches

Testing

Wrapping Up

Sending 😀 in Go

Unicode in Go

Unicode in JSON

Weird Unicode in JSON

Wrapping Up

Additional Reading

Use `crypto/rand` instead.