Forem: Charles Francoise

Working with forks in Go

Charles Francoise — Tue, 09 Jan 2018 09:59:01 +0000

Proper use of forks

Have you ever wanted to create a fork of a Go package you're using? Maybe you need to add a feature, specific to your use case, and there is no other way than to change it inside the code. Or maybe you want to fix a bug and submit a PR to an open-source project.

Whatever the case, Go, with its location-based import system, makes it hard to swap implementations of a package. If you try to just swap it into place, you'll most likely get an error along the lines of: cannot use s (type "fork".SomeStruct) as type "original".SomeStruct in argument to SomeFunc as soon as you pass your data types to a function defined in another file.

You can fix this with heavy refactoring — changing imports in your own code, and typecasting everything coming in from or going out to another package — but it's near impossible on a large project with many files.

Type aliases can help, but with a little bit of Git magic, you can seamlessly swap a package for your fork and keep hacking away.

Fork and fix

I won't go into detail on how to create a fork and submit a PR, GitHub has extensive documentation on the subject. I'll just assume you've clicked the "Fork" button on GitHub, and now have your own repository containing a duplicate of the original source.

But as we've said, you can't just go get the forked repository, since all the import paths will be wrong. What we want is to have your local branch, inside your GOPATH, track your forked remote repository, instead of the original.

Let's start by adding a remote. From the package source path, type the following:

$ git remote add forked <remote repository url>

Then, let's make sure the local branch tracks the forked remote, and not the origin any more.

$ git branch -u forked/master
Branch master set up to track remote branch master from forked.

After this line, you can start committing and pushing to your own fork, and all Go packages relying on this repo will transparently use your own fork of the code.

This is useful, for example, if you're sure of a bugfix, as you can start working with the fixed version of the code without waiting for the changes to be merged upstream.

When you want to revert to tracking the origin, perhaps because your PR has been merged and you have deleted your fork, or because you need to use the original code for some other project, just type:

$ git branch -u origin/master
Branch master set up to track remote branch master from origin.

# Reset local branch to remote HEAD
$ git reset --hard origin/master

With these few lines of git, I can quickly fork and work on an open-source Go package. It's made my collaboration on open-source projects that much easier. What is your workflow for collaborating on open-source Go projects? Do you see any mistakes or caveats? Feel free to comment below!

Hacking Go Interfaces

Charles Francoise — Wed, 26 Jul 2017 12:42:30 +0000

Introduction

Usually, in programming, interfaces precede implementation. We know what we want the pieces of our architecture to be able to do, how they'll articulate, and we only start banging out code once we're sure about this. Interfaces make it possible to reason "from the outside in", deciding the behavior of larger parts, before we break down their inner workings.

But often, we're "hacking" our way "from the inside out", building some parts we know are essential first, and putting them together or wrapping them in bigger parts later. I think there's a way to show why and how to use interfaces in Go from the inside out.

What should we hack?

I decided to hack a (very naive) logging framework. The idea is: everyone has an idea of what a logging framework should do, and that's kind of like having an interface (conceptually). Plus, I don't need to introduce any specific concepts that could distract us from what we're building. Finally, a logging framework is typically the kind of stuff we start hacking, wrap into some other part, and come back to later when we need to improve it.

So, without any prior design, let's start banging out a logging framework in Go. I commited each chapter into a GitLab repository so you can follow the progress in the code base along with the chapters. Each chapters links to the commit at that point in the reading.

Version 1 – Standard output

My very first version of the package will just have one package-level function, Log, that will output messages to standard output, preceded by the time the message was logged. (Go has a pretty fun way to format time to a string, much easier than the old strftime.)

Package:

// Package ezlog is a naive log implementation
// used as experiment and tutorial in learning Go
package ezlog

import (
    "fmt"
    "time"
)

// Log logs a message to standard output preceded
// by the time the message was emitted
func Log(msg string) {
    now := time.Now()
    fmt.Printf("[%s] %s\n", now.Format("2006-01-02 03:04:05"), msg)
}

Example:

package main

import "gitlab.com/loderunner/ezlog"

func main() {
    ezlog.Log("Hello World!")
}

$ go run examples/hello.go
[2017-07-25 06:20:55] Hello World!

It's nothing fancy, but it works. We can start logging to standard output using this package now. But of course, it's not nearly enough.

Version 2 – File logging

The next thing we're going to log to is a file. We'll need the client of the package to give the path of a file to log to, before they start calling Log. What this tells us is that our package now needs a step of initialization, before it can be used. Let's add some code.

We add an Open function to the package that opens a file and sets it to a package variable, if successful.

var logfile *os.File

// Open opens a file to append to, creating
// it if it doesn't exist
func Open(path string) error {
    f, err := os.OpenFile(
        path,
        os.O_WRONLY|os.O_APPEND|os.O_CREATE,
        0644,
    )
    if f != nil {
        logfile = f
    }
    return err
}

We then log to the file inside the Log function.

func Log(msg string) {
    // Format the output string
    now := time.Now()
    output := fmt.Sprintf("[%s] %s", now.Format("2006-01-02 03:04:05"), msg)

    fmt.Println(output)           // Log to stdout
    fmt.Fprintln(logfile, output) // Log to a file
}

We extend the test program to add the file initialization.

func main() {
    ezlog.Open("hello.log")
    ezlog.Log("Hello World!")
}

When we run this program, we find a new file hello.log in the directory. It contains the same output as we saw in the console in the console.

$ go run examples/hello.go
[2017-07-25 09:22:21] Hello World!
$ cat hello.log
[2017-07-25 09:22:21] Hello World!

"We can log to standard output and to a file and we still don't need interfaces!" Well, yes. But we're starting to see the limits of our architecture. What if we need to log to two files? What if we don't want to log to standard output? What happens if we want to log to syslog or over the network? Plus, the code is starting to look a little ugly.

Version 3 – Loggers

Loggers

The answers to the above questions are not: add package-level variables, initialization functions, and make Log a monolithic function. We want this to be modular. What we need is an arbitrary number of "loggers" that can we can configure and give to the package to do the work. Let's write two structs two encapsulate the two behaviors. One for standard output, one for files.

Standard output logger:

// StdoutLogger logs messages to stdout
type StdoutLogger struct{}

// NewStdoutLogger StdoutLogger constructor
func NewStdoutLogger() *StdoutLogger {
    return &StdoutLogger{}
}

// Log logs the msg to stdout
func (l *StdoutLogger) Log(msg string) {
    fmt.Println(msg)
}

File logger:

// FileLogger logs messages to a file
type FileLogger struct {
    f *os.File
}

// NewFileLogger opens a file to append to and
// returns a FileLogger ready to write to the file
func NewFileLogger(path string) (*FileLogger, error) {
    f, err := os.OpenFile(
        path,
        os.O_WRONLY|os.O_APPEND|os.O_CREATE,
        0644,
    )
    if err != nil {
        return nil, err
    }
    return &FileLogger{f}, nil
}

// Log logs a message to the file
func (l *FileLogger) Log(msg string) {
    fmt.Fprintln(l.f, msg)
}

We've isolated the loggers, how do we put them together in the package?

var Stdout *StdoutLogger
var File *FileLogger

// ... omitted code here ...

    if Stdout != nil {
        Stdout.Log(output) // Log to stdout
    }
    if File != nil {
        File.Log(output) // Log to a file
    }

Now this just looks silly. What if we want more files? An array of FileLoggers? And still no solution for other logger types that scales elegantly.

The interface

It's easy to see that the main package's Log function just prepares the output, then calls Log on each of the loggers. But the static typing in Go prevents us from just resolving the calls at runtime. The compiler needs to be sure that whatever we're giving it, it knows how to Log.

"Whatever this is, I need it to Log", is exactly what interfaces do in Go. When you ask for an interface type as an argument to a function, what your code is telling the compiler is: "this function takes a pointer to something, I don't care what it is, as long as it has all the methods from the interface". When returning an interface type from a function, you're saying: "don't ask what this is, all you need to know is that it does this". You can imagine an interface as a contract or requirements that the underlying type needs to fulfil.

Here, both of our loggers implement a Log method, so we can say they have a common interface. Let's hack it up.

type Logger interface {
    Log(string)
}

That's it. By writing these lines, we just defined a Logger interface. No need to change anything to StdoutLogger and FileLogger. The compiler will check if they have the Log method to determine if they uphold the interface, without any indication from our part. Anyone coming from C++ or Java will know how cool this is. And if we ever extend the interface to add new methods, it will still break at compile-time if we try to use a non-conforming type, since it won't implement the new methods.

We can now start using Logger as a type.

var loggers []Logger

// AddLogger adds a logger to the list
func AddLogger(l Logger) {
    loggers = append(loggers, l)
}

// Log logs a message to standard output preceded
// by the time the message was emitted
func Log(msg string) {
    // Format the output string
    now := time.Now()
    output := fmt.Sprintf("[%s] %s", now.Format("2006-01-02 03:04:05"), msg)

    // Log to all loggers
    for _, l := range loggers {
        l.Log(output)
    }
}

And that's it. It just works. By creating an interface that declares one method, that both of our loggers conform to, we can elegantly bring this all together in a few lines of code.

Let's make a more complex example to see what we can do now.

func main() {
    // Log to stdout
    ezlog.AddLogger(ezlog.NewStdoutLogger())

    for now := range time.Tick(1 * time.Second) {
        // The seconds of the current time ends with 5
        if (now.Second() % 5) == 0 {
            // Add a new file to log to
            filename := fmt.Sprintf("gopher-%s.log", now.Format("030405"))
            fileLogger, err := ezlog.NewFileLogger(filename)
            if err == nil {
                ezlog.AddLogger(fileLogger)
            } else {
                ezlog.Log("Couldn't open new file.")
            }
        }

        ezlog.Log("Gopher!")
    }
}

This example dynamically adds loggers to a new file every 5 seconds. Here's how it works.

What next?

Well, there's a lot to do.

This framework is by no means concurrency-safe. If several goroutines call AddLogger at the same time, we'll end up with a data race to the loggers array and could end up losing data. Heck, it's not even concurrent itself! We could probably optimize it by using goroutines and channels for different types of loggers. We could even use Go's built-in buffered channels to implement cheap spooling of logs.

There are many more loggers we might want to implement. In fact, I've already hacked together a network and a syslog logger (which was actually a fun experience in itself that I'll talk about in another post).

Here is a list of a few features I still want to add:

Leveled logs
Log formatting
Named loggers
Concurrency & optimization
... Removing loggers (trickier than it seems)

In the end, this package is also a fun playground to learn Go on. I'll intend to keep fiddling with it for a while, just to see what I can do. Check out the repository to see how it's progressing. Feel free to comment, suggest ideas, or even contribute to the package.

Go, I love you, but you're bringing me down.

Charles Francoise — Mon, 24 Jul 2017 16:00:39 +0000

I've said it before, and I'll say it again: I really like Go. It's the most fun I've had learning a language since Python. It's statically typed and compiled – and as a systems-oriented programmer, I never found anything that could really move me away from C because of this – it has a great standard library, it handles code reuse without inheritance beautifully, and those concurrency paradigms have changed my life for ever.

That being said, I quickly had a few issues with some of the language's specifics. And while I've been trying to learn its patterns and conventions as best I can, I'm pretty certain that some of them could be fixed inside the language, and not by using a specific pattern or abstraction. For the sake of simplicity.

As I was reading Russ Cox's Gophercon 2017 talk about Go 2, I decided to take him up on this:

We need your help.
Today, what we need most is experience reports. Please tell us how Go is working for you, and more importantly not working for you. Write a blog post, include real examples, concrete detail, and real experience. That's how we'll start talking about what we, the Go community, might want to change about Go.

So here are, as a new Go programmer, my 2 cents about what's missing.

Operator overloading

Got your attention? Good. Because I hate operator overloading. I've been through too many C++ code bases where developers thought it was better to be clever than to write a clean interface. I've had my share of over-enthusiastic coders re-defining fundamental semantics such as the dot(.), arrow(->) or call(()) operators to do very unexpected things. The cost in documentation and on-boarding new developers was huge, and the mental work of switching paradigms when coding made the whole thing quite error-prone. The fact that the stdlib does weird things pretty freely (std::cout << "Hello, World!", anyone?) doesn't really help.

But operators are also a great shorthand for some clearly defined semantics. In other words, if semantics precede the operator, there's a far lesser chance to abuse them. An example would be the [] operator for integer-based indexing (such as arrays or slices), or for key-value operations (such as maps). Or the usage of the range keyword to iterate over arrays, slices, maps and channels.

One way to make sure these semantics are well respected would be to make the operators as syntactic sugar to well-defined interfaces.

Key-value operator:

struct KeyValueMap interface {
    Value(interface{}) interface{}
    SetValue(interface{}, interface{})
}

We could then use the [] operator as a shorthand to these functions.

foo := keyVal["bar"]

would be equivalent to

foo := keyVal.Value("bar")

The objects used as key would need to be hashable or comparable in some way. Perhaps the compiler could enforce the same rules it does with maps.

Range iteration could be done similarly:

type Iterator interface {
    Next() (interface{}, bool)
    Reset()
}

Next would return the next value in the iteration, and a bool that would be true when the iterator is finished. Reset would be used to initialize the iterator at the beginning of the for loop.

for v := range it {

would be equivalent to

it.Reset()
for !done {
    i, done := it.Next()
    if done {
        break
    }

I think this would allow Go developers to implement much more go-ish structures and I can see many cases of memory-efficient iterators that would make use of this clean syntax.

Of course, this kind of breaks the strong typing in Go. Maybe a feature like this would go hand-in-hand with generics?

Initialization of embedded interfaces

We recently had a case of an issue that took us a while to debug. We were using type embedding to compose different services in one structure. At first we had just one handler.

type FooHandler interface {
    // some `FooHandler` methods
}

type Handler struct {
   FooHandler
}

func main() {
    // Foo is a concrete type implementing FooHandler
    h := Handler{FooHandler: &Foo{}}
    RegisterHandler(h)

We embedded a second handler, but forgot to add it in the Handler initialization. The error message was pretty cryptic.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x10871d6]

goroutine 1 [running]:
main.RegisterHandler(0x1108140, 0x1126ac8, 0x0, 0x0)

This was fixed by adding the initialization.

    h := Handler{FooHandler: &Foo{}, BarHandler: &Bar{}}

I wish the error message would have been clearer. Such a simple omission shouldn't require 30 minutes and two programmers. Maybe we're just new at Go, but I wish the error would have been more explicit. That or the compiler could warn against using zero values for embedded types. I don't know if there's a case where you would want to embed a nil value. This would probably be a breaking change for a few code bases, but the benefit in memory safety could be worth it.

Closed channels

EDIT: The issue I detailed here was only because of a feature of the language I didn't know of. It was pointed out in the comments and on Twitter (and here). I'll be showing Go way to avoid the problem I had.

In the following program, the final loop loops forever while receiving zero values from both channels, ca and cb.

func main() {
    // Create a first channel
    ca := make(chan int)
    go func() {
        // Send one integer over the channel every millisecond up to 9
        for i := 0; i < 10; i++ {
            ca <- i
            time.Sleep(1 * time.Millisecond)
        }
        close(ca)
    }()

    // Create a second channel
    cb := make(chan int)
    go func() {
        // Send one integer over the channel every millisecond up to 99
        for i := 0; i < 100; i++ {
            cb <- i
            time.Sleep(1 * time.Millisecond)
        }
        close(cb)
    }()

    // Receive from the first available channel and loop
    for {
        select {
        case n := <-ca:
            fmt.Printf("a%d ", n)
        case n := <-cb:
            fmt.Printf("b%d ", n)
        }
    }
}

This problem can be fixed by reading two values from the channel. By using n, ok := <-ca, ok will be false if the channel is closed. Once the channel has been closed, we set it to nil as receiving from a nil channel blocks.

    // Read from first available channel and loop
    // until both channels are nil
    for ca != nil || cb != nil {
        select {
        case n, ok := <-ca:
            if ok {
                fmt.Printf("a%d ", n)
            } else {
                // If the channel has been closed, set it to nil
                // Receiving from a nil channel blocks, so we know
                // this select branch will be unreachable after this
                ca = nil
            }
        case n, ok := <-cb:
            if ok {
                fmt.Printf("b%d ", n)
            } else {
                cb = nil
            }
        }
    }

I'm still pretty new to Go, so I'm fairly certain these are either misconceptions on my part, or things that have been tried and abandoned because they didn't work. Feel free to give me any documentation or pointers in the comments.

Writing safer C with Clang address sanitizer

Charles Francoise — Mon, 06 Feb 2017 15:35:17 +0000

Prologue

We wanted to improve our password strength algorithm, and decided to go for the industry-standard zxcvbn, from the people at Dropbox. Our web front-end would use the default Javascript library, and for mobile and desktop, we chose to use the C implementation as it was the lowest common denominator for all platforms.

Bootstrapping all of this together was done pretty fast. I had toyed around with a few sample passwords so I decided to run it through the test suite we had for the previous password strength evaluator. The test generates a large number of random passwords according to different rules and expects the strength to be in a given range. But the test runner kept crashing with segmentation faults.

It turns out the library has a lot of buffer overflow cases that are usually "harmless", but eventually crash your program when you run the evaluator function too much. I started fixing the cases I could see, but reading someone else's algorithms to track down tiny memory errors got old pretty fast. I needed a tool to help me.

That's when I thought of Clang's Address Sanitizer.

Meet Asan

From the Clang documentation:

AddressSanitizer is a fast memory error detector. It consists of a compiler instrumentation module and a run-time library. The tool can detect the following types of bugs:

Out-of-bounds accesses to heap, stack and globals

Use-after-free

Use-after-return

Use-after-scope

Double-free, invalid free

Memory leaks (experimental)

That covers a large portion of the crashes I've faced in C. And its impact on performance?

Typical slowdown introduced by AddressSanitizer is 2x.

Perfectly acceptable in a lot of debugging contexts. Especially if it saves you hours of trying to find the proverbial needle in a corrupt stack.

First steps

Let's try the sanitizer on a simple program. We'll allocate a buffer on the heap, copy each character of a string into it, and print it to standard output.

int main() {
    const char* hello = "Hello, World!";
    char* str = malloc(13 * sizeof(char)); // "Hello, World!" is 13 characters long

    for (int i = 0; i < 13; i++) {
        str[i] = hello[i];
    }
    str[13] = 0; // Don't forget the terminating nul character

    printf("%s\n", str);

    return 0;
}

If you compile and run this program, chances are it will work as predicted.

$ clang -o clang-asan clang-asan.c
$ ./clang-asan
Hello, World!

But it shouldn't. Or it should. It's undefined.

When assigning 0 to the character at index 13, we're writing out of the array bounds, into unallocated memory. While harmless in this case, this is a classic buffer overflow error.

Let's try compiling it again with the address sanitizer.

$ clang -fsanitize=address -g -o clang-asan clang-asan.c
$ ./clang-asan

Wow. Such output. A E S T H E T I C !

What happened?

So what can we gather from that pile of hex? Let's go through it line by line.

==36293==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200000ef3d
at pc 0x000105960da8 bp 0x7fff5a29fa30 sp 0x7fff5a29fa28

AddressSanitizer found a heap buffer overflow at 0x60200000ef3d, a seemingly valid address (not NULL or any other clearly faulty value). The rest of the addresses are the program counter, base pointer and stack pointer registers.

WRITE of size 1 at 0x60200000ef3d thread T0
    #0 0x105960da7 in main clang-asan.c:10
    #1 0x7fffa6c46254 in start (libdyld.dylib+0x5254)

Write of size 1 at line 10. What's line 10?

str[13] = 0;

We're writing outside of the heap in this instruction. And AddressSanitizer isn't having it.

0x60200000ef3d is located 0 bytes to the right of 13-byte region
[0x60200000ef30,0x60200000ef3d)

This is definitely one of my favorite indications. In addition to telling which line in the code failed and where in the memory the failure happened, you get a complete description of the closest allocated region in memory (which is probably the region you were trying to access).

Power to the debugger

We've seen that the address sanitizer has some pretty nifty tricks up its sleeve. The main benefit is that it will break your program on virtually any bad memory access. But its real power comes when used in conjunction with a debugger.

Let's run our program again, with lldb this time.

$ lldb -- ./clang-asan
(lldb) target create "./clang-asan"
Current executable set to './clang-asan' (x86_64)
(lldb) run

As expected, the program crashes again, at the same line. But let's look at what we get after the usual output.

(lldb) AddressSanitizer report breakpoint hit. Use 'thread info -s' to get extended information about the report.
Process 40049 stopped
* thread #1: tid = 0x13f073, 0x00000001000e30a0 libclang_rt.asan_osx_dynamic.dylib`__asan::AsanDie(), queue = 'com.apple.main-thread', stop reason = Heap buffer overflow detected
    frame #0: 0x00000001000e30a0 libclang_rt.asan_osx_dynamic.dylib`__asan::AsanDie()
libclang_rt.asan_osx_dynamic.dylib`__asan::AsanDie:
->  0x1000e30a0 <+0>: pushq  %rbp
    0x1000e30a1 <+1>: movq   %rsp, %rbp
    0x1000e30a4 <+4>: pushq  %rbx
    0x1000e30a5 <+5>: pushq  %rax

As we can see, the program doesn't stop on a segmentation fault in the main() function, but rather in a special __asan::AsanDie() function, in libclang_rt.asan_osx_dynamic.dylib. Obviously, AddressSanitizer is responsible for triggering the breakpoint. What is the actual stack trace?

(lldb) bt
* thread #1: tid = 0x13f073, 0x00000001000e30a0 libclang_rt.asan_osx_dynamic.dylib`__asan::AsanDie(), queue = 'com.apple.main-thread', stop reason = Heap buffer overflow detected
  * frame #0: 0x00000001000e30a0 libclang_rt.asan_osx_dynamic.dylib`__asan::AsanDie()
    frame #1: 0x00000001000e8198 libclang_rt.asan_osx_dynamic.dylib`__sanitizer::Die() + 88
    frame #2: 0x00000001000e0a29 libclang_rt.asan_osx_dynamic.dylib`__asan::ScopedInErrorReport::~ScopedInErrorReport() + 249
    frame #3: 0x00000001000e0151 libclang_rt.asan_osx_dynamic.dylib`__asan::ReportGenericError(unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned long, unsigned int, bool) + 3953
    frame #4: 0x00000001000e11e9 libclang_rt.asan_osx_dynamic.dylib`__asan_report_store1 + 57
    frame #5: 0x0000000100000da8 clang-asan`main + 328 at clang-asan.c:10
    frame #6: 0x00007fffa6c46255 libdyld.dylib`start + 1
(lldb) frame select 5
frame #5: 0x0000000100000da8 clang-asan`main + 328 at clang-asan.c:10
   7        for (int i = 0; i < 13; i++) {
   8            str[i] = hello[i];
   9        }
-> 10       str[13] = 0;
   11
   12       printf("%s\n", str);
   13

We can see that the asan library takes over as soon we attempt to store something into memory at line 10.

A couple other lines caught my attention. Let's start with:

AddressSanitizer report breakpoint hit. Use 'thread info -s' to get extended information about the report.

If we try it, we get a JSON output that nicely recaps the error, and that we can bring up if we need it again.

(lldb) thread info -s
thread #1: tid = 0x13f073, 0x00000001000e30a0 libclang_rt.asan_osx_dynamic.dylib`__asan::AsanDie(), queue = 'com.apple.main-thread', stop reason = Heap buffer overflow detected

{
  "access_size" : 1,
  "access_type" : 1,
  "address" : 105690555281181,
  "description" : "heap-buffer-overflow",
  "instrumentation_class" : "AddressSanitizer",
  "pc" : 4294970792,
  "stop_type" : "fatal_error"
}

The other line that piqued my interest was right after we launched the program:

AddressSanitizer debugger support is active. Memory error breakpoint has been installed and you can now use the 'memory history' command.

The documentation for the memory history command is:

(lldb) help memory history
     Print recorded stack traces for allocation/deallocation events associated with an address.

Syntax: memory history <address>

Sounds pretty powerful, let's give it a try.

(lldb) memory history str
  thread #4294967295: tid = 0x0001, 0x00000001000d8bf0 libclang_rt.asan_osx_dynamic.dylib`wrap_malloc + 192, name = 'Memory allocated by Thread 1'
    frame #0: 0x00000001000d8bf0 libclang_rt.asan_osx_dynamic.dylib`wrap_malloc + 192
    frame #1: 0x0000000100000c85 clang-asan`main + 37 at clang-asan.c:6
    frame #2: 0x00007fffa6c46254 libdyld.dylib`_dyld_process_info_notify_release + 44

We get the exact stack trace at the time this address in memory was allocated. In a more complex program, we would get the history of all the times the address was allocated and deallocated, making it a very powerful tool to understand cryptic memory bugs.

Putting it all to use

Back to my practical case, how did I put the address sanitizer to good use? I simply ran the test suite, compiled with the sanitizer, with lldb. Sure enough, it stopped on every line that could cause a crash. It turns out there were many cases where zxcvbn-c wrote past the end of allocated buffers, on the heap and on the stack. I fixed those cases in the C library and ran the tests again. Not a segfault in sight!

What next?

I've used memory tools in the past, but they were usually unwieldy, or put such a toll on performance that they were useless in any real-life case. Clang's address sanitizer turned out to be detailed, reliable, and surprisingly easy to use. I've heard of the miracles of Valgrind but macOS hardly supports it, making it a pain to use on my MacBook Pro.

Coupled with Clang's static analyzer, AddressSanitizer is going to become a mandatory stop for evaluating code quality. It's also going to be the first tool I grab when facing confusing memory issues. There are many more case where I could use early failure and memory history to debug my code. For example, if a program crashes when accessing member of a deallocated object, we could easily trace the event that caused the deallocation, saving hours of adding and reading logs to retrace just what happened.

Can you think of cases where the address sanitizer could be put to use? Cases where it would miss bugs or result in false positives? Tell me in the comments!

Hi, I'm Charles Francoise

Charles Francoise — Mon, 12 Dec 2016 15:39:30 +0000

You can find me on GitHub as @loderunner

I live in Paris.

I work for Dashlane