Forem: Boston Cartwright

Handling errors concurrently in Go with ErrGroup

Boston Cartwright — Mon, 13 Sep 2021 19:09:21 +0000

Preface

Concurrency is one of Go's strong points and I love working with the paradigm that the Go team has built.

It is a big topic with lots to talk about. I recommend reading through the Effective Go documentation about concurrency in Go to learn about goroutines, channels, and how they all work together.

Error handling is also done differently in Go than other languages, thanks to multiple return values. I recommend reading their blog post on error handling.

ErrGroup

If you don't need to do any further work off of the errors, use an ErrGroup!

An ErrGroup is essentially a wrapped sync.WaitGroup to catch errors out of the started goroutines.

WaitGroup

Here is an example without errors using a WaitGroup (from godoc):

package main

import (
    "sync"
)

type httpPkg struct{}

func (httpPkg) Get(url string) {}

var http httpPkg

func main() {
    var wg sync.WaitGroup
    var urls = []string{
        "http://www.golang.org/",
        "http://www.google.com/",
        "http://www.somename.com/",
    }
    for _, url := range urls {
        // Increment the WaitGroup counter.
        wg.Add(1)
        // Launch a goroutine to fetch the URL.
        go func(url string) {
            // Decrement the counter when the goroutine completes.
            defer wg.Done()
            // Fetch the URL.
            http.Get(url)
        }(url)
    }
    // Wait for all HTTP fetches to complete.
    wg.Wait()
  fmt.Println("Successfully fetched all URLs.")
}

To use a WaitGroup, first create the group:

var wg sync.WaitGroup

Next, for every goroutine, add that number to the group:

wg.Add(1)

Then whenever a goroutine is done, tell the group:

defer wg.Done()

The defer keyword:

It defers the execution of the statement following the keyword until the surrounding function returns.

Read more about it in the tour of go.

Finally, wait for the group to complete:

wg.Wait()

In this case, there are no errors that can occur. Let's look at how it changes if we needed to catch errors, using an ErrGroup.

ErrGroup

Here is the same example as above but using an ErrGroup (from godoc):

package main

import (
    "fmt"
    "net/http"

    "golang.org/x/sync/errgroup"
)

func main() {
    g := new(errgroup.Group)
    var urls = []string{
        "http://www.golang.org/",
        "http://www.google.com/",
        "http://www.somename.com/",
    }
    for _, url := range urls {
        // Launch a goroutine to fetch the URL.
        url := url // https://golang.org/doc/faq#closures_and_goroutines
        g.Go(func() error {
            // Fetch the URL.
            resp, err := http.Get(url)
            if err == nil {
                resp.Body.Close()
            }
            return err
        })
    }
    // Wait for all HTTP fetches to complete.
    if err := g.Wait(); err == nil {
        fmt.Println("Successfully fetched all URLs.")
    }
}

It looks very similar, here are the differences:

First, create the group:

var wg sync.WaitGroup

// VVV BECOMES VVV

g := new(errgroup.Group)

Next, instead of adding every goroutine to the group, call g.Go with the function to be a goroutine. The only requirement is it must have the following signature: func() error. Also, since the ErrGroup will handle when goroutines are completed, there is no need to call wg.Done().

go func(arg string) {
            // Decrement the counter when the goroutine completes.
            defer wg.Done()
            // ... work that can return error here
}(arg)

// VVV BECOMES VVV

g.Go(func() error {
  // ... work that can return error here
})

Finally, wait for the group to finish and handle the errors as needed:

wg.Wait()

// VVV BECOMES VVV

if err := g.Wait(); err == nil {
        fmt.Println("Successfully fetched all URLs.")
}

ErrGroups provide lots of opportunities on handling errors in goroutines.

That being said, ErrGroup is just another tool in the toolbox that should be used when the use case fits.
If some more complex decisions and work needs to be made based off of the errors, a channel is probably better fit.

What do you think?

Fix bugs by reading the room

Boston Cartwright — Tue, 16 Mar 2021 18:37:23 +0000

In a previous project, my team and I were working hard to get ready for a
release. This release was scheduled for the end of the week since our
application was used in an enterprise setting and the weekend is when we
would affect the least amount of users.

We scheduled the deployment for Friday (yes, I know this is a bad idea),and were working towards it.

On Thursday, I woke up sick. For my health, I didn't work and spent the day
resting in bed. However for my team, a new critical bug had surfaced.

In this application, we had what Kent calls a Soul Crushing Component. This component was complex, and had lots of different states it could be in.

To make matters worse, features for this component were added over the course of a year, by many different developers. These developers didn't always follow the same state management pattern laid out at the beginning, so state was being managed in many different ways, including Redux, local storage, React Hooks, and more.

It was the component you ran from.

And thus, the bug was in this component. In essence, if you followed a specific flow of steps, this component would get put in an impossible state, and get stuck appearing that it is loading.

This was marked as a show stopper for the production deployment, and my team quickly tried to find a solution.

I was sick the day this surfaced, and do not know the details of all the attempts to solve it that day.

However, when I came into work the next day it seems we were still working on this bug. I joined a war room talking about it, and was quickly caught up on the details that have been found:

You get stuck in an impossible state: Data has been returned from our API but component still shows loading.
It only happens when you follow a specific path (~5% of users would run into this)
As a workaround, you can refresh the page and it works again (which was unacceptable to the client)

There already was a small team of developers who were attempting to fix this. They were trying to find how the impossible state existed and how they could actually deem it impossible. After spending many hours the day before, they had no success but had come up with a design document on how state is managed in this component.

I recognized in this war room that this was putting the deploy in jeopardy, which would be bad for everyone involved. Seeing the work that had already been done, I started thinking of alternate solutions. We needed a fast solution in order to make deployment in time.

I didn't want to work a lot on refactoring the state management issues because we did not have enough time to fix them. Taking that approach would most likely equal the whole team working the weekend when a new bug was found in production due to our quick changes.

I realized one thing that was found:

As a workaround, you can refresh the page and it works again

And thus I forced a refresh anytime you followed this path. Now this wasn't super pretty, and was a little jarring for the user for about a second. But this solution was a sure-fire fix that would not cause more issues once deployed.

I was able to write this fix within minutes, had a PR open, and we were on schedule to deploy by lunch time.

Reflecting on this, I learned a lesson that I can apply to many more situations in the future: Read the room.

I understood the importance of this deployment, how many users were affected by it, and the possibility of a correct fix causing more issues due to time pressure.

By doing my simple quick fix, we were able to dedicate more time into refactoring the entire state management for this component for the next release. We took our time, and the result ended up being a much cleaner solution that is actually maintainable.

What do you think?

Dependency Injection explained simply

Boston Cartwright — Thu, 31 Dec 2020 14:48:15 +0000

Photo by Mpho Mojapelo on Unsplash

originally published on bostonc.dev

Ever since my first job as a developer, people I've worked with talked about dependency injection and how essential it was. Even with professional programming experience under my belt, it was difficult for me to understand what it really was. Online resources would use more jargon that only left me more confused.

This post will hopefully provide a simple explanation for those new to software development and architecture on what dependency injection is and why it is needed.

Let's say you have a car. For you I imagine you love to play music in your car (and especially love to sing along if you are like me 🎤 🎶).

You got a nice deal on your car. There is a problem though, the stereo in your car is special. It can only play music from a CD which is welded into the stereo. This means that if you get tired of singing Kool & The Gang, you will have to go and get a whole new stereo and then replace your current stereo with it.

class Stereo () {
  constructor(cdName, speakers) {
    this.cdName = cdName;
    this.speakers = speakers;
  }

  function play() {
    speakers.playMusic(this.cdName);
  }
}

This is a lot of manual work and will take some time. Fortunately, stereos aren't actually built this way. Instead, they have a CD reader, in which you can put many CDs in it. It can read any CD as long as the stereo and the CD speak the same language (you can't put a BluRay disk in your stereo!). You can hold CDs and change it any time you want. This is dependency injection.

class Stereo () {
  constructor(speakers) {
    this.speakers = speakers;
  }

  function play(cdName) {
    speakers.playMusic(cdName);
  }
}

This may seem like an obvious way to go about it since we have been using CDs and stereos a long time (although going away with everything being streamed!). However, software is typically written tightly coupled as described in the scenario. Its normal to have an application getting data from an outside source such as a database. But what happens when the client wants it to connect to a different database? With dependency injection, it's as simple as switching them out like a CD in a stereo.

Let's imagine we are writing an application to display current movies shown in theaters to a user. Take a look a this pseudocode:

// function to display movies to the user
function displayMovies() {
  // get movies
  var movies = getMovies()

  // display movies
  showMovies(movies)
}

// function to get movies from the database
function getMovies() {
  // connect to our DB (currently a SQLDB)
  var db = new SQLDB(url).Connect()

  // query for movies
  var movies = db.ExecuteQuery('SELECT * FROM movies')

  return movies;
}

Above we have a simple function in order to get the movies data from a SQL database by connecting to it and executing a query, then finally displaying it to the user.

This works, but is hard to test! In order to test it, we need to have a SQLDB running on any machine we test on populated with mock data. As well as in the future, if suddenly we needed to connect to a different database, such as a NOSQL database, then we have a lot of work to change.

If we wanted to be able to test this easily or change databases, we need to loose the tight coupling from the application and the database. We can do this by defining an interface for a DB with a set of methods it can call (the language spoken) and then ensure that any database we use fulfils that interface.

Let's look at how this could be done (again, psuedocode):

// movie database interface
interface MovieDB {
  GetAllMovies()
  GetMovie(name)
  // .. other methods needed to manipulate movie database
}

// class for MovieDB held on SQLDB
class SQLMovieDB implements MovieDB {

  // variable to hold db object
  var db;

  // constructor for when SQLMovieDB is created
  constructor(url) {
    // connect to DB
    db = new SQLDB(url).Connect()
  }

  // get all movies
  GetAllMovies() {
    var movies = db.ExecuteQuery('SELECT * FROM movies')
    return movies
  }

  // get a specific movie
  GetMovie(name) {
    var movie = db.ExecuteQuery('SELECT * FROM movies where movies.name is ' + name)
    return movie
  }

  // ... other methods to implement interface
}

Nice! Now we have abstracted the movie database into an interface that can be implemented by our SQLMovieDB. Now to display the movies, we can do as so:

Note: For languages that don't support interfaces, you can pass additional parameters to functions to replicate the same behavior.

// function to display movies to the user
function displayMovies() {
  // get movies
  var movies = getMovies()

  // display movies
  showMovies(movies)
}

// function to get movies from the database
function getMovies() {
  // connect to our DB
  var db = new SQLMovieDB(url)

  // query for movies
  var movies = db.GetAllMovies()

  return movies;
}

It is even easier to test! We can create a mock movie DB to be used in our tests:

class MockMovieDB implements MovieDB {
  constructor(url) {
    // don't need to do anything
  }

  // get all movies
  GetAllMovies() {
    var movies = [{ title: 'SOME SUPER COOL MOVIE', ...otheData }, ...otherMovies]
    return movies
  }

  // get a specific movie
  GetMovie(name) {
    var movie = { title: 'SOME SUPER COOL MOVIE', ...otherData }
    return movie
  }

  // ... other methods to implement interface
}

Now we can ensure for the tests that we use the MockMovieDB and easily mock the data without actually having to stand up a database server on the test machine.

Finally, it makes it easy to use another database in the future if needed, for example, using a NOSQL database:

// class for MovieDB held on NOSQLDB
class NOSQLMovieDB implements MovieDB {

  var db;

  constructor(url) {
    db = new NOSQLDB(url).Connect()
  }

  // get all movies
  GetAllMovies() {
    var movies = db.collection('movies').find({});
    return movies
  }

  // get a specific movie
  GetMovie(name) {
    var movie = db.collection('movies').find({ title: name });
    return movie
  }

  // ... other methods to implement interface
}

Now to use this NOSQLMovieDB we only have to change one line in our displayMovies function:

// function to display movies to the user
function displayMovies() {
  // get movies
  var movies = getMovies()

  // display movies
  showMovies(movies)
}

// function to get movies from the database
function getMovies() {
  // connect to our DB
  var db = new NOSQLMovieDB(url) // <---- WE ONLY HAVE TO CHANGE HERE

  // query for movies
  var movies = db.GetAllMovies()

  return movies;
}

We could even move this to an environment variable if we so wanted. We can even eliminate the need for ever rewriting the getMovies function by injecting a MovieDB implementor as a parameter!

Once I really understood dependency injection, a whole world of great software design patterns opened up to me. By using these patterns my code has become cleaner and safer.

Let me know what you think!

Special thanks to Adam Whitehurst for helping prepare this post. 🤘

How to use Git Bisect 🕵️‍♂️

Boston Cartwright — Wed, 02 Dec 2020 17:02:23 +0000

originally published on bostonc.dev

Even though I don't have years of professional programming experience, there has been a number of times where a bug is reported on my application and I cannot find out what change introduced it. This ends up being extremely frustrating and hard to work around. Once I discovered git bisect, I never experienced this frustration again.

Git Bisect

Git Bisect's documentation has a great description:

"This command uses a binary search algorithm to find which commit in your project’s history introduced a bug. You use it by first telling it a "bad" commit that is known to contain the bug, and a "good" commit that is known to be before the bug was introduced. Then git bisect picks a commit between those two endpoints and asks you whether the selected commit is "good" or "bad". It continues narrowing down the range until it finds the exact commit that introduced the change."

As it says, git bisect allows you to find which commit in your repository introduces a bug, or really any difference, such as the commit where performance increased, a specific wording was changed, or even a color change.

This is especially useful in large projects with tens or hundreds of commits a day. Finding the specific commit where a bug was introduced can make fixing it increasingly simpler. Understanding the changes that caused the issue makes it easier to solve it.

How it works

The first step in using git bisect is to find a commit where the bug did not exist yet. This can be really as far back as you want to go. Once you find that commit, keep the commit hash in a safe place. For this example, we will use abc123 as the hash.

Then you can start the git bisect session:

git bisect start
git bisect bad
git bisect good abc123

At this point, git bisect will checkout a commit in between the bad and good commits. At this point, you can test your program to see if the bug exists in this commit.

If the bug exists, you can run:

git bisect bad

If it does not, run:

git bisect good

This process is then repeated until it finds the commit that introduces the bug. You will see an output similar to (along with a description of the commit):

def456 is the first bad commit

At this point you have found the commit, can view the changes that it introduced, and hopefully squash that bug 🐞!

You can exit the git bisect by running:

git bisect reset

Example

I've prepared an example application in which we can go through this together. It is a simple React app bootstrapped with Create React App. You can see the code for this example here.

Here is what the application looks like:

Pretty simple, just has some text as a form of a landing page.

However, originally there was a cool React logo that was spinning on the page! This was not supposed to be removed but was. It looked something like this:

Now I try to keep my commit messages clear on what they are doing, so that finding where this happened would be simple but sadly this doesn't hold up in large projects. Commit messages sometimes end up being less than helpful.

With this project, I did not do a great job of writing good commit messages. Finding the commit that removed the logo will be a lot of manual work as I will have to go through every commit's changes and look for where it is removed. However, with git bisect, this is easy.

First I find a commit where the logo is present. Because this is a small project I went to the first commit of the project (66f768475cd0543370e086d6bea3a06bbc0dd2e0) and confirmed the logo is there.

I then start a git bisect session:

git-bisect-example on master
❯ git bisect start
❯ git bisect bad
❯ git bisect good 66f768475cd0543370e086d6bea3a06bbc0dd2e0
Bisecting: 2 revisions left to test after this (roughly 1 step)
[037a419be3488a3d2208abecdc49bda251b871a1] made some more changes :)

It brought me to the commit with the hash 037a419be3488a3d2208abecdc49bda251b871a1. This commit does not have the logo, thus I mark it as bad:

❯ git bisect bad
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[5e35e0026935c3c3808e87b5691d22d4650bd145] changed some text

This commit has the logo! I mark it as good:

❯ git bisect good
037a419be3488a3d2208abecdc49bda251b871a1 is the first bad commit
commit 037a419be3488a3d2208abecdc49bda251b871a1
Author: Boston Cartwright
Date:   Mon Jun 8 12:48:44 2020 -0400

    made some more changes :)

 src/App.js | 1 -
 1 file changed, 1 deletion(-)

As we can see by the above output, git bisect has successfully found the commit that introduced this bug, 037a419be3488a3d2208abecdc49bda251b871a1. Looking at the changes, I can see that it did truly remove the logo and I can plan on how to fix it.

Obviously with larger repositories and many more commits this can be a big time saver to find which commit introduces the bug. I have used git bisect a number of times now and it has been a huge help.