Forem: Chidi Williams

Partially updating an embedded Mongo document in Go

Chidi Williams — Thu, 21 May 2020 18:12:50 +0000

Let's consider a Go application that stores a user's name and address information in MongoDB.

We can represent the database model with the following structs:

type User struct {
  ID      bson.ObjectID `bson:"_id,omitempty"`
  Name    string        `bson:"name,omitempty"`
  Address Address       `bson:"address,omitempty"`
}

type Address struct {
  Street    string    `bson:"street,omitempty"`
  City      string    `bson:"city,omitempty"`
  State     string    `bson:"state,omitempty"`
  VisitedAt time.Time `bson:"visitedAt,omitempty"`
}

To save a new user document, we call the collection's Insert method, passing in the user struct.

db.C("users").Insert(User{
  ID:   bson.NewObjectID(),
  Name: "Buffy Summers",
  Address: Address{
    Street: "1630 Revello Drive",
    City:   "Sunnydale",
    State:  "CA",
  },
})

This creates a MongoDB document with an embedded document at the address field:

{
    "_id" : ObjectId("5ec50eaaceb41275cce6a0cc"),
    "name" : "Buffy Summers",
    "address" : {
        "street" : "1630 Revello Drive",
        "city" : "Sunnydale",
        "state" : "CA"
    }
}

After visiting Buffy in her home in Sunnydale, we want to update the VisitedAt field of the address embedded document. To do this, we would call the collection's UpdateId function:

db.C("users").UpdateId(id, User{Address: {VisitedAt: time.Now().UTC()}})

The Mongo document then becomes:

{
    "_id" : ObjectId("5ec50eaaceb41275cce6a0cc"),
    "name" : "Buffy Summers",
    "address" : {
        "visitedAt": ISODate("2020-05-20T11:44:08.327+0000")
    }
}

Oops—that's not what we want. Instead of setting only the visitedAt field, the UpdateId function replaced the entire embedded document.

To understand why this happened, let's inspect the equivalent Mongo command:

db.users.update(
  { _id: ObjectId('5ec50eaaceb41275cce6a0cc') },
  {
    $set: {
      address: {
        visitedAt: ISODate('2020-05-20T11:44:08.327+0000'),
      },
    },
  },
);

The command effectively set the address field to a new embedded document with a single visitedAt field, erasing the other address fields.

So how do we partially update an embedded document?

The Mongo libraries (mgo and mongo-go-driver) don't provide a way to do this easily with Go structs. But there are some workarounds we can try.

1. Copy the existing embedded document

We could copy the old embedded document into the new struct first before running the update command.

user := User{
  ID:   ObjectIDHex("5ec50eaaceb41275cce6a0cc"),
  Name: "Buffy Summers",
  Address: Address{
    Street: "1630 Revello Drive",
    City:   "Sunnydale",
    State:  "CA",
  },
}

db.C("users").UpdateId(id, User{
  Address: {
    Street: user.Address.Street,
    City:   user.Address.City,
    State:  user.Address.State,
    VisitedAt: time.Now().UTC(),
  },
})

This would update the visitedAt field and preserve the other fields we wish to leave unchanged.

Limitations

If we don't have the existing embedded document on-hand, we need to make a query to get it first.
If the document we have is stale and the nested fields have changed after we retrieved them but before the update, we would revert those fields to their previous, now-incorrect values.

2. Flatten the document

An alternative is to not use embedded documents altogether. In our case, we would merge the Address fields into the User field.

type User struct {
  ID               bson.ObjectID `bson:"_id,omitempty"`
  Name             string        `bson:"name,omitempty"`
  AddressStreet    string        `bson:"addressStreet,omitempty"`
  AddressCity      string        `bson:"addressCity,omitempty"`
  AddresState      string        `bson:"addressState,omitempty"`
  AddressVisitedAt time.Time     `bson:"addressVisitedAt,omitempty"`
}

Limitations

Depending on the application and whether or not we have existing data, changing the document schema may not be feasible.
We would also miss out on the advantages embedding provides, like reusability. For example, to add a secondary address to the user struct, we would simply add a single SecondaryAddress field with the same Address struct type, instead of duplicating all four address fields.

3. Inline the embedded document

The inline struct tag tells the mongo library to treat the fields of the tagged struct as part of the outer struct. With this tag, the document retains a nested structure in the application code but lives as a flattened document in the Mongo collection.

type User struct {
  ID      bson.ObjectID `bson:"_id,omitempty"`
  Name    string        `bson:"name,omitempty"`
  Address Address       `bson:"address,inline,omitempty"`
}

type Address struct {
  Street    string    `bson:"addressStreet,omitempty"`
  City      string    `bson:"addressCity,omitempty"`
  State     string    `bson:"addressState,omitempty"`
  VisitedAt time.Time `bson:"addressVisitedAt,omitempty"`
}

Limitations

The resulting Mongo document is the same as the flattened document in the previous solution, so this also requires a schema change.

4. Use a manual update command

We can also manually construct an update document to target specific fields:

db.C("users").UpdateId(id, bson.M{
    "$set": bson.M{"address.visitedAt": time.Now().UTC()},
})

Limitations

Though this seems like an easy way to retain the nested structure in the application and database, it becomes increasingly inconvenient with more models and types.

5. Auto-generate the update document

To fix the limitation of the manual update method, we can write a function that receives a nested struct and returns an update document containing all the "non-nil" fields.

db.C("users").UpdateId(id, F(User{Address: {VisitedAt: time.Now().UTC()}}))

// F(...) returns:
//
// map[string]interface{}{"address.visitedAt": time.Time{...}}

FlatBSON

FlatBSON is a Go package I wrote to provide this utility function.

flatbson.Flatten recursively iterates through all the fields of a given nested struct and assembles a flattened map[string]interface{}. It merges nested field names using dot syntax, e.g. address.visitedAt.

It also uses the BSON struct tags in line with the BSON codec specification. The function supports the BSON name, omitempty, inline, and skip tags.

Tag	Description
omitempty	Ignore the field if it's set to the zero value for its type or an empty slice or map.
inline	Inline the struct field, merging all the child fields to the outer struct. The keys must not conflict with the bson keys of other struct fields.
skip	Skip the struct field by setting "-" for the tag name.

The package uses reflection and is compatible with both the mgo and mongo-go-driver libraries.

chidiwilliams / flatbson

Recursively flatten a Go struct using its BSON tags

Thank you for reading. If you liked this post, please leave a ❤️ or a comment below.

Thanks to icza, whose Stack Overflow answer inspired the package. And thanks to Ope for reading initial drafts of this post.

Writing cleaner Go web servers

Chidi Williams — Fri, 01 May 2020 17:17:06 +0000

Introduction

Writing clean, high-quality code makes programs easier to understand, maintain, improve, and test.

In this post, I will share some tips for writing clean, effective Go web servers. These tips are focused on issues related to architecture and error handling in Go.

A complete project containing all the examples is available on Github.

Separate concerns with clean architecture

Clean architecture is a design pattern for separating concerns. Robert “Uncle Bob” Martin, in his book "Clean Architecture: A Craftsman’s Guide to Software Structure and Design", presents this architecture as a way of breaking up an application into loosely-coupled components.

Source: blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html

The architecture divides the application into four main components.

The entities are the business models of the application. They describe the most general requirements of the system.

The use cases layer implements all the use cases in the system. It contains the application-specific business rules and describes how data flows from and to the entities.

The interface adapters layer converts data from external agencies (like the database or the Web) to the format most suitable for the use cases and entities. This layer contains controllers, presenters, and views.

The flow of control points inwards from the external agencies through the external interface adapters and use cases to the entities.

When writing Go web servers, I use the terminology of models as entities, services as use cases, repositories as interface adapters to data sources (e.g. database, external services, etc.), and handlers as interface adapters to the Web.

The handlers depend on and communicate with services, and services depend on repositories (typically one repository to a service at a time) to store and retrieve data.

For example, consider an application that saves data for a new book to an in-memory database:

func CreateBook(w http.ResponseWriter, r *http.Request) {
    requestBody := createBookRequestBody{}
    if err := json.NewDecoder(r.Body).Decode(&requestBody); err != nil {
        writeError(w, err.Error(), http.StatusBadRequest)
        return
    }

    book := models.Book{
        ID: bson.NewObjectId(),
        Title: requestBody.Title,
        CreatedAt: time.Now().UTC(),
    }

    err := db.Update(func(tx *Tx) error {
        b, err := json.Marshal(&book)
        if err != nil {
            return err
        }

        _, _, err = tx.Set("books::"+book.ID.Hex(), string(b), nil)
        return err
    })
    if err != nil {
        writeError(w, err.Error(), http.StatusBadRequest)
        return
    }

    writeSuccess(w, book)
}

This handler function does a number of things: it decodes the HTTP request body, creates a new book, saves it to the database, and then responds to the client. Let's split this into a handler, service, and repository.

// handlers/book/handler.go
func (h bookHandler) CreateBook(w http.ResponseWriter, r *http.Request) {
    requestBody := createBookRequestBody{}
    if err := json.NewDecoder(r.Body).Decode(&requestBody); err != nil {
        writeError(w, err.Error(), http.StatusBadRequest)
        return
    }

    book, err := h.bookService.CreateBook(requestBody.Title)
    if err != nil {
        writeError(w, err.Error(), http.StatusBadRequest)
        return
    }

    writeSuccess(w, book)
}

// services/book/service.go
func (s service) CreateBook(title string) (*models.Book, error) {
   book := models.Book{
       ID: bson.NewObjectId(),
       Title: title,
       CreatedAt: time.Now().UTC(),
    }
   if err := s.repository.CreateBook(book); err != nil {
      return nil, err
   }

   return &book, nil
}

// repository/inmemory.go
func (r inMemoryRepository) CreateBook(book models.Book) error {
    return r.db.Update(func(tx *buntdb.Tx) error {
        b, err := json.Marshal(&book)
        if err != nil {
            return err
        }

        _, _, err = tx.Set("books::"+book.ID.Hex(), string(b), nil)
        return err
    })
}

Now, we've created components that each perform only one function: one decodes the HTTP request and writes the response, another creates the data model, and the last one saves the data to the database.

Admittedly, this makes the code more verbose but it provides many advantages. Each component is easy to understand, easy to maintain, and reusable.

Program to interfaces, not implementations

Instead of relying on the concrete implementation of a module, use an interface. Hide the inner workings of the module behind an interface and it becomes easier to modify the module without breaking other modules.

In the case of our application, the book service is hidden behind a Service interface and the in-memory database repository is hidden behind a Repository interface:

// handlers/book/handler.go
func NewBookHandler(bookService book.Service) BookHandler {
    return bookHandler{bookService}
}

// services/book/service.go
type Service interface {
    CreateBook(title string) (*models.Book, error)
}

func NewService(repository repository.Repository) Service {
    return service{repository}
}

// repository/repository.go
type Repository interface {
    CreateBook(book models.Book) error
}

// repository/inmemory.go
func NewInMemoryRepository(db *db.Client) Repository {
    return inMemoryRepository{db}
}

By loosening the decoupling of the different components in this way, changes to the repository module will not affect the service layer as long as the interface is satisfied, and so on. This makes the application easier to maintain.

Programming to interfaces also makes it easier to test the different layers of the application independently. For example, in a unit test for the book service, we may supply a mock implementation of the book repository instead of the concrete inMemoryDatabaseRepository.

Interfaces also make it easier to swap out dependencies. If we decide to change our data store to MongoDB, we only need to write the adapter (mongoRepository) and then change the repository implementation to be used at runtime.

// repository/mongo.go
func NewMongoRepository(db *mgo.Database) Repository {
    return mongoRepository{coll: db.C("books")}
}

func (m mongoRepository) CreateBook(book models.Book) error {
    return m.coll.Insert(book)
}

// server.go
bookRepository := repository.NewMongoRepository(mongoDB)
// bookRepository := repository.NewInMemoryRepository(inMemoryDB)
bookService := book.NewService(bookRepository)
bookHandler = books.NewBookHandler(bookService)

Simplify error handling with a custom HTTP handler

Let's revisit the book handler function. When an error occurs, the handler returns an error message to the user along with an HTTP status code.

func (h bookHandler) CreateBook(w http.ResponseWriter, r *http.Request) {
    requestBody := createBookRequestBody{}
    if err := json.NewDecoder(r.Body).Decode(&requestBody); err != nil {
        writeError(w, err.Error(), http.StatusBadRequest)
        return
    }

    book, err := h.bookService.CreateBook(requestBody.Title)
    if err != nil {
        writeError(w, err.Error(), http.StatusBadRequest)
        return
    }

    writeSuccess(w, book)
}

As we add more HTTP handlers, this explicit error handling becomes undesirably repetitive. To keep the application DRY, we can define a custom HTTP handler type that returns an error.

type Handler func(w http.ResponseWriter, r *http.Request) error

Then we can change the createBook handler to return errors:

func (h bookHandler) CreateBook(w http.ResponseWriter, r *http.Request) error {
    requestBody := createBookRequestBody{}
    if err := json.NewDecoder(r.Body).Decode(&requestBody); err != nil {
        return err
    }

    book, err := h.bookService.CreateBook(requestBody.Title)
    if err != nil {
        return err
    }

    writeSuccess(w, book)
    return nil
}

To use our Handler type with the http package, we need to implement the http.Handler interface's ServeHTTP method:

func (h Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    if err := h(w, r); err != nil {
        writeError(w, err.Error(), http.StatusBadRequest)
    }
}

Finally, to register the handler to its route, we convert the handler function to the Handler type:

http.Handle("/book", handlers.Handler(bookHandler.CreateBook))

Use custom errors for client errors

To make the application more user-friendly, we should return the error message with an appropriate HTTP status code.

We also need to differentiate errors which should be returned to the client (client errors) from those which should not (server errors).

Client errors are errors related to the request, such as validation, authentication, and permission errors.

Server errors are issues with the internal workings of the application, for example, errors that occur while connecting to the database or an external remote service.

Server errors may contain sensitive information about the database or file system, and for this reason, when they occur, we want to respond with an HTTP 500 Internal Server Error.

To do this, we may create a custom error type containing a message and a type.

type Type string

type AppError struct {
    text    string
    errType Type
}

func (e AppError) Error() string {
    return e.text
}

To create a HTTP 400-like error, we use a new AppError with a TypeBadRequestError type:

// handlers/book/handler.go
func (u handler) GetBook(w http.ResponseWriter, r *http.Request) error {
    // ...

    if ... {
        return errors.Error("invalid vendor ID")
    }

    // ...
}

// errors/error.go
const (
    TypeBadRequest Type = "bad_request_error"
    TypeNotFound Type = "not_found_error"
)

func Error(text string) error {
    return &AppError{text: text, errType: TypeBadRequest}
}

In the ServeHTTP method, we can now improve error handling. If the error matches the custom AppError type, we return the error message with the HTTP status code corresponding to the error's type. If the error is not an AppError, we assume that it is a server error, return an HTTP 500 response, and log the full error for debugging.

func (h Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    err := h(w, r)
    if err == nil {
        return
    }

    appError := new(errors2.AppError)
    if errors.As(err, &appError) { // client error
        writeError(w, err.Error(), errTypeStatusCode(appError.Type()))
        return
    }

    // server error
    log.Println("server error:", err)
    writeError(w, "Internal Server Error", http.StatusInternalServerError)
}

Standardize HTTP response format

Use a response struct to make the server response consistent and predictable.

A simple response format may contain fields for a "success" flag, a display/error message, and data to be returned to the client. Depending on your application, you may need more fields in the response body.

type response struct {
    Body       *responseBody
    StatusCode int
}

type responseBody struct {
    Success bool        `json:"success"`
    Message string      `json:"message,omitempty"`
    Data    interface{} `json:"data,omitempty"`
}

func (r response) ToJSON(w http.ResponseWriter) error {
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(r.StatusCode)
    return json.NewEncoder(w).Encode(r.Body)
}

func OK(message string, data interface{}) *response {
    return &response{&responseBody{Message: message, Data: data}, http.StatusOK}
}

func Fail(message string, statusCode int) *response {
    return &response{&responseBody{Message: message}, statusCode}
}

To use the response struct in the handlers:

// handlers/book/handler.go
type getBookResponse struct {
    Book *models.Book `json:"book"`
}

func (u handler) GetBook(w http.ResponseWriter, r *http.Request) error {
    // ...
    return responses.OK("We found your book!", getBookResponse{retrievedBook}).ToJSON(w)
}

// handlers/handler.go
func (h Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    // ...

    if errors.As(err, &appError) { // client error
        responses.Fail(err.Error(), errTypeStatusCode(appError.Type())).ToJSON(w)
        return
    }

    // server error
    log.Println("server error:", err)
    responses.Fail("Internal Server Error", http.StatusInternalServerError).ToJSON(w)
}

Conclusion

The tips contained in this post are not exhaustive. There might be alternative and better solutions depending on your application but these recommendations can serve as a guide and starting point for writing more effective Go servers.

Forem: Chidi Williams

Partially updating an embedded Mongo document in Go

So how do we partially update an embedded document?

1. Copy the existing embedded document

Limitations

2. Flatten the document

Limitations

3. Inline the embedded document

Limitations

4. Use a manual update command

Limitations

5. Auto-generate the update document

FlatBSON

chidiwilliams / flatbson

Recursively flatten a Go struct using its BSON tags

Writing cleaner Go web servers

Introduction

Separate concerns with clean architecture

Program to interfaces, not implementations

Simplify error handling with a custom HTTP handler

Use custom errors for client errors

Standardize HTTP response format

Conclusion

chidiwilliams / go-web-server-tips

Example project demonstrating tips for writing web servers in Go