Forem: Elton Minetto

App recommendation - LookAway

Elton Minetto — Fri, 30 Jan 2026 13:00:45 +0000

In this first post of 2026, I want to recommend a very simple app that has proven quite useful in my daily life: LookAway.

This application helps improve your health while using macOS. It alerts the user, at configurable intervals, to correct their posture and, most importantly, to "look away" from the monitor. This greatly helps to avoid muscle fatigue and, especially, eye strain. It's also possible to download an iOS app and sync it with your Mac, which locks your phone at the same time as it locks your computer (all customizable, of course) to prevent you from "cheating" in the game by taking your eyes off the computer and using your phone.

On the app's website, you can see screenshots and purchase a license. It's also part of the SetApp subscription, which I've been using (and talking about) since 2017. If you're not yet a subscriber, I highly recommend it. And if you decide to subscribe, use this link, and we both get a free month :)

Originally published at https://eltonminetto.dev

Go should be more opinionated

Elton Minetto — Thu, 19 Jun 2025 18:50:53 +0000

One of the perks of being a Google Developer Expert is the incredible opportunities it provides. A few weeks ago, I had the opportunity to meet Robert Griesemer, co-creator of Go, in person, as well as Marc Dougherty, Developer Advocate for the Go team at Google. At a happy hour after Google I/O, Marc asked me and another Go GDE from Korea for feedback on the language. My response was that I didn't have any specific feedback about the language but that:

Go should be more opinionated about the application layout.

It was worth writing a post to express my thoughts more clearly.

Starting from the beginning…

In 2025, I will have completed 10 years of writing code in Go. One of the things I recall from when I started is that the language was relatively simple to learn, mainly due to two reasons: its simplicity and the fact that there is only one way to do things. Go was the first language I came across that had strong opinions about several things. There is only one way to loop, and there is only one way to format files (using the 'go fmt' command). Variables with a small scope should have short names, etc. It made it much easier to read code written by other people, which is crucial for learning. The code I wrote was very similar to the Kubernetes code! Of course, the complexity of the problem was infinitely greater, but the code's structure was readable to me. Over the years, I have observed this effect in several people I have followed who were starting in the language or migrating from other environments.

But once this initial excitement has passed, the biggest challenge comes: how to adopt Go in a project larger than those used for learning? How do you structure a project that will be developed and evolved by a team? At this point, the language step aside from strong opinions, and each team or company needs to decide how to structure their projects. Over the past decade, I have worked for four companies. In all of them, it was necessary to invest the team's time in collecting examples and reading documentation and books to determine which structure they should use in the projects. At the company where I currently work, we have created a document about this.

Making an analogy with the world of games, it's as if we were having fun in the controlled and wonderful world of Super Mario World and were transported to the open world of GTA 6 (yes! I'm hyped!). It's still a fantastic universe, but the transition is quite abrupt.

Go could be more opinionated regarding these choices. We could have templates for more common projects, such as CLIs, APIs, and microservices., that teams can use to scaffold their applications. The language toolkit already allows the use of project templates, so it would be a matter of having official templates to make life easier for teams. Alternatively, we could go further and include the command in the language toolkit itself with something like go new.

A similar event occurred in the history of the language. Today, go mod dependency management is a fundamental part of our daily lives as Go developers. But it wasn't always like this. For a long time, there was no official package manager for the language; consequently, the community developed several alternatives. They all worked, but fragmentation was getting out of control, making it challenging to integrate packages. Until the language team took control of the situation and go mod was created, pacifying the issue of "package and dependency management." I believe we can apply the same approach to the structure of projects.

Another profile that would benefit from a more opinionated project structure is that formed by teams that are migrating their applications from other languages, especially Java and PHP. In these ecosystems, frameworks dictate the structure of projects, such as Spring Boot and Laravel. "Where do I start? How do I structure my project?" are common questions I hear from teams migrating from these languages. Having something that facilitates this migration would lower the barrier to entry and increase the number of teams experimenting with Go in production.

That's my biggest feedback regarding Go at the moment. What do you think, dear reader? What's your opinion on the subject? I'd love to discuss this topic in the comments or live at a conference.

Originally published at https://eltonminetto.dev on Jun 19, 2025

Creating an MCP Server Using Go

Elton Minetto — Sat, 03 May 2025 16:39:06 +0000

In November 2024, Anthropic published a blog post announcing what may be its most significant contribution to the AI ecosystem so far: the Model Context Protocol.

According to the official definition on the website:

MCP is an open protocol that standardizes how applications provide context to LLMs. Think of MCP as a USB-C port for AI applications. Just as USB-C offers a standardized way to connect your devices to various peripherals and accessories, MCP provides a standardized way to connect AI models to different data sources and tools.

Quickly, other players began announcing support for this new protocol:

To keep this post concise, I won’t discuss the architecture defined by MCP in detail, but I’ll leave some links at the end for those who want to dive deeper.

Still, some basic components are necessary to understand:

MCP Hosts: Programs like Claude Desktop, IDEs, or AI tools that want to access data via MCP
MCP Clients: Protocol clients that maintain 1:1 connections with servers
MCP Servers: Lightweight programs that expose specific resources via MCP

In this example, I’ll create a Server and use some Hosts to test access to it.

Besides the protocol definitions, the official project website offers SDKs for some languages, such as Python, TypeScript, Java, Kotlin, and C#. Although there’s an ongoing discussion about Go support in the official repository, the community has already started creating some protocol implementations. For the proof of concept presented in this post, I used one of these projects, mcp-golang.

The first step was to create a new project:

mkdir mcp-server
go mod init github.com/eminetto/mcp-server

The main.go code looked like this:

package main

import (
    "encoding/json"
    "errors"
    "fmt"
    "io"
    "net/http"
    "os"
    "strings"
    "time"

    mcp_golang "github.com/metoro-io/mcp-golang"
    "github.com/metoro-io/mcp-golang/transport/stdio"
    "github.com/ryanuber/go-filecache"
)

const cacheTime = 500

type MyFunctionsArguments struct {
    ZipCode string `json:"zip_code" jsonschema:"required,description=The zip code to be searched"`
}

// Cep is the brazilian postal code and address information
type Cep struct {
    Cep         string `json:"cep"`
    Logradouro  string `json:"logradouro"`
    Complemento string `json:"complemento"`
    Bairro      string `json:"bairro"`
    Localidade  string `json:"localidade"`
    Uf          string `json:"uf"`
    Unidade     string `json:"unidade"`
    Ibge        string `json:"ibge"`
    Gia         string `json:"gia"`
}

func main() {
    done := make(chan struct{})

    server := mcp_golang.NewServer(stdio.NewStdioServerTransport())
    err := server.RegisterTool("zipcode", "Find an address by his zip code", func(arguments MyFunctionsArguments) (*mcp_golang.ToolResponse, error) {
        address, err := getCep(arguments.ZipCode)
        if err != nil {
            return nil, err
        }

        return mcp_golang.NewToolResponse(mcp_golang.NewTextContent(fmt.Sprintf("Your address is %s!", address))), nil
    })
    if err != nil {
        panic(err)
    }
    err = server.Serve()
    if err != nil {
        panic(err)
    }

    <-done
}

func getCep(id string) (string, error) {
    cached := getFromCache(id)
    if cached != "" {
        return cached, nil
    }
    req, err := http.Get(fmt.Sprintf("http://viacep.com.br/ws/%s/json/", id))
    if err != nil {
        return "", err
    }

    var c Cep
    err = json.NewDecoder(req.Body).Decode(&c)
    if err != nil {
        return "", err
    }
    res, err := json.Marshal(c)
    if err != nil {
        return "", err
    }

    return saveOnCache(id, string(res)), nil
}

func getFromCache(id string) string {
    updater := func(path string) error {
        return errors.New("expired")
    }

    fc := filecache.New(getCacheFilename(id), cacheTime*time.Second, updater)

    fh, err := fc.Get()
    if err != nil {
        return ""
    }

    content, err := io.ReadAll(fh)
    if err != nil {
        return ""
    }

    return string(content)
}

func saveOnCache(id string, content string) string {
    updater := func(path string) error {
        f, err := os.Create(path)
        if err != nil {
            return err
        }
        defer f.Close()
        _, err = f.Write([]byte(content))
        return err
    }

    fc := filecache.New(getCacheFilename(id), cacheTime*time.Second, updater)

    _, err := fc.Get()
    if err != nil {
        return ""
    }

    return content
}

func getCacheFilename(id string) string {
    return os.TempDir() + "/cep" + strings.Replace(id, "-", "", -1)
}

Highlighting the Most Important Parts

Definition of the function parameters that will be available:

type MyFunctionsArguments struct {
    ZipCode string `json:"zip_code" jsonschema:"required,description=The zip code to be searched"`
}

Definition of the tool that will be available to the client and host:

server := mcp_golang.NewServer(stdio.NewStdioServerTransport())
    err := server.RegisterTool("zipcode", "Find an address by his zip code", func(arguments MyFunctionsArguments) (*mcp_golang.ToolResponse, error) {
        address, err := getCep(arguments.ZipCode)
        if err != nil {
            return nil, err
        }

        return mcp_golang.NewToolResponse(mcp_golang.NewTextContent(fmt.Sprintf("Your address is %s!", address))), nil
    })
    if err != nil {
        panic(err)
    }
    err = server.Serve()

The protocol defines that a Server can provide three capabilities:

Resources: Data similar to files that a client can read (such as API responses or file content)
Tools: Functions that can be called by the LLM (with user approval)
Prompts: Pre-written templates that help users perform specific tasks

As seen in the code above, I’m exposing only one tool in this proof of concept. I left out the other items to simplify the example.

The rest of the code is responsible for the tool’s logic, which in this case involves accessing an external API and returning data according to the search. Other examples could include accessing a database or a private cloud resource. The project website has some other examples.

The next step is to generate a binary using: go build.

Now that we have built our Server, we need to configure a Host to use the logic we’re exposing.

Since Anthropic created the protocol, it’s fair that their tool, Claude, is the best example of usage. With the app installed on macOS, just go to the settings area, click “Developer,” and then “Edit Config.” Claude will show you the location of the claude_desktop_config.json file, which I edited and saved with the following content:

{
  "mcpServers": {
    "golang-mcp-server": {
      "command": "/Users/eminetto/Developer/mcp-server/mcp-server",
      "args": [],
      "env": {}
    }
  }
}

There are more possible configurations, but these were sufficient for this scenario.

Now we can ask Claude a question like:

What is the full address for zip code 88034102?

It will notify that to answer this, it needs to access an external tool:

After you give permission, the tool will show the answer as expected.

In this example, I asked the same question in Portuguese and English, and Claude answered it in both languages.

As MCP proposes, we can use the same Server with different Hosts. To test this, I used Cursor, a popular AI-based IDE. Configuring it is also simple: click the settings icon. On the settings screen, an “MCP” option says, “Add new global MCP Server.” The IDE will open a mcp.json file where we fill in the same content used to configure Claude. Now, we can use the IDE’s chat to ask the same question and get the same result:

MCP paves the way for an even broader reach of existing AI tools, especially as more companies adopt the standard. It took me a while to get excited about the possibilities of AIs/GenAI and separate the hype from reality. But that excitement has finally hit me, and I see great application possibilities within companies.

I plan to write another post on the subject when Gemini launches support for MCP (maybe at this year’s Google I/O? I’ll be there and hope to bring news). The same goes for the release of the official Go SDK.

And what’s your opinion on this, dear reader? Do you think it’s just another hype wave, or does this standardization pave the way for new applications?

Links

Originally published at https://eltonminetto.dev on May 01, 2025

First impressions with the Turso database

Elton Minetto — Tue, 25 Mar 2025 11:38:17 +0000

Turso is one of those projects that you look at and think, "How has no one done something like this before?" I've been following the project since its launch, but only now have I been able to dedicate some time to testing, which I describe in this post.

Simply put, it adds a data distribution and synchronization layer using one of the most underrated databases on the market, SQLite.

To start the test, I created a free account on the company's website, which gives me access to enough resources for testing and personal projects.

The next step was to install the CLI on my macOS using the command:

brew install tursodatabase/tap/turso

The documentation shows how to install it on other operating systems.

The next step was to authenticate

turso auth signup

And create the first database:

turso db create demo-post

For this example, I created a single table using the commands:

turso db shell demo-post


CREATE TABLE books (
  ID INTEGER PRIMARY KEY AUTOINCREMENT,
  title TEXT,
  author TEXT,
  category int
);
.quit

Since I'm going to use Go for this example, I created a new project and installed the necessary dependency:

mkdir post-turso
go mod init github.com/eminetto/post-turso
go get github.com/tursodatabase/go-libsql

The simplest example to test is with code similar to:

package main

import (
    "database/sql"
    "fmt"
    "github.com/tursodatabase/go-libsql"
    "os"
    "path/filepath"
    "time"
)

func main() {
    dir, err := os.MkdirTemp("", "libsql-*")
    if err != nil {
        fmt.Printf("creating temporary directory: %w", err)
        return
    }

    dbPath := filepath.Join(dir, os.Getenv("DBNAME"))
    syncInterval := time.Second * 30

    connector, err := libsql.NewEmbeddedReplicaConnector(dbPath, os.Getenv("TURSO_DATABASE_URL"),
        libsql.WithAuthToken(os.Getenv("TURSO_AUTH_TOKEN")),
        libsql.WithSyncInterval(syncInterval),
    )

    if err != nil {
        fmt.Printf("creating connector: %w", err)
        return
    }
    db := sql.OpenDB(connector)
    defer func() {
        err := os.RemoveAll(dir)
        if err != nil {
            fmt.Printf("removing temporary directory: %w", err)
        }
        err = connector.Close()
        if err != nil {
            fmt.Printf("closing connector: %w", err)
        }
        err = db.Close()
        if err != nil {
            fmt.Printf("closing repository: %w", err)
        }
    }()
    id, err := insert(db)
    if err != nil {
        fmt.Printf("inserting: %w", err)
        return
    }
    fmt.Println("Inserted ", id)
    err = showAll(db)
    if err != nil {
        fmt.Printf("showing all: %w", err)
        return
    }
}

func insert(db *sql.DB) (int64, error) {
    stmt, err := db.Prepare(`
        insert into books (title, author, category) 
        values(?,?,?)`)
    if err != nil {
        return 0, fmt.Errorf("preparing statement: %w", err)
    }
    result, err := stmt.Exec(
        "Foundation",
        "Isaac Asimov",
        1, //read
    )
    if err != nil {
        return 0, fmt.Errorf("executing statement: %w", err)
    }
    err = stmt.Close()
    if err != nil {
        return 0, fmt.Errorf("closing statement: %w", err)
    }
    id, err := result.LastInsertId()
    if err != nil {
        return 0, fmt.Errorf("getting last insert ID: %w", err)
    }

    return id, nil
}

func showAll(db *sql.DB) error {
    rows, err := db.Query("SELECT * FROM books")
    if err != nil {
        return fmt.Errorf("selecting books: %w", err)
    }
    defer rows.Close()

    for rows.Next() {
        var id int64
        var title string
        var author string
        var category int64

        if err := rows.Scan(&id, &title, &author, &category); err != nil {
            return fmt.Errorf("scanning book: %w", err)
        }

        fmt.Println(id, title, author, category)
    }

    if err := rows.Err(); err != nil {
        return fmt.Errorf("interacting with books: %w", err)
    }
    return nil
}

As you can see in the code, we need to define some environment variables, especially TURSO_DATABASE_URL and TURSO_AUTH_TOKEN. To do this, we need to generate this data using the commands:

turso db show --url demo-post
turso db tokens create demo-post

These two commands will display the URL and the token, respectively, in the terminal. With this, we can configure the necessary variables:

export DBNAME="local.db"
export TURSO_DATABASE_URL="DATABASE_URL"
export TURSO_AUTH_TOKEN="AUTH_TOKEN"
export PORT="8082"

I will now highlight some essential points from the code above.

dir, err := os.MkdirTemp("", "libsql-*")
if err != nil {
    fmt.Printf("creating temporary directory: %w", err)
    return
}

dbPath := filepath.Join(dir, os.Getenv("DBNAME"))

In this section, we define a local database in a temporary directory. This is one of Turso's interesting details. All writes (inserts, deletes, updates) will be performed in the remote database, identified by the URL and token generated in the previous step. Meanwhile, reads are performed in this temporary database, ensuring high reading speed.

But how is the data synchronized? That's what we'll configure in the next section:

syncInterval := time.Second * 30

connector, err := libsql.NewEmbeddedReplicaConnector(dbPath, os.Getenv("TURSO_DATABASE_URL"),
    libsql.WithAuthToken(os.Getenv("TURSO_AUTH_TOKEN")),
    libsql.WithSyncInterval(syncInterval),
)

This configuration tells libsql to synchronize every specific interval, in this case, 30 seconds. As the documentation indicates, it is also possible to perform the synchronization manually.

The rest of the code is not new; it is just data manipulation in an SQLite database.

Another interesting feature is the dashboard with helpful information such as latency, lines written/read, database size, etc.:

First impressions

As the title promises, I want to share my first impressions of the product.

When I first saw the news about the startup and the product, it was an exciting idea since SQLite is almost ubiquitous among modern programming languages. "Boosting" it with features like distribution and synchronization seems like a great business opportunity to me, and it opens up a series of possibilities. On the website, you can see some testimonials and business areas that can make use of these features.

I noticed a relatively high latency when writing data, but it seems that this is because I am using the free plan, and this changes in the paid options. However, the reading performance is really attractive. I also believe that it is possible to make tunings in relation to writing, especially in environments like AWS, but I did not perform this type of test to write this text.

If your application does more reading than writing (you can think of several scenarios such as a product catalog, an application that reads user settings and saves them in a local database, use as a cache area, etc.) Turso can bring valuable features combined with simplicity to your project.

I will keep Turso on my radar, and if I manage to apply it to a "real life" project, I will write a second post improving my opinion with more information.
I also wrote a project that is a bit more structured than the example presented here, which you can access on my Github.

What did you think of Turso? Share your opinions in the comments.

Originally published at https://eltonminetto.dev on Mar 23, 2025

Creating an API with authentication using Encore.go

Elton Minetto — Tue, 25 Feb 2025 00:37:09 +0000

This text is the second part of a series of posts about the Encore.go framework :

Creating an API with a database
Creating an API with authentication (<— you are here)
Communication via Pub/Sub
Deploy

In the first part, we created a simple API that validates a user given the correct parameters. Now, let's use this functionality to increase the project's complexity: add a new API that requires authentication to be accessed.

The first step is to create a new directory to organize the code:

mkdir feedback
cd feedback
touch api.go

The first version of api.go looked like this:

package feedback

import (
    "context"
)

// API defines the API for the user service
// encore: service
type API struct {
}

// StoreFeedbackParams represents the response of the StoreFeedback function
type StoreFeedbackParams struct {
    Title string `json:"title"`
    Body  string `json:"body"`
}

// StoreFeedbackResponse represents the response of the StoreFeedback function
type StoreFeedbackResponse struct {
    ID string `json:"id"`
}

// StoreFeedback stores feedback
//
//encore:api auth method=POST path=/v1/feedback
func (a *API) StoreFeedback(ctx context.Context, p *StoreFeedbackParams) (*StoreFeedbackResponse, error) {
    return &StoreFeedbackResponse{ID: ""}, nil
}

What's new is the change we made to Encore's API definition:

//encore:api auth method=POST path=/v1/feedback

According to the documentation, there are three possible configurations for the access level of an API:

//encore:api public – defines a public API that anybody on the internet can call.
//encore:api private – defines a private API never accessible to the outside world. It can only be called from other services in your app and via cron jobs.
//encore:api auth – defines a public API that anybody can call but requires valid authentication.

Since we have configured our API with the access level auth, we need to create the logic responsible for this validation. To do this, we will create a new package:

mkdir authentication
touch authentication/handler.go

The code of authentication/handler.go is:

package authentication

import (
    "context"

    "encore.app/user"
    "encore.dev/beta/auth"
    "encore.dev/beta/errs"
    "github.com/google/uuid"
)

// Data is the auth data
type Data struct {
    Email string
}

// AuthHandler handle auth information
//
//encore:authhandler
func AuthHandler(ctx context.Context, token string) (auth.UID, *Data, error) {
    if token == "" {
        return "", nil, &errs.Error{
            Code:    errs.Unauthenticated,
            Message: "invalid token",
        }
    }
    resp, err := user.ValidateToken(ctx, &user.ValidateTokenParams{Token: token})
    if err != nil {
        return "", nil, &errs.Error{
            Code:    errs.Unauthenticated,
            Message: "invalid token",
        }
    }
    return auth.UID(uuid.New().String()), &Data{Email: resp.Email}, nil
}

The annotation //encore:authhandler indicates to the framework that it needs to execute this code whenever an API requires authentication to be accessed. The framework will automatically attempt to access a token that must be sent in the request using the Authorization header and pass this information as a parameter ( token ) to the AuthHandler function (we can choose another name since what matters is the annotation ). The documentation mentions that more advanced authentication settings, such as other variables and cookies, can be configured.

The function is required to return a value for auth.UID and can optionally return more data, as I did in this example.

Now we can change our API so that it makes use of the authentication data:

// StoreFeedback stores feedback
//
//encore:api auth method=POST path=/v1/feedback
func (a *API) StoreFeedback(ctx context.Context, p *StoreFeedbackParams) (*StoreFeedbackResponse, error) {
    eb := errs.B().Meta("store_feedback", p.Title)
    var email string
    data := auth.Data()
    if data != nil {
        email = data.(*authentication.Data).Email
    }
    if email == "" {
        return nil, eb.Code(errs.Unauthenticated).Msg("unauthenticated").Err()
    }
    f := &Feedback{
        Email: email,
        Title: p.Title,
        Body:  p.Body,
    }
    id, err := a.Service.Store(ctx, f)
    if err != nil {
        return nil, eb.Code(errs.Internal).Msg("internal error").Err()
    }
    return &StoreFeedbackResponse{ID: id}, nil
}

The following example shows how to invoke the API with a token generated by the API I developed in the last post:

curl '127.0.0.1:4000/v1/feedback' \
-H 'Authorization:Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6ImVtaW5ldHRvQGVtYWlsLmNvbSIsImV4cCI6MTc0MDIzNTcyMiwiaWF0IjoxNzQwMjMyMDkyLCJuYmYiOjE3NDAyMzIwOTJ9._7BZwT3rveDV8gN9f2pBCy1D6_ZA17uRKIOAd7GVKLU' \
-d '{"title":"title","body":"body of feedback"}'

We can also write a test to cover this functionality:

package feedback_test

import (
    "context"
    "encore.app/authentication"
    "encore.app/feedback"
    "encore.dev/et"
    "github.com/google/uuid"
    "testing"
)

type ServiceMock struct{}

func (s *ServiceMock) Store(ctx context.Context, f *feedback.Feedback) (string, error) {
    return uuid.New().String(), nil
}

func TestStoreFeedback(t *testing.T) {
    api := feedback.API{
        Service: &ServiceMock{},
    }
    et.OverrideAuthInfo("uuid", &authentication.Data{Email: "eminetto@email.com"})
    p := feedback.StoreFeedbackParams{
        Title: "title",
        Body:  "body",
    }

    resp, err := api.StoreFeedback(context.Background(), &p)
    if err != nil {
        t.Fatalf("unexpected error: %v", err)
    }
    if resp.ID == "" {
        t.Fatalf("expected ID to be non-empty")
    }
}

The critical part is the use of the function et.OverrideAuthInfo("uuid", &authentication.Data{Email: "eminetto@email.com"}), which is a facility that the framework provides for writing tests.

You can see the complete code for this functionality in this repository.

Conclusion

I really liked this feature because it is something common and potentially repetitive, so it is valuable that the framework helps in this process. You can see more complex examples in the documentation, with integration with authentication services such as Auth0.

Another aspect that I found very interesting goes beyond the code. When I was writing the code for the test I presented here, I had doubts about how to pass the authentication data. After reading the documentation and not finding the solution, I joined the project's Discord and asked a question about the subject. On a Saturday morning, in less than 30 minutes, a member of the framework team helped me solve the problem. Extra points for responsiveness and kindness, but here is a suggestion for a review of the documentation to include an example of how to implement this test.

I'm still excited about the framework and the series's following text. What about you, dear reader? What do you think of Encore?

Originally published at https://eltonminetto.dev on Feb 22, 2025

Getting Started with Encore.go

Elton Minetto — Sun, 26 Jan 2025 15:28:26 +0000

Encore.go has been on my radar for quite some time, when its beautiful website and examples caught my attention in a news article on Hacker News. But my excitement really increased after this post was published in December 2024. It announced that the framework would become an independent project, separated from the Encore Cloud tool.
This decision can make the framework more attractive to companies and developers who want to use it in their existing environments. I have nothing against Encore Cloud, which seems to be a very interesting and robust solution, but this freedom of choice favors adoption in companies of different sizes.

Given this initial context, I decided to port a project that I use to write texts and lectures about microservices to Encore, and the result is a series of posts, this being the first. The initial idea is to divide the series as follows:

Creating an API with a database (<— you are here)
Interservice communication and authentication
Communication via Pub/Sub
Deploy

I may create new posts while writing the following parts, but the initial plan is defined. So, let's go to the first part.

Creating an API with a database

To use Encore, we need to install its CLI, which we will use throughout the development cycle. Since I use macOS, I installed it using the command:

brew install encoredev/tap/encore

In the documentation, you can see the other possible installation methods.

With the CLI installed, we can use it to create the project:

Since I chose the option to create a project from scratch, the created directory only contains the files with the dependencies (go.mod and go.sum) and encore.app, which will be used by the CLI to manipulate the project. Its initial content is straightforward:

{
    // The app is not currently linked to the encore.dev platform.
    // Use "encore app link" to link it.
    "id": "",
}

For now, we won't need to change anything in encore.app, so let's create the structure of the first microservice:

❯ cd post-encore/
❯ mkdir user
❯ touch user/api.go

In addition to creating the directory, we initialize a file called api.go, where we will define our API. The content of the first version was as follows:

package user

import (
    "context"
)

// API defines the API for the user service
// encore: service
type API struct{}

func initAPI() (*API, error) {
    return &API{}, nil
}

// AuthParams are the parameters to the Auth method
type AuthParams struct {
    Email    string `json:"email"`
    Password string `json:"password"`
}

// AuthResponse is the response to the Auth method
type AuthResponse struct {
    Token string `json:"token"`
}

// Auth authenticates a user and returns a token
//
//encore:api public method=POST path=/v1/auth
func (a *API) Auth(ctx context.Context, p *AuthParams) (*AuthResponse, error) {
    var response AuthResponse
    return &response, nil
}

// ValidateTokenParams are the parameters to the ValidateToken method
type ValidateTokenParams struct {
    Token string `json:"token"`
}

// ValidateTokenResponse is the response to the ValidateToken method
type ValidateTokenResponse struct {
    Email string `json:"email"`
}

// ValidateToken validates a token
//
//encore:api public method=POST path=/v1/validate-token
func (a *API) ValidateToken(ctx context.Context, p *ValidateTokenParams) (*ValidateTokenResponse, error) {
    response := ValidateTokenResponse{}
    return &response, nil
}

Encore extensively uses the concept of annotations to define the application's behavior and generate the code necessary for execution. The first example of this is the creation of a service:

// API defines the API for the user service
// encore: service
type API struct{}

Another common feature of frameworks like Encore is the existence of conventions. The first one we will find here is the initialization of services. Since we have defined a service called API, we can create a function called initAPI that will be invoked by the framework. In this function, we will inject the service dependencies, as we will do later in the project.

The following annotation we see in the code is the one that defines an API, as in the example:

// Auth authenticates a user and returns a token
//
//encore:api public method=POST path=/v1/auth
func (a *API) Auth(ctx context.Context, p *AuthParams) (*AuthResponse, error) {
    var response AuthResponse
    return &response, nil
}

This declaration tells the framework that this is a public API (we'll see more about this in the following parts of this series) whose path is /v1/auth and that it will be accessed via the POST method.
An API is a function that receives a context and a parameter struct (in this case, AuthParams) and returns a response struct (in this case, AuthResponse).

One of the interesting features of the framework is the ease of accessing the parameter values: we can use the values as p.Email, which is part of the received struct, without needing to convert the received JSON. More details about parameter handling and examples can be found in the official documentation.

We can now run the project using the CLI:

❯ encore run
  ✔ Building Encore application graph... Done!
  ✔ Analyzing service topology... Done!
  ✔ Generating boilerplate code... Done!
  ✔ Compiling application source code... Done!
  ✔ Starting Encore application... Done!

  Encore development server running!

  Your API is running at:     http://127.0.0.1:4000
  Development Dashboard URL:  http://127.0.0.1:9400/wst7a

  New Encore release available: v1.46.1 (you have v1.45.6)
  Update with: encore version update

11:07AM INF registered API endpoint endpoint=Auth path=/v1/auth service=user
11:07AM INF registered API endpoint endpoint=ValidateToken path=/v1/validate-token service=user
11:07AM INF listening for incoming HTTP requests

We can see that the API is available at the URL http://127.0.0.1:4000 and the development dashboard, which is one of the biggest attractions of the framework. How it works can be seen below:

In the video, you can see the interesting features available, such as testing the API and viewing traces and application components. The dashboard will become even more useful as we add layers of complexity to the project.

Setting up the database

The next step in developing the application is to define a database. We need to have Docker installed and running to do this since Encore will use it to create the database image. At the time of writing, the available database is PostgreSQL. In the api.go, we will make the following changes:

package user

import (
    "context"

    "encore.dev/storage/sqldb"
)

var db = sqldb.NewDatabase("user", sqldb.DatabaseConfig{
    Migrations: "./migrations",
})

As you can see in the excerpt, Encore uses the concept of migrations, which is very useful. It was necessary to create the directory user/migrations and the file user/migrations/1_create_tables.up.sql with the following content:

create table users (id varchar(50) PRIMARY KEY ,email varchar(255),password varchar(255),first_name varchar(100), last_name varchar(100), created_at date, updated_at date);
INSERT INTO users (id, email, password, first_name, last_name, created_at, updated_at) values ('8cb2237d0679ca88db6464eac60da96345513964','eminetto@email.com','8cb2237d0679ca88db6464eac60da96345513964', 'Elton', 'Minetto', now(), null);

The database is created in the file, and a record is also inserted for use in tests via the dashboard. More details about migrations can be seen in the documentation.

Another interesting feature of the command encore run is that it auto-reloads the application. Whenever something is changed in one of the project files, the application is recompiled and executed again, so your database should have been created successfully.

To finalize this first version of the project, I created other files to complement the functionality. Using a framework like Encore makes repetitive tasks much more straightforward, such as configuring routes, converting parameters and responses, etc. However, it does not eliminate the need to use good development practices, such as abstractions, decoupling, etc. With this in mind, I created other files that are important for our project:

user.go, which defines what a "user" in the application is
service.go, which contains the application's business rule and will be used by the API
security/jwt.go, which includes the logic for generating and validating JWT tokens

With these auxiliary files, the final version of our API looks like this:

package user

import (
    "context"

    "encore.app/user/security"
    "encore.dev/beta/errs"
    "encore.dev/storage/sqldb"
)

var db = sqldb.NewDatabase("user", sqldb.DatabaseConfig{
    Migrations: "./migrations",
})

// API defines the API for the user service
// encore: service
type API struct {
    Service UseCase
}

func initAPI() (*API, error) {
    return &API{Service: NewService(db)}, nil
}

// AuthParams are the parameters to the Auth method
type AuthParams struct {
    Email    string `json:"email"`
    Password string `json:"password"`
}

// AuthResponse is the response to the Auth method
type AuthResponse struct {
    Token string `json:"token"`
}

// Auth authenticates a user and returns a token
//
//encore:api public method=POST path=/v1/auth
func (a *API) Auth(ctx context.Context, p *AuthParams) (*AuthResponse, error) {
    // Construct a new error builder with errs.B()
    eb := errs.B().Meta("auth", p.Email)

    err := a.Service.ValidateUser(ctx, p.Email, p.Password)
    if err != nil {
        return nil, eb.Code(errs.Unauthenticated).Msg("invalid credentials").Err()
    }
    var response AuthResponse
    response.Token, err = security.NewToken(p.Email)
    if err != nil {
        return nil, eb.Code(errs.Internal).Msg("internal error").Err()
    }
    return &response, nil
}

// ValidateTokenParams are the parameters to the ValidateToken method
type ValidateTokenParams struct {
    Token string `json:"token"`
}

// ValidateTokenResponse is the response to the ValidateToken method
type ValidateTokenResponse struct {
    Email string `json:"email"`
}

// ValidateToken validates a token
//
//encore:api public method=POST path=/v1/validate-token
func (a *API) ValidateToken(ctx context.Context, p *ValidateTokenParams) (*ValidateTokenResponse, error) {
    // Construct a new error builder with errs.B()
    eb := errs.B().Meta("validate_token", p.Token)
    t, err := security.ParseToken(p.Token)
    if err != nil {
        return nil, eb.Code(errs.Internal).Msg("internal error").Err()
    }
    tData, err := security.GetClaims(t)
    if err != nil {
        return nil, eb.Code(errs.Internal).Msg("internal error").Err()
    }
    response := ValidateTokenResponse{
        Email: tData["email"].(string),
    }
    return &response, nil
}

In this new version, you can see that we initialize the service (in the function initAPI) with the injection of the business rule and the error handling provided by the framework.

Tests

Another advantage of using a framework like Encore is that it provides features that help with the critical task of writing tests. In this first version, we have two crucial components to test:

service_test.go

package user_test

import (
    "context"
    "testing"

    "encore.app/user"
    "encore.dev/et"
)

func TestService(t *testing.T) {
    ctx := context.Background()
    et.EnableServiceInstanceIsolation()
    testDB, err := et.NewTestDatabase(ctx, "user")
    if err != nil {
        t.Fatalf("unexpected error: %v", err)
    }

    s := user.NewService(testDB)
    t.Run("valid user", func(t *testing.T) {
        err := s.ValidateUser(ctx, "eminetto@email.com", "12345")
        if err != nil {
            t.Fatalf("unexpected error: %v", err)
        }
    })
    t.Run("invalid user", func(t *testing.T) {
        err := s.ValidateUser(ctx, "e@email.com", "12345")
        if err == nil {
            t.Fatalf("unexpected error: %v", err)
        }
    })
    t.Run("invalid password", func(t *testing.T) {
        err := s.ValidateUser(ctx, "eminetto@email.com", "111")
        if err == nil {
            t.Fatalf("unexpected error: %v", err)
        }
    })
}

The highlight is the use of the package encore.dev/et, which provides a way to ensure that tests can be executed in parallel (et.EnableServiceInstanceIsolation()) and the ease of using an exclusive database for tests (testDB, err:= et.NewTestDatabase(ctx, "user")).

The interesting thing is that migrations are used automatically, making the test easier to write and execute.

api_test.go

package user_test

import (
    "context"
    "testing"

    "encore.app/user"
)

type ServiceMock struct{}

func (s *ServiceMock) ValidateUser(ctx context.Context, email string, password string) error {
    return nil
}

func (s *ServiceMock) ValidatePassword(ctx context.Context, u *user.User, password string) error {
    return nil
}

func TestIntegration(t *testing.T) {
    api := &user.API{
        Service: &ServiceMock{},
    }
    email := "eminetto@email.com"
    resp, err := api.Auth(context.Background(), &user.AuthParams{
        Email:    email,
        Password: "12345",
    })
    if err != nil {
        t.Fatalf("unexpected error: %v", err)
    }
    if resp.Token == "" {
        t.Fatalf("expected token to be non-empty")
    }
    r, err := api.ValidateToken(context.Background(), &user.ValidateTokenParams{
        Token: resp.Token,
    })
    if err != nil {
        t.Fatalf("unexpected error: %v", err)
    }
    if r.Email != email {
        t.Fatalf("expected email to be %q, got %q", email, r.Email)
    }
}

In this test, nothing from the framework was necessary; just good old Go, with its native advantages, was used.

An important detail: to run the tests, you must use the CLI. Therefore, instead of running them using the command:

go test ./...

You must use:

encore test ./...

There is a plugin for Goland that allows execution directly through the IDE, but the same is not yet true for VSCode, as can be seen in the official testing documentation.

Conclusion

The objective of this first post was to present the basics of the framework and spark your curiosity about the next chapters in this series.

I'll leave my opinions on the framework for the last part of the series so I can provide more arguments to say whether I liked the experience or not. But I can say that I'm having a lot of fun with the first steps. What about you, dear reader? What do you think of Encore so far? Leave your impressions in the comments.

Originally published at https://eltonminetto.dev on Jan 25, 2025

Using CloudEvents in Go

Elton Minetto — Fri, 29 Nov 2024 11:52:34 +0000

Adopting an event-driven architecture (EDA) to increase scalability and reduce coupling between components/services is relatively common in complex environments.

While this approach solves a number of problems, one of the challenges faced by teams is standardizing events to ensure compatibility between all components. To mitigate this challenge, we can use the CloudEvents project.

The project aims to be a specification for standardizing and describing events, bringing consistency, accessibility, and portability. Another advantage is that the project provides a series of SDKs to accelerate team adoption in addition to being a specification.

In this post, I want to demonstrate the use of the Go SDK (with a special appearance by the Python SDK ) in a fictitious project.

Let's consider an environment composed of two microservices: a user, which manages users (CRUD), and an audit service, which stores important events in the environment for future analysis.

The service code of the user service is as follows:

package main

import (
    "context"
    "encoding/json"
    "log"
    "net/http"
    "time"

    cloudevents "github.com/cloudevents/sdk-go/v2"
    "github.com/cloudevents/sdk-go/v2/protocol"
    "github.com/go-chi/chi/v5"
    "github.com/go-chi/httplog"
    "github.com/google/uuid"
)

const auditService = "http://localhost:8080/"

func main() {
    logger := httplog.NewLogger("user", httplog.Options{
        JSON: true,
    })
    ctx := context.Background()
    ceClient, err := cloudevents.NewClientHTTP()
    if err != nil {
        log.Fatalf("failed to create client, %v", err)
    }

    r := chi.NewRouter()
    r.Use(httplog.RequestLogger(logger))
    r.Post("/v1/user", storeUser(ctx, ceClient))

    http.Handle("/", r)
    srv := &http.Server{
        ReadTimeout:  30 * time.Second,
        WriteTimeout: 30 * time.Second,
        Addr:         ":3000",
        Handler:      http.DefaultServeMux,
    }
    err = srv.ListenAndServe()
    if err != nil {
        logger.Panic().Msg(err.Error())
    }
}

type userRequest struct {
    ID       uuid.UUID
    Name     string `json:"name"`
    Password string `json:"password"`
}

func storeUser(ctx context.Context, ceClient cloudevents.Client) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        oplog := httplog.LogEntry(r.Context())

        var ur userRequest
        err := json.NewDecoder(r.Body).Decode(&ur)
        if err != nil {
            w.WriteHeader(http.StatusBadRequest)
            oplog.Error().Msg(err.Error())
            return
        }
        ur.ID = uuid.New()
        //TODO: store user in a database

        // Create an Event.
        event := cloudevents.NewEvent()
        event.SetSource("github.com/eminetto/post-cloudevents")
        event.SetType("user.storeUser")
        event.SetData(cloudevents.ApplicationJSON, map[string]string{"id": ur.ID.String()})

        // Set a target.
        ctx := cloudevents.ContextWithTarget(context.Background(), auditService)

        // Send that Event.
        var result protocol.Result
        if result = ceClient.Send(ctx, event); cloudevents.IsUndelivered(result) {
            oplog.Error().Msgf("failed to send, %v", result)
            w.WriteHeader(http.StatusInternalServerError)
            return
        }

        return
    }
}

In the code, you can see the creation of an event and its sending to the audit service, which looks like this:

package main

import (
    "context"
    "fmt"
    "log"

    cloudevents "github.com/cloudevents/sdk-go/v2"
)

func receive(event cloudevents.Event) {
    // do something with event.
    fmt.Printf("%s", event)
}

func main() {
    // The default client is HTTP.
    c, err := cloudevents.NewClientHTTP()
    if err != nil {
        log.Fatalf("failed to create client, %v", err)
    }
    if err = c.StartReceiver(context.Background(), receive); err != nil {
        log.Fatalf("failed to start receiver: %v", err)
    }
}

By running both services, you can see how they work by sending a request to the user :

curl -X "POST" "http://localhost:3000/v1/user" \
     -H 'Accept: application/json' \
     -H 'Content-Type: application/json' \
     -d $'{
  "name": "Ozzy Osbourne",
  "password": "12345"
}'

The user output is:

{"level":"info","service":"user","httpRequest":{"proto":"HTTP/1.1","remoteIP":"[::1]:50894","requestID":"Macbook-Air-de-Elton.local/3YUAnzEbis-000001","requestMethod":"POST","requestPath":"/v1/user","requestURL":"http://localhost:3000/v1/user"},"httpRequest":{"header":{"accept":"application/json","content-length":"52","content-type":"application/json","user-agent":"curl/8.7.1"},"proto":"HTTP/1.1","remoteIP":"[::1]:50894","requestID":"Macbook-Air-de-Elton.local/3YUAnzEbis-000001","requestMethod":"POST","requestPath":"/v1/user","requestURL":"http://localhost:3000/v1/user","scheme":"http"},"timestamp":"2024-11-28T15:52:27.947355-03:00","message":"Request: POST /v1/user"}
{"level":"warn","service":"user","httpRequest":{"proto":"HTTP/1.1","remoteIP":"[::1]:50894","requestID":"Macbook-Air-de-Elton.local/3YUAnzEbis-000001","requestMethod":"POST","requestPath":"/v1/user","requestURL":"http://localhost:3000/v1/user"},"httpResponse":{"bytes":0,"elapsed":2.33225,"status":0},"timestamp":"2024-11-28T15:52:27.949877-03:00","message":"Response: 0 Unknown"}

The output of the audit service demonstrates the receipt of the event.

❯ go run main.go
Context Attributes,
  specversion: 1.0
  type: user.storeUser
  source: github.com/eminetto/post-cloudevents
  id: 5190bc29-a3d5-4fca-9a88-85fccffc16b6
  time: 2024-11-28T18:53:17.474154Z
  datacontenttype: application/json
Data,
  {
    "id": "8aadf8c5-9c4e-4c11-af24-beac2fb9a4b7"
  }

To validate the portability goal, I used the Python SDK to implement a version of the audit service:

from flask import Flask, request

from cloudevents.http import from_http

app = Flask(__name__)


# create an endpoint at http://localhost:/3000/
@app.route("/", methods=["POST"])
def home():
    # create a CloudEvent
    event = from_http(request.headers, request.get_data())

    # you can access cloudevent fields as seen below
    print(
        f"Found {event['id']} from {event['source']} with type "
        f"{event['type']} and specversion {event['specversion']}"
    )

    return "", 204


if __name__ == "__main__":
    app.run(port=8080)

The application output shows the receipt of the event without the need for changes to the service user:

(.venv) eminetto@Macbook-Air-de-Elton audit-python % python3 main.py
 * Serving Flask app 'main'
 * Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:8080
Press CTRL+C to quit
Found ce1abe22-dce5-40f0-8c82-12093b707ed7 from github.com/eminetto/post-cloudevents with type user.storeUser and specversion 1.0
127.0.0.1 - - [28/Nov/2024 15:59:31] "POST / HTTP/1.1" 204 -

The previous example introduces the CloudEvents SDKs, but it violates a principle of event-based architectures: loosen coupling. The application user is aware of and tied to the auditing application, which is not a good practice. We can improve this situation by using other CloudEvents features, such as pub/sub, or by adding something like Kafka. The following example uses Kafka to decouple the two applications.

The first step was to create one docker-compose.yaml to use Kafka:

services:
  kafka:
    image: bitnami/kafka:latest
    restart: on-failure
    ports:
      - 9092:9092
    environment:
      - KAFKA_CFG_BROKER_ID=1
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_CFG_NUM_PARTITIONS=3
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper

  zookeeper:
    image: bitnami/zookeeper:latest
    ports:
      - 2181:2181
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes

The following change was in the service user:

package main

import (
    "context"
    "encoding/json"
    "log"
    "net/http"
    "time"

    "github.com/IBM/sarama"
    "github.com/cloudevents/sdk-go/protocol/kafka_sarama/v2"
    cloudevents "github.com/cloudevents/sdk-go/v2"
    "github.com/go-chi/chi/v5"
    "github.com/go-chi/httplog"
    "github.com/google/uuid"
)

const (
    auditService = "127.0.0.1:9092"
    auditTopic   = "audit"
)

func main() {
    logger := httplog.NewLogger("user", httplog.Options{
        JSON: true,
    })
    ctx := context.Background()

    saramaConfig := sarama.NewConfig()
    saramaConfig.Version = sarama.V2_0_0_0

    sender, err := kafka_sarama.NewSender([]string{auditService}, saramaConfig, auditTopic)
    if err != nil {
        log.Fatalf("failed to create protocol: %s", err.Error())
    }

    defer sender.Close(context.Background())

    ceClient, err := cloudevents.NewClient(sender, cloudevents.WithTimeNow(), cloudevents.WithUUIDs())
    if err != nil {
        log.Fatalf("failed to create client, %v", err)
    }

    r := chi.NewRouter()
    r.Use(httplog.RequestLogger(logger))
    r.Post("/v1/user", storeUser(ctx, ceClient))

    http.Handle("/", r)
    srv := &http.Server{
        ReadTimeout:  30 * time.Second,
        WriteTimeout: 30 * time.Second,
        Addr:         ":3000",
        Handler:      http.DefaultServeMux,
    }
    err = srv.ListenAndServe()
    if err != nil {
        logger.Panic().Msg(err.Error())
    }
}

type userRequest struct {
    ID       uuid.UUID
    Name     string `json:"name"`
    Password string `json:"password"`
}

func storeUser(ctx context.Context, ceClient cloudevents.Client) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        oplog := httplog.LogEntry(r.Context())

        var ur userRequest
        err := json.NewDecoder(r.Body).Decode(&ur)
        if err != nil {
            w.WriteHeader(http.StatusBadRequest)
            oplog.Error().Msg(err.Error())
            return
        }
        ur.ID = uuid.New()
        //TODO: store user in a database

        // Create an Event.
        event := cloudevents.NewEvent()
        event.SetID(uuid.New().String())
        event.SetSource("github.com/eminetto/post-cloudevents")
        event.SetType("user.storeUser")
        event.SetData(cloudevents.ApplicationJSON, map[string]string{"id": ur.ID.String()})

        // Send that Event.
        if result := ceClient.Send(
            // Set the producer message key
            kafka_sarama.WithMessageKey(context.Background(), sarama.StringEncoder(event.ID())),
            event,
        ); cloudevents.IsUndelivered(result) {
            oplog.Error().Msgf("failed to send, %v", result)
            w.WriteHeader(http.StatusInternalServerError)
            return
        }

        return
    }
}

A few changes were needed, mainly to make a connection with Kafka, but the event itself did not change.

I made a similar change to the audit service:

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/IBM/sarama"

    "github.com/cloudevents/sdk-go/protocol/kafka_sarama/v2"
    cloudevents "github.com/cloudevents/sdk-go/v2"
)

const (
    auditService = "127.0.0.1:9092"
    auditTopic   = "audit"
    auditGroupID = "audit-group-id"
)

func receive(event cloudevents.Event) {
    // do something with event.
    fmt.Printf("%s", event)
}

func main() {
    saramaConfig := sarama.NewConfig()
    saramaConfig.Version = sarama.V2_0_0_0

    receiver, err := kafka_sarama.NewConsumer([]string{auditService}, saramaConfig, auditGroupID, auditTopic)
    if err != nil {
        log.Fatalf("failed to create protocol: %s", err.Error())
    }

    defer receiver.Close(context.Background())

    c, err := cloudevents.NewClient(receiver)
    if err != nil {
        log.Fatalf("failed to create client, %v", err)
    }

    if err = c.StartReceiver(context.Background(), receive); err != nil {
        log.Fatalf("failed to start receiver: %v", err)
    }
}

The output of the applications stays the same.

With the inclusion of Kafka, we decoupled the applications, no longer violating the principles of EDA while maintaining the advantages provided by CloudEvents.

The goal of this post was to introduce the standard and demonstrate the ease of implementation using the SDKs. I could cover the subject in more depth, but I hope I have achieved the objective and inspired research and use of the technology.

It would be very useful if you already use/have used CloudEvents and wanted to share your experiences in the comments.

You can find the codes I presented in this post in the repository on GitHub.

Originally published at https://eltonminetto.dev on Nov 29, 2024.

Must-have apps and services in 2024

Elton Minetto — Tue, 05 Nov 2024 00:05:30 +0000

This post is part of a tradition I started 10 years ago! Every now and then, I make a list of the apps and services I use on a daily basis and share them in these posts. Discovering new apps is something I love, so I can help someone who has the same taste for new things.

Interestingly, some apps I've been using since 2014, like iA Writer (which I'm using to write this post) and Buffer. I'll talk about both of them in a moment, but first, I want to talk about another tool that's crucial to my setup: SetApp (horrible pun, but I couldn't resist).

I've been using Setapp since 2017 and am still a huge fan. Its monthly subscription gives you access to over 250 apps, some for macOS and some for iOS and iPadOS.

PS: if you sign up through the link, you and I will receive some benefits (which may change depending on the subscription date).

PS2: Since I use several Setapp apps, I'll mark them in italics for easy identification.

Without further ado, let's get to my list, separated into some categories, as I did in the last post (in portuguese) in this series.

Software development

iTerm + fish. I wrote a post explaining my environment settings.
Visual Studio Code. No introduction is needed, right? I use it mainly for development in Go.
Zed is my second editor. I use it as much as I can because it's a very lightweight and well-built tool. VSCode wins when I need to debug in Go because it's better prepared for that.
OrbStack is a fast and free Docker Desktop replacement for personal use. It comes with a configured K8s cluster, which is useful for local testing.
OpenLens. Although I like using kubectl to manage the local k8s cluster, OpenLens is helpful, especially when I want to show something in a lecture because it is more visual.

Productivity

1Password. Password and secret vault. It's a subscription I'd pay for without thinking twice.
CleanMyMac. Cleans up macOS and optimizes several performance settings.
Paste. This very useful clipboard manager app was a recent addition to my list.
Soulver. Note-taking tool that automatically performs calculations based on the text. Very useful for quick calculations.
Lungo. For those times when I don't want macOS to go into sleep mode.
Spark. Email client with superb usability and some handy features. I'm curious to try Notion Mail, and maybe Spark is in danger of being dropped from my list by 2025
Bartender. Helps organize the clutter of icons in your macOS menu bar.
Notion. My second brain, I write everything down in it.
Notion Calendar. I've been using it since Notion bought the company when it was called Cron.
Todoist. I adopted it this year after using paper notebooks for years and Apple's Reminders for a while. It's a great tool and well worth the subscription.
Google Workspace. I pay for the plan for one user with my own domain. With this, I have an email and access to Drive, Docs, and other tools.
Safari is my main browser, but I still have Google Chrome for some emergencies, like using Google Meet, which is faster in Google's browser.
Freeform. I use this native Apple app to take notes on iPad and MacOS.
Readwise. I recently subscribed to it to centralize the notes I take in e-books and blog posts. It's still too early to know if I'll renew the subscription in 2025.

Posts and Lectures

iA Writer. I've been a fan of the product and company for over 10 years. It's my favorite Markdown editor.
Buffer. This is another tool I've been using for a decade to schedule social media posts.
Grammarly is another tool that I happily renew every year. I use it to help me proofread my English posts.
Google Slides. I have used the excellent Deckset for years, but recently, I started using Google Slides to produce slides for my presentations. The advantage is that it is easy to share presentations, but I still miss the practicality of Deckset. Maybe I will rethink this decision at some point.
For my personal site, I use Hugo, and I host the markdown files on GitHub. Publishing is done whenever I push to the repository using Netlify.
To manage the newsletters A Semana Go and Mais que Senior, I use Substack.

Of course, I always have Apple Music open while I work because I can't live without music. Another must-have category is communication apps like WhatsApp and Telegram, which are always open to chatting with friends and family.

You may have noticed the absence of AI tools like ChatGPT, right? Although I installed the tool, it has yet to become part of my daily usage flow. However, the vast majority of tools I use have AI features, so I can say that I use technology in the best way I can imagine: without even realizing it.

If you have any suggestions, you can share them here in the comments.
If you find it useful for me to make a similar list of the apps I use on the iPad, use the comments to give me this feedback.

Originally published at https://eltonminetto.dev on November 4, 2024.

Introduction to TypeSpec

Elton Minetto — Tue, 15 Oct 2024 00:05:58 +0000

I'll start this post with a bit of history. Back in the early 2010s, the buzz was the concepts of APIs and API-first. It seems trivial today, but we must remember that the technology used before was SOAP and its giant XML files. So, lightweight APIs using JSON and respecting the REST concepts, which had been invented a few years earlier, were a considerable evolution.

In this context, a "dispute" began between some API specification languages; as more and more teams adopted APIs, something was needed that would make it possible to define, document, and detail them. Among the competitors, I can remember RAML and the one I was betting my chips on, API Blueprint, but the winner was Swagger, later renamed to OpenAPI.

Fast forward to 2024, and everyone is happily writing and maintaining YAML files of hundreds, sometimes thousands of lines. Right?

I'm not the only one with this impression because, in early 2024, Microsoft launched a new Open Source project called TypeSpec. It is a format based heavily on TypeScript, also created by Microsoft and simplifies the definition of APIs.

Anyone who closely follows the history of the OpenAPI standard may find this move quite strange since Microsoft is one of the standard's member entities. In a talk, one of the leaders of the TypeSpec project commented on the reasons for creating the new standard:

OpenAPI was perceived by Microsoft teams as being hard to write, review, and maintain.
It was difficult to follow API guidelines in this context, especially when working with many new teams that did "know" OpenAPI.
OpenAPI does not rule the Microsoft world in terms of API description languages and protocols, with others such as gRPC being supported.

At this point, some of the readers may be asking themselves something like:

I don't see any problems with OpenAPI; I generate the documentation from my code!

Indeed, this is an approach some teams adopt, using tools like swaggo. I see some problems with this approach, such as code being "polluted" with comments and annotations and the fact that the API is heavily influenced by the team's programming style.

However, there is another approach, which, in my opinion, is the best: design-first. In this approach, teams first define the API design, considering its consumers, the resources that will be exposed, and its evolution. With this in mind, it is necessary to have a simple way to document this design, which will later be implemented and used by clients. And I wouldn't use the word "simple" to define a document written in the OpenAPI standard…

Let's get our hands dirty and test this out.

The first step is to install the TypeSpec CLI using the command:

npm install -g @typespec/compiler

In the next step, I created a directory and initialized a project:

mkdir post-typespec
cd post-typespec/
tsp init

I selected Generic REST API in the options menu and accepted the default options.

The following structure was created:

❯ ls -lha
total 32
drwxr-xr-x   6 eminetto  staff   192B 13 Out 09:51 .
drwxr-xr-x  80 eminetto  staff   2,5K 13 Out 09:50 ..
-rw-r--r--   1 eminetto  staff   102B 13 Out 09:51 .gitignore
-rw-r--r--   1 eminetto  staff    79B 13 Out 09:51 main.tsp
-rw-r--r--   1 eminetto  staff   417B 13 Out 09:51 package.json
-rw-r--r--   1 eminetto  staff    31B 13 Out 09:51 tspconfig.yaml

The file package.json contains the project dependencies, which can be installed with the command:

tsp install

Now we have the dependencies installed, we can describe our API in the file main.tsp. In this case, I used the example from the official documentation:

import "@typespec/http";

using TypeSpec.Http;

model Pet {
  name: string;
  age: int32;
}

model Store {
  name: string;
  address: Address;
}

model Address {
  street: string;
  city: string;
}

@route("/pets")
interface Pets {
  list(@query filter: string): Pet[];
  create(@body pet: Pet): Pet;
  read(@path id: string): Pet;
}

@route("/stores")
interface Stores {
  list(@query filter: string): Store[];
  read(@path id: string): Store;
}

With this definition, we can generate the specifications using the concept of Emitters. I will return to this subject shortly, but we will use it to generate the OpenAPI specification for now. To do this, we will use the file tspconfig.yaml, which has the following generated code:

emit:
  - "@typespec/openapi3"

When using the following command, we have the document generated inside the directory tsp-output/@typespec/openapi3/openapi.yaml:

tsp compile .

You can also use the following command to have the compilation run automatically as files within the project change:

tsp compile . --watch

Doing a file size comparison, the 31 lines of TypeSpec become 125 lines of OpenAPI in YAML format! The two files can be compared in the official documentation.

Getting back to Emitters. In addition to the OpenAPI we saw in action, there are options for generating specifications in JSON Schema and Protobuf. But more importantly, it is possible to create new emitters for generating codes, SDKs, other documentation formats, etc. For example, we could have an Emitter that reads the specification in TypeSpec and generates configurations for some API Gateway like Kong or Traefik (I gave this example because it is something I want to test).

One argument against adopting TypeSpec could be the downfalls of including a new component in the team's stack. Another could be the low maturity of the project, as it is less than a year old (as an Open Source project), despite its rapid evolution and use within Microsoft.

These valid arguments should be considered whenever a new technology is considered for a project. My goal with this post was to present this new option and suggest we take a closer look at it, as it can be very valuable in large projects. I will keep it on my list of tools to test and validate.

What is your opinion on the subject? I would love to hear opinions, especially from #teamOpenAPI :)

Originally published at https://eltonminetto.dev on October 12, 2024

Resilience in communication between microservices using the failsafe-go lib

Elton Minetto — Mon, 26 Aug 2024 14:14:16 +0000

Let's start at the beginning. What is resilience? I like the definition in this post:

The intrinsic ability of a system to adjust its functioning prior to, during, or following changes and disturbances, so that it can sustain required operations under both expected and unexpected conditions

As it is a broad term, I will focus on communication between microservices in this post. To do this, I created two services using Go: serviceA and serviceB (my creativity was not high when writing this post).

The initial code for both was as follows:

package main

// serviceA
import (
    "encoding/json"
    "io"
    "log/slog"
    "net/http"
    "os"

    "github.com/go-chi/chi/v5"
    slogchi "github.com/samber/slog-chi"
)

func main() {
    logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
    r := chi.NewRouter()
    r.Use(slogchi.New(logger))
    r.Get("/", func(w http.ResponseWriter, r *http.Request) {
        type response struct {
            Message string `json:"message"`
        }
        resp, err := http.Get("http://localhost:3001")
        if err != nil {
            w.WriteHeader(http.StatusInternalServerError)
            w.Write([]byte(err.Error()))
            return
        }
        body, err := io.ReadAll(resp.Body)
        if err != nil {
            w.WriteHeader(http.StatusInternalServerError)
            w.Write([]byte(err.Error()))
            return
        }
        defer resp.Body.Close()
        var data response
        err = json.Unmarshal(body, &data)
        if err != nil {
            w.WriteHeader(http.StatusInternalServerError)
            w.Write([]byte(err.Error()))
            return
        }
        w.Header().Set("Content-Type", "application/json")
        w.Write([]byte(`{"messageA": "hello from service A","messageB": "` + data.Message + `"}`))
    })
    http.ListenAndServe(":3000", r)
}

package main

//serviceB
import (
    "net/http"

    "github.com/go-chi/chi/v5"
    "github.com/go-chi/chi/v5/middleware"
)

func main() {
    r := chi.NewRouter()
    r.Use(middleware.Logger)
    r.Get("/", func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json")
        w.Write([]byte(`{"message": "hello from service B"}`))
    })
    http.ListenAndServe(":3001", r)
}

As you can see in the code, if serviceB has a problem, it will affect the functioning of serviceA, as it does not handle any communication failure. We will improve this by using lib failsafe-go.

According to the documentation on the official website:

Failsafe-go is a library for building resilient, fault tolerant Go applications. It works by wrapping functions with one or more resilience policies, which can be combined and composed as needed.

Let's start by applying some available policies and testing their composition.

Timeout

The first policy we will test is the simplest, including a timeout to ensure that the connection is interrupted if serviceB takes too long to respond and the client knows why.

The first step was to change the serviceB so that it includes a delay to make it easier to demonstrate the scenario:

package main
//serviceB
import (
    "net/http"
    "time"

    "github.com/go-chi/chi/v5"
    "github.com/go-chi/chi/v5/middleware"
)

func main() {
    r := chi.NewRouter()
    r.Use(middleware.Logger)
    r.Get("/", func(w http.ResponseWriter, r *http.Request) {
        time.Sleep(5 * time.Second) //add a delay to simulate a slow service
        w.Header().Set("Content-Type", "application/json")
        w.Write([]byte(`{"message": "hello from service B"}`))
    })
    http.ListenAndServe(":3001", r)
}

After installing failsafe-go, using the commands:

❯ cd serviceA
❯ go get github.com/failsafe-go/failsafe-go

The code of serviceA/main.go is:

package main

import (
    "encoding/json"
    "io"
    "log/slog"
    "net/http"
    "os"
    "time"

    "github.com/failsafe-go/failsafe-go"
    "github.com/failsafe-go/failsafe-go/failsafehttp"
    "github.com/failsafe-go/failsafe-go/timeout"
    "github.com/go-chi/chi/v5"
    slogchi "github.com/samber/slog-chi"
)

func main() {
    logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
    r := chi.NewRouter()
    r.Use(slogchi.New(logger))
    r.Get("/", func(w http.ResponseWriter, r *http.Request) {
        type response struct {
            Message string `json:"message"`
        }
        // Create a Timeout for 1 second
        timeout := newTimeout(logger)

        // Use the Timeout with a failsafe RoundTripper
        roundTripper := failsafehttp.NewRoundTripper(nil, timeout)
        client := &http.Client{Transport: roundTripper}
        resp, err := client.Get("http://localhost:3001")
        if err != nil {
            w.WriteHeader(http.StatusInternalServerError)
            w.Write([]byte(err.Error()))
            return
        }
        body, err := io.ReadAll(resp.Body)
        if err != nil {
            w.WriteHeader(http.StatusInternalServerError)
            w.Write([]byte(err.Error()))
            return
        }
        defer resp.Body.Close()
        var data response
        err = json.Unmarshal(body, &data)
        if err != nil {
            w.WriteHeader(http.StatusInternalServerError)
            w.Write([]byte(err.Error()))
            return
        }
        w.Header().Set("Content-Type", "application/json")
        w.Write([]byte(`{"messageA": "hello from service A","messageB": "` + data.Message + `"}`))
    })
    http.ListenAndServe(":3000", r)
}

func newTimeout(logger *slog.Logger) timeout.Timeout[*http.Response] {
    return timeout.Builder[*http.Response](1 * time.Second).
        OnTimeoutExceeded(func(e failsafe.ExecutionDoneEvent[*http.Response]) {
            logger.Info("Connection timed out")
        }).Build()
}

To test how it works, I used curl to access the serviceA:

❯ curl -v http://localhost:3000
* Host localhost:3000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:3000...
* Connected to localhost (::1) port 3000
> GET / HTTP/1.1
> Host: localhost:3000
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 500 Internal Server Error
< Date: Fri, 23 Aug 2024 19:43:23 GMT
< Content-Length: 45
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host localhost left intact
Get "http://localhost:3001": timeout exceeded⏎

And the following output is generated by serviceA:

go run main.go
{"time":"2024-08-20T08:37:36.852886-03:00","level":"INFO","msg":"Connection timed out"}
{"time":"2024-08-20T08:37:36.856079-03:00","level":"ERROR","msg":"500: Internal Server Error","request":{"time":"2024-08-20T08:37:35.851262-03:00","method":"GET","host":"localhost:3000","path":"/","query":"","params":{},"route":"/","ip":"[::1]:63409","referer":"","length":0},"response":{"time":"2024-08-20T08:37:36.856046-03:00","latency":1004819000,"status":500,"length":45},"id":""}

This way, it is possible to see that the client (curl in this case) had an effective response and that serviceA was no significant impact.

Let's improve the resilience of our application by investigating another beneficial policy: retry.

Retry

Again, it was necessary to make a change to serviceB to add random errors:

package main

import (
    "math/rand"
    "net/http"
    "strconv"
    "time"

    "github.com/go-chi/chi/v5"
    "github.com/go-chi/chi/v5/middleware"
)

func main() {
    r := chi.NewRouter()
    r.Use(middleware.Logger)
    r.Get("/", func(w http.ResponseWriter, r *http.Request) {
        retryAfterDelay := 1 * time.Second
        if fail() {
            w.Header().Add("Retry-After", strconv.Itoa(int(retryAfterDelay.Seconds())))
            w.WriteHeader(http.StatusServiceUnavailable)
            return
        }
        w.Header().Set("Content-Type", "application/json")
        w.Write([]byte(`{"message": "hello from service B"}`))
    })
    http.ListenAndServe(":3001", r)
}

func fail() bool {
    if flipint := rand.Intn(2); flipint == 0 {
        return true
    }
    return false
}

To make it easier to understand, I am showing one policy at a time, which is why serviceA was changed to the original version and not to the version with a timeout. Later, we will examine how to compose several policies to make the application more resilient.

The code serviceA/main.go looked like this:

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "log/slog"
    "net/http"
    "os"
    "time"

    "github.com/failsafe-go/failsafe-go"
    "github.com/failsafe-go/failsafe-go/failsafehttp"
    "github.com/failsafe-go/failsafe-go/retrypolicy"
    "github.com/go-chi/chi/v5"
    slogchi "github.com/samber/slog-chi"
)

func main() {
    logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
    r := chi.NewRouter()
    r.Use(slogchi.New(logger))
    r.Get("/", func(w http.ResponseWriter, r *http.Request) {
        type response struct {
            Message string `json:"message"`
        }
        // Create a RetryPolicy that only handles 500 responses, with backoff delays between retries
        retryPolicy := newRetryPolicy(logger)

        // Use the RetryPolicy with a failsafe RoundTripper
        roundTripper := failsafehttp.NewRoundTripper(nil, retryPolicy)
        client := &http.Client{Transport: roundTripper}

        resp, err := client.Get("http://localhost:3001")
        if err != nil {
            w.WriteHeader(http.StatusInternalServerError)
            w.Write([]byte(err.Error()))
            return
        }
        body, err := io.ReadAll(resp.Body)
        if err != nil {
            w.WriteHeader(http.StatusInternalServerError)
            w.Write([]byte(err.Error()))
            return
        }
        defer resp.Body.Close()
        var data response
        err = json.Unmarshal(body, &data)
        if err != nil {
            w.WriteHeader(http.StatusInternalServerError)
            w.Write([]byte(err.Error()))
            return
        }
        w.Header().Set("Content-Type", "application/json")
        w.Write([]byte(`{"messageA": "hello from service A","messageB": "` + data.Message + `"}`))
    })
    http.ListenAndServe(":3000", r)
}

func newRetryPolicy(logger *slog.Logger) retrypolicy.RetryPolicy[*http.Response] {
    return retrypolicy.Builder[*http.Response]().
        HandleIf(func(response *http.Response, _ error) bool {
            return response != nil && response.StatusCode == http.StatusServiceUnavailable
        }).
        WithBackoff(time.Second, 10*time.Second).
        OnRetryScheduled(func(e failsafe.ExecutionScheduledEvent[*http.Response]) {
            logger.Info(fmt.Sprintf("Retry %d after delay of %d", e.Attempts(), e.Delay))
        }).Build()
}

This way, if serviceB returns the status StatusServiceUnavailable (code 503), the connection will be attempted again at progressive intervals, thanks to the function configuration WithBackoff. The output of serviceA, when accessed via curl, should be something similar to:

go run main.go
{"time":"2024-08-20T08:43:38.297621-03:00","level":"INFO","msg":"200: OK","request":{"time":"2024-08-20T08:43:38.283715-03:00","method":"GET","host":"localhost:3000","path":"/","query":"","params":{},"route":"/","ip":"[::1]:63542","referer":"","length":0},"response":{"time":"2024-08-20T08:43:38.297556-03:00","latency":13840708,"status":200,"length":71},"id":""}
{"time":"2024-08-20T08:43:39.946562-03:00","level":"INFO","msg":"200: OK","request":{"time":"2024-08-20T08:43:39.943394-03:00","method":"GET","host":"localhost:3000","path":"/","query":"","params":{},"route":"/","ip":"[::1]:63544","referer":"","length":0},"response":{"time":"2024-08-20T08:43:39.946545-03:00","latency":3151000,"status":200,"length":71},"id":""}
{"time":"2024-08-20T08:43:40.845862-03:00","level":"INFO","msg":"Retry 1 after delay of 1000000000"}
{"time":"2024-08-20T08:43:41.85287-03:00","level":"INFO","msg":"Retry 2 after delay of 2000000000"}
{"time":"2024-08-20T08:43:43.860694-03:00","level":"INFO","msg":"200: OK","request":{"time":"2024-08-20T08:43:40.841468-03:00","method":"GET","host":"localhost:3000","path":"/","query":"","params":{},"route":"/","ip":"[::1]:63545","referer":"","length":0},"response":{"time":"2024-08-20T08:43:43.860651-03:00","latency":3019287458,"status":200,"length":71},"id":""}

In this example, it is possible to see that errors occurred when accessing the serviceB, and the lib executed the connection again until it was successful. If the connection continues to give an error, the client will receive an error message ”http://localhost:3001": retries exceeded.

Let's go deeper into resilience by adding a circuit breaker to our project.

Circuit breaker

The circuit breaker concept is a more advanced policy that provides greater control over access to services. The pattern circuit breaker works in three states: closed (no errors), open (with errors, interrupts transmission), and semi-open (sends a limited number of requests to the service in difficulty to test its recovery).

To use this policy, I made a new version of serviceB so that it could generate more error scenarios and delays:

package main

import (
    "math/rand"
    "net/http"
    "strconv"
    "time"

    "github.com/go-chi/chi/v5"
    "github.com/go-chi/chi/v5/middleware"
)

func main() {
    r := chi.NewRouter()
    r.Use(middleware.Logger)
    r.Get("/", func(w http.ResponseWriter, r *http.Request) {
        retryAfterDelay := 1 * time.Second
        if fail() {
            w.Header().Add("Retry-After", strconv.Itoa(int(retryAfterDelay.Seconds())))
            w.WriteHeader(http.StatusServiceUnavailable)
            return
        }
        if sleep() {
            time.Sleep(1 * time.Second)
        }
        w.Header().Set("Content-Type", "application/json")
        w.Write([]byte(`{"message": "hello from service B"}`))
    })
    http.ListenAndServe(":3001", r)
}

func fail() bool {
    if flipint := rand.Intn(2); flipint == 0 {
        return true
    }
    return false
}

func sleep() bool {
    if flipint := rand.Intn(2); flipint == 0 {
        return true
    }
    return false
}

And the code of serviceA:

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "log/slog"
    "net/http"
    "os"
    "time"

    "github.com/failsafe-go/failsafe-go/circuitbreaker"
    "github.com/failsafe-go/failsafe-go/failsafehttp"
    "github.com/go-chi/chi/v5"
    slogchi "github.com/samber/slog-chi"
)

func main() {
    logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
    r := chi.NewRouter()
    r.Use(slogchi.New(logger))
    r.Get("/", func(w http.ResponseWriter, r *http.Request) {
        type response struct {
            Message string `json:"message"`
        }
        // Create a CircuitBreaker that handles 503 responses and uses a half-open delay based on the Retry-After header
        circuitBreaker := newCircuitBreaker(logger)

        // Use the RetryPolicy with a failsafe RoundTripper
        roundTripper := failsafehttp.NewRoundTripper(nil, circuitBreaker)
        client := &http.Client{Transport: roundTripper}

        sendGet := func() (*http.Response, error) {
            resp, err := client.Get("http://localhost:3001")
            return resp, err
        }
        maxRetries := 3
        resp, err := sendGet()
        for i := 0; i < maxRetries; i++ {
            if err == nil && resp != nil && resp.StatusCode != http.StatusServiceUnavailable && resp.StatusCode != http.StatusTooManyRequests {
                break
            }
            time.Sleep(circuitBreaker.RemainingDelay()) // Wait for circuit breaker's delay, provided by the Retry-After header
            resp, err = sendGet()
        }
        if err != nil {
            w.WriteHeader(http.StatusInternalServerError)
            w.Write([]byte(err.Error()))
            return
        }
        body, err := io.ReadAll(resp.Body)
        if err != nil {
            w.WriteHeader(http.StatusInternalServerError)
            w.Write([]byte(err.Error()))
            return
        }
        defer resp.Body.Close()
        var data response
        err = json.Unmarshal(body, &data)
        if err != nil {
            w.WriteHeader(http.StatusInternalServerError)
            w.Write([]byte(err.Error()))
            return
        }
        w.Header().Set("Content-Type", "application/json")
        w.Write([]byte(`{"messageA": "hello from service A","messageB": "` + data.Message + `"}`))
    })
    http.ListenAndServe(":3000", r)
}

func newCircuitBreaker(logger *slog.Logger) circuitbreaker.CircuitBreaker[*http.Response] {
    return circuitbreaker.Builder[*http.Response]().
        HandleIf(func(response *http.Response, err error) bool {
            return response != nil && response.StatusCode == http.StatusServiceUnavailable
        }).
        WithDelayFunc(failsafehttp.DelayFunc).
        OnStateChanged(func(event circuitbreaker.StateChangedEvent) {
            logger.Info(fmt.Sprintf("circuit breaker state changed from %s to %s", event.OldState.String(), event.NewState.String()))
        }).
        Build()
}

As we can see in the output of serviceA, the circuit breaker is working:

❯ go run main.go
{"time":"2024-08-20T08:51:37.770611-03:00","level":"INFO","msg":"circuit breaker state changed from closed to open"}
{"time":"2024-08-20T08:51:38.771682-03:00","level":"INFO","msg":"circuit breaker state changed from open to half-open"}
{"time":"2024-08-20T08:51:38.776743-03:00","level":"INFO","msg":"circuit breaker state changed from half-open to open"}
{"time":"2024-08-20T08:51:39.777821-03:00","level":"INFO","msg":"circuit breaker state changed from open to half-open"}
{"time":"2024-08-20T08:51:39.784897-03:00","level":"INFO","msg":"circuit breaker state changed from half-open to open"}
{"time":"2024-08-20T08:51:40.786209-03:00","level":"INFO","msg":"circuit breaker state changed from open to half-open"}
{"time":"2024-08-20T08:51:40.792457-03:00","level":"INFO","msg":"circuit breaker state changed from half-open to closed"}
{"time":"2024-08-20T08:51:40.792733-03:00","level":"INFO","msg":"200: OK","request":{"time":"2024-08-20T08:51:37.756947-03:00","method":"GET","host":"localhost:3000","path":"/","query":"","params":{},"route":"/","ip":"[::1]:63699","referer":"","length":0},"response":{"time":"2024-08-20T08:51:40.792709-03:00","latency":3036065875,"status":200,"length":71},"id":""}

This policy allows greater control over errors, allowing serviceB to recover if it is experiencing a problem.

But what do you do when serviceB can no longer return, for whatever reason? In these cases, we can use a fallback.

Fallback

The idea of this policy is to have an alternative if the desired service has a more severe problem and takes a long time to return. To do this, we will change the code serviceA:

package main

import (
    "bytes"
    "encoding/json"
    "io"
    "log/slog"
    "net/http"
    "os"

    "github.com/failsafe-go/failsafe-go"
    "github.com/failsafe-go/failsafe-go/failsafehttp"
    "github.com/failsafe-go/failsafe-go/fallback"
    "github.com/go-chi/chi/v5"
    slogchi "github.com/samber/slog-chi"
)

func main() {
    logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
    r := chi.NewRouter()
    r.Use(slogchi.New(logger))
    r.Get("/", func(w http.ResponseWriter, r *http.Request) {
        fallback := newFallback(logger)

        roundTripper := failsafehttp.NewRoundTripper(nil, fallback)
        client := &http.Client{Transport: roundTripper}

        resp, err := client.Get("http://localhost:3001")
        if err != nil {
            w.WriteHeader(http.StatusInternalServerError)
            w.Write([]byte(err.Error()))
            return
        }
        body, err := io.ReadAll(resp.Body)
        if err != nil {
            w.WriteHeader(http.StatusInternalServerError)
            w.Write([]byte(err.Error()))
            return
        }
        defer resp.Body.Close()
        type response struct {
            Message string `json:"message"`
        }
        var data response
        err = json.Unmarshal(body, &data)
        if err != nil {
            w.WriteHeader(http.StatusInternalServerError)
            w.Write([]byte(err.Error()))
            return
        }
        w.Header().Set("Content-Type", "application/json")
        w.Write([]byte(`{"messageA": "hello from service A","messageB": "` + data.Message + `"}`))
    })
    http.ListenAndServe(":3000", r)
}

func newFallback(logger *slog.Logger) fallback.Fallback[*http.Response] {
    resp := &http.Response{
        StatusCode: http.StatusOK,
        Header:     map[string][]string{"Content-Type": {"application/json"}},
        Body:       io.NopCloser(bytes.NewBufferString(`{"message": "error accessing service B"}`)),
    }
    return fallback.BuilderWithResult[*http.Response](resp).
        HandleIf(func(response *http.Response, err error) bool {
            return response != nil && response.StatusCode == http.StatusServiceUnavailable
        }).
        OnFallbackExecuted(func(e failsafe.ExecutionDoneEvent[*http.Response]) {
            logger.Info("Fallback executed result")
        }).
        Build()
}

In the function newFallback, we can see the creation of one http.Response that the lib will use if the user serviceB does not respond.

This feature allows us to respond to the client while the team responsible for serviceB have time to get the service up and running again.

The output of serviceA is something similar to this:

❯ go run main.go
{"time":"2024-08-20T08:55:27.326475-03:00","level":"INFO","msg":"200: OK","request":{"time":"2024-08-20T08:55:27.31306-03:00","method":"GET","host":"localhost:3000","path":"/","query":"","params":{},"route":"/","ip":"[::1]:63772","referer":"","length":0},"response":{"time":"2024-08-20T08:55:27.326402-03:00","latency":13343208,"status":200,"length":71},"id":""}
{"time":"2024-08-20T08:55:31.756765-03:00","level":"INFO","msg":"200: OK","request":{"time":"2024-08-20T08:55:31.754348-03:00","method":"GET","host":"localhost:3000","path":"/","query":"","params":{},"route":"/","ip":"[::1]:63774","referer":"","length":0},"response":{"time":"2024-08-20T08:55:31.756753-03:00","latency":2404750,"status":200,"length":71},"id":""}
{"time":"2024-08-20T08:55:34.091845-03:00","level":"INFO","msg":"200: OK","request":{"time":"2024-08-20T08:55:33.086273-03:00","method":"GET","host":"localhost:3000","path":"/","query":"","params":{},"route":"/","ip":"[::1]:63775","referer":"","length":0},"response":{"time":"2024-08-20T08:55:34.091812-03:00","latency":1005580625,"status":200,"length":71},"id":""}
{"time":"2024-08-20T08:55:37.386512-03:00","level":"INFO","msg":"Fallback executed result"}
{"time":"2024-08-20T08:55:37.386553-03:00","level":"INFO","msg":"200: OK","request":{"time":"2024-08-20T08:55:37.38415-03:00","method":"GET","host":"localhost:3000","path":"/","query":"","params":{},"route":"/","ip":"[::1]:63777","referer":"","length":0},"response":{"time":"2024-08-20T08:55:37.386544-03:00","latency":2393916,"status":200,"length":76},"id":""}

In the next step, we will combine the concepts we've seen to create a more resilient application.

Policy composition

To do this, we need to change the code of serviceA so that it makes use of the policies we have seen so far:

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io"
    "log/slog"
    "net/http"
    "os"
    "time"

    "github.com/failsafe-go/failsafe-go"
    "github.com/failsafe-go/failsafe-go/circuitbreaker"
    "github.com/failsafe-go/failsafe-go/failsafehttp"
    "github.com/failsafe-go/failsafe-go/fallback"
    "github.com/failsafe-go/failsafe-go/retrypolicy"
    "github.com/failsafe-go/failsafe-go/timeout"
    "github.com/go-chi/chi/v5"
    slogchi "github.com/samber/slog-chi"
)

func main() {
    logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
    r := chi.NewRouter()
    r.Use(slogchi.New(logger))
    r.Get("/", func(w http.ResponseWriter, r *http.Request) {
        type response struct {
            Message string `json:"message"`
        }
        retryPolicy := newRetryPolicy(logger)
        fallback := newFallback(logger)
        circuitBreaker := newCircuitBreaker(logger)
        timeout := newTimeout(logger)

        roundTripper := failsafehttp.NewRoundTripper(nil, fallback, retryPolicy, circuitBreaker, timeout)
        client := &http.Client{Transport: roundTripper}

        sendGet := func() (*http.Response, error) {
            resp, err := client.Get("http://localhost:3001")
            return resp, err
        }
        maxRetries := 3
        resp, err := sendGet()
        for i := 0; i < maxRetries; i++ {
            if err == nil && resp != nil && resp.StatusCode != http.StatusServiceUnavailable && resp.StatusCode != http.StatusTooManyRequests {
                break
            }
            time.Sleep(circuitBreaker.RemainingDelay()) // Wait for circuit breaker's delay, provided by the Retry-After header
            resp, err = sendGet()
        }
        if err != nil {
            w.WriteHeader(http.StatusInternalServerError)
            w.Write([]byte(err.Error()))
            return
        }
        body, err := io.ReadAll(resp.Body)
        if err != nil {
            w.WriteHeader(http.StatusInternalServerError)
            w.Write([]byte(err.Error()))
            return
        }
        defer resp.Body.Close()
        var data response
        err = json.Unmarshal(body, &data)
        if err != nil {
            w.WriteHeader(http.StatusInternalServerError)
            w.Write([]byte(err.Error()))
            return
        }
        w.Header().Set("Content-Type", "application/json")
        w.Write([]byte(`{"messageA": "hello from service A","messageB": "` + data.Message + `"}`))
    })
    http.ListenAndServe(":3000", r)
}

func newTimeout(logger *slog.Logger) timeout.Timeout[*http.Response] {
    return timeout.Builder[*http.Response](10 * time.Second).
        OnTimeoutExceeded(func(e failsafe.ExecutionDoneEvent[*http.Response]) {
            logger.Info("Connection timed out")
        }).Build()
}

func newFallback(logger *slog.Logger) fallback.Fallback[*http.Response] {
    resp := &http.Response{
        StatusCode: http.StatusOK,
        Header:     map[string][]string{"Content-Type": {"application/json"}},
        Body:       io.NopCloser(bytes.NewBufferString(`{"message": "error accessing service B"}`)),
    }
    return fallback.BuilderWithResult[*http.Response](resp).
        HandleIf(func(response *http.Response, err error) bool {
            return response != nil && response.StatusCode == http.StatusServiceUnavailable
        }).
        OnFallbackExecuted(func(e failsafe.ExecutionDoneEvent[*http.Response]) {
            logger.Info("Fallback executed result")
        }).
        Build()
}

func newRetryPolicy(logger *slog.Logger) retrypolicy.RetryPolicy[*http.Response] {
    return retrypolicy.Builder[*http.Response]().
        HandleIf(func(response *http.Response, _ error) bool {
            return response != nil && response.StatusCode == http.StatusServiceUnavailable
        }).
        WithBackoff(time.Second, 10*time.Second).
        OnRetryScheduled(func(e failsafe.ExecutionScheduledEvent[*http.Response]) {
            logger.Info(fmt.Sprintf("Retry %d after delay of %d", e.Attempts(), e.Delay))
        }).Build()
}

func newCircuitBreaker(logger *slog.Logger) circuitbreaker.CircuitBreaker[*http.Response] {
    return circuitbreaker.Builder[*http.Response]().
        HandleIf(func(response *http.Response, err error) bool {
            return response != nil && response.StatusCode == http.StatusServiceUnavailable
        }).
        WithDelayFunc(failsafehttp.DelayFunc).
        OnStateChanged(func(event circuitbreaker.StateChangedEvent) {
            logger.Info(fmt.Sprintf("circuit breaker state changed from %s to %s", event.OldState.String(), event.NewState.String()))
        }).
        Build()
}

In the code:

roundTripper := failsafehttp.NewRoundTripper(nil, fallback, retryPolicy, circuitBreaker, timeout)

It is possible to view the use of all defined policies. The lib will execute it in the "rightmost" order, that is:

timeout -> circuitBreaker -> retryPolicy -> fallback

We can see the execution of the policies by observing the serviceA output:

go run main.go
{"time":"2024-08-19T10:15:29.226553-03:00","level":"INFO","msg":"circuit breaker state changed from closed to open"}
{"time":"2024-08-19T10:15:29.226841-03:00","level":"INFO","msg":"Retry 1 after delay of 1000000000"}
{"time":"2024-08-19T10:15:30.227941-03:00","level":"INFO","msg":"circuit breaker state changed from open to half-open"}
{"time":"2024-08-19T10:15:30.234182-03:00","level":"INFO","msg":"circuit breaker state changed from half-open to open"}
{"time":"2024-08-19T10:15:30.234258-03:00","level":"INFO","msg":"Retry 2 after delay of 2000000000"}
{"time":"2024-08-19T10:15:32.235282-03:00","level":"INFO","msg":"circuit breaker state changed from open to half-open"}
{"time":"2024-08-19T10:15:42.23622-03:00","level":"INFO","msg":"Connection timed out"}
{"time":"2024-08-19T10:15:42.237942-03:00","level":"INFO","msg":"circuit breaker state changed from half-open to closed"}
{"time":"2024-08-19T10:15:42.238043-03:00","level":"ERROR","msg":"500: Internal Server Error","request":{"time":"2024-08-19T10:15:29.215709-03:00","method":"GET","host":"localhost:3000","path":"/","query":"","params":{},"route":"/","ip":"[::1]:52527","referer":"","length":0},"response":{"time":"2024-08-19T10:15:42.238008-03:00","latency":13022704750,"status":500,"length":45},"id":""}
{"time":"2024-08-19T10:15:56.53476-03:00","level":"INFO","msg":"circuit breaker state changed from closed to open"}
{"time":"2024-08-19T10:15:56.534803-03:00","level":"INFO","msg":"Retry 1 after delay of 1000000000"}
{"time":"2024-08-19T10:15:57.535108-03:00","level":"INFO","msg":"circuit breaker state changed from open to half-open"}
{"time":"2024-08-19T10:15:57.53889-03:00","level":"INFO","msg":"circuit breaker state changed from half-open to open"}
{"time":"2024-08-19T10:15:57.538911-03:00","level":"INFO","msg":"Retry 2 after delay of 2000000000"}
{"time":"2024-08-19T10:15:59.539948-03:00","level":"INFO","msg":"circuit breaker state changed from open to half-open"}
{"time":"2024-08-19T10:15:59.544425-03:00","level":"INFO","msg":"circuit breaker state changed from half-open to open"}
{"time":"2024-08-19T10:15:59.544575-03:00","level":"ERROR","msg":"500: Internal Server Error","request":{"time":"2024-08-19T10:15:56.5263-03:00","method":"GET","host":"localhost:3000","path":"/","query":"","params":{},"route":"/","ip":"[::1]:52542","referer":"","length":0},"response":{"time":"2024-08-19T10:15:59.544557-03:00","latency":3018352000,"status":500,"length":245},"id":""}
{"time":"2024-08-19T10:16:11.044207-03:00","level":"INFO","msg":"Connection timed out"}
{"time":"2024-08-19T10:16:11.046026-03:00","level":"ERROR","msg":"500: Internal Server Error","request":{"time":"2024-08-19T10:16:01.043317-03:00","method":"GET","host":"localhost:3000","path":"/","query":"","params":{},"route":"/","ip":"[::1]:52544","referer":"","length":0},"response":{"time":"2024-08-19T10:16:11.045601-03:00","latency":10002596334,"status":500,"length":45},"id":""}

Conclusion

One of the advantages of microservices architecture is that we can break a complex domain into smaller, specialized services that communicate with each other to complete the necessary logic. Ensuring that this communication is resilient and will continue to work even in the face of failures and unforeseen events is fundamental. Using libraries such as failsafe-go makes this process easier.

You can find the codes presented in this post on my Github.

Originally published at https://eltonminetto.dev on August 24, 2024

JSON vs FlatBuffers vs Protocol Buffers

Elton Minetto — Wed, 07 Aug 2024 11:53:59 +0000

When we think about communication between services/microservices, the first option that comes to mind is good old JSON. And this is not without reason, as the format has advantages, such as:

is easily readable by both computers and humans;
all modern programming languages can read and generate JSON;
it is much less verbose than the previous alternative, Jurassic XML.

Using JSON is the recommendation for the vast majority of APIs developed in companies' daily lives. But in some cases, where performance is critical, we may need to consider other alternatives. This post aims to show two alternatives to JSON when it comes to communication between applications.

But what's the problem with JSON? One of its advantages is that it is "easily readable by humans," but this can be a weak point in performance. The fact is that we need to convert the JSON content to some structure that is known by the programming language we are using. An exception to this rule is if we use JavaScript, as JSON is native to it. But if you are using another language, Go, for example, we need to parse the data, as we can see in the (incomplete) code example below:



type event struct {
    ID      uuid.UUID
    Type    string `json:"type"`
    Source  string `json:"source"`
    Subject string `json:"subject"`
    Time    string `json:"time"`
    Data    string `json:"data"`
}

var e event
err := json.NewDecoder(data).Decode(&e)
if err != nil {
    http.Error(w, err.Error(), http.StatusBadRequest)
}

To solve this problem, we can test two alternatives, Protocol Buffers and Flatbuffers.

Protocol Buffers

Protobuf (Protocol Buffers), created by Google, is, according to the official website :

Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data—think XML, but smaller, faster, and more straightforward. You define how you want your data to be structured once. Then, you can use specially generated source code to quickly write and read your structured data to and from various data streams using a variety of languages.

Generally used in conjunction with gRPC (but not necessarily), Protobuf is a binary protocol that significantly increases performance compared to the text format of JSON. But it "suffers" from the same problem as JSON: we need to parse it to a data structure of our language. For example, in Go:



//generated code
type Event struct {
    state         protoimpl.MessageState
    sizeCache     protoimpl.SizeCache
    unknownFields protoimpl.UnknownFields

    Type    string `protobuf:"bytes,1,opt,name=type,proto3" json:"type,omitempty"`
    Subject string `protobuf:"bytes,2,opt,name=subject,proto3" json:"subject,omitempty"`
    Source  string `protobuf:"bytes,3,opt,name=source,proto3" json:"source,omitempty"`
    Time    string `protobuf:"bytes,4,opt,name=time,proto3" json:"time,omitempty"`
    Data    string `protobuf:"bytes,5,opt,name=data,proto3" json:"data,omitempty"`
}

e := Event{}
err := proto.Unmarshal(data, &e)
if err != nil {
    http.Error(w, err.Error(), http.StatusBadRequest)
}

Adopting a binary protocol gives us a performance gain, but we still need to solve the problem of data parsing. Our third competitor focuses on solving this problem.

Flatbuffers

According to the official website:

FlatBuffers is an efficient cross-platform serialization library for C++, C#, C, Go, Java, Kotlin, JavaScript, Lobster, Lua, TypeScript, PHP, Python, Rust, and Swift. It was initially created at Google for game development and other performance-critical applications.

Although initially created for game development, it fits perfectly into the environment we study in this post. Its advantage is that we do not need to parse the data in addition to being a binary protocol. For example, in Go:



//generated code
e := events.GetRootAsEvent(data, 0)

//we can use the data directly
saveEvent(string(e.Type()), string(e.Source()), string(e.Subject()), string(e.Time()), string(e.Data()))

But how much more performant are the two alternatives to JSON? Let's investigate...

Application

The first question that came to my mind was, "how can I apply this in a real scenario?". I imagined the following scenario:

A company with a mobile application, accessed daily by millions of customers, with an internal microservices architecture and which needs to save events generated by users and systems for auditing purposes.

This is a genuine scenario. So real that I live with it every day at the company where I work :)

Note: the scenario above is a simplification and does not represent the actual complexity of the team's application. It serves educational purposes.

The first step is defining an event in Protocol Buffers and Flatbuffers. Both have their own language for defining schemas, which we can then use to generate code in the languages we will use. I will not delve into the details of each scheme as this is easily found in the documentation.

The file event.proto has the Protocol Buffer definition:



syntax = "proto3";
package events;

option go_package = "./events_pb";

message Event {
    string type = 1;
    string subject = 2;
    string source = 3;
    string time = 4;
    string data = 5;
}

And the file event.fbs has the equivalent in Flatbuffers:



namespace events;

table Event {
    type: string;
    subject:string;
    source:string;
    time:string;
    data:string;
}

root_type Event;

The next step is to use these definitions to generate the necessary code. The following commands install the dependencies on macOS:



go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
brew install protobuf
protoc -I=. --go_out=./ event.proto
brew install flatbuffers
flatc --go event.fbs

The result is the creation of Go packages to manipulate data in each format.

With the requirements met, the next step was implementing the event API. The main.go looked like this:



package main

import (
    "fmt"
    "net/http"
    "os"

    "github.com/go-chi/chi/v5"
    "github.com/go-chi/chi/v5/middleware"
    "github.com/google/uuid"
)

func main() {
    r := handlers()
    http.ListenAndServe(":3000", r)
}

func handlers() *chi.Mux {
    r := chi.NewRouter()
    if os.Getenv("DEBUG") != "false" {
        r.Use(middleware.Logger)
    }
    r.Post("/json", processJSON())
    r.Post("/fb", processFB())
    r.Post("/pb", processPB())
    return r
}

func saveEvent(evType, source, subject, time, data string) {
    if os.Getenv("DEBUG") != "false" {
        id := uuid.New()
        q := fmt.Sprintf("insert into event values('%s', '%s', '%s', '%s', '%s', '%s')", id, evType, source, subject, time, data)
        fmt.Println(q)
    }
    // save event to database
}

For better organization, I created files to separate each function, which looked as follows:



package main

import (
    "encoding/json"
    "net/http"

    "github.com/google/uuid"
)

type event struct {
    ID      uuid.UUID
    Type    string `json:"type"`
    Source  string `json:"source"`
    Subject string `json:"subject"`
    Time    string `json:"time"`
    Data    string `json:"data"`
}

func processJSON() http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        var e event
        err := json.NewDecoder(r.Body).Decode(&e)
        if err != nil {
            http.Error(w, err.Error(), http.StatusBadRequest)
        }
        saveEvent(e.Type, e.Source, e.Subject, e.Time, e.Data)
        w.WriteHeader(http.StatusCreated)
        w.Write([]byte("json received"))
    }
}



package main

import (
    "io"
    "net/http"

    "github.com/eminetto/post-flatbuffers/events_pb"
    "google.golang.org/protobuf/proto"
)

func processPB() http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        body := r.Body
        data, _ := io.ReadAll(body)

        e := events_pb.Event{}
        err := proto.Unmarshal(data, &e)
        if err != nil {
            http.Error(w, err.Error(), http.StatusBadRequest)
        }
        saveEvent(e.GetType(), e.GetSource(), e.GetSubject(), e.GetTime(), e.GetData())
        w.WriteHeader(http.StatusCreated)
        w.Write([]byte("protobuf received"))
    }
}



package main

import (
    "io"
    "net/http"

    "github.com/eminetto/post-flatbuffers/events"
)

func processFB() http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        body := r.Body
        data, _ := io.ReadAll(body)
        e := events.GetRootAsEvent(data, 0)
        saveEvent(string(e.Type()), string(e.Source()), string(e.Subject()), string(e.Time()), string(e.Data()))
        w.WriteHeader(http.StatusCreated)
        w.Write([]byte("flatbuffer received"))
    }
}

In the functions processPB() and processFB(), we can see how the generated packages are used to manipulate the data.

Benchmark

The last step of our proof of concept is generating the benchmark to compare the formats. I used the Go stdlib benchmark package for this.

The file main_test.go has tests for each format:



package main

import (
    "bytes"
    "fmt"
    "net/http"
    "net/http/httptest"
    "os"
    "strings"
    "testing"

    "github.com/eminetto/post-flatbuffers/events"
    "github.com/eminetto/post-flatbuffers/events_pb"
    flatbuffers "github.com/google/flatbuffers/go"
    "google.golang.org/protobuf/proto"
)

func benchSetup() {
    os.Setenv("DEBUG", "false")
}

func BenchmarkJSON(b *testing.B) {
    benchSetup()
    r := handlers()
    payload := fmt.Sprintf(`{
        "type": "button.clicked",
        "source": "Login",
        "subject": "user1000",
        "time": "2018-04-05T17:31:00Z",
        "data": "User clicked because X"}`)
    for i := 0; i < b.N; i++ {
        w := httptest.NewRecorder()
        req, _ := http.NewRequest("POST", "/json", strings.NewReader(payload))
        r.ServeHTTP(w, req)
        if w.Code != http.StatusCreated {
            b.Errorf("expected status 201, got %d", w.Code)
        }
    }
}

func BenchmarkFlatBuffers(b *testing.B) {
    benchSetup()
    r := handlers()
    builder := flatbuffers.NewBuilder(1024)
    evtType := builder.CreateString("button.clicked")
    evtSource := builder.CreateString("service-b")
    evtSubject := builder.CreateString("user1000")
    evtTime := builder.CreateString("2018-04-05T17:31:00Z")
    evtData := builder.CreateString("User clicked because X")

    events.EventStart(builder)
    events.EventAddType(builder, evtType)
    events.EventAddSource(builder, evtSource)
    events.EventAddSubject(builder, evtSubject)
    events.EventAddTime(builder, evtTime)
    events.EventAddData(builder, evtData)
    evt := events.EventEnd(builder)
    builder.Finish(evt)

    buff := builder.FinishedBytes()
    for i := 0; i < b.N; i++ {
        w := httptest.NewRecorder()
        req, _ := http.NewRequest("POST", "/fb", bytes.NewReader(buff))
        r.ServeHTTP(w, req)
        if w.Code != http.StatusCreated {
            b.Errorf("expected status 201, got %d", w.Code)
        }
    }
}

func BenchmarkProtobuffer(b *testing.B) {
    benchSetup()
    r := handlers()
    evt := events_pb.Event{
        Type:    "button.clicked",
        Subject: "user1000",
        Source:  "service-b",
        Time:    "2018-04-05T17:31:00Z",
        Data:    "User clicked because X",
    }
    payload, err := proto.Marshal(&evt)
    if err != nil {
        panic(err)
    }
    for i := 0; i < b.N; i++ {
        w := httptest.NewRecorder()
        req, _ := http.NewRequest("POST", "/pb", bytes.NewReader(payload))
        r.ServeHTTP(w, req)
        if w.Code != http.StatusCreated {
            b.Errorf("expected status 201, got %d", w.Code)
        }
    }
}

It generates an event in each format and sends it to the API.

When we run the benchmark, we have the following result:



Running tool: /opt/homebrew/bin/go test -benchmem -run=^$ -coverprofile=/var/folders/vn/gff4w90d37xbfc_2tn3616h40000gn/T/vscode-gojAS4GO/go-code-cover -bench . github.com/eminetto/post-flatbuffers/cmd/api -failfast -v

goos: darwin
goarch: arm64
pkg: github.com/eminetto/post-flatbuffers/cmd/api
BenchmarkJSON
BenchmarkJSON-8               658386          1732 ns/op        2288 B/op         26 allocs/op
BenchmarkFlatBuffers
BenchmarkFlatBuffers-8       1749194           640.5 ns/op      1856 B/op         21 allocs/op
BenchmarkProtobuffer
BenchmarkProtobuffer-8       1497356           696.9 ns/op      1952 B/op         21 allocs/op
PASS
coverage: 77.5% of statements
ok      github.com/eminetto/post-flatbuffers/cmd/api    5.042s

If this is the first time you have analyzed the results of a Go benchmark, I recommend reading this post, where the author describes the details of each column and its meaning.

To make it easier to visualize, I created graphs for the most critical information generated by the benchmark:

‌Number of iterations (higher is better)

Nanoseconds per operation (lower is better)

Number of bytes allocated per operation (lower is better)

Number of allocations per operation (lower is better)

Conclusion

The numbers show a great advantage of binary protocols over JSON, especially Flatbuffers. This advantage is that we do not need to parse the data into structures of the language we are using.

Should you refactor your applications to replace JSON with Flatbuffers? Not necessarily. Performance is just one factor that teams must consider when selecting a communication protocol between their services and applications. But if your application receives billions of requests per day, performance improvements like those presented in this post can make a big difference in terms of costs and user experience.

The codes presented here can be found in this repository. I made the examples using the Go language, but both Protocol Buffers and Flatbuffers support different programming languages, so I would love to see other versions of these comparisons. Additionally, other benchmarks can be used, such as network consumption, CPU, etc. (since we only compare memory here).

I hope this post serves as an introduction to these formats and an incentive for new tests and experiments.

Originally published at https://eltonminetto.dev on August 05, 2024

Go is a platform

Elton Minetto — Sat, 15 Jun 2024 12:42:25 +0000

Thanks to the Google Developer Experts program, I had the opportunity to participate in Google I/O in Mountain View, California, in May this year. Among the various talks I watched, one of my favorites was ‌ Boost performance of Go applications with profile-guided optimization, which you can watch on Youtube.

Although Profile Guided Optimization (PGO) is one of the most exciting features of the language, what caught my attention the most was the first part of the talk, which Cameron Balahan, a Group Product Manager at Google Cloud, presented.

The part that blew my mind was the statement:

He starts by talking about the "DevOps Life Cycle":

And he goes on to highlight the qualities of Go in three crucial aspects:

development speed;
security;
performance.

My comments on each item mentioned in the images:

Easy concurrency: this is one of the great appeals of the language thanks to goroutines and channels.
IDE integrations: We have great IDEs, like Jetbrains' Goland and the Go plugin for Visual Studio Code.
Dependency Management: The language took a while to adopt this, but having dependency management built into the toolset is very useful. All it takes is one go get or go mod tidy to install the project's dependencies.
Static Binaries: this speeds up application deployment, as all you need to do is compile, and we have a self-contained executable ready to run.
Delve Debugger: having a powerful debugger already configured in all IDEs speeds up code development and maintenance.
Built-in Test Framework: The test library being built into the standard library accelerates the adoption of techniques such as TDD.
Cross-compilation: I think it's fantastic that my macOS can generate a binary for Windows, Linux, and other platforms in such a simple way.
Built-in Formatting: There is no time wasted discussing how to format your code as the linter is already in the language's toolset, and all IDEs can format your code when saving it.

Module Mirror Checksum DB: This feature was launched in 2019 and helps ensure the authenticity of the modules the application uses as dependencies.
Memory Safe Code: "Memory Safe is a property of some programming languages that prevents programmers from introducing certain types of bugs related to memory usage." source
Compatibility Promise: This makes choosing Go much safer, especially for companies, as it avoids the need for significant future refactorings, as happened from PHP 4 to PHP 5 or from Python 2 to Python 3, for example—more details on the language blog.
Vulnerability Scanning: The govulncheck tool is essential to the language's toolset and helps us find and fix vulnerabilities in our projects.
Built-in Fuzz Testing: Fuzz testing is an advanced technique that increases the test coverage surface with arbitrary values, improving code quality.
SBOM Generation: SBOM stands for Software Bill of Materials and presents a detailed inventory of all software components and dependencies within a project. This post outlines some ways to generate this resource for applications.
Source Code Analysis: Most commercial static code analysis solutions, such as Sonar, support Go, but there are also open-source projects, such as golangci-lint.

Rich Standard Library: if the motto "with batteries included" had not been used by Python, the Go community could have adopted it. The language's native library has practically everything modern software development needs, from HTTP server/client, tests, JSON parsers, data structures, etc.
Built-in profiling: The language's native library features pprof, which allows us to analyze application performance. I wrote about this in another post.
Runtime Tracing: The team behind the language development recently improved the tracing feature, increasing the details we can collect from applications to understand their behavior.
Self-tuning GC: Go has one of the most modern Garbage Collector implementations, and more details are available on the language blog.
Dynamic Race Detector: Another feature built into the language that helps detect problems faster in the development and testing phase.
Profile-guided Optimization: This feature has significantly reduced resource consumption. The talk that generated this post provides more details.

I liked this way of presenting the language because it briefly shows the entire ecosystem around it and all the benefits we receive when adopting it.
What do you think of this vision? Have you ever felt this way?

Originally published at https://eltonminetto.dev on Jun 12, 2024