Forem: Calvin McLean

Experiment with gopls MCP: Improving Agent Context for Go Development

Calvin McLean — Tue, 09 Sep 2025 03:24:42 +0000

When working with LLMs, managing the context size is really important. I previously learned that a chat session with an LLM or agent is stored as a continuous array of messages and is re-processed by the model for each prompt. As the conversation or coding sessions goes on, the amount of data the model has to process grows. This eventually erodes the responses since they are processing too much information. It is best to keep sessions short, concise, and on-topic.

When you are using an agent to write code, it constantly has to search directories and read files. A lot of this information is not relevant to the goal and contributes to the context rot. This is something that the experimental Model Context Protocol (MCP) server in gopls aims to fix. This tool provides direct access to some parts of the language server protocol (LSP) for Go. Some of these tools, like go_package_api, which fetches the exported functions, types, and comments for a Go package, can be really useful to learn how to use a package. Additionally, go_search provides a way to search a codebase and its dependencies without reading all of the implementation code. go_symbol_references allows the agent to search where a function is used so it can understand the impact of a change without searching a large number of files.

I want to take a deeper dive into gopls MCP and how (or if) it can improve coding agents. Specifically, I want to see how it can improve an agent's ability to use external packages where the source code is not readily available. One way to get around this is to use go mod vendor to get a local copy of dependencies. However, this is not efficient because the agent really only cares about how to use a package. Reading in all of the implementation source code is a waste of the precious context. An abundance of source code in the context can also cause an agent to start copying pieces of code and style choices from dependencies which is not ideal. If you are using a popular package with tons of open source examples in the LLMs training set, you might not notice this issue. However, it is a huge drawback when trying to use new versions of packages, private internal libraries, or more obscure packages.

The Experiment

In order to learn more about the gopls MCP and see how it improves an agent, I will do an experiment. This is not the most scientific experiment and is largely based on user experience and vibes. The experiment will work by attempting to create a new program using a lesser-known library. This library is babyapi, which I wrote to simplify REST API development. This package is pretty unknown and has an eclectic feature set based around things I found interesting or useful.

I will start with a simple project with the module initialized and github.com/calvinmclean/babyapi imported.

go.mod

module customers

go 1.24

require (
    github.com/calvinmclean/babyapi v0.31.0
)

main.go

package main

import _ "github.com/calvinmclean/babyapi"

Here is my initial prompt:

Use github.com/calvinmclean/babyapi to create a REST API for customer relationship management. This should support babyapi's feature for end-dating resources. It should also use babyapi's CLI and support saving to JSON file.

This prompt doesn't provide any specific function names and generally requests features that I know are provided by babyapi.

I am using Github Copilot's free tier to access the GPT-4.1 model.

First, I will run the prompt without the gopls MCP server. Then, I will start fresh, setup the MCP server and system instruction, and run the prompt again.

Attempt 1: VS Code without gopls MCP

I was surprised to see that the first thing the agent did is fetch the documentation from pkg.go.dev/github.com/calvinmclean/babyapi. It then implemented this perfectly on its first attempt.

The end-dating and JSON storage features are directly mentioned and used in examples in my project's README file, so this was easy to implement. Interestingly, the agent also enabled babyapi's built-in MCP server option on the customers API. This feature is also mentioned in the README and probably had a high relevance due to the agent's interest in MCP.

This is the first time I saw an agent fetch documentation like this, so I was impressed to see how well it worked. Unfortunately, it doesn't leave much room for improvement provided by the MCP. The project documentation is somewhat large, so maybe the MCP will be able to reduce the context size by only loading relevant information. Fetching the documentation might not be as effective in larger packages or sparsely-documented ones, so the gopls MCP is still worth exploring.

Attempt 2: VS Code with gopls MCP

Next, I start fresh in a new project, setup the MCP server, the default system instruction, and provide the same prompt. The first attempt here did not work well at all.

It started by using go_package_api to describe the babyapi package and then implemented the base functionality. Then, it spent a lot of time using the go_search tool to figure out how to end-date and save to file storage. It used a variety of search queries like "file storage", "storage", "NewFileDB", and "NewKVStorage". Each got closer and closer to the correct function which is kv.NewFileDB. Then, it started searching for a way to end-date the resources. Eventually, it timed out with a nearly-correct implementation. The only problem was using the incorrect type for the NewFileDB config input. Althought I cannot see the context size, it seems like this likely used more tokens due to the long iteration.

The agent with the MCP was at a disadvantage since the KV Storage example is explicitly spelled-out in the package README, but the go_package_api only gets documentation from the Go file comments. However, it was interesting to see how the agent was able to iterate using the search tools.

I was curious about how the context usage compares between these two experiments, but VS Code does not show the token usage. For the next attempts, I will switch over to my favorite editor: Zed.

Attempt 3: Zed without gopls MCP

Zed is able to use the same model provided by Github, so the only difference here will be the agent implementation by Zed. This time, the agent did not start by reading the documentation and came up with a totally halucinated implementation:

func main() {
    api := babyapi.NewAPI()

    // Register the Customer resource with end-dating enabled
    api.RegisterResource(babyapi.ResourceConfig[Customer]{
        Name:      "customer",
        Store:     babyapi.NewJSONStore[Customer]("customers.json"),
        EndDating: true, // Enables end-dating (soft delete)
    })

    // Start the API server (default on :8080)
    api.Run()
}

When the documentation is not fetched, it's clearly worse than the agent using gopls MCP. Since the agents operated by VS Code and Zed have different system prompts and tooling around them, they tackle this problem differently. The gopls MCP will enable general functionality that is independent of the agent being used. This means a lesser, and probably cheaper agent can reach similar functionality to more advanced ones.

In order to estimate the size of the context that comes from reading the documentation, the prompt is modified to improve the agent's behavior:

Use github.com/calvinmclean/babyapi to create a REST API for customer relationship management. This should support babyapi's feature for end-dating resources. It should also use babyapi's CLI and support saving to JSON file. Start by reading the documentation.

With the imprved prompt, it implemented the program nearly identically to the first attempt with VS Code. This used approximately 26k tokens.

Attempt 4: Zed with gopls MCP

After struggling a bit to get the gopls MCP working with Zed, I continued the next step of the experiment. This had a similar problem compared to VS Code and did a lot of iteration using go_search. It eventually used 35k tokens to get a mostly correct solution. It is unfortunate to see that the gopls MCP is not reducing the context size so far.

I spent some more time iterating on the prompt and analyzing the agent logs. Eventually, I came up with this addition to the base prompt:

Start by using Gopls mcp to describe the babyapi package. Instead of searching files by regex, use gopls MCP's go_search.

Before this change, the agent wasn't always using the go_package_api tool, so I instruced it to do so. It was using regex searches on the codebase when trying to find babyapi functions which was not working. The go_search tool will also search dependencies, so it should use that instead. With the new prompt, it provided a working solution with only 23k tokens. Interestingly, instead of using babyapi's built-in KVStorage, the agent fully implemented the babyapi.Stoage interface with its own file storage mechanism. Had it used the built-in solution, it might have used even fewer tokens.

Conclusion

While the gopls MCP sounds really promising for improving an agent's ability to write Go code, it isn't an absolute game changer right now. While working on these experiments, I learned more about general strategies for using agents rather than learning directly about the impacts of the gopls MCP. Some of these lessons are:

The prompt is really important. Even with a repository instruction/rule describing how to use the gopls MCP, it was useful to explicitly tell the agent to prioritize it over built-in tools
When you know a specific function that you want to use, it's best to spell it out directly instead of describing it generally. With the addition of the gopls MCP, this strategy can become even more effective since the agent can effeciently search for this symbol to learn how to use it
Agents can consume a lot of context. Instructing it to simply read documentation can be really effective if the documentation is well-written. This of course depends on your goal. If you are looking to use one specific function, it might be overkill to consume the whole documentation
Agents are best at iterating. Rather than attempting to get a perfect one-shot prompt, pay attention to the agent's output. Interrupt it and correct it early in the process if you see something incorrect

In regards to the gopls MCP, my example usecase is probably not the most useful one. As demonstrated, simply reading the documentation is sufficient for smaller, well-documented libraries. LLMs are trained on existing code and like to copy, so reading examples is more effective than analyzing function signatures.

I expect that the gopls MCP will work better to augment development in large codebases where the agent can benefit from the go_search and go_symbol_references tools. Also, in a more extended and iterative development example, loading up the whole documentation might pollute the context too much, especially when using multiple packages. I plan to continue using it in my real development flow to observe how it works in a larger real-world usecase. While it didn't crush this example, it is still promising.

Another interesting strategy for working with external libraries is to use a multi-agent workflow. A second agent can be used to browse a variety of relevant documentation. Then, it will extract or summarize the most important pieces and provide that smaller set of data to the main agent.

Ultimately, I learned that good prompting is still the best way to get good results from an agent. The gopls MCP is another tool that, when used properly, can improve the workflow, but is not a perfect solution. It can be used along with good prompts to guide an agent to the best solution.

Next Steps

Before I heard about the gopls MCP, I started theorizing about and working on an idea to improve an agent's ability to get relevant context about a library. This idea is to use retrieval augmented generation (RAG) by vectorizing and semantic-searching documentation. This will provide documentation that is directly relevant to what the agent is looking for. My initial idea was to use information from pkg.go.dev, but I ended up parsing Go files directly to read documentation comments. This was easier to parse and link directly to the code.

After hearing about the gopls MCP, I felt discouraged about this idea and thought that it might no longer be relevant. This experiment proved otherwise. The agent that read documentation was really effective. While reading the whole documentation worked well, using the vectorized data could reduce overall context since the agent won't need to load the whole documentation into each session. Additionally, it should pair really well with the gopls MCP. The semantic search will find relevant symbol and file names, and gopls can be used to look them up and get more details if needed. I will continue working on this and rerun the experiments to see if there are any benefits. I learned here that I should also parse a library's README since it can be more useful than the code comments. Keep an eye out for my next post where I will share these results!

Thank you for reading! If you learned anything new or have any advice, please let me know in the comments.

How to use gopls MCP with VS Code

Calvin McLean — Tue, 02 Sep 2025 04:14:43 +0000

The Go team recently added an official Model Context Protocol (MCP) server to gopls, the language server protocol (LSP) implementation for Go. This allows IDEs and other AI coding agents to directly access some features of the LSP. While most of the features provided by the MCP already exist in AI coding tools, this can improve the efficiency of token use by avoiding searching and reading whole files.

Enable gopls MCP in VS Code

Now, let's get to the reason you're here! With a few simple steps, we can start using this new MCP feature in VS Code:

1. Install/update `gopls`

go install golang.org/x/tools/gopls@latest

I currently have Go 1.24.0 with v0.20.0 of gopls, but newer versions exist at the time of writing this

2. Enable VS Code to run the server

VS Code automatically runs the gopls server for Go projects. A simple update to the config will also start the MCP server.

Navigate to Settings and find Go: Language Server Flags
Click Edit in settings.json and add:

  "go.languageServerFlags": [
      "-mcp.listen=localhost:8092",
  ]

Restart/reload VS Code

3. Enable the MCP server for agents

Use the MCP: Add Server... command
Select HTTP
Input the address: http://localhost:8092
Alternatively, just edit .vscode/settings.json with this content:

  {
    "servers": {
      "gopls": {
        "url": "http://localhost:8092",
        "type": "http"
      }
    }
  }

4. Setup instructions for using the tools

The Go team provides a base instruction to improve how agents use the feature. Setting up VS Code to use this feature will greatly improve how it uses the MCP tools.

First, create the instructions directory

  mkdir -p .github/instructions

Then, start by telling VS Code to use this for Go files:

  cat <<EOF > .github/instructions/gopls.instructions.md
  ---
  applyTo: "**/*.go"
  ---
  EOF

Finally, add the gopls instruction:

  gopls mcp -instructions >> .github/instructions/gopls.instructions.md

Use it!

Now you should have gopls MCP setup in your Go project! Open up the agent chat and start coding!

The simplest place to start is to ask about references for a function or type in your code: "Describe where MyFunction is used."

This should use the go_symbol_references tool to find references to this function.

Why MCP?

One of the most impactful features is the ability to use the go_diagnostics tool. Besides the usual information about diagnostic issues, this also allows the agent to automatically review and apply fixes suggested by the LSP. This greatly improves consistency and accuracy of code fixes since it uses deterministic analysis provided directly by the Go maintainers.

If you are regularly programming with newer versions of Go, you might be familiar with the "modernize" suggestions. I most-commonly see these with typical for i := 0; i < 10; i++ loops that could use the new for i := range 10 syntax. This feature is a big win for the language in the world of AI coding. By definition, LLMs always output "old" code since they are trained on existing codebases. This means any new language features won't be used by LLMs until enough human coders have introduced them into the training data.

The new analyze and modernize tools allow coding agents to automatically update to new language features that they don't even know about yet. This improves the overall quality of a codebase even when using AI agents. This is an important feature helping to keep Go relevant as software development is rapidly changing.

To demonstrate this tool, write a simple function using a for loop:

func Example() {
    for i := 0; i < 10; i++ {
        println("Hello, World!")
    }
}

Then, ask the agent to run diagnostics on it! It should look something like this:

How to Implement LLM Tool-calling with Go and Ollama

Calvin McLean — Mon, 16 Jun 2025 00:43:11 +0000

Tool-calling is a feature of LLMs and LLM APIs that enables the model to access external information and perform actions. Instead of responding to a prompt directly, the model generates inputs to a function, calls that function (more on this later), and then generates a response based on the function's output. This significantly improves the functionality of LLMs by accessing real-time information like weather, sports scores, or recent events. Additionally, the model can interact with external systems like databases, APIs, or even execute scripts.

LLMs and tool-calling introduced a new type of program where the operation and flow is controlled by the LLM rather than rigid code. This creates highly flexible and dynamic programs, but they can also be less reliable and consistent. Like always, it's important to balance functionality and reliability.

How it works

LLMs are only able to generate text. In order to use tools, the model needs to be prompted through another program or controller. Instead of actually "calling" a function directly, the the model generates structured data to describe the function to use and which inputs to use with it. The controller program is responsible for parsing the input data, finding the function, and executing it. Then, it provides the output to the LLM (often as JSON). The LLM is finally able to respond to the user's prompt using the output of the function in addition to the data it was trained on.

Before using tools, the LLM needs to know about the tools that it can use. It needs the function's name, a description of what it does, and a schema defining the function's inputs. This can be done with system prompts, but many model APIs now include a separate input field for tool definitions.

Interacting with the model directly, with only a thin chat abstraction around it, effectively demonstrates how this works. Ollama allows creating a Modelfile to extend a model with a specific system prompt or other configurations. Here is a system prompt that instructs a model how to use "tools" (but really it will just be me responding with JSON):

FROM gemma3:4b

SYSTEM """
A program is used to interact with you and the user. The program allows you to use functions/tools
to get external information. When you use a tool, the program will respond with JSON data. Anything else is
coming from the user and should be treated as human interaction.

To use a tool, respond with a JSON object with the following structure:
{
    "tool": <name of the called tool>,
    "tool_input": <parameters for the tool matching the above JSON schema>
}
Do not include any other text in your response.

If the user's prompt requires a tool to get a valid answer, use the above format to use the tool.
After receiving a tool response, continue to answer the user's prompt using the tool's response.
If you don't have a relevant tool for the prompt, answer it normally. Be fun and interesting.

Here are your available tools:

getCurrentWeather
Get the current weather in a given location
{
    "type": "object",
    "properties": {
        "location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"},
        "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
    },
    "required": ["location", "unit"]
}
"""

Now that the model is ready, let's chat:

Input:

what is the weather like in new york city?

Model's response:

{
    "tool": "getCurrentWeather",
    "tool_input": {
        "location": "New York City",
        "unit": "fahrenheit"
    }
}

Input:
```
{"temperature": 78, "unit": "F"}
```
Model's response:

The temperature in New York City is currently 78°F! It’s a beautiful, warm day – perfect for a stroll through Central Park or grabbing a hot dog.

Instead of attempting to hallucinate a response, the model generates tool input using the specified format. Then, I respond with JSON emulating the tool output. With this new information, the model is able to accurately answer the prompt with a friendly flair.

Although this isn't real tool calling, it is interesting to see how impactful the system prompt is on the model's behavior. This is helpful to demystify the LLM's behavior and show what is really happening behind the scenes. This foundational understanding beneficial when developing more complex tools and building applications around LLMs.

Implement Tool-calling with Ollama + Go

The next step is to integrate tool calling into a real program instead of chatting with the model. In order to stay at a low-level and learn more about LLM interactions, the program uses Ollama's API and Go client instead of a more generalized library.

Here is the request used to prompt a model with a tool defined:

messages := []api.Message{
    api.Message{
        Role:    "user",
        Content: "What is the weather like in New York City?",
    },
}

req := &api.ChatRequest{
        Model: model,
        Messages: messages,
        Tools: api.Tools{
            api.Tool{
                Type: "function",
                Function: api.ToolFunction{
                    Name:        "getCurrentWeather",
                    Description: "Get the current weather in a given location",
                    Parameters: api.ToolFunctionParameters{
                        Type:     "object",
                        Required: []string{"location"},
                        Properties: map[string]api.ToolFunctionProperty{
                            "location": {
                                Type:        api.PropertyType{"string"},
                                Description: "The city and state, e.g. San Francisco, CA",
                            },
                        },
                    },
                },
            },
        },
    }

The client program sends the ChatRequest and handles the ToolCall in the model's response:

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "log"

    "github.com/ollama/ollama/api"
)

const model = "llama3.2:3b"

func main() {
    client, err := api.ClientFromEnvironment()
    if err != nil {
        log.Fatal(err)
    }

    messages := []api.Message{
        api.Message{
            Role:    "user",
            Content: "What is the weather like in New York City?",
        },
    }

    ctx := context.Background()
    req := &api.ChatRequest{
        // ... request from above
    }

    handler := func(resp api.ChatResponse) error {
        // If there is no tool call, just print the message content
        if len(resp.Message.ToolCalls) == 0 {
            fmt.Print(resp.Message.Content)
            return nil
        }

        // Otherwise, process the tool call
        tc := resp.Message.ToolCalls[0].Function
        switch tc.Name {
        case "getCurrentWeather":
            output, err := getCurrentWeather(tc.Arguments)
            if err != nil {
                log.Fatal(err)
            }

            messages = append(messages, api.Message{
                Role:    "tool",
                Content: output,
            })
        default:
            log.Fatal(fmt.Errorf("invalid function: %q", tc.Name))
        }

        return nil
    }

    // Send initial chat message
    err = client.Chat(ctx, req, handler)
    if err != nil {
        log.Fatal(err)
    }

    // The model should have responded with a tool call
    // and the handler would have appended a response to messages.
    // Now, call the tool again with the response
    req.Messages = messages
    err = client.Chat(ctx, req, handler)
    if err != nil {
        log.Fatal(err)
    }
}

func getCurrentWeather(input map[string]any) (string, error) {
    location, ok := input["location"].(string)
    if !ok {
        log.Fatalf("bad args: %v", input)
    }

    weatherInfo := map[string]any{
        "location":    location,
        "temperature": "80",
        "unit":        "fahrenheit",
        "forecast":    []string{"sunny", "windy"},
    }

    b, err := json.Marshal(weatherInfo)
    if err != nil {
        return "", err
    }
    return string(b), nil
}

The output should be something like:

The current weather in New York City is mostly sunny, with a windy condition. The temperature is at 80°F.

Going a step further

I happen to have a program that is perfect for demonstrating simple LLM integration using tools. The walks-of-italy was originally created to watch for new tour availability from Walks of Italy tour company. I started with a grander vision for this program: a tool that could watch, search, and observe tour availability. After implementing the base feature of notifying on new availability, I decided it was good enough for my use-case.

Now, with LLM tools, this program can be extended in any number of ways. Here are the tools it currently has:

getAllTours: list the tour names and their UUIDs from the database
getTourDetails: use the API to access the tour descriptions
getTourAvailability: use the API to check availability, prices, and capacity at certain times

With just a few tools, the LLM has these features:

Simply query a date range with natural language
- "what is the availability of the Pristine Sistine tour next week?"
Ask a more general question about multiple tours
- "what is the availability of vatican tours on July 5th?"
Get travel recommendations
- "I am going to be in Rome on August 3rd and 4th. Which tours can I go on?"
- "If I can only do one tour in Rome, which should I do?"

When asked "what is the availability of the Pristine Sistine tour next week?" the model responds:

Here's an overview of the availability for the tours:

June 20, 2025:

There are limited spots available for the 7:15 AM and 7:45 AM tours.

The 7:15 AM tour has 1 vacancy out of a capacity of 20.

The 7:45 AM tour also has 1 vacancy out of a capacity of 48.

The 6:45 AM tour is fully booked.

June 21, 2025:

The 6:45 AM tour has 10 available spots out of 10 total.

The 7:15 AM tour has 3 vacancies out of a capacity of 34.

The 7:45 AM tour has 1 vacancy out of a capacity of 28.

Other dates:

On June 19, 2025, the 7:45 AM tour is fully booked.

On June 18, 2025, the 7:15 AM tour is fully booked, and the 7:45 AM tour has limited availability.

On June 17, 2025, the 7:15 AM tour is fully booked, and the 7:45 AM tour has limited availability.

On June 16, 2025, the 7:45 AM tour is fully booked.

If you're looking for availability, the best options are on June 20 and June 21, with limited spots available for the morning tours. Let me know if you'd like help booking!

These are just a few examples that I can think of. A huge benefit to this type of program is that its functionality isn't only limited by the features added by a developer. Instead, it's limited by the LLM and the tools that it can use. The user might come up with a usecase that works out of the box due to the program's flexibility. In this example, I justed wanted to check specific tours and dates for availability, but can now do a lot more.

The effectiveness and accuracy of the responses here of course varies and is dependent on the model that is used. More complex questions that require more tool calls will generally decrease the accuracy of responses. When a lot of information is required, larger and more capable models are required.

In this example, the functionality could be made even more flexible. Currently, it just is able to use tours that I manually add. If it has a new function to get tours from the Walks of Italy website, it could be even more dynamic (but less predictable).

MCP

Model Context Protocol, or MCP, is the next evolution of tool calling. Instead of requiring a custom program to provide tools, it moves this to the server-side. The model is used with a generic client that supports the protocol and accesses functions provided by any number of external servers. If this previous example used MCP, Walks of Italy would provide an MCP server any model would be able to access it when used with an MCP client. It would eliminate almost all of the work done to implement my walks-of-italy program.

Thoughts

Until recently, I primarily interacted with LLMs using ChatGPT or Gemini to answer questions, discuss ideas, and sometimes generate SQL queries or code. As AI Agents become more popular and integrated into all parts of our lives and jobs, it starts to seem like LLMs can do anything. It was refreshing to start from the barebones model interactions and really see how tool-calling works. In reality, LLMs are still just reading input and generating output. It's the code around them, built by software engineers, that enables them to actually do things.

Building an application around an LLM and tools can be really beneficial. If the model has access to a set of tools with general functionality like querying APIs and DBs (with pre-defined SQL of course), the program's functionality can change and expand on-demand. This is especially useful for non-technical users. Instead of asking engineers to build new features for a specific usecase, they can just adjust their prompts and interactions with the LLM. In the previous example, the walks-of-italy program was originally designed to watch and alert for new tour availability or run specific date range queries with the CLI. The introduction of a few tools and an LLM drastically expanded the scope of the program and makes it more useful for non-technical users. Now that the LLM can query any date range and get descriptions about tours, you can:

Get help deciding which tours to go based on real pricing, descriptions, and availability
Ask about general availability next month for multiple tours
Compare availability and prices for weekdays or weekends

It is important to remember that the model can behave unpredictably and the tools should not provide unlimited access. For example, instead of providing a tool to execute any generated SQL, engineers should expose a few functions like "createResource" and "updateResource" with pre-defined queries.

System prompts and tool definitions provide more structure and control over how an LLM operates. An engineer can build a relatively reliable and useful program with these techniques. However, as complexity of the system grows, the potential for the LLM to do unpredictable things also grows.

Normally, code is very predictable. A set of functions exists and engineers define exactly when and how they are used. Now, programs become a collection of tools and the actual control is being relinquished to LLMs. Engineers always have to balance complexity, maintainability, and functionality. Using LLMs just makes this balance more delicate.

Middleware and RoundTrippers in Go

Calvin McLean — Sun, 01 Jun 2025 03:46:10 +0000

Check out the full code referenced in this article on Github!

Middleware and RoundTrippers are common web-development tools used in Go programs. Middleware operate on the server-side, while RoundTrippers are client-side. They are both reusable components that execute code before and/or after a request and response are processed.

In this article, I will provide some details about Middleware and RoundTrippers before providing interesting examples of each. Keep in mind that these are intended to be easily-understood examples and are not meant to be used in real applications as-is.

Then, tests will be used to demonstrate the correct and incorrect usage of these tools.

Middleware

A middleware is just an HTTP handler (defined as ServeHTTP(ResponseWriter, *Request) in Go's standard library) that calls another HTTP handler before and/or after doing some other actions. This is commonly used for logging and authentication. In the case of auth, it allows you to wrap every handler with the same auth middleware. If the middleware is applied to every handler in the application, the handlers can all operate assuming the request is already authenticated. A logging middleware allows you to observe every request's details as well as the response's status and duration. Go's standard library even provides a few middleware already: StripPrefix and TimeoutHandler.

Cache Middleware

This middleware caches responses on the server-side to prevent duplicate work. Implementing this simple middleware can result in a significant performance boost in applications with frequent GET requests. It can even reduce operational costs in cloud environments that charge for compute time, bandwidth, or traffic to databases. This can also be combined with an external cache, like Redis, to be functional across multiple instances.

func CacheMiddleware(next http.HandlerFunc) http.HandlerFunc {
    cache := map[string][]byte{}

    return func(w http.ResponseWriter, r *http.Request) {
        cacheKey := fmt.Sprintf("%s_%s", r.Method, r.URL.String())

        // use cached response of the key matches
        if cached, ok := cache[cacheKey]; ok {
            serverLogger.Info("SERVER using cached response")
            w.Header().Add("X-Server-Cached", "true")
            w.Write(cached)
            w.WriteHeader(http.StatusOK)
            return
        }

        var buf bytes.Buffer
        // responseWriter is an implementation of the http.ResponseWriter
        // interface that can intercept the response body
        writer := &responseWriter{ResponseWriter: w, buffer: &buf}

        // Call the main HTTP handler
        next(writer, r)

        // Cache the response
        cache[cacheKey] = buf.Bytes()
    }
}

Logger Middleware

This example simply logs request and response details.

func LoggerMiddleware(next http.HandlerFunc) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        serverLogger.Info(fmt.Sprintf("SERVER Request: %s %s", r.Method, r.URL.Path))

        var buf bytes.Buffer
        // responseWriter is an implementation of the http.ResponseWriter
        // interface that can intercept the response body
        writer := &responseWriter{ResponseWriter: w, buffer: &buf}

        // Call the main HTTP handler
        next(writer, r)

        serverLogger.Info(fmt.Sprintf("SERVER Response: %s %s %d %s", r.Method, r.URL.Path, writer.statusCode, time.Since(start)))
    }
}

Auth Middleware

This is an simplified example of a middleware that reads the Authorization header and authenticates requests. It also removes the header so the secret/password is not at risk of logging or caching.

func AuthMiddleware(next http.HandlerFunc) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        auth := r.Header.Get("Authorization")

        if auth != "password" {
            serverLogger.Warn("SERVER Authentication failed")
            w.WriteHeader(http.StatusForbidden)
            fmt.Fprintln(w, "Forbidden")
            return
        }

        // remove header to avoid logging
        r.Header.Del("Authorization")
        serverLogger.Info("SERVER Authentication successful")

        // Call the main HTTP handler
        next(w, r)
    }
}

Recovery Middleware

This middleware is essential in any webserver applications. Go runs a goroutine for each incoming request, but a panic will cause the whole application to crash. This middleware uses recover() to "catch" a panic, log it, and avoid crashing.

func RecoveryMiddleware(next http.HandlerFunc) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        defer func() {
            if err := recover(); err != nil {
                serverLogger.Error(fmt.Sprintf("SERVER panic: %v", err))
                http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
            }
        }()

        // Call the main HTTP handler
        next(w, r)
    }
}

Sequential Middleware

This final example is a middleware that simplifies the composition of multiple middleware. Instead of using nested function calls, all of the middleware can be provided as a variadic argument.

func SequentialMiddleware(middleware ...func(http.HandlerFunc) http.HandlerFunc) func(http.HandlerFunc) http.HandlerFunc {
    return func(final http.HandlerFunc) http.HandlerFunc {
        for _, mw := range slices.Backward(middleware) {
            final = mw(final)
        }
        return final
    }
}

Here is what it looks like to use it:

handler := SequentialMiddleware(
    RecoveryMiddleware,
    AuthMiddleware,
    LoggerMiddleware,
    CacheMiddleware,
)(helloHandler)

// easier to read than:
RecoveryMiddleware(AuthMiddleware(LoggerMiddleware(CacheMiddleware(helloHandler))))

In addition to being easier to read and format, this can be used to create a slice of middleware ([]func(http.HandlerFunc) http.HandlerFunc) based on configuration values.

RoundTrippers

RoundTrippers seem to be less common than middleware, but are very useful for client-side HTTP requests. It is called a RoundTripper because it handles the round-trip from the client to the server. This is shown by the interface's single method signature: RoundTrip(*Request) (*Response, error). It takes a request and provides the response.

RoundTrippers are used by setting the http.Client's Transport field. Unless you want to re-implement the code for establishing network connections between the client and the server, your RoundTripper implementations must, at some point, wrap Go's DefaultTransport. This, in my opinion, is a key distinction that makes RoundTrippers a bit more difficult to use than middleware. A middleware always uses your own code, but a custom RoundTripper needs to have the DefaultTransport somewhere in the chain. Therefore, almost any custom RoundTripper must wrap another RoundTripper.

RoundTripperFunc

The RoundTripperFunc is similar to Go's http.HandlerFunc and simplfies the creation of RoundTrippers. Instead of creating a struct and implementing RoundTrip, this type allows implementing just the function.

type RoundTripperFunc func(*http.Request) (*http.Response, error)

func (rt RoundTripperFunc) RoundTrip(r *http.Request) (*http.Response, error) {
    if rt == nil {
        return http.DefaultTransport.RoundTrip(r)
    }
    return rt(r)
}

Cache RoundTripper

This RoundTripper reduces network traffic and latency by using cached responses instead of making another round-trip to the server. Similar to the CacheMiddleware, this can provide a significant performance boost to applications with frequent external requests. When interfacing with vendor APIs that charge per-request, this can also signficantly reduce operational costs.

func CacheRoundTripper(next http.RoundTripper) http.RoundTripper {
    cache := map[string]cachedResponse{}

    return RoundTripperFunc(func(r *http.Request) (*http.Response, error) {
        cacheKey := fmt.Sprintf("%s_%s", r.Method, r.URL.String())

        cachedResponse, ok := cache[cacheKey]
        if ok {
            clientLogger.Info("CLIENT using cached response")
            return cachedResponse.Response(), nil
        }

        // send the actual request
        resp, err := next.RoundTrip(r)

        // cache the response
        cache[cacheKey] = newCachedResponse(resp)
        return resp, err
    })
}

The cachedResponse type is excluded here to keep the example short. This is just a new struct that copies the http.Response and its body. You can check out the full example code here.

Log RoundTripper

This is very similar to the log middleware and just logs the request and response details.

func LogRoundTripper(next http.RoundTripper) http.RoundTripper {
    return RoundTripperFunc(func(r *http.Request) (*http.Response, error) {
        clientLogger.Info(fmt.Sprintf("CLIENT Request: %s %s %v", r.Method, r.URL, r.Header))
        start := time.Now()

        // Send the actual request
        resp, err := next.RoundTrip(r)

        duration := time.Since(start)
        if err != nil {
            clientLogger.Error(fmt.Sprintf("CLIENT Request failed after %s: %v", duration, err))
            return nil, err
        }
        clientLogger.Info(fmt.Sprintf("CLIENT Response: %s %s %d in %s", r.Method, r.URL, resp.StatusCode, duration))
        return resp, nil
    })
}

Auth RoundTripper

This is a simplified example of using a RoundTripper for adding auth to a request. In this case, it just adds a password to the Authorization header. Even though this is a simple case, it is still useful because only one part of your application needs to be aware of the password/secret and the rest will reuse the RoundTripper.

func AuthRoundTripper(next http.RoundTripper) http.RoundTripper {
    return RoundTripperFunc(func(r *http.Request) (*http.Response, error) {
        r.Header.Add("Authorization", "password")
        return next.RoundTrip(r)
    })
}

In a real-world scenario, an auth RoundTripper can handle more complex tasks like using a refresh token to get an updated access token when it expires.

Go's documentation on RoundTripper says:

RoundTrip should not attempt to handle higher-level protocol details such as redirects, authentication, or cookies.
RoundTrip should not modify the request, except for consuming and closing the Request's Body

However, we'll ignore that here since it is overly-cautious and severely limits the RoundTripper's functionality. This is likely a constraint because the *http.Request is a pointer and modifying it may have unexpected consequences. In production-ready RoundTrippers, r.Clone(r.Context()) can be used to create a new *http.Request that is safe to modify.

Sequential RoundTripper

Similar to the SequentialMiddleware, this is a helper that simplifies the composition of multiple RoundTrippers. It is especially useful with the RoundTripperFunc. It also ensures that the last step in the RoundTripping is http.DefaultTransport.

func SequentialRoundTripper(rts ...func(http.RoundTripper) http.RoundTripper) http.RoundTripper {
    result := http.DefaultTransport

    for _, rt := range slices.Backward(rts) {
        result = RoundTripperFunc(rt(result).RoundTrip)
    }

    return result
}

Now, we can use a few RoundTrippers like this:

SequentialRoundTripper(
    LogRoundTripper,
    CacheRoundTripper,
    AuthRoundTripper,
)

// easier to read than:
LogRoundTripper(CacheRoundTripper(AuthRoundTripper(http.DefaultTransport)))

In addition to being easier to read and format, this can be used to create a slice of RoundTrippers based on configuration values.

Tests

Now we have a toolkit of middleware and RoundTrippers that can be used for an HTTP server and a client. In order to demonstrate this, we can write some Go tests.

Correct Setup

func TestGood(t *testing.T) {
    handler := SequentialMiddleware(
        RecoveryMiddleware,
        AuthMiddleware,
        LoggerMiddleware,
        CacheMiddleware,
    )(helloHandler)

    server := httptest.NewServer(handler)
    defer server.Close()

    t.Run("SuccessfulResponse", func(t *testing.T) {
        client := NewClientWithRoundTrippers(
            LogRoundTripper,
            CacheRoundTripper,
            AuthRoundTripper,
        )

        body, resp, err := doTestRequest(client, server.URL)
        if err != nil {
            t.Errorf("unexpected error: %v", err)
        }

        expectations{
            StatusCode: http.StatusOK,
            Body:       "Hello, World!\n",
        }.assert(t, body, resp)

        // request again to use client cache
        body, resp, err = doTestRequest(client, server.URL)
        if err != nil {
            t.Errorf("unexpected error: %v", err)
        }

        expectations{
            StatusCode:      http.StatusOK,
            Body:            "Hello, World!\n",
            ClientCacheUsed: true,
        }.assert(t, body, resp)
    })

    t.Run("DisableClientCacheToDemoServerCache", func(t *testing.T) {
        client := NewClientWithRoundTrippers(
            LogRoundTripper,
            AuthRoundTripper,
        )

        body, resp, err := doTestRequest(client, server.URL)
        if err != nil {
            t.Errorf("unexpected error: %v", err)
        }

        expectations{
            StatusCode:      http.StatusOK,
            Body:            "Hello, World!\n",
            ServerCacheUsed: true,
        }.assert(t, body, resp)
    })

    t.Run("ForbiddenWithoutAuthRoundTripper", func(t *testing.T) {
        client := NewClientWithRoundTrippers(
            LogRoundTripper,
        )

        body, resp, err := doTestRequest(client, server.URL)
        if err != nil {
            t.Errorf("unexpected error: %v", err)
        }

        expectations{
            StatusCode: http.StatusForbidden,
            Body:       "Forbidden\n",
        }.assert(t, body, resp)
    })
}

This test starts by setting up the HTTP server with the middleware in the correct order:

RecoveryMiddleware: this is the first code to run. It defers recover() to catch any panic that happens down the line
LoggerMiddleware: logs the request and response details. Running this second ensures that every request and response is logged, even ones that fail auth
AuthMiddleware: before doing anything else, the request should be authenticated
CacheMiddleware: finally, cache the response. Any requests that use a cached response will still be authenticated and logged

Next, a few different subtests are used to show different client scenarios. These all use the same general ordering, but exclude the cache or auth RoundTrippers to demonstrate different outcomes:

LogRoundTripper: always logs request/response
CacheRoundTripper: cache the response. This is removed to demonstrate that the server-side cache also works
AuthRoundTripper: adds the Authorization header to the request. It is removed to show that the request fails without it

The output of this test shows that the client and server both log the request and response.

Incorrect Setup

func TestBad_ServerCacheExposesAccess(t *testing.T) {
    // The server will cache responses before checking authentication. After caching a response, the next request
    // will receive this response even if it does not authenticate successfully
    handler := SequentialMiddleware(
        CacheMiddleware,
        LoggerMiddleware,
        AuthMiddleware,
        RecoveryMiddleware,
    )(helloHandler)

    server := httptest.NewServer(handler)
    defer server.Close()

    t.Run("PopulateCacheWithAuth", func(t *testing.T) {
        client := NewClientWithRoundTrippers(
            LogRoundTripper,
            AuthRoundTripper,
        )

        body, resp, err := doTestRequest(client, server.URL)
        if err != nil {
            t.Errorf("unexpected error: %v", err)
        }
        expectations{
            StatusCode: http.StatusOK,
            Body:       "Hello, World!\n",
        }.assert(t, body, resp)
    })

    // Server cache is used before AuthMiddleware, allowing unrestricted access
    t.Run("RequestWithoutAuth", func(t *testing.T) {
        client := NewClientWithRoundTrippers(
            LogRoundTripper,
        )

        body, resp, err := doTestRequest(client, server.URL)
        if err != nil {
            t.Errorf("unexpected error: %v", err)
        }
        expectations{
            StatusCode:      http.StatusOK,
            Body:            "Hello, World!\n",
            ServerCacheUsed: true,
        }.assert(t, body, resp)
    })
}

In this test, the middleware are initialized in reverse order. This has a few issues:

While the RecoveryMiddleware will still handle panics in the main handler, it will not handle panics in any of the other middleware
Since the CacheMiddleware is the first to run, it will immediately respond with a cached response before checking auth. This means that unauthorized request will be able to access the server
Additionally, since the LoggerMiddleware runs after the cache, we won't even know that these unauthorized requests are occurring

This is a trivial example, but we can see how a simple mistake like incorrect middleware order can have catatrophic results. Similar, but less-impactful errors can occur with the RoundTrippers. If we use the AuthRoundTripper before the CacheRoundTripper, our authentication credentials will be saved in the cache, potentially leaking sensitive details. Also, we will lose a benefit of the cache if the AuthRoundTripper is refreshing access tokens.

Conclusion

Middleware and RoundTrippers are simple and common tools in Go. While the topic is familiar for many seasoned Gophers, it's beneficial to revisit the basics and remember to carefully consider how they are used. These are just a few of the usecases and possible pitfalls of using middleware and RoundTrippers. Here are a few more possible ways to leverage them:

Automatic retries for 500-level errors (RoundTripper). Make sure to use this after the CacheRoundTripper so you can use the cache before encountering and retrying any errors
OpenTelemetry tracing and metrics (RoundTripper and middleware)
Throttling and rate-limiting. Use with a middleware to protect your server from abuse, or implement in a RoundTripper to avoid overloading another server. Similar to the retries, this RoundTripper should run after the CacheRoundTripper since the cache will already reduce the number of requests
Client-side load balancing by choosing from a pool of upstream URLs
Testing. The go-vcr library leverages a RoundTripper to record interactions with external servers and then replay them using httptest.Server. Make sure to run this RoundTripper before the AuthRoundTripper so your Authorization header isn't saved in a test fixture! Different tokens would also cause requests to not match

Let me know some of your favorite or novel usecases for middleware and RoundTrippers in Go!

Check out the full code referenced in this article on Github!

Improve Your Go Applications with Byte-level Data Optimization

Calvin McLean — Fri, 09 May 2025 05:59:44 +0000

Check out the full code referenced in this article on Github!

Almost any application requires a way to store or transfer data. We often choose formats like JSON or YAML that are easy to read and modify. These formats are built to be generic to work in a variety of usecases, which means they are not heavily optimized for data size or fast encoding/decoding.

I recently started making a sourdough starter. This requires dividing, feeding, and discarding every 12 or 24 hours. To ensure accuracy and consistency, I record the time and amounts for each feeding. This data is just numbers and time (which is also a number). Since I already had data optimizations on my mind, I saw a real-world usecase for implementing binary data optimizations. My goal is to create a hyper-specific and optimized time-series data. Rather than defaulting to JSON for storing data in a file, I will store it in a binary format.

Intro to bytes

Let's start with some basics about binary data in Go:

The most basic types here are uint8 and byte, which are the same
These contain 8 bits, which is 1 byte

You might already be familiar with byte and []byte from encoding/decoding JSON or reading/writing files. This data type is use by Go for any filesystem interactions or encoding/decoding data. With 8 bits, we have 2⁸ = 256 distinct values. Since we start at zero, we use the range 0-255. Each byte represents a number in this range and can be converted to characters from the extended ASCII table. The lower-case latin alphabet starts at 97 for 'a' and ends at 122 for 'z'. Therefore, []byte{104, 101, 108, 108, 111} is "hello" and 'b' - 'a' == 1 (see it for yourself!). Instead of using the ASCII table to convert bytes into human-readable strings, we can assign our own meaning to the data.

The following JSON data holds two numbers and is 20 characters long, so it is 20 bytes:

{"a": 114, "b": 125}

A lot of this data is included just to make the data generically parseable by both humans and computers:

": marks the start and end of the string data type
:: separates keys from values
,: separates one key-value pair from another
{/}: shows the start and end of objects
(space): just for human-readability

A lot of this data is not strictly necessary if we are able to make specific assumptions about the data. After removing the brackets, quotes, and colons, we are left with:

a255b127

Now this is only 8 bytes. However, it is no longer a flexible format. In order to be parseable, we have to introduce a few constraints: keys are always alphabetic (a-z, A-Z) and values are numeric 0-9. These constraints allow distinguishing the keys from values when parsing the data instead of relying on commas, colons, and quotation marks. Even this simple reduction introduces significant restrictions on what the data can represent, which is why JSON is so verbose.

These numbers are each 3 characters since they are meant to be human-readable, so they use up 3 bytes of data. You might have noticed that these are within the 0-255 range, so they can be represented by a single byte each. As a human-readable character from the ASCII table, 114 is r and 125 is }. If we are able to assume that 'a' and 'b' are in order, then we can represent this data with only 2 bytes:

r}

(check it out here!)

This data is no longer a generic key-value format and instead represents a specific type of data that just has two uint8 values. It also requires custom programming to encode and decode. In many usecases, these strict rules might not be worth the benefit. However, the tenfold reduction from 20 bytes to just 2 can result in significant storage and bandwidth savings.

Now that we are familiar with byte and uint8, let's introduce a few more types:

uint16: this is two bytes and holds up to 2¹⁶-1 (65,535)
uint32: 4 bytes, 2³²-1 (4,294,967,295)
uint64: 8 bytes, 2⁶⁴-1 (18,446,744,073,709,551,615)

We want to use the smallest type possible for each piece of data without compromising too much on the constraints. With such big jumps in size, there can be a lot of seemingly-wasted space. If we need a range of 0-300, we have to upsize from uint8 to uint16 which goes all the way to 65,535! It just uses one additional byte, so it's not a big deal. However, what if we could just use one additional bit to reach 300 instead of a whole byte? This, of course, is entirely possible. It requires more complex parsing using bit shifting and masking, but we'll get more into that later.

Requirements

In the previous section, we learned that we can significantly reduce the size of data by ditching JSON in favor of a more specific binary format. However, this requires a lot more knowledge about your data and the ability to follow strict constraints. It will increase the time it takes to program the encoding/decoding whenever new data fields are added. With JSON, you don't care if a number is int, int64, uint16, or any other value. In our case, these differences are very important and decisions between them must be data-driven. Let's get back to the sourdough use case and define some requirements:

Sourdough starter measurement: this is the amount of existing starter that we are using. The process is started with just flour and water, so this measurement starts at 0. Once the process gets started, I'll be using 25-50 grams of starter.
Flour measurement: we will always have a non-zero amount of flour, generally 50-100 grams.
Water measurement: we will also always have non-zero amount of water. I aim to use the same amount of flour and water, but am sometimes off by a few grams so I can't always assume it's the same.
Flour type: we can feed the starter with different types of flour: whole wheat, all purpose, bread, rye, or others
Datetime: we want to know the time of feeding. Since feeding is a process that takes some time, we aren't going to pinpoint the time down to the second. Truncating to the minute is plenty. We could even do 5 minute granularity if that buys us any additional optimization. I don't need to backfill historical data, so dates after 2025 are fine.

Now that we have some base requirements, we can start designing and optimizing our data format.

Measurements

Since our measurements all have similar constraints and numeric values are easy, let's start there.

I follow a low-waste recipe that uses small amounts of each ingredient, so we can easily use uint8 which maxes out at 255 grams. If we need to allow for a larger-scale bakery to use this data format, we can measure in decagrams or kilograms which gives us up to 2,550 or 255,000 grams but loses some granularity. A uint16 would give us a much larger range at the cost of 1 additional byte, so this is an easy change to make if the requirements for measurements were different. In this case, let's stick with uint8.

Can we simplify it further? Ratios are big in the world of sourdough baking, so maybe we can use them to reduce data sizes. Unfortunately, since we start with 0 starter and non-zero water/flour, a ratio won't work here. Water and flour are usually in a 1:1 or 1:2 ratio which could easily be stored with a single bit. However, this doesn't account for inaccuracy in measurements. I often add a little too much water or a little too much flour so the values are slightly different and I want to record that discrepancy. Additionally, I might want to experiment with different ratios like 1:1.5 or others, so this constraint doesn't fit.

Since we are already using 1 byte with the uint8, any additional reductions introduce a lot of complexity. Taking off a single bit gives us up to 127g, which is enough for the scale I am operating at, but 1 bit isn't worth the effort and results in the same on-disk space in the end since filesystems operate at the byte-level. Removing another bit and limiting our measurements to 63g still works for my very small scale, but sometimes I might want to make a larger batch of bread that doesn't fit in this constraint. Therefore, I'll stick with the uint8.

In code, this looks like:

buf := make([]byte, dataSize)

// encode to bytes
buf[0] = data.StarterGrams
buf[1]] = data.FlourGrams
buf[2] = data.WaterGrams

// decode bytes
data.StarterGrams = in[0]
data.FlourGrams = in[1]
data.WaterGrams = in[2]

Flour type

Starters can be fed with different types of flour which will impact how it grows. I am only aware of four relevant types: all purpose, whole wheat, bread, rye, but I am sure there are more. In Go, enums are often created using iota constants, which is an untyped integer. Each flour type can be assigned to a different integer value, so the type used will determine the range of values.

uint8 allows up to 255 which goes way beyond the five types. 2³ has the range 0-7 which fits our 5 types, but is not very extensible if I learn about and start using new types. However, Go does not allow having variables or data smaller than a byte and neither do filesystems. We would have to pad out the remaining 5 bits with zeros and lose any benefit from the optimization. Therefore, we'll stick with the uint8, but keep 2³ in mind.

Once again, the code is very simple:

// encode
buf[3] = byte(data.FlourType)

// decode
data.FlourType = FlourType(in[3])

Time

Go represents time with a struct that can easily be converted to seconds since an epoch (January 1, 1970 UTC). This is an int64 which is already compact at 8 bytes, but we can do better. 1,745,010,295 represents a timestamp in April 2025 which is less than the max uint32 (4,294,967,295). Since we don't care about seconds, we can convert this to minutes by dividing by 60: 29,083,504. This is already way smaller, but doesn't fit in our next size down (uint16). Since we don't need historical data, we can count the minutes since 2025 as the epoch instead of 1970. That 55 year difference converts to 28,908,000 minutes. While this is a large number, it is less than 1% of uint32's max. This optimization would just extend our max date by 55 years which is not important here.

Since this data is sequential in a time-series, we can optimize by using time offsets. The initial time is recorded using uint32 (4 bytes) and then each new data point stores an amount of time passed since the previous. The options for recording the offsets are:

uint8: maxes at 255 minutes, or 10.625 hours which is not enough for a 12-24 hour feeding interval
uint16: significantly larger range of 0-65,535 minutes, or about 45 days

uint16 is a good choice since the starter can go for a week or more without feeding if it's refrigerated. There is a low risk of overflowing the 45 days, but it's not an excessively large limit.

The flaw with this implementation is that we always have to read our data from the beginning if we want to see the latest date. Also, the first entry is a different format which can be challenging to deal with. There is no way to randomly-access data with this optimization. This would significantly reduce the data size from 4 bytes to only 2, so it may be worth it, especially since parsing the data should be fast. This flaw could also be mitigated by saving data ranges in chunks, where a full timestamp is written after every N data points. This would be interesting to implement, but I am focusing on parsing individual data points for now.

There's one more option that I am considering. Take a look at this simple datetime: 2025-04-16 12:00. Breaking it down, we have:

Year (2025+): This is the largest number, but we do not plan to insert historical data, so we can assume it starts at 2025 and only record an offset from there. Using a uint8 gives us 255 years before it overflows
Month (1-12): Our smallest data type, uint8 goes up to 255 which is way overkill. Half of that, 2⁴, allows 0-15, so we can use 4 bits instead
Day (1-31): Once again, uint8 is overkill for this. 2⁵ allows 0-31 so it's a perfect fit
Hour (1-24): This is another case for 2⁵
Minute (0-60): This one needs 2⁶, which is 0-63

Overall, we have 8 + 4 + 5 + 5 + 6 = 28 bits. This is another partial-byte idea that requires padding out to 32 bits/4 bytes anyways. Instead of padding with empty bits, the 3-bit option for flour type could be stored in those extra bits. This requires more complex bitwise operations to encode and decode, so I'll put it on the backburner for now.

Ultimately, I'll move forward with the simplest option first: uint32 for a minutes epoch instead of the usual seconds epoch. This approach has a significant edge for its simplicity and already halves the data size compared to int64. Keep the other ideas in mind for future optimizations.

Since we have multiple bytes now, the code is a bit different:

// encode
// convert to minutes since epoch, then write across 4 bytes
minutes := sd.Time.Unix()/60
binary.LittleEndian.PutUint32(buf[4:], uint32(minutes))

// decode
// read 4 bytes into a uint32 and convert back to epoch time
minutes := binary.LittleEndian.Uint32(in[4:])
sd.Time = time.Unix(int64(minutes)*60, 0).UTC()

Putting it all together

Time: 4 bytes (32 bits)
Measurements: 1 byte each, for a total of 3 bytes (24 bits)
FlourType: 1 byte (8 bits)
Total: 8 bytes (64 bits)

This conveniently fits in a single int64! However, [8]byte is more practical and efficient to work with. If I feed my sourdough starter twice daily for 100 years, that's just 570.3 KiB!

Two data points per day is 2 * 365 = 730 records each year
Each record is 8 bytes total, so 8 * 730 = 5,840 bytes per year
After 100 years, its 584,000 bytes
584,000 / 1,024 ~= 570.3 KiB

I averaged out the size of 100,000 JSON data points to be about 97.71074 bytes each. The JSON representation of this data would be 730 * 100 * 97.71074 = 7,132,884.02 bytes ~= 6.8 MiB! While this data is still pretty small, it significantly larger than the binary representation of data.

Here is the final code responsible for these optimizations:

const dataSize = 8

type Data struct {
    Time         time.Time
    StarterGrams uint8
    FlourGrams   uint8
    WaterGrams   uint8
    FlourType    FlourType
}

func (sd *Data) MarshalBinary() ([]byte, error) {
    buf := make([]byte, dataSize)

    // The uint8 fields can be directly insert to the byte slice
    buf[0] = sd.StarterGrams
    buf[1] = sd.FlourGrams
    buf[2] = sd.WaterGrams
    buf[3] = byte(sd.FlourType)

    // Convert Unix time to minutes
    minutes := sd.Time.Unix()/60

    // The encoding/binary library can be used to insert the uint32
    // time representation into the byte slice
    binary.LittleEndian.PutUint32(buf[4:], uint32(minutes))

    return buf, nil
}

func (s *Data) UnmarshalBinary(in []byte) error {
    // Simply read uint8 fields from the byte slice in the correct order
    sd.StarterGrams = in[0]
    sd.FlourGrams = in[1]
    sd.WaterGrams = in[2]
    sd.FlourType = FlourType(in[3])

    // Get uint32 minutes from the byte slice and convert back into time.Time
    minutes := binary.LittleEndian.Uint32(in[4:])
    sd.Time = time.Unix(int64(minutes)*60, 0).UTC()

    return nil
}

Similar to marshal and unmarshal in encoding/json, implementing MarshalBinary and UnmarshalBinary enables encoding and decoding with the encoding/gob package. This is useful when a struct is part of another data structure, such as a slice. With these implemented, we can encode and decode binary data with the same convenience as JSON.

Go's built-in benchmarking provides a way to test the performance compared to JSON. In order to make the tests more fair, the init function is used to generate a slice of data:

var data []sourdough.Data

func init() {
    size := 10_000
    data = make([]sourdough.Data, size)
    for i := 0; i < size; i++ {
        data[i].Time = time.Date(
            2025+rand.Intn(50),          // Year: 2025 + up to 49 years
            time.Month(rand.Intn(12)+1), // Month
            rand.Intn(28)+1,             // Day
            rand.Intn(24)+1,             // Hour
            rand.Intn(60)+1,             // Minute
            0, 0, time.UTC,
        )
        data[i].StarterGrams = uint8(rand.Intn(256))
        data[i].FlourGrams = uint8(rand.Intn(256))
        data[i].WaterGrams = uint8(rand.Intn(256))
        data[i].FlourType = sourdough.FlourType(rand.Intn(int(sourdough.FlourTypeBread + 1)))
    }
}

The benchmarks for each implementation encode the data to bytes, then decode it back into a new slice of data. Finally, it is compared to the initial data to make sure it encoded and decoded successfully. Here are the results of the benchmark:

> go test -bench . -benchmem
goos: darwin
goarch: arm64
pkg: sourdough
cpu: Apple M3 Pro
BenchmarkJSON-11                            50          23172882 ns/op   5747196 B/op      30054 allocs/op
BenchmarkBinaryUnixMinute-11        1000000000           0.01411 ns/op         0 B/op          0 allocs/op
BenchmarkBinaryUnix-11              1000000000           0.01398 ns/op         0 B/op          0 allocs/op
BenchmarkBinaryCompact-11           1000000000           0.01439 ns/op         0 B/op          0 allocs/op
PASS
ok      sourdough   2.433s

The first number in each benchmark row is the total number of times that the test was executed. By default, Go will limit the benchmark time to 1 second and run as many iterations of each test as it takes to fill this time. It looks like each of the binary tests max out at 1,000,000,000 iterations while the JSON test only runs a miniscule 50 times. The next value shows how many nanoseconds each iteration of the benchmark took to execute, which shows a clear advantage to the binary implementations. Finally, we can see the memory usage and allocations for the tests which is another big win for binary.

There are slight variances each time the benchmark is executed, so I encourage you to try it for yourself.

In this output, there are a few additional tests different binary implementations:

BenchmarkBinaryUnixMinute uses uint32 for minutes since the epoch
BenchmarkBinaryUnix uses uint64 for seconds since the epoch (time.Unix())
BenchmarkBinaryCompact uses a more compact encoding that we'll explore in the next section

The benchmark results for the different binary implementations are too close to call, so comparing these will come down to storage size rather than performance.

Overall, each of the binary implementations is way faster than JSON, uses less memory, and much less storage space. In the short test execution time, the binary benchmarks had 20,000,000 times more iterations than the JSON benchmark. To be clear, that's 20,000,000 times the number of iterations, not 20,000,000 more iterations. Then, each JSON operation takes 23,172,882 nanoseconds, or about 23ms. The binary operations are taking less than a nanosecond, which is incomprehensible and 1,639,977,495 times faster than the JSON. We can scale things up a bit and run 1000 executions of encoding for each implementation. This takes over 9 seconds for JSON and about 477 milliseconds for the binary data. 9 seconds is definitely a loading time that you notice, while 477ms is more tolerable.

Optimizing it further

Benchmarking already revealed that this doesn't have an impact on speed, but let's look back at the partial-byte optimizations for time (28 bits) and flour type (3 bits). Speed isn't everything, so we'll focus on how this can reduce the data size. Since there are 31 bits, we can extend the flour type to use 4 bits and have an even 32 (4 bytes). Now the uint32 size that previously held the time can hold up to 16 different flour types!

This reduces the overall size of data to 7 bytes instead of 8. This small difference is more significant than it initially seems. The 12.5% reduction in size results in 71.3 KiB savings over 100 years. More importantly, it's an interesting technique to learn about and implement.

Sharing a byte

The tricky part now is implementing the shared bytes.

4 bytes look like this:

0x00 0x00 0x00 0x00

Zoomed in a bit more:

00000000 00000000 00000000 00000000

Now we need to assign some meaning to these bits.

 00000000 0000 0000 0 00000 00 0000 0000
|   A    | B  | C    |  D  |   E   | F  |

A - Year offset (8 bits, 0-255): this is the entire first byte
B - Month (4 bits, 0-15): this is half of the second byte
C - Day (5 bits, 0-31): the other half of the second byte + one bit from the third byte
D - Hour (5 bits, 0-31): the next five bits of the third byte
E - Minute (6 bits, 0-63): two remaining bits of the third byte and first half of the last byte
F - Flour Type (4 bits, 0-15): last four bits of the fourth byte

This is how we can access data within the same byte, or even as parts of multiple bytes. When you have a normal 8-bit uint8, it may look something like this when converted to binary representation: 11110000. This is the base ten number 240. This byte can be split into two parts (nibbles): 1111 and 0000, or 15 and 0 respectively. How can we extract these values? In Go, the smallest integer is uint8, so these would still have to be represented by 8 bits: 00001111 and 00000000. This is where bit shifting and masking come in:

11110000 can be changed to 00001111 by shifting the 1's to the right by 4 positions:
```
var x uint8 = 0b11110000
fmt.Println(x >> 4)
```
11110000 can be changed to 00000000 by replacing the 1's with 0's by masking
```
var x uint8 = 0b11110000
fmt.Println(x & 0b00001111)
```
- This works by performing a bitwise "and" operation on the data. We put a 1 in the positions that we want to keep since 1 & 1 == 1 and 1 & 0 == 0. Anywhere on the original data with a 0 will remain 0, and anywhere with a 1 will remain 1
If we want the middle 4 bits (1100, or 12), we need to combine shifting and masking:
```
var x uint8 = 0b11110000
fmt.Println((x & 0b00_1111_00) >> 2)
```
- This works by "selecting" the middle 4 bits with the mask, and then shifts them over to be the right-most bits

Check out this simple example. This isn't intended to be an in-depth tutorial of bitwise operations. If this is a new concept for you, I encourage you to continue researching and learning about it.

Now we need to apply these techniques to extract six different pieces of data from four bytes.

Here's how we can encode the datetime and flour type into four bytes:

func encodeCompactDateAndFlourType(sd Data) []byte {
    // Year is simply one byte/uint8
    year := uint8(sd.Time.Year() - 2025)

    // Month is the left four bits of the 2nd byte
    month := uint8(sd.Time.Month() << 4)

    // Day is 5 bits split across the 2nd and 3rd bytes
    day := sd.Time.Day()
    dayPart1 := (uint8(day) & 0b000_1111_0) >> 1
    dayPart2 := (uint8(day) & 0b0000000_1) << 7

    // Hour is 5 bits in the middle of the 3rd byte
    hour := uint8(sd.Time.Hour()) << 2

    // Minute is 6 bits split across the 3rd and 4th bytes
    minute := sd.Time.Minute()
    minutePart1 := (uint8(minute) & 0b00_11_0000) >> 4
    minutePart2 := (uint8(minute) & 0b0000_1111) << 4

    // FlourType is just the final 4 bits
    flourType := uint8(sd.FlourType) & 0b0000_1111

    return []byte{
        year,
        month | dayPart1,
        dayPart2 | hour | minutePart1,
        minutePart2 | flourType,
    }
}

It can then be decoded from bytes with this function, which simply reverses the previous bitwise operations:

func decodeCompactDateAndFlourType(data []byte, sd *Data) {
    year := int(data[0]) + 2025

    month := data[1] >> 4

    dayPart1 := (data[1] << 1) & 0b000_1111_0
    dayPart2 := (data[2] >> 7) & 0b0000000_1
    day := dayPart1 | dayPart2

    hour := (data[2] >> 2) & 0b000_11111

    minutePart1 := (data[2] << 4) & 0b00_11_0000
    minutePart2 := (data[3] >> 4) & 0b0000_1111
    minute := minutePart1 | minutePart2

    sd.Time = time.Date(year, time.Month(month), int(day), int(hour), int(minute), 0, 0, time.UTC)
    sd.FlourType = FlourType(data[3] & 0b0000_1111)
}

While this code isn't the most complex, it's generally unapproachable for a lot of developers. Even when revisiting this to write about it, I had to spend some extra time figuring out how it works. All of this saves just one byte of data. At a large scale, this can save a decent amount of storage space, but comes at a cost of maintainability.

The previous solution that operated at the byte-level already provided a significant boost in performance and reduced storage space with a minimal impact to maintainability. It's usually not worth the effort to take this next step to the bit level. If there's a specific use case to store many 4-bit values, that is a case where bit-level operations would be simple and halves the data size compared to uint8.

Conclusion

In this exploration of data optimization, we learned about the different data types and how they can be manipulated to represent more than just basic numbers. Well-defined constraints are the key to optimizing data. If we know exactly what our data can look like and how it needs to be used, we can make informed decisions about how to represent that data.

The tradeoff is that this format is very rigid and challenging to modify. Also, it cannot be parsed outside of our program, so debugging and analyzing data becomes more difficult.

Now what if you wanted to balance flexibility, extensibility, and performance? This is the reason Protobuf exists. It uses a schema/definition for structured data and generates the code for encoding and decoding it. This allows the defined types to be compatible with different progamming languages and spares engineers from low-level programming. I created a proto definition for this data and added another benchmark. There are a few small tradeoffs here. First, the smallest integer is uint32, so the protobuf data isn't quite as compact as the custom solution. Additionally, it is slightly slower in the benchmark, but when we are looking at thousandths of a nanosecond, the difference isn't significant.

If you have the ability to use rigidly-structured data, I recommend using Protobuf before rolling your own custom binary solution. However, it's still good to learn about how it works from scratch, and you can still squeeze out more optimizations if it's really necessary. Low-level concepts like binary data and bytes can be challenging to work with, but understanding them can increase your understanding of how programs work and how memory impacts performance. Knowing how things work will give you more tools and options for improving performance and efficiency of your applications.

Check out the full code and tests on Github!

Effortless HTTP Client Testing in Go

Calvin McLean — Thu, 11 Jul 2024 23:16:03 +0000

Introduction

As a software engineer, you are probably familiar with writing code to interact with external HTTP services. After all, it is one of the most common things we do! Whether it's fetching data, processing payments with a provider, or automating social media posts, our applications almost always involve external HTTP requests. In order for our software to be reliable and maintainable, we need a way to test the code responsible for executing these requests and handling the errors that could occur. This leaves us with a few options:

Implement a client wrapper that can be mocked by the main application code, which still leaves a gap in testing
Test response parsing and handling separate from actual request execution. While it's probably a good idea to test this lower-level unit individually, it'd be nice if that could easily be covered along with the actual requests
Move tests to integration testing which can slow down development and is unable to test some error scenarios and may be impacted by the reliability of other services

These options aren't terrible, especially if they can all be used together, but we have a better option: VCR testing.

VCR testing, named after the videocassette recorder, is a type of mock testing that generates test fixtures from actual requests. The fixtures record the request and response to automatically reuse in future tests. Although you might have to modify the fixtures afterwards to handle dynamic time-based inputs or remove credentials, it is much simpler than creating mocks from scratch. There are a few additional benefits to VCR testing:

Execute your code all the way down to the HTTP level, so you can test your application end-to-end
You can take real-world responses and modify the generated fixtures to increase response time, cause rate limiting, etc. to test error scenarios that don't often occur organically
If your code uses an external package/library for interacting with an API, you might not know exactly what a request and response look like, so VCR testing can automatically figure that out
Generated fixtures can also be used for debugging tests and making sure your code executes the expected request

Deeper Dive using Go

Now that you see the motivation behind VCR testing, let's dig deeper into how to implement it in Go using dnaeon/go-vcr.

This library integrates seamlessly into any HTTP client code. If your client library code doesn't already allow setting the *http.Client or the Client's http.Transport, you should add that now.

For those that aren't familiar, an http.Transport is an implementation of http.RoundTripper, which is basically a client-side middleware that can access the request/response. It is useful for implementing automatic retries on 500-level or 429 (rate-limit) responses, or adding metrics and logging around requests. In this case, it allows go-vcr to re-reoute requests to its own in-process HTTP server.

URL Shortener Example

Let's get started on a simple example. We want to create a package that makes requests to the free https://cleanuri.com API. This package will provide one function: Shorten(string) (string, error)

Since this is a free API, maybe we can just test it by making requests directly to the server? This might work, but can result in a few problems:

The server has a rate limit of 2 requests/second which could be an issue if we have a lot of tests
If the server goes down or takes awhile to respond, our tests could fail
Although the shortened URLs are cached, we have no guarantee that we will get the same output every time
It's just rude to send unnecessary traffic to a free API!

Ok, what if we create an interface and mock it? Our package is incredibly simple, so this would overcomplicate it. Since the lowest-level thing we use is *http.Client, we would have to define a new interface around it and implement a mock.

Another option is to override the target URL to use a local port served by httptest.Server. This is basically a simplified version of what go-vcr does and would be sufficient in our simple case, but won't be maintainable in more complex scenarios. Even in this example, you'll see how managing generated fixtures is easier than managing different mock server implementations.

Since our interface is already defined and we know some valid input/output from trying the UI at https://cleanuri.com, this is a great opportunity to practice test-driven development. We'll start by implementing a simple test for our Shorten function:

package shortener_test

func TestShorten(t *testing.T) {
    shortened, err := shortener.Shorten("https://dev.to/calvinmclean")
    if err != nil {
        t.Errorf("unexpected error: %v", err)
    }

    if shortened != "https://cleanuri.com/7nPmQk" {
        t.Errorf("unexpected result: %v", shortened)
    }
}

Pretty easy! We know that the test will fail to compile because shortener.Shorten is not defined, but we run it anyways so fixing it will be more satisfying.

Finally, let's go ahead and implement this function:

package shortener

var DefaultClient = http.DefaultClient

const address = "https://cleanuri.com/api/v1/shorten"

// Shorten will returned the shortened URL
func Shorten(targetURL string) (string, error) {
    resp, err := DefaultClient.PostForm(
        address,
        url.Values{"url": []string{targetURL}},
    )
    if err != nil {
        return "", err
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK {
        return "", fmt.Errorf("unexpected response code: %d", resp.StatusCode)
    }

    var respData struct {
        ResultURL string `json:"result_url"`
    }
    err = json.NewDecoder(resp.Body).Decode(&respData)
    if err != nil {
        return "", err
    }

    return respData.ResultURL, nil
}

Now our test passes! It's just as satisfying as I promised.

In order to start using VCR, we need to initialize the Recorder and override shortener.DefaultClient at the beginning of the test:

func TestShorten(t *testing.T) {
    r, err := recorder.New("fixtures/dev.to")
    if err != nil {
        t.Fatal(err)
    }
    defer func() {
        require.NoError(t, r.Stop())
    }()

    if r.Mode() != recorder.ModeRecordOnce {
        t.Fatal("Recorder should be in ModeRecordOnce")
    }

    shortener.DefaultClient = r.GetDefaultClient()

    // ...

Run the test to generate fixtures/dev.to.yaml with details about the test's request and response. When we re-run the test, it uses the recorded response instead of reaching out to the server. Don't just take my word for it; turn off your computer's WiFi and re-run the tests!

You might also notice that the time it takes to run the test is relatively consistent since go-vcr records and replays the response duration. You can manually modify this field in the YAML to speed up the tests.

Mocking Errors

To further demonstrate the benefits of this kind of testing, let's add another feature: retry after 429 response due to rate-limiting. Since we know the API's rate limit is per second, Shorten can automatically wait a second and retry if it receives a 429 response code.

I tried to reproduce this error using the API directly, but it seems like it responds with existing URLs from a cache before considering the rate limit. Rather than polluting the cache with bogus URLs, we can create our own mocks this time.

This is a simple process since we already have generated fixtures. After copy/pasting fixtures/dev.to.yaml to a new file, duplicate the successful request/response interaction and change first response's code from 200 to 429. This fixture mimics a successful retry after rate-limiting failure.

The only difference between this test and the original test is the new fixture filename. The expected output is the same since Shorten should handle the error. This means we can throw the test in a loop to make it more dynamic:

func TestShorten(t *testing.T) {
    fixtures := []string{
        "fixtures/dev.to",
        "fixtures/rate_limit",
    }

    for _, fixture := range fixtures {
        t.Run(fixture, func(t *testing.T) {
            r, err := recorder.New(fixture)
            if err != nil {
                t.Fatal(err)
            }
            defer func() {
                require.NoError(t, r.Stop())
            }()

            if r.Mode() != recorder.ModeRecordOnce {
                t.Fatal("Recorder should be in ModeRecordOnce")
            }

            shortener.DefaultClient = r.GetDefaultClient()

            shortened, err := shortener.Shorten("https://dev.to/calvinmclean")
            if err != nil {
                t.Errorf("unexpected error: %v", err)
            }

            if shortened != "https://cleanuri.com/7nPmQk" {
                t.Errorf("unexpected result: %v", shortened)
            }
        })
    }
}

Once again, the new test fails. This time due to the unhandled 429 response, so let's implement the new feature to pass the test. In order to maintain simplicity, our function handles the error using time.Sleep and a recursive call rather than dealing with the complexity of considering max retries and exponential backoffs:

func Shorten(targetURL string) (string, error) {
    // ...
    switch resp.StatusCode {
    case http.StatusOK:
    case http.StatusTooManyRequests:
        time.Sleep(time.Second)
        return Shorten(targetURL)
    default:
        return "", fmt.Errorf("unexpected response code: %d", resp.StatusCode)
    }
    // ...

Now run the tests again and see them pass!

Take it a step further on your own and try adding a test for a bad request, which will occur when using an invalid URL like my-fake-url.

The full code for this example (and the bad request test) is available on Github.

Conclusion

The benefits of VCR testing are clear from just this simple example, but they are even more impactful when dealing with complex applications where the requests and responses are unwieldy. Rather than dealing with tedious mocks or opting for no tests at all, I encourage you to give this a try in your own applications. If you already rely on integration tests, getting started with VCR is even easier since you already have real requests that can generate fixtures.

Check out more documentation and examples in the package's Github repository: https://github.com/dnaeon/go-vcr

Test-driven API Development in Go

Calvin McLean — Mon, 08 Jul 2024 23:36:15 +0000

Introduction

Test-driven development is an effective method for ensuring well-tested and refactorable code. The basic idea is that you start development by writing tests. These tests clearly document expectations and create a rubric for a successful implementation. When done properly, you can clearly define the expected input/output of a function before writing any code. This has a few immediate benefits:

You carefully consider the interface for interacting with your code and design it to be testable
When you begin writing code, your flow isn't interrupted by manual testing or stepping through execution logic to predict the outcome. Instead, you just run the tests
Making a test pass becomes a goal that is satisfying to achieve. Breaking down the process into a series of well-defined and achieveable milestones makes the work more enjoyable
Avoid post-implementation laziness and over-confidence that could prevent you from testing your code

Now that you're convinced of the benefits, you can get started with test-driven development (TDD) by following these steps:

Write or modify tests
Check if test fails
Write the minimum amount of code to make tests pass

These steps are followed in a cycle so you are always adding more tests to challenge the current implementation.

The last step, which specifies writing the minimum amount of code, is where things can get tedious if followed rigidly. It's important to understand why this rule exists before you can determine when it's appropriate to stray from it.

Simple Example

You're tasked with implementing the function Add(x, y int) int. Before you jump to the implementation and just return x + y, write the simplest test: 1 + 1 == 2. Then, what is the simplest implementation that would pass the test? It's just return 2. Now your tests pass!

At this point, you realize that you need more tests, so you pick up the pace and add a few more:

1 + 2 == 3
100 + 5 == 105

Now your tests fail, so you need to fix the implementation. You can't just return 3 or return 105 this time, so you need to find a solution that works for all tests. This leads to the implementation: return x + y.

While this feels overly tedious in the trivial example, strict adherence to this method caused you to write multiple tests instead of just trusting your implementation. Of course, your initial idea to return x + y would have worked, but the point is to re-train yourself to rely on tests rather than your own understanding of the code. In the real world, you're not the only one working on this piece of code and will inevitably forget implementation details. This process forces you to write more tests and think of more ways to break the simple implementation.

Eventually, you'll gain experience and learn to find the balance that works in the different scenarios that you encounter. You'll get back to full-speed implementation of features and find that you have fewer bugs and write more maintanable code.

Step by step TDD for an HTTP API

Let's get into a more complicated example using TDD for an HTTP REST API. This step-by-step guide uses my Go framework, babyapi, but the concepts can be applied anywhere.

babyapi uses generics to create a full CRUD API around Go structs, making it super easy to create a full REST API and client CLI. In addition to this, the babytest package provides some tools for creating end-to-end API tables tests. Using TDD at the API-level allows for fully testing the HTTP and storage layers of a new API or feature all at once.

Disclaimer: Since babyapi handles most of the implementation and also is used to generate test boilerplate, we aren't technically starting with TDD. However, we'll see how beneficial it is when adding support for PATCH requests to our API.

Create a new Go project
Create initial main.go using babyapi's simple example
Use the CLI to generate a test boilerplate
Implement each test by filling in the placeholders with expected JSON
Run the tests and see that they pass!
Since PUT is idempotent, it requires all fields to be included. To avoid this, we want to add support for toggling Completed with PATCH requests. We start by adding a simple test for what we expect this feature to look like
This test fails since babyapi doesn't support PATCH by default. We can fix it by implementing Patch for the TODO struct. Since we defined our feature with two tests, our simplest implementation isn't just setting Completed = true and we have to use the value from the request
Now we can change the Completed status of a TODO, but we still cannot use PATCH to modify other fields as show by this new set of tests
Update Patch to set the remaining fields
Our tests still fail since we always update the TODO with the request fields, even if they're empty. Fix this by updating the implementation to check for empty values
The new UpdateWithPatch test passes, but our previous tests fail. Since we changed Completed to be *bool, TODOs created with an empty value will show as null
Implement Render for TODO so we can treat nil as false

Implementing the PATCH feature with test-driven development resulted in a robust set of tests and a well-implemented feature. Since we started by defining the expected input and output of a PATCH request in tests, it was easy to see the issues caused by not checking for empty values in the request. Also, our pre-existing tests were able to protect from breaking changes when changing the type of Completed to *bool.

Conclusion

Test-driven development is an effective approach for creating fully tested and correct code. By starting with tests in mind, we can ensure that every piece of code is designed to be testable instead of letting tests be an afterthought.

If you're hesitant about adopting TDD, here are a few ideas to get started:

Try it in simple scenarios where a function's input/output is clear and the implementation is not overly complicated. You can write a robust table test for the variety of input/output that could be encountered. Having a clear visual of the different scenarios can simplify implementation
If you're fixing a new bug, you have already identified a gap in your testing. Start by writing a test that would have identified this bug in the first place. Then, make this test pass without breaking any existing tests.
Similar to the babyapi example, you can use TDD for high-level API tests. Once you have a definition of the expected request/response, you can resume your usual development flow for more detail-oriented parts of the implementation

Even if TDD isn't a good fit for the way you write code, it's still a powerful tool to have in your belt. I encourage you to at least commit some time to trying it out and see how it affects your development process.

Let's Talk About REST...

Calvin McLean — Wed, 01 May 2024 02:19:37 +0000

REST, or Representational State Transfer, is a concept that guides the implementation of an application's interface. It is typically used in HTTP communication with JSON data to define create, read, update and delete (CRUD) operations on resources. However, by definition, REST APIs can exist without JSON and even without HTTP. Incorrectly claiming that any HTTP JSON API as REST is harmful to the standard since the assumptions about the API's implementation might not be true. In addition to this, non-standard APIs require extensive documentation in order to be usable. If we correctly implement REST and use accurate terminology when discussing APIs, we can build applications that are much easier to use and maintain.

Basics of REST

Roy Fielding defined the term REST in his 1999-2000 PhD dissertation. Fundamentally, REST requires:

Client-server architecture
Statelessness between requests
Uniform interface
Identifiable resources
Hypermedia representations

The first few requirements are often achieved using HTTP methods (GET, POST, PUT, DELETE) and URIs (Uniform Resource Identifiers) as the uniform interface and identification of resources. This clear and consistent interaction builds confidence in which resource is affected by the request and what that effect is. The hypermedia can be JSON, HTML, or anything else that can provide a consistent representation of the application’s resources.

As we dig deeper into REST, the examples will describe an HTTP and JSON API for the sake of familiarity and wide-appeal, but keep in mind that these are not defining characteristics of REST.

REST API Maturity

To better commmunicate the level of RESTfulness that an HTTP API achieves, the Richardson maturity model describes four levels:

Level 0: Use a URI and POST for client-server interaction
Level 1: Use separate URIs for each resource
Level 2: Introduce more HTTP methods for different operations on resources. Follow established standards for the behavior of each method
Level 3: Use hypermedia

Notice that we are now including HTTP-specifics like methods and URIs into the requirements since this maturity model is specific for REST APIs on the web, not the more general definition of REST.

Level 2 is the most impactful level since it introduces a lot of functionality and consistency to the API. It also dictates the use of standard HTTP response codes for communicating the success or failure of requests, such as 200 OK and 400 Bad Request. Once your API reaches this level, it demonstrates a fully-functioning application that allows for all necessary interactions between the client and the server. In most cases, this level of API maturity is sufficient, but what does it look like to take it one step further?

The last, and often overlooked, piece is hypermedia linking, or Hypertext As The Engine of Application State (HATEOAS). Despite the horrendous acronym, this level introduces sophistacation and friendliness to an already-useful API. By adding links to the responses, the server creates a self-documenting API where the client is free to navigate the entire interface using the base path and reasonable assumptions about HTTP methods.

For example, if we are navigating an API for blog posts, we can start with a request to GET /posts. If the API is RESTful, it will respond with a list of Post resources containing details and a link to /posts/{id}. The server will only provide the client with Posts that it has permission to access, so the client can follow the link to GET /posts/1, which will respond with details of the Post and, ideally, links to more related resources such as Likes and Comments:

{
    "id": 1,
    "title": "Post Title",
    ...,
    "links": {
        "comments": "/posts/1/comments",
        "likes": "/posts/1/likes"
    }
}

As you can see from this example, the basic knowledge of the /posts endpoint and the GET method is all the client needs to fully navigate the Posts API. A level-2 API would not enable the client to discover the Comments or Likes related to a Post. If you look beyond the unpronounceable acronym, HATEOAS is an essential feature of a robust REST API.

This example uses the more-familiar JSON API, but the same thing can be achieved using HTML as the hypermedia. In this case, the hypermedia would reveal the child resources by actual clickable links or UI elements instead of just the URIs. Implementations of REST clearly go beyond structural JSON representations of objects and instead show us more generally how to interact with an application.

Self Documenting

You can create a very functional API without level 3, but only the root-level resources will be available to clients without some sort of external documentation or brute-force search. In most cases, this is achieved using an OpenAPI specification.

OpenAPI specifications are a generic and expressive way to document APIs, but they are tedious to write and keep in sync with actual API behavior. The extra step of maintaining API specifications on top of writing code and managing other responsibilities allows the implementation and specification to drift apart. It is worth considering if we can skip this step altogether when our APIs are intuitive and standardized.

Query parameters and request headers are overlooked by self-documenting REST principles. OpenAPI Specifications come out ahead in this aspect, but self-documenting APIs can still be very usable if they adhere to a few assumptions:

A resource's root endpoint (like GET /posts) can filter/search by any field of a Post, like title or author
There might be more fields that aid in querying like order and limit
User-friendly errors will make troubleshooting easy

Before you start deleting all of your API specifications, remember every scenario has unique considerations. We can't realistically expect to learn every API by trial-and-error, so external documentation is still valuable. Thoughtful implementation of REST builds a solid foundation for a usable and well-documented application. Then, API specifications are a valuable add-on to provide more clarity about the API. However, the foundations of REST come first and a specification is not an excuse for eschewing standards. Ultimately, you should find a balance that works for you, your peers, and users.

babyapi

Now this is the part where I tell you about a library I created, babyapi, and how it makes all of this tedious REST stuff easy!

I created babyapi with the goal of providing the simplest path to a full REST API. It started as a humble package that used generics to reduce code duplication in my automated-garden API. After breaking it out into its own repository, I added the optional HATEOAS extension which sets up automatic hypermedia linking. Now you can easily create a REST API which achieves all levels in the Richardson maturity model using babyapi.

Check it out and let me know what you think!

Conclusion

REST is a useful standard for creating intuitive application interfaces, especially in the case of HTTP web applications. In modern web development, the understood definition of REST has drifted to just mean any JSON HTTP interface that loosely interacts with resources. In reality, these are not adhering to the strict definitions of REST and claiming RESTfulness is harmful to usability and maintainability of the APIs. While API specifications are a powerful tool for describing complex APIs, I hope you will first consider designing a self-documenting REST API. This will enable your users to quickly get started with your API before getting bogged down by specifications. Of course, specifications still have an appropriate role in many complex APIs, but we should not be relying on them to make an API usable in the first place.

Run Go + HTMX in the Cloud with Acorn

Calvin McLean — Mon, 18 Dec 2023 02:45:53 +0000

Introduction

Note: Since writing this article, the Acorn SaaS product has been shutdown.

In a recent article, I demonstrated how babyapi, a library I created, makes it easy to write a TODO app with a RESTful API and HTMX frontend using only 150 lines of code. babyapi abstracts the HTTP handling based on a provided struct and serves HTMX templates for a dynamic frontend.

It works great in a tutorial, but you might have been left thinking: "what about persistent storage and running in the cloud?"

This simple tutorial will show you how to connect your babyapi application to Redis storage and quickly run in the cloud using the free Sandbox from Acorn.

Storage

The babyapi.Storage interface and the SetStorage modifier allow implementing any storage backend for your application. As mentioned in the previous article, the babyapi/storage package provides a generic implementation of the interface with helpers for setting up local file or Redis storage. This time, since the goal is to run in the cloud rather than just locally, we'll use the Redis version:

db, err := storage.NewRedisDB(redis.Config{
    Server:   host + ":6379",
    Password: password,
})
if err != nil {
    return fmt.Errorf("error setting up redis storage: %w", err)
}

api.SetStorage(storage.NewClient[*TODO](db, "TODO"))

The full example code for this tutorial is available in the babyapi GitHub repository

With this simple addition, the TODO application is ready to connect to a Redis instance and run in the cloud.

Acorn

If you're not already familiar with Acorn, I recommend checking out the official docs to learn more about it! Basically, it is an app platform that makes it easy to deploy cloud applications and their dependencies by describing them in a simple Acornfile. Instead of configuring all of the required Kubernetes manifests to run our application in the cloud, we can just use an Acornfile.

First, we need a Dockerfile to create our app container. Then, instead of building and pushing this image to a container registry and writing a Kubernetes manifests (by copy/pasting from an online example if we're being honest), let's take a look at Acorn.

Before deploying the updated TODO app, we need the Redis database dependency. If you search "run redis in k8s", all of the top results look exhausting. Alternatively, the Acorn documentation for Redis looks much simpler.

We can even use the Acornfile from the documentation's example with one little change to build and run the TODO app with Redis database:

services: db: {
    image: "ghcr.io/acorn-io/redis:v7.#.#-#"
}

containers: app: {
    build: {
        context: "."
    }
    consumes: ["db"]
    ports: publish: "8080/http"
    env: {
        REDIS_HOST: "@{service.db.address}"
        REDIS_PASS: "@{service.db.secrets.admin.token}"
    }
}

Not only did this save us the effort of configuring everything for the TODO app, we even have the entire Redis dependency with a persistent volume and a random password. This is already way better than the no-volume, no-password K8s manifests I would thrown together for this example.

Run it for real

Now that you have seen how it all works, you can run it for yourself using this button:

At the time of posting this, Acorn offers free Sandbox accounts to anyone with a GitHub account.

Once it has finished deploying, open the endpoint and append /todos in the URL to reveal the HTMX UI! You can even use the babyapi CLI from your terminal to create a new TODO and watch it show up in the UI automatically:

export ACORN_ADDR=http://COPY_ENDPOINT_FROM_ACORN

go run -mod=mod \
  github.com/calvinmclean/babyapi/examples/todo-htmx \
  -address $ACORN_ADDR \
  post TODOs '{"title": "use babyapi on Acorn!"}'

Conclusion

The availability of app platforms like Acorn and easy-to-use libraries like babyapi take care of the boring and tedious parts of software engineering and let you focus on the things that interest you. Try them out for yourself and let me know what you have any feature requests or issues with babyapi.

Thanks for reading!

How To Build a Web Application with HTMX and Go

Calvin McLean — Tue, 12 Dec 2023 04:56:11 +0000

This UI and backend are implemented with only 150 lines of code, including the HTML!
The full example code for this tutorial is available in the babyapi GitHub repository if you're eager to get into it.

In my recent article, The Easiest Way to Create a REST API With Go, I demonstrated how babyapi can jumpstart REST API creation. This time, I will walk you through some additional babyapi features and show how to create an easy and dynamic frontend using HTMX. If you're not already familiar, HTMX is a library that basically extends HTML with functionality that normally requires Javascript.

HTMX was designed with REST backends in mind, which makes it a perfect companion for babyapi. HTMX provides a snappy SPA-like feel by replacing and rendering individual components from server responses instead of the entire page.

Since babyapi uses chi/render package for requests and responses, it automatically supports input from HTML forms instead of only JSON. Additionally, babyapi defines an HTMLer interface which allows resources to define custom HTML string responses when the client requests text/html. We can take advantage of these features to easily create and serve an HTMX frontend.

Create the TODOs API

In the last article, we created a simple TODO resource. This struct allows babyapi to serve the necessary HTTP endpoints for interacting with the data:



package main

import "github.com/calvinmclean/babyapi"

type TODO struct {
    babyapi.DefaultResource

    Title       string
    Description string
    Completed   bool
}

func main() {
    api := babyapi.NewAPI[*TODO](
        "TODOs", "/todos",
        func() *TODO { return &TODO{} },
    )

    api.RunCLI()
}

In order to add an HTMX UI on top of this, all we need to do is:

Implement babyapi.HTMLer interface for the TODO resource
Use api.SetGetAllResponseWrapper with a new babyapi.HTMLer type to render HTML for the /todos response
Write templates for HTMX frontend with these features:
- List all TODOs in a table
- Buttons to mark items as completed and delete
- Server-sent events automatically append new TODOs
- Form to create TODOs

Respond With HTML Instead of JSON

By default, babyapi is designed to marshal structs to the requested response type (usually JSON). HTML responses are not as straightforward, so babyapi provides an HTMLer interface that enables creating HTML responses from resource structs:



type HTMLer interface {
    HTML(*http.Request) string
}

The HTML(*http.Request) method can be implemented as follows:



const todoRowTemplate = `...`

func (t *TODO) HTML(*http.Request) string {
    tmpl := template.Must(
        template.New("todoRow").Parse(todoRowTemplate),
    )
    return babyapi.MustRenderHTML(tmpl, t)
}

This method simply renders a template from a string and uses babyapi.MustRenderHTML to execute it with the TODO data.

The HTML template looks like this:



<tr hx-target="this" hx-swap="outerHTML">
    <td>{{ .Title }}</td>
    <td>{{ .Description }}</td>
    <td>{{ $disabled := "" }}{{ if .Completed }}{{ $disabled = "disabled" }}{{ end }}<button
            hx-put="/todos/{{ .ID }}"
            hx-headers='{"Accept": "text/html"}'
            hx-include="this"
            {{ $disabled }}>

            <!-- Include entire TODO item for idempotent PUT -->
            <input type="hidden" name="Completed" value="true">
            <input type="hidden" name="Title" value="{{ .Title }}">
            <input type="hidden" name="Description" value="{{ .Description }}">
            <input type="hidden" name="ID" value="{{ .ID }}">
            Complete
        </button>

        <button hx-delete="/todos/{{ .ID }}" hx-swap="swap:1s">
            Delete
        </button>
    </td>
</tr>

NOTE: some minor details, like CSS classes, are excluded from the example for simplicity. The full example can be found on GitHub

This template creates an HTML table row (<tr>) to display the TODO's title and description. The row also contains buttons to mark the item as complete or to delete it.

On top of the regular HTML, we use HTMX attributes to control interactions with the backend:

hx-target and hx-swap tell HTMX to replace the entire row with the contents of a successful response. These attributes are inherited by the buttons
The "Complete" button uses hx-put to make a PUT request to /todos/{{ .ID }}. The request sets Accept: text/html to request an HTML response from the server. hx-include="this" uses the child input fields to make the request body. This sets Completed=true and leaves the other fields unchanged
The "Delete" button uses hx-delete to send a DELETE request to /todos/{{ .ID }}. Then, hx-swap="swap:1s" tells HTMX to swap new contents over 1 second instead of immediately. Since the response content will be empty, this creates a fade-out effect

Now that we have a simple HTML implementation, run the server, create a TODO item, and fetch it:



go build -o todo-app

# start the server in a separate terminal
./todo-app serve

# create new TODO
./todo-app post TODOs '{"title": "use babyapi!"}'

# get the previously-created TODO by ID with HTML response
./todo-app -H "Accept: text/html" get TODOs clnvnt5o402av6j1oal0

Loading http://localhost:8080/todos/{ID} in the web browser will show the browser-rendering of the HTML, but it's not very impressive yet without style or HTMX working. The next step will be rendering the full HTML page to display all TODOs and enable HTMX.

Create the All TODOs Page

Logically, the UI view for all TODOs will come from our base endpoint, /todos. However, babyapi is setup to use the default response type of *babyapi.ResourceList[*TODO] to create a JSON-compatible response. As you probably guessed, babyapi has a way to deal with this.

All we have to do is use api.SetGetAllResponseWrapper to set a function that accepts []*TODO and returns a new render.Renderer. For this example, we create the AllTODOs type to satisfy the render.Renderer and babyapi.HTMLer interfaces:



type AllTODOs []*TODO

func (at AllTODOs) Render(w http.ResponseWriter, r *http.Request) error {
    return nil
}

const allTODOsTemplate = `...`

func (at AllTODOs) HTML(*http.Request) string {
    tmpl := template.Must(
        template.New("todoRow").Parse(todoRowTemplate),
    )
    tmpl = template.Must(
        tmpl.New("allTODOs").Parse(allTODOsTemplate),
    )
    return babyapi.MustRenderHTML(tmpl, at)
}

func main() {
    api := babyapi.NewAPI[*TODO]("TODOs", "/todos", func() *TODO { return &TODO{} })

    api.SetGetAllResponseWrapper(
        func(todos []*TODO) render.Renderer {
            return AllTODOs(todos)
        },
    )

    // ...
}

The AllTODOs type implements babyapi.HTMLer in the same way as the previous example: just parse and execute HTML templates. Then, api.SetGetAllResponseWrapper function simply returns AllTODOs from the provided slice of TODOs.

The allTODOs template contains the full HTML page which imports HTMX scripts and UIKit CSS. The <body> portion of this new template looks like:



<body>
    <table>
        <thead>
            <tr>
                <th>Title</th>
                <th>Description</th>
                <th></th>
            </tr>
        </thead>

        <tbody>
            <form hx-post="/todos" hx-swap="none" hx-on::after-request="this.reset()">
                <td>
                    <input name="Title" type="text">
                </td>
                <td>
                    <input name="Description" type="text">
                </td>
                <td>
                    <button type="submit">Add TODO</button>
                </td>
            </form>

            {{ range .Items }}
            {{ template "todoRow" . }}
            {{ end }}
        </tbody>
    </table>
</body>

NOTE: some styling and other details that are not important to the example are excluded for brevity. The full example can be found on GitHub

This template creates the structure of an HTML table and populates the first row with a simple HTML form for creating new TODO items. Then, the todoRow template from the previous section is used to create rows for all existing TODOs.

The form uses hx-post="/todos" to send a POST request with the form contents to the API. hx-swap="none" disables swapping the response contents since we will use server-sent events in the next section to append new rows. Then we use hx-on::after-request="this.reset()" to reset the form.

Implement Server-Sent Events

Server-sent events allow one-way communication from the backend to the frontend. In this case, we will use the feature to push new TODO rows to the UI, even if they are created from the CLI or other sources. Luckily, both babyapi and HTMX make it super easy to use server-sent events.

HTMX has an SSE extension which can be used in this UI by replacing the plain <tbody> with the following:



<tbody
    hx-ext="sse"
    sse-connect="/todos/listen"
    sse-swap="newTODO"
    hx-swap="beforeend">
    ...
</tbody>

This new tbody works with server-sent events using the new HTMX attributes:

hx-ext="sse" enables the extension
sse-connect="/todos/listen" opens the connection to the server's SSE endpoint (not implemented yet)
sse-swap="newTODO" instructs HTMX to only use events with the type newTODO
hx-swap="beforeend" will append rows before the </tbody> closing tag

Now that the frontend is ready to receive new TODO rows, the backend just needs to send them! babyapi makes it unbelievably simple to handle SSE connections, even if this concept is completely new to you. The api.AddServerSentEventHandler modifier adds a new route to handle SSE connections and returns a channel for sending events.

Since we want to push new TODOs after they are created, we can use api.SetOnCreateOrUpdate to push events to the channel after successful POST requests:



todoChan := api.AddServerSentEventHandler("/listen")

api.SetOnCreateOrUpdate(func(r *http.Request, t *TODO) *babyapi.ErrResponse {
    if r.Method != http.MethodPost {
        return nil
    }

    select {
    case todoChan <- &babyapi.ServerSentEvent{
        Event: "newTODO",
        Data: t.HTML(r),
    }:
    default:
    }
    return nil
})

This function first makes sure the request method is POST and then pushes a new event on the channel. The event has the newTODO type that is expected by the frontend and uses the TODO's HTML method to create the row. Additionally, it uses select with a default case instead of simply pushing to the channel so the function will not block if there is no frontend receiving from the channel.

Final Touches

With the main structure and functionality in place, only a few minor tweaks are required to finish the seamless integration between babyapi and HTMX.

By default, babyapi is setup to respond to successful DELETE requests with 204 No Content, but HTMX treats this response as the server requesting to disable the swap. This means our deleted rows will not disappear until the page is refreshed, which is not okay in a modern web application. In order to enable swaps, HTMX needs a 200 OK response. Once again, babyapi provides the necessary flexibility to achieve this with the api.SetCustomResponseCode modifier:



api.SetCustomResponseCode(http.MethodDelete, http.StatusOK)

This simply instructs the default DELETE handler to respond with http.StatusOK.

All that's left is adding a little style. I won't claim to be a frontend engineer or a UI designer, so I just used UIKit to easily add modern-looking style to the HTML table and buttons. As mentioned throughout the article, the CSS classes and other small details are excluded since they are not directly relevant to the tutorial. See the full example on GitHub to try running it for yourself.

Storage Layer

At this point, we have a fully functioning web app for managing TODOs. If you want to take it one step further and use it for a real TODO tracker instead of a demo, you just need to add persistent storage. babyapi also simplifies this with the babyapi/storage package. This package provides a generic implementation of the Storage interface with helpers for setting up local file or Redis storage. Add the following to the main function to save the TODOs in a JSON file:



db, err := storage.NewFileDB(hashmap.Config{

    Filename: "todos.json",

})

if err != nil {

    panic(err)

}

api.SetStorage(storage.NewClientdb, "TODO"*TODO)

Conclusion

In this tutorial, we extended the super simple babyapi introduction example to implement an HTMX frontend without adding too much complexity. This shows how babyapi's RESTful API and HTMX, a frontend library designed for RESTful backends, are a perfect fit together. It's easier than ever to create a responsive and dynamic web application.

The steps here also demonstrate the flexibility of babyapi. Although the default API provided is sufficient for all CRUD functionality, our real-world usecases often introduce more requirements and variance which babyapi is able to handle gracefully. Hopefully this has revealed how babyapi can be used to implement your next API-driven application!

I encourage you to continue experimenting with babyapi and HTMX to learn more! Here are some ideas to get you started with extending this example:

Add a toggle to show complete, incomplete, or all TODOs
Allow "un-completing" a TODO
Add a CreatedAt field to the TODO struct and sort by this date (you can implement render.Binder for TODO to automatically set CreatedAt on new items)

Thanks for reading!

The Easiest Way to Create a REST API With Go

Calvin McLean — Sun, 03 Dec 2023 01:42:29 +0000

Introduction

Go is known for being easy to learn and providing the fastest path to production. With HTTP functionality built-in to the standard library, you have everything you need without any external dependencies. The lower-level access to HTTP can be a breath of fresh air if you are accustomed to bloated frameworks, but it results in a lot of boilerplate code which eventually becomes tiring to write over and over. This is where babyapi comes in. I created this library to enable super simple creation of REST APIs without forcing a specific structure on your whole application. It's so easy that a baby could do it!

Baby API

All you need to get started with babyapi is a struct that implements the babyapi.Resource interface. This is easily achieved by extending the babyapi.DefaultResource struct. After completing this simple step, you will have an HTTP API capable of create, read, update, and delete operations. Additionally, babyapi provides an HTTP client to interact with the API, a CLI to easily use this client, and some shortcuts for unit testing.

babyapi can also take you beyond the basics. It provides plenty of options for extending the API behavior, adding validation, and even custom API routes. Any storage backend can be integrated by implementing the simple Storage interface.

This article just scratches the surface of what babyapi is capable of. If you are interested in learning more, keep an eye out for my future articles and star the GitHub repository.

Getting Started

The goal of babyapi is to be so easy that a baby could do it. As previously mentioned, babyapi.DefaultResource already implements the required interface, so it can be used as a starting point for simple resource types. Besides simply implementing the interface, this default struct implements some validations around the ID and uses rs/xid to create a unique identifer on new resources.

Here is a simple example that extends the default to create an API for TODO items:

package main

import "github.com/calvinmclean/babyapi"

type TODO struct {
    babyapi.DefaultResource
    Title       string
    Description string
    Completed   bool
}

func main() {
    api := babyapi.NewAPI[*TODO](
        "TODOs", "/todos",
        func() *TODO { return &TODO{} },
    )
    api.RunCLI()
}

Next, all you need to do is run the server and use the CLI to interact with it:

go run main.go serve

There are too many features to cover in this article, so the best ways to learn more are:

Client

The struct used to create the server also enables a built-in client for interacting with the babyapi server. The client can also be used in the application or other Go applications interacting with your API. It supports top-level and nested API resources and allows easy access to all CRUD endpoints.

// Create a client from an existing API struct (mostly useful for unit testing):
client := api.Client(serverURL)

// Create a client from the Resource type:
client := babyapi.NewClient[*TODO](addr, "/todos")

// Create a new TODO item
todo, err := client.Post(context.Background(), &TODO{Title: "use babyapi!"})

// Get an existing TODO item by ID
todo, err := client.Get(context.Background(), todo.GetID())

// Get all incomplete TODO items
incompleteTODOs, err := client.GetAll(context.Background(), url.Values{
    "completed": []string{"false"},
})

// Delete a TODO item
err := client.Delete(context.Background(), todo.GetID())

The client provides more general methods for interacting with server: MakeRequest and MakeRequestWithResponse. The client is also cutomizable. You can replace the underlying http.Client to use a custom RoundTripper or use SetRequestEditor to add a pre-request function that modifies the http.Request.

Storage

You can bring any storage backend to babyapi by implementing the Storage interface. By default, the API will use the built-in MapStorage which just uses an in-memory map.

In an effort to provide actual persistent storage out of the box, the babyapi/storage package uses madflojo/hord to support a variety of key-value store backends. Additionally, babyapi/storage provides helper functions for initializing the hord client for Redis or file-based storage.

db, err := storage.NewFileDB(hashmap.Config{
    Filename: "storage.json",
})
db, err := storage.NewRedisDB(redis.Config{
    Server: "localhost:6379",
})

api.SetStorage(storage.NewClient[*TODO](db, "TODO"))

Conclusion & Roadmap

babyapi is a very new project and still has a lot of potential for new features and improvements. However, my goal is to keep it as simple and barebones as possible, so I plan to focus on usability and testing improvements. I will add new features if use-cases or feature requests come up. These are some of the things coming next:

Improve CLI to have better usage instructions and implement flags when multiple IDs are required for nested APIs
Improve default logging and implement log-levels
Allow CLI to access non-CRUD endpoints (endpoints added by AddCustomRoute or AddCustomIDRoute)
Create more automated testing. Ideally, I will create a generator that reads your source file using babyapi and generates a test structure with some default tests and room to extend them

Also keep an eye out for more articles that dig deeper into some of the core features of babyapi!

How I Learned Generics in Go

Calvin McLean — Sun, 12 Nov 2023 22:52:32 +0000

Introduction

Added in Go 1.18, generics were a controversial and long-awaited addition to the language. Go purists feared the change would encourage lazy practices and ruin Go's reputation for favoring simplicity over complexity. More recent Gophers were missing the utility of generics from other languages and eagerly awaited this update. I found myself somewhere in the middle.

I didn't have any desire to use generics initially, but understood that it could be useful in some cases. After working with Go long enough, I was already accustomed to thinking in a way that didn't involve generics and never thought twice about rewriting a GetKeys(map[string]interface{}) function for the hundredth time. It wasn't until reading the release notes for Go 1.21, which added slices and maps packages to the standard library, that I started thinking more about generics. These packages are all built around generic type parameters and add common operations for the slice and map data structures. maps.Keys is currently in the experimental version!

Seeing how the standard library was able to use generics to expand its own functionality and alleviate some common pain points really opened my eyes to the potential of this feature. Originally, it seemed like generics were added to placate a few users, but these recent additions made me realize that generics are being taken seriously in Go and are worth a fresh look. I started thinking about how I could leverage generics in my own programs to reduce code duplication and make other improvements.

First Attempt: API Layer

When thinking about duplicated code in my automated-garden project, the first thing that comes to mind is all of my API handlers. The server side of this application implements a few straightforward CRUD APIs following RESTful principles. Each resource type implements handlers for the different HTTP verbs and mostly interacts with the storage layer. I created a very simple setup for the API handlers following this formula:

Each resource has an ID, which is included in the URL
Middleware is used to fetch a resource from storage and put it into the request context
GET endpoints just use go-chi/render to create the HTTP response
Other endpoints, like PATCH and DELETE, read the resource from context and use it to perform additional actions

The only difference between the GET handlers is the type that is read from context and the response created. This, of course, sounds like a great use-case for generics! However, after thinking about it more and trying out a basic implementation, I realized it is possible to implement this using the render.Renderer interface that my types already implement:

func getGardenFromContext(ctx context.Context) render.Renderer {
    return ctx.Value(gardenCtxKey).(*GardenResponse)
}

func get(getter func(context.Context) render.Renderer) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        resource := getter(r.Context())

        err := render.Render(w, r, resource)
        if err != nil {
            render.Render(w, r, ErrRender(err))
        }
    }
}

func routes(r *chi.Mux) {
    r.With(gardenContextMiddleware).Get("/", get(getGardenFromContext))
}

In order to implement this, I had to change the middleware to store the GardenResponse in the context instead of the raw Garden struct. The getGardenFromContext now has to return a render.Renderer so it can satisfy the input to get without additional type assertions. This makes the function less usable in other scenarios. One solution is replacing the getter input with just the context key, so the handler can use a more generalized getRendererFromContext function. Another option is to use a generic type parameter:

func getGardenFromContext(ctx context.Context) *GardenResponse {
    // function body unchanged
}

func get[T render.Renderer](getter func(context.Context) T) http.HandlerFunc {
    // function body unchanged
}

func routes(r *chi.Mux) {
    r.With(gardenContextMiddleware).Get("/", get[*GardenResponse](getGardenFromContext))
}

The introduction of the generic type parameter adds a bit of clarity about the types being used and actually reduces the abstraction when compared to the interface-only approach. This is going to be appreciated by those concerned about unnecessary increases to complexity. Additionally, the render.Renderer interface is still used as a constraint for our generic type, so all of the benefits are still available. Finally, the getGardenFromContext function is returned to its original type so it can still be used outside of this response rendering scenario. In the end, it just comes down to personal preference.

The hidden truth for both implementations is that everything that isn't generalized is moved into the specific middlewares or the Render methods of each type. This is already how my application was designed, so it made the refactor pretty easy. However, this approach is not going to work as well for the more complicated handlers like POST and PATCH which do more than just returning the resource and have drastically different behaviors for each resource.

Ultimately, I decided to scrap this generic API idea, at least for now. While it slightly reduced code duplication, it doesn't work in all cases and furthers the separation between the application code and HTTP layer, which I see as an unnecessary abstraction. However, if I took the time to implement it as a generic API framework and rebuild my automated-garden API around it, that might be a different story.

Next Attempt: Storage Layer

Determined to implement generics and delete some lines of code, I turned my sights to the storage layer of my application. I designed my storage around key-value pairs since I started by storing resources in YAML files. This eventually evolved to use madflojo/hord to interact with key-value data stores like Redis. The function to read a *pkg.Garden from storage looks like this:

func (c *Client) getGarden(key string) (*pkg.Garden, error) {
    dataBytes, err := c.db.Get(key)
    if err != nil {
        return nil, fmt.Errorf("error getting Garden: %w", err)
    }

    var result pkg.Garden
    err = json.Unmarshal(dataBytes, &result)
    if err != nil {
        return nil, fmt.Errorf("error parsing Garden data: %w", err)
    }

    return &result, nil
}

This uses the hord client's Get(key) which returns a JSON []byte. This is unmarshalled into the pkg.Garden struct. As you can probably imagine, the getters for other types are the same except the var result pkg.Garden line specifies a different type. Another potential usecase for generics! Or, once again, maybe it can be abstracted with interfaces.

In this case, the var result pkg.Garden line can be relocated outside of the function and passed in as a destination pointer. Instead of returning a *pkg.Garden, get will pass the destination pointer to json.Unmarshal. The function can now be used more generally:

func (c *Client) GetGarden(id xid.ID) (*pkg.Garden, error) {
    var result *pkg.Garden
    err := c.get(gardenPrefix+id.String(), &result)
    return result, err
}

func (c *Client) get(key string, target interface{}) error {
    dataBytes, err := c.db.Get(key)
    if err != nil {
        return fmt.Errorf("error getting data: %w", err)
    }

    err = json.Unmarshal(dataBytes, &target)
    if err != nil {
        return fmt.Errorf("error parsing data: %w", err)
    }

    return nil
}

This is basically a wrapper around json.Unmarshal and using it will be familiar for any Go engineer. The same thing will work for save since it is similar to json.Marshal. Then, the only other function that touches the storage library is getMultipleGardens (and same for other types). Since I am using a simple key-value store instead of SQL tables, I have to get a list of keys and then fetch each one, which looks like this if I convert to use interface{}:

func (c *Client) GetGardens() ([]*pkg.Garden, error) {
    var result []*pkg.Garden
    err := c.getMultiple(gardenPrefix, &result)
    return result, err
}

func (c *Client) getMultiple(prefix string, results []interface{}) error {
    keys, err := c.db.Keys()
    if err != nil {
        return fmt.Errorf("error getting keys: %w", err)
    }

    for _, key := range keys {
        if !strings.HasPrefix(key, gardenPrefix) {
            continue
        }

        var result interface{}
        err := c.get(key, &result)
        if err != nil {
            return fmt.Errorf("error getting Garden: %w", err)
        }
        if result == nil {
            continue
        }

        results = append(results, result)
    }

    return nil
}

This results in a compiler error: cannot use result (variable of type []*pkg.Garden) as []interface{} value in argument to c.getMultiple. This error occurs because []interface{} does not behave in the same way as interface{}. Since it is a slice, each individual element needs to be typed separately. Even if this was valid, it would not work because the c.get call is using an emtpy var result interface{} and never knows which type to unmarshal into.

Now, this should be a case where generics can help since it is an abstraction on a data structure. Here is how it looks:

func getOne[T any](c *Client, key string) (*T, error) {
    if c.db == nil {
        return nil, fmt.Errorf("error missing database connection")
    }

    dataBytes, err := c.db.Get(key)
    if err != nil {
        return nil, fmt.Errorf("error getting data: %w", err)
    }

    var result T
    err = json.Unmarshal(dataBytes, &result)
    if err != nil {
        return nil, fmt.Errorf("error parsing data: %w", err)
    }

    return &result, nil
}

func getMultiple[T any](c *Client, prefix string) ([]T, error) {
    keys, err := c.db.Keys()
    if err != nil {
        return nil, fmt.Errorf("error getting keys: %w", err)
    }

    results := []T{}
    for _, key := range keys {
        if !strings.HasPrefix(key, prefix) {
            continue
        }

        result, err := getOne[T](c, key)
        if err != nil {
            return nil, fmt.Errorf("error getting data: %w", err)
        }
        if result == nil {
            continue
        }

        results = append(results, *result)
    }

    return results, nil
}

Using generics here even simplifies the calling code:

func (c *Client) GetGarden(id xid.ID) (*pkg.Garden, error) {
    return getOne[pkg.Garden](c, gardenKey(id))
}

func (c *Client) GetGardens(getEndDated bool) ([]*pkg.Garden, error) {
    return getMultiple[*pkg.Garden](c, getEndDated, gardenPrefix)
}

The result is a generic implementation that leaves the external interface of the storage client unchanged, but significantly reduces duplication internally. Take a look at the full PR here. This is an ideal use of generics because it takes advantage of code reuse, but doesn’t expose the added complexity and abstraction to consumers of the package. Additionally, it was a very straightforward refactor because it strictly follows the advice of using generics "where the only difference between the copies is that the code uses different types."

Another Use-Case: Request Retries

This final example is the result of solving a problem that occurred organically rather than an attempt to brute-force generics into my existing application. While building a tool to publish markdown files to DEV as articles using GitHub Actions, I ran into rate limiting from the DEV API. Normally, HTTP client retries are done using a custom http.Transport, but this program uses a client library that doesn't allow setting the Transport. Instead, I turned to creating a wrapper function using generic type parameters for the response struct:

type response interface {
    StatusCode() int
}

func doWithRetry[T response](do func() (T, error), numRetries int, initialWait time.Duration) (T, error) {
    for i := 1; i <= numRetries; i++ {
        result, err := do()
        if err != nil {
            return *new(T), err
        }

        if result.StatusCode() == http.StatusTooManyRequests {
            time.Sleep(initialWait * time.Duration(i))
            continue
        }

        return result, err
    }

    return *new(T), fmt.Errorf("exhausted retry limit %d", numRetries)
}

func (c *Client) getPublishedArticles() {
    resp, err := doWithRetry(func() (*api.GetUserPublishedArticlesResponse, error) {
        return c.GetUserPublishedArticlesWithResponse(context.Background(), nil)
    }, 5, 1*time.Second)
    if err != nil {
        return nil, fmt.Errorf("error getting articles: %w", err)
    }
}

You might notice that I don't include any type parameters when using the doWithRetry function. This is a nice feature of generics in Go that allows the type parameter to be inferred when it is used in multiple places. In this case, the return type of the do function has to be defined, so there is no need to duplicate this in the type parameter.

Another interesting aspect of this this example is that it takes advantage of Go's implicit interfaces to define an interface outside of the client library the defines all the structs and StatusCode() methods. The generic type parameter requires types that implement the interface so it can check response codes without additional type assertions. Using generics does not create an "either-or" scenario with interfaces, and often when using generics, interfaces are extremely useful to introduce constraints.

Conclusion

I chose to learn about generics in Go by finding how the feature can improve my automated-garden project. In this process, I learned how to use generics and more importantly, when to use generics. I was reminded of the pitfalls that come from overusing abstraction and why I originally appreciated the language discouraging these practices, but I also see a lot of potential for this new feature.

Generic type parameters enable the simplification of small, tedious repetition and open the door to whole new levels of abstraction. In some cases, the feature provides a second way to achieve the same things that were already possible. In other cases, it unlocks whole new options that were previously unachievable. Most of the time, this level of abstraction is unnecessary, but it also allows for new and creative solutions. Although the potential to create overly-complex and abstracted code is increased, it is important for a programming language to provide these necessary tools. Ultimately, it is up to the programmer to decide how and when to use them. This reminds me of the famous quote: "If all you have is a hammer, everything looks like a nail." Luckily, with the addition of more tools to the language, we have much more than a hammer at our disposal.

Forem: Calvin McLean

Experiment with gopls MCP: Improving Agent Context for Go Development

The Experiment

Attempt 1: VS Code without gopls MCP

Attempt 2: VS Code with gopls MCP

Attempt 3: Zed without gopls MCP

Attempt 4: Zed with gopls MCP

Conclusion

Next Steps

How to use gopls MCP with VS Code

Enable gopls MCP in VS Code

1. Install/update gopls

2. Enable VS Code to run the server

3. Enable the MCP server for agents

4. Setup instructions for using the tools

Use it!

Why MCP?

How to Implement LLM Tool-calling with Go and Ollama

How it works

Implement Tool-calling with Ollama + Go

Going a step further

MCP

Thoughts

Middleware and RoundTrippers in Go

Middleware

Cache Middleware

Logger Middleware

Auth Middleware

Recovery Middleware

Sequential Middleware

RoundTrippers

RoundTripperFunc

Cache RoundTripper

Log RoundTripper

Auth RoundTripper

Sequential RoundTripper

Tests

Correct Setup

Incorrect Setup

Conclusion

Improve Your Go Applications with Byte-level Data Optimization

Intro to bytes

Requirements

Measurements

Flour type

Time

Putting it all together

Optimizing it further

Sharing a byte

Conclusion

Effortless HTTP Client Testing in Go

Introduction

Deeper Dive using Go

URL Shortener Example

Mocking Errors

Conclusion

Test-driven API Development in Go

Introduction

Simple Example

Step by step TDD for an HTTP API

Conclusion

Let's Talk About REST...

Basics of REST

REST API Maturity

Self Documenting

babyapi

Conclusion

Run Go + HTMX in the Cloud with Acorn

Introduction

Storage

Acorn

Run it for real

Conclusion

How To Build a Web Application with HTMX and Go

Create the TODOs API

Respond With HTML Instead of JSON

Create the All TODOs Page

Implement Server-Sent Events

Final Touches

Storage Layer

1. Install/update `gopls`