Forem: Eke Enyinnaya Diala

Resource Cleanup in Go Web Servers: Lessons from Nursery School

Eke Enyinnaya Diala — Fri, 18 Jul 2025 11:17:45 +0000

Introduction

Think back to nursery school or to Robert Fulghum's "All I Really Need to Know I Learned in Kindergarten" [if you read it] one of the first lessons we learned was simple:

Clean up your own mess
Put things back where you found them
Share everything

As kids, this meant picking up toys after playtime. As engineers, it means making sure our programs don’t leave behind open connections, stuck processes, or unfinished business. In Go web servers, these lessons are more than just good manners, they’re essential for building reliable, maintainable software.

Let’s walk through a Go web server and see how these childhood rules guide our approach to resource cleanup.

The Mess We Make

Every time our server starts, it opens connections to databases, listens for HTTP requests, and spins up goroutines to handle these requests. If we don’t clean up, these resources pile up like toys left on the floor. Eventually, someone trips.

And just like a classroom with toys everywhere, a production system with resource leaks can become a minefield. Outages creep in, and the cause is often hard to spot until someone steps on a stray toy, or a database refuses new connections, or a server freezes on shutdown, forcing a risky manual kill.

Context: The Teacher’s Bell

In nursery school, the teacher rings a bell to signal the end of playtime. In Go, the context package is our bell. It tells every part of our program when it’s time to stop, clean up, and go home.

Our server listens for signals from the operating system—SIGINT, SIGTERM, SIGQUIT—just like listening for the teacher’s bell. When a signal arrives, we use signal.NotifyContext to broadcast the message: “Clean up time!”

ctx, cancel := signal.NotifyContext(
    context.Background(),
    syscall.SIGINT,
    syscall.SIGTERM,
    syscall.SIGQUIT,
)
defer cancel()

This ctx is passed everywhere: to the database, to HTTP handlers, to background jobs. When it’s cancelled, everyone knows it’s time to wrap up.

With context as our bell, we don’t need to shout across the room or rely on everyone to notice the clock. Every goroutine listens for the same signal [or derives their own signal from it], so the cleanup is coordinated and calm. This makes the code easier to reason about and less prone to subtle bugs.

Cleaning Up the Database Toys

When we connect to the database, we use our context. If the teacher says "Stop!" (the process is killed or our timeout expires), the connection attempt is cancelled. No waiting forever for a broken database to never connect or leaving behind an open connection after the server receives a signal to shutdown.

func ConnectDB(ctx context.Context, dbURL string) (*sql.DB, error) {
    db, err := sql.Open("postgres", dbURL)
    if err != nil {
        return nil, fmt.Errorf("opening database connection: %w", err)
    }

    timeoutCtx, cancel := context.WithTimeout(ctx, time.Second*30)
    defer cancel()

    if err := db.PingContext(timeoutCtx); err != nil {
        return nil, fmt.Errorf("pinging database: %w", err)
    }

    return db, nil
}

And when we’re done, we put the connection back where we found it:

conn, err := ConnectDB(ctx, dbURL)
if err != nil {
    log.Fatal(err)
}
defer conn.Close()

Just like putting toys back on the shelf, closing the connection ensures the next group can play without running into a mess. And if you ever scale your service horizontally, proper cleanup means new instances can start without waiting for old connections to be released. Rolling deployments and zero-downtime upgrades become smooth transitions, not frantic scrambles.

Sharing Toys: The HTTP Client

Just like we don’t need a new ball for every child, we don’t always need a new http.Client for every request. We can share one client, so connections are reused and the playground isn’t cluttered with abandoned toys.

This means faster requests, fewer resources wasted, and a happier playground for everyone. And when you want to set timeouts, proxies, or custom transports, you do it once and all requests benefit. Testing and mocking become easier too—just swap out the client for a test double and the whole system adapts. I wrote about mocking HTTP requests here.

func PingURL(ctx context.Context, client *http.Client, url string) error {
    // use context here again to make sure our request does not take forever.
    req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
    if err != nil {
        return fmt.Errorf("creating request: %w", err)
    }

    res, err := client.Do(req)
    if err != nil {
        return fmt.Errorf("making request: %w", err)
    }
    defer res.Body.Close()

    if res.StatusCode >= http.StatusBadRequest {
        b, err := io.ReadAll(res.Body)
        if err != nil {
            return fmt.Errorf("reading error response: %w", err)
        }
        return fmt.Errorf("response from %s: %s", url, b)
    }

    return nil
}

Furthermore, because of the shared context, every HTTP request can have its own timer derived from the parent signal, like saying, "You have 30 seconds to finish your game." If the time runs out, or if the teacher calls for cleanup, the request stops.

handler.HandleFunc("/ping/{url}", func(w http.ResponseWriter, r *http.Request) {
    pingCtx, cancel := context.WithTimeout(ctx, time.Second*30)
    defer cancel()
    url := r.PathValue("url")
    if err := PingURL(pingCtx, client, url); err != nil {
        http.Error(w, err.Error(), http.StatusBadGateway)
        return
    }
    w.Write([]byte("OK"))
})

This keeps the playground fair—no one hogs the toys, and everyone gets a turn. And if a third-party API is down, your server won’t hang forever waiting for a response. The bell rings, the game ends, and the playground stays lively and responsive.

The Final Bell: Graceful Shutdown

When playtime is really over, we wait for everyone to finish their last game, but not forever. We give them some time to wrap up, then we close the playground.

<-ctx.Done()
shutdownCtx, cancel := context.WithTimeout(context.Background(), time.Second*30)
defer cancel()

if err := server.Shutdown(shutdownCtx); err != nil {
    log.Fatal(err)
}

This way, no one is left behind, and the playground is ready for the next group. Deploying updates or restarting servers doesn’t mean dropped requests or angry users. It’s a professional responsibility, and your ops team will thank you.

Why It Matters

If we don’t clean up, the next group can’t play—the database is full, the server is stuck, the system is a mess, and someone has to stop reading a book to his son at night. By listening for signals, using context, and always putting things back, our Go server stays tidy, reliable, and ready for the next round.

And these patterns aren’t just for big companies. Even a small side project benefits from being easy to restart, scale, and debug.

Conclusion

The lessons from nursery school (clean up your mess, put things back, share everything, listen for the teacher) are the same lessons that keep our web servers healthy. With Go’s context and careful resource management, we make sure our playground is always ready for the next game.

Read the Full Code

Want to see the complete implementation?

Check out the repo: github.com/ekediala/resource_cleanup

From ReadAll to CopyBuffer: A Go Developer’s Guide to Efficient Data Copying

Eke Enyinnaya Diala — Tue, 15 Jul 2025 21:37:59 +0000

A practical, evidence-based guide to choosing between io.ReadAll, io.Copy, and io.CopyBuffer for file and stream operations in Go.

Introduction

When working with files, network streams, or other I/O in Go, developers often face a choice:

Should you load all data into memory, or process it as a stream?

This article presents real benchmark and memory profiling results for Go’s core data copying strategies, clarifies their tradeoffs, and offers clear guidance for real-world applications.

Approaches Compared

1. `io.ReadAll`: Load Everything Into Memory

func readBodyIoReadAll(r io.Reader, w io.Writer) error {
    b, err := io.ReadAll(r)
    if err != nil {
        return fmt.Errorf("reading data: %w", err)
    }

    // process data. we just use a write here as a placeholder
    if _, err := w.Write(b); err != nil {
        return fmt.Errorf("writing data: %w", err)
    }
    return nil
}

Reads the entire input into memory before writing.
Simple and idiomatic when you need the whole payload at once.

2. `io.Copy`: Stream Data in Chunks

func readBodyBuffered(r io.Reader, w io.Writer) error {
    _, err := io.Copy(w, r)
    if err != nil {
        return fmt.Errorf("reading data: %w", err)
    }
    return nil
}

Streams data using a fixed-size buffer (default: 32KB).
Memory usage is constant, regardless of file size.

3. `io.CopyBuffer` + `sync.Pool`: Optimized Streaming

var bufferPool = sync.Pool{
    New: func() any {
        return make([]byte, 32*1024)
    },
}

func readBodyBufferedPool(r io.Reader, w io.Writer) error {
    buf := bufferPool.Get().([]byte)
    defer bufferPool.Put(buf)
    _, err := io.CopyBuffer(w, r, buf)
    if err != nil {
        return fmt.Errorf("reading data: %w", err)
    }
    return nil
}

Reuses buffers across operations.
Reduces allocations in high-throughput or concurrent scenarios.

Benchmark Results

Benchmarks were run using Go’s testing package on an Apple M1 Pro (darwin/arm64):

goos: darwin
goarch: arm64
cpu: Apple M1 Pro

Benchmark	ns/op	B/op	allocs/op
IoReadAll_SmartRaise_Dash.jpg	1081	519	1
io.Copy_SmartRaise_Dash.jpg	2908	32784	3
io.CopyBuffer_SmartRaise_Dash.jpg	2975	32784	3
IoReadAll_README.md	1082	512	1
io.Copy_README.md	2915	32784	3
io.CopyBuffer_README.md	2870	32784	3

Key Takeaways:

io.ReadAll is extremely fast and allocates very little memory for small files.
io.Copy and io.CopyBuffer are slightly slower and always allocate a 32KB buffer, regardless of file size.
All approaches have minimal allocations, but streaming methods always allocate at least the buffer.

Memory Profiling Results

Memory profiling with pprof (using a 1.4mb file) showed:

io.ReadAll allocates memory proportional to the file size.
io.Copy and io.CopyBuffer (file-to-file) may show zero Go-level allocations due to OS-level optimizations (e.g., sendfile).
When copying to a buffer (not a file), allocations are visible and reflect buffer growth.

When to Use Each Approach

Use `io.ReadAll` when:

Your logic requires the entire payload in memory (e.g., validation, transformation, scanning, or random access).
The data fits comfortably in memory (small/medium files or payloads).

Use streaming (`io.Copy`, `io.CopyBuffer`) when:

You can process data incrementally and do not need the whole payload at once.
You are saving uploads directly to disk, forwarding to another service, or building pipelines.
Memory efficiency is critical, especially for large files or high concurrency.

Summary:
Choose io.ReadAll for scenarios where full in-memory access is required, and prefer streaming for large data or when memory efficiency is critical and whole-data access is unnecessary.

Benchmarking vs. Profiling: What’s the Difference?

Benchmarking (via go test -bench) measures per-operation speed and allocations in isolation.
Memory profiling (via pprof) shows real-world, end-to-end memory usage, including effects of OS-level optimizations and actual file I/O.
Both are essential: benchmarks reveal micro-level costs, while profiling shows whole-program behavior.

Practical Guidance

Profile and benchmark with real workloads: Synthetic tests can be misleading; always measure with realistic scenarios.
io.ReadAll is efficient for small/medium data: It’s fast and allocates only what’s needed.
Streaming (io.Copy, io.CopyBuffer) is best for large data: Memory usage is constant and independent of file size.
OS-level optimizations matter: File-to-file copying may use zero Go memory due to system calls like sendfile.
Buffer reuse (sync.Pool) helps at scale: For high-throughput or concurrent workloads, pooling buffers can further reduce allocations.

Analysis conducted with Go 1.24.2 on macOS (Apple M1 Pro). Results may vary across different operating systems and Go versions. Source code and profiles available in this repository.

Complete Memory Profile Data

For those interested in the raw memory profiling results, here are the pprof outputs used in this analysis:

io.ReadAll Profile (readall.prof)

File: bin
Type: inuse_space
Time: 2025-07-15 21:53:20 WAT
Showing nodes accounting for 2468.87kB, 100% of 2468.87kB total
      flat  flat%   sum%        cum   cum%
    1026kB 41.56% 41.56%     1026kB 41.56%  runtime.allocm
  930.82kB 37.70% 79.26%   930.82kB 37.70%  io.ReadAll
  512.05kB 20.74%   100%  1442.87kB 58.44%  runtime.main
         0     0%   100%   930.82kB 37.70%  main.main
         0     0%   100%   930.82kB 37.70%  main.readBodyIoReadAll

Function-level allocation:
ROUTINE ======================== main.readBodyIoReadAll
         0   930.82kB (flat, cum) 37.70% of Total
         .   930.82kB     20:   b, err := io.ReadAll(r)

io.Copy Profile (copy.prof)

File: bin
Type: inuse_space
Time: 2025-07-15 21:53:25 WAT
Showing nodes accounting for 1546.28kB, 100% of 1546.28kB total
      flat  flat%   sum%        cum   cum%
 1033.28kB 66.82% 66.82%  1033.28kB 66.82%  runtime.procresize
     513kB 33.18%   100%      513kB 33.18%  runtime.allocm

No application-level allocations visible - OS optimization in effect

sync.Pool Profile (syncpool.prof)

File: bin
Type: inuse_space
Time: 2025-07-15 21:53:30 WAT
Showing nodes accounting for 513kB, 100% of 513kB total
      flat  flat%   sum%        cum   cum%
     513kB   100%   100%      513kB   100%  runtime.allocm

Minimal runtime overhead only - perfect buffer reuse

Note: Only the three profiles above were actually generated from the test code. All analysis is based on these real measurements.

Testing HTTP Requests with http.RoundTripper

Eke Enyinnaya Diala — Fri, 04 Jul 2025 14:06:56 +0000

Last year at work we were working on a B2B SAAS product that other businesses could build upon by calling our APIs. We handle some of the requests and for some we forward the requests to an external provider. Our requirements meant we had to have a very robust testing suite. However, to avoid running out of API credits, rate limit issues, and for our tests to be resilient and fast we have to mock the external API calls. As a recovering JavaScripter, I did the reasonable thing: pull in a library, this time httpmock. This is not a criticism of httpmock, I love and use it extensively.

Recently, I read GoByExample and learned a different way to mock the http calls using the http.RoundTripper interface. The interface is tailor made for this purpose:

RoundTripper is an interface representing the ability to execute a
single HTTP transaction, obtaining the [Response] for a given
[Request].

This tells us we can decide how we want our http.Client to process our requests. Say, we have a function named pingUrl that sends a head request to the provided URL that we need to test:

func pingUrl(ctx context.Context, url string, client *http.Client) error {
    req, err := http.NewRequestWithContext(ctx, http.MethodHead, url, nil)
    if err != nil {
        return fmt.Errorf("setting up request: %w", err)
    }

    res, err := client.Do(req)
    if err != nil {
        return fmt.Errorf("sending request to %s: %w", url, err)
    }
    defer res.Body.Close()

    if res.StatusCode >= http.StatusBadRequest {
        buf := &bytes.Buffer{}
        _, err = io.Copy(buf, res.Body)
        if err != nil {
            return fmt.Errorf("copying response from %s: %w", url, err)
        }
        return fmt.Errorf("%s", buf.String())
    }

    return nil
}

We could also write it without explicitly passing in a http.Client pointer:

func pingUrl(ctx context.Context, url string) error {
    client := http.DefaultClient
    req, err := http.NewRequestWithContext(ctx, http.MethodHead, url, nil)
    if err != nil {
        return fmt.Errorf("setting up request: %w", err)
    }

    res, err := client.Do(req)
    if err != nil {
        return fmt.Errorf("sending request to %s: %w", url, err)
    }
    defer res.Body.Close()

    if res.StatusCode >= http.StatusBadRequest {
        buf := &bytes.Buffer{}
        _, err = io.Copy(buf, res.Body)
        if err != nil {
            return fmt.Errorf("copying response from %s: %w", url, err)
        }
        return fmt.Errorf("%s", buf.String())
    }

    return nil
}

The first function is preferred because it has explicit dependencies and therefore easier to test. We can "inject" different clients with different roundTrippers depending on context. It is also faster because the http client can reuse existing connections to make requests to the same URLs. There are different ways to test this function:

With a mocking library

If you are from the Javascript ecosystem, this is probably what you do. We have a mocking library intercept the http request and respond with what we want. This is simple and effective. It has the added benefit of having a built in clean up and ease of setting up responders. We can also reduce the number of arguments we pass to the function (remove the "client", it's cleaner 😁).

func TestPingURLWithMockLibrary(t *testing.T) {
    client := &http.Client{}

    t.Run("mock success", func(t *testing.T) {
        httpmock.Activate(t)
        defer httpmock.DeactivateAndReset()
        // Exact URL match
        httpmock.RegisterResponder(http.MethodHead, url,
            httpmock.NewStringResponder(http.StatusOK, `[{"id": 1, "name": "My Great Article"}]`))
        if err := pingUrl(context.Background(), url, client); err != nil {
            t.Fatalf("expected error to be nil, got %v", err)
        }
    })

    t.Run("mock failure", func(t *testing.T) {
        httpmock.Activate(t)
        defer httpmock.DeactivateAndReset()
        // Exact URL match
        httpmock.RegisterResponder(http.MethodHead, url,
            httpmock.NewStringResponder(http.StatusBadRequest, `[{"id": 1, "name": "My Great Article"}]`))

        if err := pingUrl(context.Background(), url, client); err == nil {
            t.Fatal("expected error to not be nil")
        }
    })
}

With a http.RoundTripper

Alternatively, we could dispense with the external dependency and have more control over the request. This is where http.RoundTripper comes in. We pass a client with a custom roundtripper and then test the behaviour we want. As Casey Muratori makes clear every now and again, the effect of dependencies on the reliability of your software cannot be overstated. Each dependency we add makes our software disproportionately flakier. However, this is not just about reducing dependencies, it is also about reducing "magic" and simplicity.

type Tripper func(*http.Request) (*http.Response, error)

func (t Tripper) RoundTrip(request *http.Request) (*http.Response, error) {
    return t(request)
}

var successTripper = Tripper(func(*http.Request) (*http.Response, error) {
    return &http.Response{StatusCode: http.StatusOK}, nil
})

var failureTripper = Tripper(func(*http.Request) (*http.Response, error) {
    body := io.NopCloser(strings.NewReader(`{"error": "something went wrong"}`))
    return &http.Response{
        StatusCode: http.StatusBadRequest,
        Body:       body,
    }, nil
})

var url = "https://api.mybiz.com/articles"

func TestPingURLWithRoundTripper(t *testing.T) {
    client := http.Client{}

    t.Run("roundtrip success", func(t *testing.T) {
        client.Transport = successTripper
        if err := pingUrl(context.Background(), url, &client); err != nil {
            t.Fatalf("expected error to be nil, got %v", err)
        }
    })

    t.Run("roundtrip failure", func(t *testing.T) {
        client.Transport = failureTripper
        if err := pingUrl(context.Background(), url, &client); err == nil {
            t.Fatal("expected error to not be nil")
        }
    })
}

It looks like more code for the same functionality but is it? We are essentially swapping out multi-purpose code written by someone else for our own custom code written for our own particular use case. We can create the helpers if we want.

Anywho, I asked our AI overlords what they thought and here from Claude 4:

Use httpmock when you need:

Complex URL pattern matching

Request verification/counting

Testing multiple endpoints in the same test

Your team prefers declarative mocking

You're already using it extensively

Use RoundTripper when you want:

Zero dependencies

Maximum control and transparency

Simple, focused unit tests

Performance-critical test suites

I'd say that's pretty good advice. Ciao!

TDD and Software Design

Eke Enyinnaya Diala — Fri, 13 Jun 2025 15:59:08 +0000

Last week I was watching a video on YouTube. It was a conversation about Clean Code with Primagen, Teej, Casey Muratori, and Carson Gross. The conversation touched test driven development and they offered their opinions which I won't rehash here. Watch the video, it is as informative as it is entertaining.

After watching the video, I went back to reading "The Go Programming Language". I am fairly proficient in Go but I want to have an even better understanding of the language. In Chapter 4, there was a function called dedup. Please keep in mind that I am not criticising this bit of code, it works and it was used by the author for illustrative purposes:

func main() {
    seen := make(map[string]bool) // a set of strings
    input := bufio.NewScanner(os.Stdin)
    for input.Scan() {
        line := input.Text()
        if !seen[line] {
            seen[line] = true
            fmt.Println(line)
        }
    }

    if err := input.Err(); err != nil {
        fmt.Fprintf(os.Stderr, "dedup: %v\n", err)
        os.Exit(1)
    }
}

My policy around reading programming texts is to write out every bit of code in it so I set about rewriting this function but I had one tiny problem. I was tired of remembering how to run each function. This code is in a main function which means I have to remember what folder it is in to run it. I have been doing this for 4 chapters now and I am just tired of typing out the file path. I decided to instead write tests for it so I just need to remember the test name. Another upside to using tests to test code is the ease of adding more test cases.

This decision exposed me to one of the interesting positives of test driven development: Code Modularity. Ideally, we want our software components to be independent modules that we can compose in different ways to achieve our business goals. To test the code above is going to be difficult as it is tightly coupled with standard input and error; you have to mock os.Stdin or do some other shenanigan to capture what is inside it. Writing the test for the function however forced me to rethink the structure of the function. What does this function really do and does it really care about os.Stdin and os.Stderr? The answer to the second question is an emphatic NO. The answer to the first question is we just want something to read from and another thing to write to, ergo, a reader and a writer. See alternative implementation:

func dedup(r io.Reader, w io.Writer) error {
    seen := make(map[string]bool)
    scanner := bufio.NewScanner(r)

    for scanner.Scan() {
        line := scanner.Text()
        if !seen[line] {
            seen[line] = true
            fmt.Fprint(w, line)
        }
    }

    return scanner.Err()
}

Same functionality but decoupled from its dependencies using interfaces and a bit less fragile. It is self contained and can be used with any reader or writer. Making changes to the code can be done with a bit more confidence plus we do not have to manually run the code to see check for regression. Testing it becomes easy because you can make any reader and writer and it works:

func TestDedup(t *testing.T) {
    data := "hello"
    r := &bytes.Buffer{}
    r.WriteString(data)
    r.WriteByte('\n')
    r.WriteString(data)

    w := &bytes.Buffer{}
    if err := dedup(r, w); err != nil {
        t.Fatalf("dedup() expected nil error, got %v", err)
    }

    if count := strings.Count(w.String(), "hello"); count != 1 {
        t.Errorf("dedup() writes %s, want %s", w.String(), data)
    }
}

In my opinion, this is one of those situations where TDD shines. The process of thinking about how to test a function facilitates better design decisions. As always, there are no right or wrong answers regarding TDD, just trade offs and use cases that fit or not.

Thanks for reading.

Streaming Large Files Between Microservices: A gRPC Implementation

Eke Enyinnaya Diala — Fri, 16 May 2025 14:19:02 +0000

Efficient Large Data Transfer with gRPC Streaming

This project demonstrates an efficient method for transferring large files or data payloads between microservices using gRPC server streaming in Go. It serves as a gRPC-based alternative to traditional HTTP Range request solutions (like the one detailed in https://dev.to/dialaeke/streaming-large-files-between-microservices-a-go-implementation-4n62). It directly addresses the challenge posed by Sumit Mukhija in this tweet:

"Your microservice needs to transfer large amounts of data (e.g., files, images) between services. How do you design the communication to avoid performance bottlenecks and manage large payloads efficiently?"

The core challenge, often faced in microservice architectures, is transferring significant amounts of data (files, images, datasets) without causing performance bottlenecks, excessive memory consumption, or network congestion. This implementation leverages gRPC's strengths to provide a robust and efficient solution.

See project repo here

Why gRPC for Large File Transfers?

While HTTP Range requests offer a viable solution, gRPC provides several advantages in this context:

Native Streaming: gRPC has first-class support for server-side streaming, making it natural to send data in chunks without complex application-level logic for chunk management.
Efficiency (HTTP/2): gRPC typically runs over HTTP/2, benefiting from features like multiplexing and header compression, which can improve network efficiency compared to HTTP/1.1.
Type Safety & Schema Definition: Using Protocol Buffers (.proto files) enforces a clear contract between client and server, reducing integration errors.
Performance: The binary nature of Protocol Buffers and the efficiency of HTTP/2 often lead to better performance than text-based protocols like JSON over HTTP/1.1.

This project specifically showcases:

Memory Efficiency: Avoids loading large files into memory on either client or server using streaming and efficient buffer management.
Resumable Downloads: Client tracks progress and requests the next chunk from the correct offset.
Performance: Optimizes I/O using io.CopyN and pooled bytes.Buffer.

Key Features (gRPC Implementation)

gRPC Server Streaming: The server streams file chunks to the client using a single RPC call (StreamFile).
Protocol Buffers: Defines the service contract (TransferService) and message formats (StreamFileRequest, StreamFileResponse, etc.) in proto/transfer.proto.
Chunked Transfer: Data is inherently sent in chunks managed by the server loop. Chunk size is configurable but capped on the server.
Resumable Downloads: The client requests the download starting from the last received byte offset (Start field in StreamFileRequest).
Memory Efficiency: Uses a sync.Pool of bytes.Buffer on the server combined with io.CopyN to efficiently read and buffer data without excessive allocations. The client uses io.Copy for efficient disk writes.
Type Safety: Leverages Go types generated from the .proto definition.
Selective gzip compression: The server intelligently compresses individual chunks based on file type, only when beneficial and supported by the client.

How it works

The client sends the first RPC call to get the file size. This is useful to implement resumability and to ensure we do not waste bandwidth on already downloaded files:

// os.O_APPEND allows us to append to the end of the file without
    // manually seeking to it
    file, err := os.OpenFile(fileName, os.O_CREATE|os.O_RDWR|os.O_APPEND, 0666)
    if err != nil {
        exit("os.OpenFile", err)
    }
    defer file.Close()

    info, err := file.Stat()
    if err != nil {
        exit("file.Stat", err)
    }

    fileSizeCtx, cancel := context.WithTimeout(ctx, time.Second*10)
    defer cancel()

    client := transferpb.NewTransferClient(conn)
    res, err := client.GetFileSize(fileSizeCtx, &transferpb.GetFileSizeRequest{FileName: fileName})
    if err != nil {
        exit("client.GetFileSize()", err)
    }

    if info.Size() >= res.Size {
        slog.Info("file already downloaded")
        return
    }

Now, we know we haven't downloaded the file before, we can make the second RPC call to stream the data from the server. We set up the stream and inform the server where we want to resume the download from, our preferred chunk size, the file name, and if we support compression.

stream, err := client.StreamFile(ctx, &transferpb.StreamFileRequest{
        Start:         info.Size(),
        ChunkSize:     ChunkSize,
        FileName:      fileName,
        CanDecompress: decompress,
    })
    if err != nil {
        exit("client.StreamFile", err)
    }

Then we start streaming. We use io.Copy to write to the file in order to take advantage of its internal buffer. This helps us reduce the number of syscalls we make while writing to the file. This allows us to optimize CPU usage.

for {
        r, streamErr = stream.Recv()
        if streamErr != nil {
            break
        }

        // if file is compressed, we decompress it first before reading
        if r.Compressed {
            gr, err := gzip.NewReader(bytes.NewReader(r.Chunk))
            if err != nil {
                slog.Error("gzip decompression failed", "offset", bytesDownloaded, "err", err)
                exit("gzip.NewReader", err)
            }
            defer gr.Close()
            chunkReader = gr
        } else {
            chunkReader = bytes.NewReader(r.Chunk)
        }

        _, streamErr = io.Copy(file, chunkReader)
        if streamErr != nil {
            break
        }

    }

Server

Client sends us an RPC call and we respond with the size of the file. This response is used by the client for different purposes but mainly to know if this is a partially downloaded file so we can resume the download. It can theoretically be used to choose the optimal chunk size to optimize the overhead of network requests.

func (t *TransferService) GetFileSize(ctx context.Context, req *transferpb.GetFileSizeRequest) (*transferpb.GetFileSizeResponse, error) {
    if strings.Contains(req.FileName, "..") {
        return nil, status.Error(codes.InvalidArgument, "Invalid file name")
    }

    dir, err := os.Getwd()
    if err != nil {
        return nil, status.Error(codes.Internal, err.Error())
    }

    fileName := path.Join(dir, ContentFolderName, req.FileName)
    file, err := os.Open(fileName)
    if err != nil {
        return nil, status.Error(codes.InvalidArgument, err.Error())
    }
    defer file.Close()

    info, err := file.Stat()
    if err != nil {
        return nil, status.Error(codes.Internal, err.Error())
    }

    return &transferpb.GetFileSizeResponse{Size: info.Size()}, nil
}

The client can now send us a subsequent RPC call to ask us to stream the file to them. The client will tell us the name of the file, where to resume the stream from, their preferred chunk size, and if they support compression. To keep the system memory efficient, we do not always use the client's preferred chunk size. We have a max chunk size that we allow.

To reduce memory consumption and reduce GC pressure, we use a sync pool for our buffers. The pool allows us to reuse allocated buffers. The buffers are the temporary data structures we use to extract chunks of the file before sending it to the stream. These buffers are key to ensuring we never read all of the file into memory at any time.

Define pool

var bufPool = sync.Pool{
    New: func() any {
        return &bytes.Buffer{}
    },
}

Each time a client request comes in, we go to our pool to "get" a buffer. After streaming we put the buffer back in the pool so other requests can reuse them. I have read that this is not a silver lining so use these pools with caution. The buffer we get has a default size. This is by design. We want to be precise in our allocations, we do not want to over-allocate, or allocate multiple times. We are going to allocate precisely what we need depending on the chunk size.

buf, _ := bufPool.Get().(*bytes.Buffer)
    defer func() {
        buf.Reset()
        bufPool.Put(buf)
    }()

chunkSize := min(req.ChunkSize, MaxChunkSize)
    buf.Grow(int(chunkSize))

Now, we begin to read the file in chunks. Each chunk we read we send to the client. If the file benefits from compression, we compress it before sending:

for {
        if shouldCompress {
            w.Reset(buf) // reset gzipWriter
            n, err := copyData(w)
            if err != nil && n <= 0 {
                if err == io.EOF {
                    return nil
                }
                return status.Error(codes.Internal, err.Error())
            }

            err = w.Close() // flush contents to buf
            if err != nil {
                return status.Error(codes.Internal, err.Error())
            }

            err = sendResponse(true)
            if err != nil {
                if err == io.EOF {
                    return nil
                }
                return status.Error(codes.Internal, err.Error())
            }

        } else {
            n, err := copyData(buf)
            if err != nil && n <= 0 {
                if err == io.EOF {
                    return nil
                }
                return status.Error(codes.Internal, err.Error())
            }

            err = sendResponse(false)
            if err != nil {
                if err == io.EOF {
                    return nil
                }
                return status.Error(codes.Internal, err.Error())
            }
        }

        buf.Reset()
    }

The key to all these working is the copyData function. This is where we efficiently read data from the file into our buffer for transmission. Because io.CopyN has an internal buffer, we leverage it to efficiently read from the file system. In each iteration, we efficiently read the specified chunk size amount of data into the our buffer and send it to the client. This way we again reduce the number of syscalls due to file reads while never reading the entire file into memory.

copyData := func(w io.Writer) (int64, error) {
        n, err := io.CopyN(w, file, int64(chunkSize))
        if err != nil && n <= 0 {
            if err == io.EOF {
                return n, io.EOF
            }
            return n, status.Error(codes.Internal, err.Error())
        }
        return n, nil
    }

Memory Efficiency: Real-World Profiling

To validate the efficiency of this approach, we profiled the server's memory usage during the transfer of a 2GB+ file. The results were outstanding:

Showing nodes accounting for 512.05kB, 100% of 512.05kB total
      flat  flat%   sum%        cum   cum%
  512.05kB   100%   100%   512.05kB   100%  google.golang.org/grpc/internal/transport.(*writeQuota).get (inline)
         0     0%   100%   512.05kB   100%  ... (other stack frames)

What does this mean?

Extremely Low Memory Usage:

Even when transferring a file over 2GB in size, the server's peak memory usage attributable to the transfer was only about 512KB. This confirms that the implementation streams data in small chunks and never loads the entire file into memory.
No Memory Leaks:

The profile shows no evidence of memory leaks or excessive buffering. All allocations are related to gRPC's internal flow control, which is expected and minimal.
Scalability:

Because memory usage does not scale with file size, this service can handle many concurrent large transfers without running into memory exhaustion.

Conclusion

This project demonstrates how gRPC server streaming, combined with efficient buffer management (sync.Pool, bytes.Buffer, io.CopyN), provides a powerful and performant alternative to HTTP-based methods for large file transfers between services. It offers type safety, potential performance benefits from HTTP/2, and a natural way to handle streaming data.

Streaming Large Files Between Microservices: A Go Implementation

Eke Enyinnaya Diala — Fri, 04 Apr 2025 11:30:50 +0000

Efficient Large Data Transfer Between Microservices (Go Demo)

This project provides a practical demonstration in Go for efficiently transferring large files or data payloads between microservices. It seeks to directly address the challenge posed by Sumit Mukhija in this tweet:

"Your microservice needs to transfer large amounts of data (e.g., files, images) between services. How do you design the communication to avoid performance bottlenecks and manage large payloads efficiently?"

This implementation showcases a solution using standard HTTP protocols, focusing on:

Memory Efficiency: Avoids loading entire large files into memory on either the client or server side.
Bandwidth Efficiency: Uses selective compression to reduce network traffic for suitable content types.
Resilience: Implements resumable downloads to handle network interruptions gracefully.
Performance: Optimizes I/O using buffered readers/writers and standard library features.

The entire code for this implementation can be found here.

Key Features

Chunked Transfers: Uses HTTP Range requests to transfer data in manageable chunks (configurable size).
Resumable Downloads: The client can resume downloads from where they left off if interrupted.
Selective Gzip Compression: The server intelligently compresses individual chunks based on content type and chunk size, only when beneficial and supported by the client (Accept-Encoding: gzip).
Memory Efficiency: Leverages io.Reader, io.Writer, io.LimitReader, and bufio for streaming data without high memory allocation.
Standard HTTP Protocols: Relies entirely on standard HTTP/1.1 features (HEAD, GET, Range headers, Content-Encoding) for maximum interoperability.
Graceful Shutdown: Both client and server handle OS interrupt signals (SIGINT, SIGTERM, SIGHUP) for clean termination.
Content Type Detection: Server uses file extension and content sniffing (http.DetectContentType) to determine MIME types for compression decisions.

How it Works

PS: A lot of folks in the replies to the tweet suggested different variations of "Put it on S3" and while I do not disagree- I actually do agree it is quite an efficient solution- but there's no fun in that and it doesn't relieve you of the JIRA ticket for getting the file from one service to another [S3 is a service].

Server

Client sends us a Head request and we respond with the size of the file. This response is used by the client for different purposes but mainly to know if this is a partially downloaded file so we can resume the download. It can theoretically be used to choose the optimal chunk size to optimize the overhead of network requests.

fileName := r.PathValue("fileName")
    // ensure this request is not trying to do something fishy
    if strings.Contains(fileName, "..") {
        http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest)
        return
    }

    dir, err := os.Getwd()
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    fileName = path.Join(dir, ContentFolderName, fileName)
    file, err := os.Open(fileName)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    stat, err := file.Stat()
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    // if it is a head request, we send back the file size. The client will use
    // that data for resumability i.e to tell us if parts of the file have already
    // been downloaded and therefore where to resume from. The client can also use
    // the size to decide how to chunk the data to achieve a balance between number
    // of http calls and download speed per chunk.
    if r.Method == http.MethodHead {
        w.Header().Set("Content-Length", fmt.Sprintf("%d", stat.Size()))
        return
    }

The client now sends further GET requests to the same endpoint with a range header. This range header is what we will use -like my favorite software, the browser- to stream the response back to the client:

// Parse range header (required for our implementation)
    rangeHeader := r.Header.Get("Range")
    if rangeHeader == "" {
        http.Error(w, "Range header required", http.StatusBadRequest)
        return
    }

    // Parse the range
    var start, end int64
    n, err := fmt.Sscanf(rangeHeader, "bytes=%d-%d", &start, &end)
    if err != nil || n != 2 {
        http.Error(w, "Invalid range format", http.StatusBadRequest)
        return
    }

    // Validate range
    if start < 0 || end < start || end >= stat.Size() {
        http.Error(w, "Invalid range", http.StatusRequestedRangeNotSatisfiable)
        return
    }

    chunkSize := end - start + 1
    contentType := getContentType(fileName, file)
    w.Header().Set("Content-Type", contentType)

Now, that we have parsed the range headers and have the chunk size, this is where it gets interesting. We have to find the requested chunk in the file but we must also ensure not to put the entire chunk in memory. We also have to make a decision on compression. If it is a compression friendly file eg., text based files, we compress the chunk else we just stream the chunk back to the client:

// Set the file offset to the provided start point.
    // We do not want to read from the start of the file.
    // We want to "resume" from where they stopped
    _, err = file.Seek(start, io.SeekStart)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    // Create a limited reader for just this chunk.
    // Why? This is exactly why it exists. Alternatively, we could create a
    // buffer to minimize syscalls but it won't be necessary here.
    // We use io.copy to stream the response to the client. Given we also don't
    // know what is contained in the file, we will have to read the whole chunk size
    // into memory:
    //
    // data := make([]byte, chunkSize)
    // reader := bufio.NewReader(file)
    // n, _ := reader.Read(data)
    //
    // we have already added 516kb of memory to the program [512 for chunk size
    // and 4kb for the buffer] and lost the benefits of streaming. Just 10 concurrent
    // requests and we are already at 5mb of memory.
    //
    // Meanwhile we already have io.Copy with its buffer that streams 32kb chunks from the file to the connection.
    reader := io.LimitReader(file, chunkSize)

Now, we make a decision on compression. First, we check if the client supports compression- we use just gzip to keep it simple- and the file is compression friendly, and it is above 8kb which is the threshold where compression has benefits (according to our AI overlords btw. Gemini, GPT 4, and Claude 3.7 Sonnet all said the same thing so it must be true). If it is compression friendly, we compress it and use io.Copy to stream the chunk from the file (through the LimitReader) to the client. At no point, do we have the entire chunk or file in memory or in transit. The LimitReader will read from the file up till the chunk size and then return EOF.

acceptsGzip := strings.Contains(r.Header.Get("Accept-Encoding"), "gzip")
    shouldCompress := acceptsGzip &&
        isCompressibleType(contentType) &&
        chunkSize >= MinCompressionSize // Only compress chunks >= 8KB

    if shouldCompress {
        // For compressed chunks
        w.Header().Set("Content-Encoding", "gzip")
        // Cannot predict final Content-Length after compression
        // Set partial content status
        w.Header().Set("Content-Range", fmt.Sprintf("bytes %d-%d/%d", start, end, stat.Size()))
        w.WriteHeader(http.StatusPartialContent)

        // Create gzip writer with fast compression
        gz, err := gzip.NewWriterLevel(w, gzip.BestSpeed)
        if err != nil {
            http.Error(w, err.Error(), http.StatusInternalServerError)
            return
        }
        defer gz.Close()

        // Send compressed chunk
        _, err = io.Copy(gz, reader)
        if err != nil {
            http.Error(w, err.Error(), http.StatusInternalServerError)
            return
        }
        return
    }

    // For uncompressed chunks
    w.Header().Set("Content-Length", fmt.Sprintf("%d", chunkSize))
    w.Header().Set("Content-Range", fmt.Sprintf("bytes %d-%d/%d", start, end, stat.Size()))
    w.WriteHeader(http.StatusPartialContent)

    // Send uncompressed chunk
    _, err = io.Copy(w, reader)
    if err != nil {
        log.Printf("Error sending chunk: %v", err)
        return
    }

Client

The client is a simpler implementation. We get the file size from the server with a HEAD request and check to see if it is a fresh download or a partial download.

fileName := r.PathValue("fileName")
    if strings.Contains(fileName, "..") {
        http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest)
        return
    }

    file, err := os.OpenFile(fileName, os.O_CREATE|os.O_RDWR, 0666)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    defer file.Close()

    // Get file info to check existing size
    stat, err := file.Stat()
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    fileSize := stat.Size()
    client := http.DefaultClient
    url := fmt.Sprintf("%s/download/%s", serviceUrl, fileName)

    // make head request to get the file size. this helps with resumability
    req, err := http.NewRequestWithContext(r.Context(), http.MethodHead, url, nil)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    res, err := client.Do(req)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    defer res.Body.Close()

    if res.StatusCode != http.StatusOK {
        var b bytes.Buffer
        _, err := io.Copy(&b, res.Body)
        if err != nil {
            http.Error(w, err.Error(), http.StatusInternalServerError)
            return
        }

        http.Error(w, b.String(), http.StatusInternalServerError)
        return
    }

    totalSize := res.ContentLength
    if fileSize >= totalSize {
        w.Write([]byte("File already downloaded"))
        return
    }

Now, we also want to stream the chunks to the file. We will use a 64kb buffer to do this. 64kb is a small price to pay for the chance to reduce our syscalls for writing to the file by half. We have to benchmark this to know but for now I am using my intuition and I also asked our AI overlords to confirm my bias [systematic error as the late Daniel Kahnemann calls it]. This is also where we add resumability. We are going to seek to the last place the file was written to.

    // set file offset to last offset we wrote to, this is where
    // we do resumability
    _, err = file.Seek(fileSize, io.SeekStart)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    // create a 64kb buffer to stream the response to the file
    // why? we want to minimize the amount of system calls we make
    // we want to hit the sweet spot between memory consumption 
    // and cpu usage.
    writer := bufio.NewWriterSize(file, bufferSize)
    defer writer.Flush()

Finally, we request those chunks in multiple GET requests.

// we make http requests to get each chunk of data
    for start := fileSize; start < totalSize; start += chunkSize {
        end := start + chunkSize - 1
        if end > totalSize {
            end = totalSize - 1
        }

        statusCode, err := downloadChunk(r.Context(), client, writer, url, start, end)
        if err != nil {
            http.Error(w, err.Error(), statusCode)
            return
        }
    }

There was a temptation to parallelize this with SectionReaders but I decided against it because I can, the complexity tradeoff wasn't worth it for me, and getting it to work with resumability is beyond my skillset- so yeah me, me, me.

Alternative Approaches

gRPC streaming
Message queues with chunked messages
S3/blob storage as intermediary
Dedicated file transfer protocols

Why the HTTP approach was chosen

My skill issues.

Thanks for reading. Comments and opinions [that argue in my favour] appreciated.

Day 3 of Elm- Records

Eke Enyinnaya Diala — Thu, 04 Aug 2022 23:55:00 +0000

Records

According to Richard Feldman,

Records are plain, immutable data.

They are syntactically similar to Javascript objects but not exactly the same. They have no prototype and no this for example. What I really want to talk about is record updates.

Record update simply refers to how we mutate the properties of a record in Elm. The term "mutate" here is sort of a misnomer. A record update does not mutate the old record, they instead create new records based off of the old record. For example, we have this bit of elm code:

record = { a = 1, b = 2 }

newRecord = { record | a = 5 }

The code above does not mutate record. It looks a lot like the . operator for mutating objects in Javascript but it is not the same. This is a lot closer to spreading record into newRecord and updating the fields you need. The way I think about this behaviour is along these lines in Javascript:

const record = Object.freeze({ a: 1, b: 2 });
const newRecord = Object.freeze({ ...record, a: 5 });

We are using Object.freeze here because we need the objects returned in both cases to be immutable. You see that in both cases, no actual mutation does happen. This is good because it saves us a lot of trouble with side effects resulting from passing values by reference.

This is very very interesting!

Day 2 of Elm- Partial Application

Eke Enyinnaya Diala — Wed, 03 Aug 2022 13:50:22 +0000

Today, we learned about Lists and Partial Application. Lists are similar to Javascript arrays except that they cannot contain elements of the same type. It is defined similarly as Javascript arrays with square brackets []. Boring stuff.

The interesting thing we learned today was Partial Application. A partially applied function is a function called with some of its arguments omitted. This is a fairly standard issue in Javascript, PHP, and I imagine other programming languages. What I find interesting is how Elm deals with the problem as a functional programming language.

The difficulty in learning Functional Programming as say, a Javascript or PHP developer, is the conflict between your mental model of different concepts and how they are actually applied in FP.

In Javascript, when an argument is omitted, the function uses a default argument in its place. The default argument is undefined [unless the developer specifies one] and the function executes with this default argument. Boring stuff. For example:

function pluralize(singular, plural, quantity){
  // ...
}

pluralize('boy', 'boys') // the third argument here will be `undefined`.

function pluralize(singular, plural, quantity = 1){
  // ...
}

pluralize('boy', 'boys') // the value 1 will be passed to pluralize as the third argument.

In Elm, the concept is entirely different. If you omit an argument, the function will be partially executed and another function will be returned to you that takes the missing argument and completes the evaluation. Let's see the same example in Elm:

pluralize singular plural quantity = 
  if quantity === 1 then
    singular
  else
    plural

pluralize "boy" "boys" -- will return an anonymous function that I will express as `f` that takes a number and finishes the evaluation of pluralize:

f quantity = pluralize "boy" "boys" quantity.

This is so interesting and different from what I am used to I have written various versions of this to make it stick.

Another example I adapted from the the intro to elm to make it sink in:

Elm is great. Pass it on!

Day 1 of Elm

Eke Enyinnaya Diala — Wed, 03 Aug 2022 13:14:42 +0000

Started learning functional programming in Elm yesterday and it has been fun. Tried learning with Lisp and Haskell but didn't make much progress. Decided to go with Elm and it has been easier and better for me.

The most difficult bit about transitioning into Functional Programming is wrapping your head around everything being an expression that evaluates to a result.

Conditionals

Take the concept of conditionals for example. In Javascript land, conditionals are statements but in Elm they are expressions. This means that conditionals in Elm must resolve (evaluate?) to a value and therefore must also have an else block. In a sense, as Richard Feldman explains, conditionals in elm are better thought about as ternary operators in Javascript. Since the conditional is an expression and must evaluate to a value, it makes sense that the else block must be defined.

so this block of code in Elm:

pluralise singular plural quantity = 
  if quantity == 1 then
    singular
  else 
    plural

and this in JS are equivalent:

const pluralize = (singular, plural, quantity) => quantity === 1 ? singular : plural;

but not exactly equivalent to this even if they achieve the same result:

const pluralize = (singular, plural, quantity) => {
  if (quantity === 1) return singular;
  return plural;
};

because in the above code you can omit the else block and the code will still compile and you will run into runtime exceptions later. Elm massively frowns at runtime exceptions. It does not allow its possibility.

Parentheses

The role of parenthesis in Elm is interesting. Here, I will just quote Richard Feldman:

Parenthesis disambiguate between nested expressions.

This is interesting because in Javascript, parenthesis can be used to disambiguate but they are also used for different other purposes including calling functions, creating functions, conditionals for loops and if statements, etc.

In Elm, it does what it does in Math; disambiguate between nested expressions.

That's all for yesterday. Will write about what I learned in Elm today later.

Using Chakra UI, React Hook Form, and Yup for React Form Validation.

Eke Enyinnaya Diala — Sun, 13 Dec 2020 12:59:17 +0000

When building our react applications, we more often than not have to validate user generated content before we send it to our api endpoints for processing and storage.

Recently, we started development on a product at my workplace. We decided to go with React. I insisted we use Chakra for our UI components. Partly because I am Nigerian and it absolutely gladdens my heart to see my compatriots doing great work and partly because Chakra is really good. For one, Chakra gives you accessibility by default. I cannot overemphasise how important that was for us.

One stumbling block with using Chakra was the lack of resources on the internet for when you inevitably run into trouble. This is understandable because it is still pretty young. This is my attempt to fill some of that gap.

Validating a login form with react-hooks-form, and yup.

I am assuming the reader has some knowledge setting up react. If you do not, please visit the react website for instructions on how to set up a react app.

Here is what we expect to achieve at the end of this:

After creating a new project with React, we need to install Chakra, React Hook form, and Yup. For that we open our terminal, navigate to our project's folder.

install chakra

npm i @chakra-ui/react @emotion/react @emotion/styled framer-motion

# or

yarn add @chakra-ui/react @emotion/react @emotion/styled framer-motion

install yup, react-hook-form, and @hookform/resolvers. We install @hookform/resolvers because react-hook-form supports multiple schema libraries besides yup.

npm i yup react-hook-form @hookform/resolvers

# or

yarn add yup react-hook-form @hookform/resolvers

next we wrap our app with the Chakra provider so we can use Chakra components in our app. This we will do in our app's entry point, index.tsx in my case.

import React from 'react';
import ReactDOM from 'react-dom';
import './index.css';
import App from './App';
import reportWebVitals from './reportWebVitals';
import { ChakraProvider } from '@chakra-ui/react';

ReactDOM.render(
  <React.StrictMode>
    <ChakraProvider>
      <App />
    </ChakraProvider>
  </React.StrictMode>,
  document.getElementById('root')
);

Now, we have our set up complete, let's now get down to building the login form.

First, we import the Form components provided by Chakra. These components give us nice, accessible form components. I won't go into explaining what each do because I do not want this to become very long. Please visit the Chakra documentation for more information on these components.

import {
  FormControl,
  FormLabel,
  FormErrorMessage,
  FormHelperText,
  Input,
  Button
} from '@chakra-ui/react';

Then we import yup, yupResolver, and useForm to manage validation for our forms. Again, please visit React Hook Form docs and Yup docs for more information on these imports.

import * as yup from 'yup';
import { useForm } from 'react-hook-form';
import { yupResolver } from '@hookform/resolvers/yup';

We create the form schema and the type definitions for our form inputs. The type definitions are for Typescript users.

const schema = yup.object().shape({
  email: yup.string().email().required(),
  password: yup.string().min(8).required(),
});

type LoginFormInputs = {
  email: string;
  password: string;
};

Now, let's use all of these to create our login form. We will register our inputs into react hook form using useForm. The reason is clearly stated in the docs:

One of the key concepts in React Hook Form is to register your uncontrolled component into the hook. This will make its value available for both the form validation and submission.

Note: Each field is required to have a unique name as a key for the registration process.

useForm returns to us an errors object containing our validation errors [if any], a handleSubmit function that takes callback which is executed if validation is successful, and a register function we use to register our inputs. We set the mode to onBlur so our inputs are validated when the user leaves the input field

const { register, handleSubmit, errors } = useForm<LoginFormInputs>({
    mode: 'onBlur',
    resolver: yupResolver(schema),
  });

Now, we register our inputs. We use FormControl because it:

provides context such as isInvalid, isDisabled, and isRequired to form elements.

We use errortext to display the errors from our errors object, if any.

<FormControl
        isInvalid={!!errors?.email}
        errortext={errors?.email?.message}
        p='4'
        isRequired
      >
        <FormLabel>Email</FormLabel>
        <Input type='email' name='email' placeholder='Email' ref={register} />
        <FormErrorMessage>{errors?.email?.message}</FormErrorMessage>
        <FormHelperText>
          We are obviously giving this straight to Facebook.
        </FormHelperText>
      </FormControl>

Here's what the full component looks like:

import {
  FormControl,
  FormLabel,
  FormErrorMessage,
  FormHelperText,
  Input,
  Button
} from '@chakra-ui/react';
import * as yup from 'yup';
import { useForm } from 'react-hook-form';
import { yupResolver } from '@hookform/resolvers/yup';

const schema = yup.object().shape({
  email: yup.string().email().required(),
  password: yup.string().min(8).required(),
});

type LoginFormInputs = {
  email: string;
  password: string;
};

export default function LoginForm() {
  const { register, handleSubmit, errors } = useForm<LoginFormInputs>({
    mode: 'onBlur',
    resolver: yupResolver(schema),
  });

  const onSubmit = (values: LoginFormInputs) => console.log(values);

  return (
    <form style={{ width: 350 }}>
      <FormControl
        isInvalid={!!errors?.email?.message}
        errortext={errors?.email?.message}
        p='4'
        isRequired
      >
        <FormLabel>Email</FormLabel>
        <Input type='email' name='email' placeholder='Email' ref={register} />
        <FormErrorMessage>{errors?.email?.message}</FormErrorMessage>
        <FormHelperText>
          We are obviously giving this straight to Facebook.
        </FormHelperText>
      </FormControl>
      <FormControl
        isInvalid={!!errors?.password?.message}
        errortext={errors?.password?.message}
        px='4'
        pb='4'
        isRequired
      >
        <FormLabel>Password</FormLabel>
        <Input
          ref={register}
          type='password'
          placeholder='Password'
          name='password'
        />
        <FormErrorMessage>{errors?.password?.message}</FormErrorMessage>
      </FormControl>
      <Button
        onClick={handleSubmit(onSubmit)}
        p='4'
        mx='4'
        mt='6'
        w='90%'
        colorScheme='blue'
        variant='solid'
        disabled={!!errors.email || !!errors.password}
      >
        Login
      </Button>
    </form>
  );
}

Let's import the LoginForm component into our App.tsx file so we can use it.

import { Center, Flex } from '@chakra-ui/react';
import LoginForm from './LoginForm';

function App() {
  return (
    <Flex justify='center' h='100vh' w='100vw' align='center'>
      <Center w='100%'>
        <LoginForm />
      </Center>
    </Flex>
  );
}

export default App;

And that's it folks. That is how we validate a form using React Hook Form, Yup, and Chakra.

You can try a live demo on the sandbox below:

Intro to Nuxt

Eke Enyinnaya Diala — Fri, 23 Aug 2019 13:16:42 +0000

Nuxt is a server side rendering framework built on top of Vue.

You might think 'but Vue is a framework, why a framework for a framework?'

The thing is Vue is beautiful and easy to pick up and use, but Vue Single Page Applications (SPA) -as other front end framework SPAs- have problems with Search Engine Optimization (SEO) because when a SPA url is visited, the server sends the index.html which has no content. The pages are rendered by Javascript on the browser. This reduces server load but search engine crawlers are not very good at waiting for JavaScript to render content.

Enter Server Side Rendering (SSR). SSR enables us to render our content on the fly before it hits the browser. When a user and / or crawler hits a server side rendered web application url, the server sends a fully rendered page the first time, subsequent navigation is done on the client as with regular SPAs.

This has lots of advantages and disadvantages.

Here from the Vue Docs:

Pros:

Better SEO, as the search engine crawlers will directly see the fully rendered page.

Note that as of now, Google and Bing can index synchronous JavaScript applications just fine. Synchronous being the key word there. If your app starts with a loading spinner, then fetches content via Ajax, the crawler will not wait for you to finish. This means if you have content fetched asynchronously on pages where SEO is important, SSR might be necessary.

Faster time-to-content, especially on slow internet or slow devices. Server-rendered markup doesn't need to wait until all JavaScript has been downloaded and executed to be displayed, so your user will see a fully-rendered page sooner. This generally results in better user experience, and can be critical for applications where time-to-content is directly associated with conversion rate.

Cons:

Development constraints. Browser-specific code can only be used inside certain lifecycle hooks; some external libraries may need special treatment to be able to run in a server-rendered app.

More involved build setup and deployment requirements. Unlike a fully static SPA that can be deployed on any static file server, a server-rendered app requires an environment where a Node.js server can run.

More server-side load. Rendering a full app in Node.js is obviously going to be more CPU-intensive than just serving static files, so if you expect high traffic, be prepared for corresponding server load and wisely employ caching strategies.

If a Server Side Rendered Vue App serves your purpose Nuxt makes this a breeze.

Nuxt makes the whole process as simple as possible. You only need to run 'npx create-nuxt-app' and you have a SSR app ready.

Nuxt introduces a folder structure that makes it easy to work with it. Your pages are stored in a pages folder and Nuxt adds them to the router automatically.

Your plugins are contained in a plugins folder and are run before Vue is booted.

There is a huge community around it and lots of modules to handle most of the everyday problems.

Here's a small project I made with it, Laravel, and TailwindCSS. The code for the project is on github (laravel api and nuxt frontend).

PWAs for beginners

Eke Enyinnaya Diala — Sat, 10 Aug 2019 10:29:39 +0000

I was discussing with my step brother a few days ago (he is taking a course on Web development) and I told him about PWAs and much to my surprise he said he hadn't heard much of it neither did he really understand it. A few weeks earlier, I was also discussing with two of my classmates from the university who were also getting into web development and they also didn't really understand what it was. So here's my attempt to explain it.

What exactly are PWAs?

PWA is an acronym for Progressive Web Applications. They are Web applications that behave like mobile applications- they can be used offline and can be installed on mobile devices like mobile apps. They're making waves because;
a) they are cheaper to build than mobile applications, and
b) they are cross platform-- they can run wherever the browser runs. So, if I, an online store owner, wants my users to have a mobile experience, I don't have to build a mobile application. I can have the same web application running as a mobile application on mobile devices. That saves cost and time. Granted, you wouldn't have all the access to the device native features but support for that is growing, besides it's not like an online store needs access to lots of native features.

How do PWAs achieve this?

They use what is known as Service Workers. In simple terms, service workers decide what to fetch for the browser when a url is visited. Service workers intercept the request and check if they already have a copy of the response for that request. If they don't, they fetch the page, cache it for future use, and serve it to the browser. If they do, they send the browser the cached response. Behind the scenes, they also check if the copy of the response they have is the same as what is on the server and will let the browser know if it is not. It is up to the developer to listen for this event and decide the best way to let the user know. Most people use a snack bar.

So if you visit a url for the first time, the service worker will fetch the page, cache it, and send the page to the browser. Next time, you request the same page, it will not try to fetch it from the server, it will serve you the local copy. This means that with or without Internet, you can still access the page because the beautiful things have it cached.

For further reading, I found this article really helpful.

Forem: Eke Enyinnaya Diala

Resource Cleanup in Go Web Servers: Lessons from Nursery School

Introduction

The Mess We Make

Context: The Teacher’s Bell

Cleaning Up the Database Toys

Sharing Toys: The HTTP Client

The Final Bell: Graceful Shutdown

Why It Matters

Conclusion

Read the Full Code

From ReadAll to CopyBuffer: A Go Developer’s Guide to Efficient Data Copying

Introduction

Approaches Compared

1. io.ReadAll: Load Everything Into Memory

2. io.Copy: Stream Data in Chunks

3. io.CopyBuffer + sync.Pool: Optimized Streaming

Benchmark Results

Memory Profiling Results

When to Use Each Approach

Use io.ReadAll when:

Use streaming (io.Copy, io.CopyBuffer) when:

Benchmarking vs. Profiling: What’s the Difference?

Practical Guidance

Complete Memory Profile Data

io.ReadAll Profile (readall.prof)

io.Copy Profile (copy.prof)

sync.Pool Profile (syncpool.prof)

Testing HTTP Requests with http.RoundTripper

Use httpmock when you need:

Use RoundTripper when you want:

TDD and Software Design

Streaming Large Files Between Microservices: A gRPC Implementation

Efficient Large Data Transfer with gRPC Streaming

Why gRPC for Large File Transfers?

Key Features (gRPC Implementation)

How it works

Server

Memory Efficiency: Real-World Profiling

Conclusion

Streaming Large Files Between Microservices: A Go Implementation

Efficient Large Data Transfer Between Microservices (Go Demo)

Key Features

How it Works

Server

Client

Alternative Approaches

Why the HTTP approach was chosen

Day 3 of Elm- Records

Records

Day 2 of Elm- Partial Application

Day 1 of Elm

Conditionals

Parentheses

Using Chakra UI, React Hook Form, and Yup for React Form Validation.

Validating a login form with react-hooks-form, and yup.

Intro to Nuxt

PWAs for beginners

1. `io.ReadAll`: Load Everything Into Memory

2. `io.Copy`: Stream Data in Chunks

3. `io.CopyBuffer` + `sync.Pool`: Optimized Streaming

Use `io.ReadAll` when:

Use streaming (`io.Copy`, `io.CopyBuffer`) when: