Forem: Harrison Guo

Node Turns Waiting Into Events. Go Moves Context Switching Into User Space.

Harrison Guo — Tue, 28 Apr 2026 18:01:12 +0000

Most discussions of TypeScript/Node vs Go concurrency stop at the surface: Node is async, Go is threaded. That framing isn't wrong — it just isn't deep enough to be useful when you're picking a runtime, debugging a tail-latency problem, or explaining to your team why one of the services keeps falling over under CPU load.

The real difference is not async vs threaded. It's a question about where, in the system, suspended work lives — and what shape it takes when it's resumed.

tl;dr — Both Node and Go refuse to let the CPU sit idle while a request waits on I/O. They disagree on the unit of scheduling. Node's unit is the continuation — the tail of an async function captured as a heap closure. Go's unit is the goroutine — a full call stack the runtime can suspend and resume in user space. That single decision cascades into every other property of each runtime.

The Wrong Question

"Async vs threaded" is the wrong frame because it makes you think the choice is between paradigms. It isn't. Both runtimes have already made the same fundamental decision: do not block an OS thread waiting for slow external work. The interesting choice is how they implement that.

The actually useful question is:

When a request is waiting for I/O — for a database, an HTTP call, a Redis round-trip, a file read — what does the CPU do, and where does the suspended state of that request live?

Once you frame it that way, Node and Go aren't opposites. They're two answers to the same question — and each answer cascades into a different language shape, a different library style, and a different failure mode under load.

The naive blocking model answers the question with "an OS thread waits for the syscall to return." That model collapses around a few thousand concurrent connections — memory per thread, scheduler overhead, kernel context-switch cost. By 40,000 connections you're out of RAM, not CPU. Node and Go both refuse to do this. They diverge on which resource gets freed up and how the suspended work is captured for later resumption.

Node's Answer: Turn Waiting Into an Event

Node's model can be summarized in one line: the JS main thread only executes code that's already ready to run.

Look at this:

const user = await db.getUser(id);
return user;

It reads as if the function is paused, blocking on the database. It isn't. Here's what V8 actually does at the bytecode level when it compiles an async function: it rewrites the body into a state machine, with each await becoming a state transition.

The function above gets transformed into something equivalent to:

function asyncFn() {
  const promise = new Promise((resolve) => {
    let state = 0;
    const closure = {};                  // heap object holding locals

    function step(value) {
      switch (state) {
        case 0:
          state = 1;
          db.getUser(id).then(step);     // await → register continuation
          return;                         // ← function POPS here
        case 1:
          closure.user = value;           // resume: locals live in closure
          resolve(closure.user);
          return;
      }
    }
    step();
  });
  return promise;
}

Three things to notice:

await is not a pause. It's the point at which V8 returns from the function and pops the JS stack frame. The "rest of the function" is captured as a continuation registered on the awaited Promise via .then.
Local variables move to the heap. Because the stack frame is gone, locals (user here) live in a heap closure, accessible only when the state machine resumes.
Each await slices the function into another state. A function with two awaits runs in three event-loop turns, with three independently-pushed JS frames, with all live state stored in heap closures between them.

That third point is the most non-obvious. A single async function is not one unit of execution — it's a sequence of fresh frames separated by event-loop turns:

There is no "paused" function. There are only captured continuations and fresh frames that resume them. The event loop is the dispatcher: it watches for I/O readiness via libuv, for resolved Promises (via V8's microtask queue), for timers — and pulls the corresponding continuation onto the JS thread when it's ready to run. One thread can manage tens of thousands of concurrent connections, because at any moment only a handful of them have work that's actually ready.

This is event-driven concurrency in its precise sense — the runtime turns "waiting" into a registered event, and only resumes the captured continuation when the event fires.

The Visible Side Effect: Function Color

Because the suspension point has to be marked at compile time, async-ness becomes part of the function's type. A function that does I/O returns Promise<T>. Its callers must await it. Once they await, they themselves return Promise<T>. The "color" propagates up the call stack until you hit an async-aware entry point — typically the top of an HTTP handler or the event loop itself.

Bob Nystrom named this the function color problem in 2015. It's not a notation choice — it's a logical consequence of the stackless coroutine model. V8 cannot save and restore arbitrary JS call stacks. The only way to express suspension is "return a Promise and be marked async," and once one function does that, every function on the way up has to do the same.

The Hard Limit

The model fails the moment your code stops waiting. A single CPU-bound operation:

while (true) { /* heavy work */ }

…holds the JS main thread, and every other request on this process is dead until it returns. The event loop has nowhere else to go. Worker threads, child processes, or splitting CPU work into a separate service are real fixes, but they're escape hatches — they exist because the core model has only one main thread executing JS, and there is exactly one of it.

Go's Answer: Move Context Switching Into User Space

Go writes synchronous code:

user := db.GetUser(id)
sendResponse(user)

There is no await. There is no callback. The function looks like it blocks on the database. And yet the program scales to hundreds of thousands of concurrent operations on modest hardware.

The trick is that the scheduling boundary has been moved. Where Node has the programmer mark the suspension point with await and the runtime captures a continuation, Go lets the programmer write straight-line code and has the runtime suspend the entire goroutine when it hits a blocking I/O call.

This is the central insight, and the cleanest one-line statement of Go's concurrency model:

Go's essence is the user-space-ification of context switching.

A goroutine isn't an OS thread. It's a small (initially 2 KB) growable stack and a register snapshot, managed by the Go runtime. The runtime maps a large number of goroutines (G) onto a small number of OS threads (M) using scheduling contexts (P). This is the GMP model:

G — a goroutine. The unit of scheduling. Cheap to create, cheap to suspend.
M — an OS thread. Usually only GOMAXPROCS of them.
P — a scheduling context. Decides which G runs on which M.

many G  →  Go scheduler  →  few M  →  CPU cores

When a goroutine hits a blocking syscall or a channel wait, the Go runtime suspends the goroutine — saves its stack and registers — detaches it from the current M, and schedules another runnable goroutine onto that M. When the original goroutine's wait completes, it's marked runnable again, and some M eventually picks it up and resumes execution from the suspension point. None of this enters the kernel. No clone(2), no kernel-mediated thread switch, no kernel scheduler queue. The bookkeeping is all in user space.

That's the user-space-ification. The CPU still has to switch contexts when work shifts between goroutines, but the cost is roughly a function call plus a stack swap — not a kernel-mediated thread switch.

The key contrast with Node's model is in where the suspended state lives:

In Node, the JS call stack is shared and almost always near-empty — every async function in flight has already popped, with its state sitting in a heap closure. In Go, every goroutine owns its full call chain on its own heap-allocated stack; suspended goroutines look like frozen frames waiting for the runtime to resume them on some OS thread.

This is also why neither language can simply borrow the other's model. Node runs on V8, which was designed in 2008 for browser JS — single call stack, synchronous semantics, no concept of saving stacks across yields. Adding stackful coroutines would mean rewriting the engine, which is roughly what Java's Project Loom did to the JVM at huge cost. Go was designed from scratch with a runtime that owns stacks, can grow them, and can save them. The choice is locked in by runtime architecture, not language taste.

What "User-Space" Actually Buys You

The slogan only matters if user-space context switching is meaningfully cheaper than the kernel-mediated kind. It is — by more than an order of magnitude.

Two goroutines pinned to one OS thread (GOMAXPROCS=1), ping-ponging via runtime.Gosched() and via an unbuffered channel. Two pthreads pinned to one core (taskset -c 0), ping-ponging via pthread_mutex + pthread_cond. (Reproduction code at the end of the post.)

Measured on Intel N100, Ubuntu 24.04 (kernel 6.8.0), Go 1.23.4, gcc 13.3:

Operation	ns / switch
Goroutine yield (`runtime.Gosched`, GOMAXPROCS=1)	~102 ns
Goroutine round-trip via unbuffered channel	~436 ns (≈218 ns per G-switch + channel coordination)
pthread switch (mutex+cond ping-pong, single core)	~2,900 ns (range 2,818–3,611 across 5 runs of 2M iterations)

Ratio: roughly 28× cheaper for the bare scheduler yield, ~13× cheaper for the apples-to-apples synchronized round-trip.

Where the gap comes from:

Mode switch. The user → kernel → user round-trip alone is ~100 ns of entry/exit and ABI-mandated register save/restore. A goroutine switch never crosses that line.
Scheduler work in kernel space. Linux CFS maintains a red-black tree of runnable threads with locked, cross-CPU runqueues. The Go scheduler does the same job in user space with per-P local runqueues and lock-free fast paths — and skips the kernel locks entirely.
Cache and TLB effects. A kernel scheduler may migrate a thread to a different core, costing you cold L1/L2 and an instruction-cache reload. Goroutines normally stay on the same M, so the cache stays warm.

What the model does not buy you: a goroutine that makes a real blocking syscall still pays for a real OS thread switch — the runtime detaches the G from its M and may spin up another M so the rest of the goroutines keep running. Async preemption (Go 1.14+, signal-based) is the runtime's answer to tight loops that never yield, and it has its own cost. Once you saturate GOMAXPROCS, the user-space runqueue itself starts to show up in profiles.

The "user-space-ification" buys you cheap G-to-G switching on a hot M. That's where the order-of-magnitude lives. The syscalls, the M-to-M handoffs, the actual kernel work — those are still as expensive as they always were. The model wins by making the common case — many concurrent goroutines, mostly waiting, occasionally running — almost free.

(N100 is a low-power Alder Lake-N E-core; absolute numbers will be smaller on a server-class Xeon or EPYC, but the ratio is expected to hold.)

The Unit of Scheduling

The cleanest comparison is to ask what each runtime actually schedules:

	Node / TypeScript	Go
Unit of scheduling	callback / Promise continuation	goroutine
What's captured at suspension	tail of an async function as a heap closure	full call stack + registers
How code looks	explicit `async`/`await`	straight-line synchronous
Suspension marked by	the programmer (`await`)	the runtime (any blocking op)
Suspended state lives in	V8 microtask queue + heap closure	goroutine stack on the user-space heap
Kernel involvement	epoll/kqueue/IOCP via libuv	epoll/kqueue/IOCP via netpoller
CPU parallelism	one main JS thread; needs workers/cluster for cores	M:N scheduler runs goroutines across cores natively
Function color	yes (Promise infects up the call stack)	no (any function may block)
What breaks under CPU load	the entire event loop	nothing — scheduler runs another G on another M

The two columns describe deeply different mental models, but they belong to the same family. They are both user-space concurrency runtimes that avoid kernel thread-per-request. They differ in where the suspension is captured (the language vs. the call stack) and how broad the scheduler's mandate is.

Where the Boundaries Diverge: CPU-Bound Work

Node and Go look interchangeable on I/O-bound workloads. They diverge sharply the moment CPU work enters the picture.

Node's event loop has one job: dispatch ready callbacks onto a single JS thread. If a callback runs for 200 ms doing JSON parsing or hashing, the loop is frozen for those 200 ms. Every other suspended continuation has to wait. Throughput collapses.

Go's runtime has a different mandate. It doesn't only manage waiting — it also manages execution. If you spawn:

go task1()
go task2()
go task3()

…the scheduler is happy to put each goroutine on a different M, run them on different cores in true parallel, and preempt long-running goroutines so they don't starve the rest of the runtime. CPU-bound goroutines aren't a special case to work around. They're just goroutines.

That's why Go's concurrency model covers more ground:

Node's model mainly solves non-CPU-bound concurrency — network I/O, database waits, downstream API calls. Go's model solves I/O waiting and CPU parallelism with the same primitive.

This isn't a knock on Node. The event loop is brilliant at what it's designed for: lots of slow waits, light per-request CPU. It's the natural shape of API gateways, BFFs, websocket hubs, real-time aggregation, and most of the JSON-shuffling that makes up modern web backends. But sustained CPU work, mixed CPU + I/O pipelines, long-lived infrastructure services — those are workloads where Go's scheduler-driven model has more headroom built in.

Two Answers to the Same Question

Strip away the implementation details and the two runtimes are answering the same question with different abstractions:

Concurrency at scale is the problem of what to do with the CPU while a request waits on I/O.

Node's answer: turn the wait into an event, capture the rest of the function as a continuation, resume the continuation when the event fires. One thread cycling through ready continuations.

Go's answer: run the request on a goroutine, suspend the goroutine in user space when it blocks, schedule another runnable goroutine onto the OS thread, resume the original when its wait completes.

Two ways of solving the same waste. One state-machines it. The other lowers the cost of context switching far enough that you can afford to keep one execution flow per request.

Two answers to one question: one is events, implemented as a state machine. The other is low-cost user-space context switching.

But there's a deeper layer worth surfacing. The two answers also disagree about whether suspension should be visible in the type system. Node says yes — Promise<T> is part of the signature, async is part of the contract, function color propagates. Go says no — any function may block, and the type doesn't carry that information.

This visibility-vs-uniformity trade-off shows up far beyond Node and Go. It's the same shape as monadic IO vs implicit IO in Haskell, checked vs unchecked exceptions in Java, capability-based security vs ambient authority. Each pair makes the same trade: composable static reasoning vs ergonomic uniform code. Node and Go are picking sides of a much bigger question.

You see the consequence in the libraries. Node libraries publish fs.readFile and fs.readFileSync, two retry helpers (one for sync ops, one for async), p-limit-style bounded-concurrency wrappers around Promise.all. Go libraries publish os.ReadFile (one function), one Retry(op func() error, n int) error, twenty lines of chan + WaitGroup for bounded concurrency. The Go versions aren't simpler because Go developers are smarter — they're simpler because the runtime hides the same complexity that Node's type system insists on exposing.

The Closing Line

If you remember one thing from this:

Node turns waiting into events. Go turns execution flows into schedulable units. Both refuse to let the CPU sit idle while I/O blocks — they just disagree on what the unit of scheduling should be.

Or, if you want the deeper layer:

Node makes "this function might suspend" visible at the type level. Go makes it invisible.

That's the whole story. Everything else — await vs go, libuv vs the netpoller, V8's microtask queue vs GMP, single-thread bottleneck vs CPU-bound resilience, libraries that look complicated vs libraries that look simple — falls out of that one disagreement.

Appendix: Reproduce the Benchmark

goroutine_switch_test.go — GOMAXPROCS=1 go test -bench=. -benchtime=5s -count=5:

package bench

import (
    "runtime"
    "sync"
    "testing"
)

// Channel ping-pong: each iter is a full round-trip = 2 G-switches.
func BenchmarkGoroutineSwitchChannel(b *testing.B) {
    ch := make(chan struct{})
    done := make(chan struct{})
    go func() {
        for {
            select {
            case <-done:
                return
            case <-ch:
                ch <- struct{}{}
            }
        }
    }()
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        ch <- struct{}{}
        <-ch
    }
    b.StopTimer()
    close(done)
}

// Bare scheduler yield. Each iter ≈ 1 G-switch.
func BenchmarkGoroutineSwitchGosched(b *testing.B) {
    var wg sync.WaitGroup
    wg.Add(1)
    half := b.N / 2
    go func() {
        for i := 0; i < half; i++ {
            runtime.Gosched()
        }
        wg.Done()
    }()
    b.ResetTimer()
    for i := 0; i < half; i++ {
        runtime.Gosched()
    }
    wg.Wait()
}

pthread_switch.c — gcc -O2 -o pthread_switch pthread_switch.c -lpthread && taskset -c 0 ./pthread_switch 2000000:

#define _GNU_SOURCE
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <stdint.h>

static pthread_mutex_t mu  = PTHREAD_MUTEX_INITIALIZER;
static pthread_cond_t  cv  = PTHREAD_COND_INITIALIZER;
static volatile int    turn = 0;
static long            iters;

static void *worker(void *arg) {
    int my_turn = (int)(intptr_t)arg;
    pthread_mutex_lock(&mu);
    for (long i = 0; i < iters; i++) {
        while (turn != my_turn) pthread_cond_wait(&cv, &mu);
        turn = 1 - my_turn;
        pthread_cond_broadcast(&cv);
    }
    pthread_mutex_unlock(&mu);
    return NULL;
}

static double now_ns(void) {
    struct timespec ts;
    clock_gettime(CLOCK_MONOTONIC, &ts);
    return (double)ts.tv_sec * 1e9 + (double)ts.tv_nsec;
}

int main(int argc, char **argv) {
    iters = (argc > 1) ? atol(argv[1]) : 1000000L;
    pthread_t t0, t1;
    double start = now_ns();
    pthread_create(&t0, NULL, worker, (void *)(intptr_t)0);
    pthread_create(&t1, NULL, worker, (void *)(intptr_t)1);
    pthread_join(t0, NULL); pthread_join(t1, NULL);
    double end = now_ns();
    printf("ns / switch: %.1f\n", (end - start) / (2.0 * iters));
    return 0;
}

GOMAXPROCS=1 forces both goroutines onto the same M so we measure pure G-to-G switching, not cross-core migration. taskset -c 0 pins both pthreads to one CPU so they actually have to context-switch (otherwise they run in parallel on two cores and there is nothing to measure). Both benches do the simplest possible synchronized hand-off — no I/O, no real work — so what is left is the cost of the switch itself.

gRPC Interceptors in Production: Design Patterns That Survive Real Load

Harrison Guo — Mon, 20 Apr 2026 17:02:20 +0000

gRPC interceptors are the middleware pattern, specialized for gRPC. If you've written HTTP middleware before, the shape is familiar — a function that wraps a call, can observe or modify the request, pass to the next handler, then observe or modify the response. The difference: gRPC's type system makes the flavors (unary, server-stream, client-stream, bidi) explicit, and chain ordering matters more than most people realize.

Most online examples show a single toy interceptor. Production systems stack five to ten of them per service. Getting the composition right — ordering, concern separation, testability — is half of running a gRPC-based microservice well.

tl;dr — gRPC interceptors are middleware with more explicit types. Chain them outside-in: observability wraps everything, then throttling, then auth, then retry, then the actual service. Keep each interceptor focused on one concern; the moment an interceptor does two things you're writing coupled middleware. Stream interceptors are trickier than unary — don't copy-paste unary logic into stream without thinking. Test the chain composition with bufconn, not just each interceptor in isolation.

The Four Interceptor Types

gRPC has four interceptor signatures, two for client, two for server:

Unary server interceptor: wraps a single request → single response call.
Stream server interceptor: wraps streaming RPCs (server-stream, client-stream, bidi).
Unary client interceptor: wraps the client side of a unary call.
Stream client interceptor: wraps the client side of a streaming call.

Unary interceptors are easy. Stream interceptors are harder because you're wrapping a bidirectional wire, not a single call.

Example unary server interceptor:

func loggingInterceptor(ctx context.Context, req interface{},
    info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
    start := time.Now()
    resp, err := handler(ctx, req)
    log.Printf("method=%s duration=%s err=%v", info.FullMethod, time.Since(start), err)
    return resp, err
}

s := grpc.NewServer(grpc.UnaryInterceptor(loggingInterceptor))

Straightforward. Now stack five of them.

Chaining and Order

Real services need multiple interceptors. gRPC's standard library gives you grpc.ChainUnaryInterceptor(...) (since 1.25), or you can use google.golang.org/grpc/interceptor helpers:

s := grpc.NewServer(
    grpc.ChainUnaryInterceptor(
        observabilityInterceptor,  // outermost
        rateLimitInterceptor,
        authInterceptor,
        validationInterceptor,
        businessLogicContextInterceptor, // innermost
    ),
)

Chain order matters enormously. Interceptors execute outside-in on the way to the handler, inside-out on the way back. Put the wrong interceptor outside the wrong one and you get bugs that are hard to debug.

Canonical order I use:

Outside-in on the way to the handler, inside-out on the way back. Observability must wrap everything — so it sees every rejection, every rate-limit hit, every failed auth — otherwise you have operational blind spots. Details:

Observability (tracing + metrics + logging) — outermost. You want to see every request, including the ones that get rejected by later interceptors. If observability is inside auth, unauth'd attempts are invisible — a security-relevant blind spot.
Rate limiting / quota — before auth. Why? Because auth involves token verification (DB lookup, JWT parsing, external identity service), and you don't want unauthenticated requests to cost you CPU. Rate-limit first, authenticate second.
Auth (authentication + authorization) — before business logic. Reject unauthenticated/unauthorized requests early.
Validation (request shape, basic sanity) — before business logic. Catches malformed requests before they hit service code.
Retry / idempotency handling — closer to business. Only retry what actually made it through auth.
Request context enrichment (trace IDs, user metadata) — innermost. Populate context with validated data for the service to use.

Inverted order produces real bugs. I've seen auth outside observability (auth failures weren't logged). Retry outside rate limiter (a retry storm blew through the rate limit). Validation outside observability (validation failures invisible in metrics). Each one a real incident.

Keeping Interceptors Focused

The rule: one concern per interceptor. The moment you have an "auth-and-logging" interceptor, you're coupling concerns that should evolve separately.

Concretely:

Don't: single "observability" interceptor that does tracing, metrics, and logging in one function.
Do: three interceptors (tracingInterceptor, metricsInterceptor, loggingInterceptor), chained.

Cost: three function-call overheads instead of one. Marginal.

Benefit: you can swap tracing backends without touching logging. You can disable metrics in tests without disabling tracing. Each interceptor is testable in isolation.

This is the same argument for Unix pipes over monolithic commands. Composition beats monoliths.

Common Interceptor Recipes

Real interceptors I've written variants of many times:

Tracing (OpenTelemetry)

Use the otelgrpc integration from go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc. Don't write your own — the ecosystem is mature. Current idiomatic setup uses a StatsHandler, which hooks deeper than the interceptor chain and captures stream events correctly:

import "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"

s := grpc.NewServer(
    grpc.StatsHandler(otelgrpc.NewServerHandler()),
    grpc.ChainUnaryInterceptor( /* your app interceptors */ ),
)

Older codebases still use otelgrpc.UnaryServerInterceptor() and otelgrpc.StreamServerInterceptor() — those are deprecated but still work. Migrate when convenient; don't rewrite in a panic.

Metrics

Prometheus histogram of request duration per method:

var (
    reqDuration = promauto.NewHistogramVec(
        prometheus.HistogramOpts{
            Name: "grpc_server_request_duration_seconds",
            Buckets: prometheus.DefBuckets,
        },
        []string{"method", "code"},
    )
)

func metricsInterceptor(ctx context.Context, req interface{},
    info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
    start := time.Now()
    resp, err := handler(ctx, req)
    code := status.Code(err).String()
    reqDuration.WithLabelValues(info.FullMethod, code).Observe(time.Since(start).Seconds())
    return resp, err
}

Note: cardinality of method is bounded (you know your service's methods). Cardinality of code is bounded (gRPC codes are a fixed enum). Don't add user-id or request-id as labels — that's cardinality-explosion territory.

Auth

Extract bearer token from metadata, verify, inject user context:

func authInterceptor(ctx context.Context, req interface{},
    info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
    md, ok := metadata.FromIncomingContext(ctx)
    if !ok {
        return nil, status.Error(codes.Unauthenticated, "no metadata")
    }
    tokens := md.Get("authorization")
    if len(tokens) == 0 {
        return nil, status.Error(codes.Unauthenticated, "no auth token")
    }

    claims, err := verifyToken(tokens[0])
    if err != nil {
        return nil, status.Error(codes.Unauthenticated, "invalid token")
    }

    // Skip certain public methods
    if isPublic(info.FullMethod) {
        return handler(ctx, req)
    }

    ctx = context.WithValue(ctx, userCtxKey{}, claims)
    return handler(ctx, req)
}

Key detail: add the user context here, near the boundary. Service code reads it from context. You don't pass claims as argument through every service method.

Rate limiting

Token bucket per caller or per method:

func rateLimitInterceptor(limiter *rate.Limiter) grpc.UnaryServerInterceptor {
    return func(ctx context.Context, req interface{},
        info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
        if !limiter.Allow() {
            return nil, status.Error(codes.ResourceExhausted, "rate limited")
        }
        return handler(ctx, req)
    }
}

Production rate limiting is fancier — per-tenant, distributed state in Redis, burst capacity — but the shape is the same. Reject with ResourceExhausted before doing work.

Retry (client-side)

Client interceptor that retries on transient errors:

func retryClientInterceptor(attempts int) grpc.UnaryClientInterceptor {
    return func(ctx context.Context, method string, req, reply interface{},
        cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
        var err error
        for i := 0; i < attempts; i++ {
            err = invoker(ctx, method, req, reply, cc, opts...)
            if err == nil {
                return nil
            }
            if !isRetryable(err) {
                return err
            }
            backoff := time.Duration(1<<uint(i)) * 100 * time.Millisecond
            select {
            case <-time.After(backoff):
            case <-ctx.Done():
                return ctx.Err()
            }
        }
        return err
    }
}

Retry is one of the most dangerous interceptors. Get it wrong (no idempotency keys, retry non-idempotent operations, retry storm during outage) and it causes more production incidents than it prevents. Pair with grpc-middleware/retry if you can; it's battle-tested.

The Stream Interceptor Trap

Stream interceptors are harder. The interceptor signature gives you a grpc.ServerStream, which is a bidirectional channel. Logging becomes:

func loggingStreamInterceptor(srv interface{}, ss grpc.ServerStream,
    info *grpc.StreamServerInfo, handler grpc.StreamHandler) error {
    start := time.Now()
    err := handler(srv, ss)
    log.Printf("stream=%s duration=%s err=%v", info.FullMethod, time.Since(start), err)
    return err
}

This only logs at stream-end, not per message. If you want per-message observability, you need to wrap the ServerStream itself:

type observedStream struct {
    grpc.ServerStream
    sent, recv int64
}

func (s *observedStream) SendMsg(m interface{}) error {
    atomic.AddInt64(&s.sent, 1)
    return s.ServerStream.SendMsg(m)
}

func (s *observedStream) RecvMsg(m interface{}) error {
    err := s.ServerStream.RecvMsg(m)
    if err == nil {
        atomic.AddInt64(&s.recv, 1)
    }
    return err
}

Then pass the wrapper to the handler. This is the pattern for any stream interceptor that needs per-message visibility.

Common mistakes:

Forgetting to propagate context to the wrapper. The wrapped stream's Context() should be the enriched context.
Per-message overhead blows up long streams. A message-level log line is fine at 100 msgs/sec. At 100K msgs/sec, it's your dominant cost.
State in the wrapper not thread-safe. Streams can be concurrent on the Send and Recv sides. Protect counters.

Testing Interceptor Chains

Unit test each interceptor in isolation:

func TestAuthInterceptor_NoToken(t *testing.T) {
    ctx := context.Background() // no metadata
    info := &grpc.UnaryServerInfo{FullMethod: "/my.Service/Method"}
    handler := func(ctx context.Context, req interface{}) (interface{}, error) {
        t.Fatal("handler should not be called")
        return nil, nil
    }

    _, err := authInterceptor(ctx, nil, info, handler)
    require.Equal(t, codes.Unauthenticated, status.Code(err))
}

Integration-test the chain end-to-end using bufconn (in-memory connection):

func TestChain_Ordering(t *testing.T) {
    lis := bufconn.Listen(1024 * 1024)
    defer lis.Close()

    s := grpc.NewServer(grpc.ChainUnaryInterceptor(observability, auth, business))
    pb.RegisterMyServer(s, &realImpl{})
    go s.Serve(lis)
    defer s.Stop()

    conn, _ := grpc.Dial("bufnet",
        grpc.WithContextDialer(func(ctx context.Context, _ string) (net.Conn, error) {
            return lis.DialContext(ctx)
        }),
        grpc.WithTransportCredentials(insecure.NewCredentials()),
    )
    defer conn.Close()

    client := pb.NewMyClient(conn)
    resp, err := client.Method(ctx, req)
    // assert on behavior end-to-end
}

Integration tests catch bugs that unit tests don't: metadata propagation, interceptor ordering, context enrichment visible to the handler. Don't skip them.

Patterns That Save Time

Use grpc-middleware/v2 (github.com/grpc-ecosystem/go-grpc-middleware/v2) for chain helpers, recovery, and batteries-included interceptors. Don't reinvent every wheel.
Keep error semantics consistent. Every interceptor should return status.Error(code, msg) for failures. Don't return raw Go errors — clients can't parse them properly.
Skip-list for public methods. Auth and rate limit often need to skip health check and reflection endpoints. Keep the skip list in one place.
Per-service vs global interceptors. Most interceptors are global (tracing, metrics, auth). A few might be per-service (e.g., a bespoke rate limiter for a specific hot endpoint). Compose accordingly.
Panic recovery at the outermost layer. A panic in a handler shouldn't kill the server. Use the recovery middleware from grpc-middleware or write your own, and put it first in the chain.

The Discipline That Makes This Work

Interceptors are the right tool for cross-cutting concerns — the things every RPC needs but the service code shouldn't have to think about. The discipline is: one concern per interceptor, careful ordering, consistent error semantics, tested end-to-end.

The services I've seen do this well have clean business logic (because the cross-cutting stuff is outside it) and reliable operational behavior (because the interceptor chain is tested as a unit, not just piece-by-piece). The services that do it poorly have auth logic sprinkled through their handlers, tracing that randomly misses requests, and rate limiters that let certain code paths bypass.

Interceptor order is one of those details that looks tactical and turns out to be architectural. Get it right once; the service's behavior improves every release.

Go Context in Distributed Systems: What Actually Works in Production — the context that flows through every interceptor.
RPC vs NATS: It's Not About Sync vs Async — It's About Who Owns Completion — the shape of gRPC calls as one side of the bigger messaging picture.
Observability and Cost Attribution: Why One Pipeline Isn't Enough — why tracing interceptors alone aren't enough for business attribution.

Go Generics, One Year In: Which Promises Held, Which Didn't

Harrison Guo — Mon, 20 Apr 2026 16:30:05 +0000

Go 1.18 shipped generics in March 2022. The two years before that were dominated by hopeful blog posts ("finally, a real type system!") and the two years after by the predictable backlash ("why did we even bother, Go was simpler"). I've written production Go before and after. The honest answer is somewhere in the middle and closer to "useful for a narrower set of problems than we expected."

This is a look back from someone who has shipped generic code in anger and reviewed a lot more of it. What held up. What didn't. What habits to adopt and which to avoid.

tl;dr — Go generics are genuinely valuable for parametric operations on container-shaped types — slices, maps, channels, any-key lookup tables, min/max/sum utilities. Less valuable for "clever abstractions" that dress up control flow as type magic. The clearest gains are in the standard library itself (slices, maps) and in domain-specific utility packages. Most application code didn't need generics before and doesn't need them after. The mistake is not using generics; it's using them for things interfaces already handled fine.

What Generics Actually Are

Go generics are type parameters on functions and types. A function like slices.Contains can be written once, work for any slice element type, and still be type-checked at compile time:

func Contains[S ~[]E, E comparable](s S, v E) bool {
    for _, x := range s {
        if x == v {
            return true
        }
    }
    return false
}

Three features you should know:

Type parameters: the [E any] or [E comparable] in brackets.
Constraints: tell the compiler what operations the type parameter supports. any, comparable, or custom interfaces like constraints.Ordered.
Approximate constraints: ~[]E means "any type whose underlying type is []E" — lets you be flexible about named slice types.

What they aren't: Java-style wildcards, C++ SFINAE, or anything that mimics variance. The design is deliberately narrower than most prior languages. It's more like Rust's generics, minus the trait system's complexity.

Where Generics Clearly Win

Standard-library style container and utility functions

The slices and maps packages in the standard library are the canonical example:

slices.Contains(users, "alice")
slices.Sort(numbers)
maps.Keys(config)
maps.Values(settings)

Before generics, these were either hand-written per-type (tedious, error-prone), done via interface{} (type-unsafe, slow), or done via reflect (slow and error-prone). Generics are strictly better for these.

The same pattern shows up in third-party libraries: samber/lo (JS-style utilities), thoas/go-funk (functional helpers), and many domain-specific ones. If you reach for lodash-style helpers in JavaScript, you'll want similar in Go, and generics made that workable.

Concurrency helpers

Generic worker pools, futures, result types — these all benefit from generics:

type Future[T any] struct {
    done chan struct{}
    val  T
    err  error
}

func (f *Future[T]) Get() (T, error) {
    <-f.done
    return f.val, f.err
}

Before generics, you'd have had an interface{} return and a type assertion at the call site. Now you can express "this future produces a T" in the type. Cleaner at the boundary, safer at the call site.

Typed collections

If your system has a genuinely typed container use case — say, an ordered map keyed by a domain ID — generics let you write it once:

type OrderedMap[K comparable, V any] struct {
    order []K
    data  map[K]V
}

This is a rare case where "custom generic container" is the right tool. The majority of code doesn't need this. But when you do need it, the generics version is much better than the interface{} alternative.

Numerical / algorithmic code

constraints.Ordered (or its post-1.21 replacement cmp.Ordered) is the key constraint for "works for any numeric or ordered type":

func Max[T cmp.Ordered](a, b T) T {
    if a > b { return a }
    return b
}

Math helpers, min/max, sum, average — all cleanly generic. Readable, type-safe, performant.

Where Generics Don't Help, Or Hurt

"Generic services" and similar framework-y code

I've seen codebases where someone wrote a generic "repository" type:

type Repository[T any] struct { /* ... */ }
func (r *Repository[T]) FindByID(id string) (T, error) { /* ... */ }
func (r *Repository[T]) Save(t T) error { /* ... */ }

The instinct — "all repositories do the same thing" — is mostly wrong. Real repositories differ in query shape, error cases, caching rules, transaction boundaries. Forcing them behind a generic interface either (a) produces a lowest-common-denominator API that doesn't fit any actual use, or (b) gets so many type parameters that readability collapses.

The Go idiom is usually better: one non-generic UserRepository, one OrderRepository, etc. Each concrete, each tuned to its domain.

Over-constrained helpers

If your "generic" function has five type parameters with custom constraints each, readability dies:

func Complicated[
    T comparable,
    K Hashable,
    V any,
    F func(T, K) (V, error),
    M map[K]V,
](items []T, f F, cache M) error { /* ... */ }

This is technically legal. Reading it, you realize it's a glorified map-with-cache-and-error. Interfaces or function types would have been clearer. Generics don't make complex APIs simple; they just let you make them complex in a type-checked way.

Behavioral polymorphism

Interfaces are still the right tool when different types have different behavior. A generic Process[T any](x T) error doesn't help if you actually want different logic per type. You want an interface with a method.

// Good use of interface
type Processor interface {
    Process(ctx context.Context) error
}

// Bad use of generics
func ProcessGeneric[T any](x T) error {
    // can't actually differentiate behavior
}

The separation: generics for parametric operations (same logic, any type), interfaces for polymorphic behavior (different logic per type).

Performance: Usually a Wash

The performance story is more nuanced than either "generics are slow" or "generics are free."

Go's current generic implementation uses GCShape stenciling — one compiled version per "GC shape" (roughly, per memory layout). This is between full monomorphization (one version per type, like Rust) and type-erased dispatch (one version total, like Java's reified-erased hybrid).

Practical implications:

Small primitive types (int, int64) often get specialized versions. Competitive with hand-written.
Pointer-sized types (most structs, interfaces) share code. Slightly slower than hand-written but usually faster than interface-based dispatch.
Call overhead is similar to function calls, not interface dispatch. No devirtualization issue.
Compile times increase, especially for libraries with many instantiations. This is the real cost.

Benchmarks I've seen: generic versions are within 5% of hand-written equivalents, and consistently faster than interface{}-based alternatives. Performance is almost never the deciding factor — readability and design fit matter more.

Idioms That Emerged

Over the years since 1.18, a few conventions have stuck:

Prefer `any` to `interface{}`

any is a type alias for interface{} added in 1.18. Shorter, clearer. Use it everywhere.

Single-letter type parameters for simple cases, descriptive for complex

T, K, V for the obvious cases. More descriptive when the role is specific:

func Reduce[In any, Out any](items []In, f func(Out, In) Out, initial Out) Out

Put constraints in a dedicated package

If you have several custom constraints, group them:

package constraints

type Ordered interface { ~int | ~int64 | ~string | ~float64 }
type Numeric interface { ~int | ~int64 | ~float64 }

The standard golang.org/x/exp/constraints (and later cmp.Ordered in 1.21) set the pattern.

Use `~T` approximations for flexibility

~[]E includes named slice types. ~int includes type MyInt int. Almost always the right choice for generic parametric code; refuses arbitrary extension.

Never overload generic helpers to do too much

Each generic function should do one parametric thing. Generic helpers that try to be many things at once collapse under type-parameter weight.

The Standard Library Won

The clearest vindication of Go generics is what happened to the standard library. slices, maps, cmp.Ordered — these additions are uncontroversially better than the pre-1.18 alternatives. A lot of code that used to be hand-rolled or based on sort.Interface has cleaner replacements.

The user-land picture is more mixed. Libraries that benefit from generics genuinely use them well (samber/lo, kelindar/column, many others). Libraries that don't need them mostly haven't been retrofitted with them.

What I Do Now

A few simple rules I apply:

Prefer standard library generic helpers over hand-rolled. slices.Contains, slices.Sort, maps.Keys — use them.
Write a generic helper only when I have at least two concrete use cases for it. One use case is a pattern waiting to be born, not necessarily a generic.
Prefer functions to methods on generic types when possible. Generic methods have more friction (can't overload by type, can't add methods outside the defining package).
Keep constraints simple. any, comparable, cmp.Ordered, and domain-specific single-type-union constraints cover 95% of cases. More complex constraints usually mean the abstraction is wrong.
Never turn interfaces into generics just because you can. If the types have genuinely different behavior, an interface is right.

Where Generics Actually Sit Now

Generics were oversold before they landed ("Go finally becomes a real language!") and oversampled in the aftermath ("generics everywhere!"). The truth is narrower and more boring: they're a useful addition for a specific class of problems, mostly centered on parametric operations over containers and numerics. They improved the standard library. They haven't changed the shape of most Go code.

If you've been writing Go and wondering whether you're missing out by not using generics, the answer is almost certainly no. Code without them is still idiomatic. Code with them, when the use case fits, is cleaner. Neither is dominant. Both are fine.

The one concrete thing I'd say: learn the generic parts of the standard library. slices, maps, cmp.Ordered. Use them reflexively. Stop hand-rolling indexOf and contains. Everything else can wait until you have a real problem that generics solve.

Go Profiling in Anger: pprof, Escape Analysis, and Inlining Without Magic — the performance toolchain that tells you whether your generic code actually matches the hand-written version.
sync.Pool in Go: When It Actually Helps, and When It Quietly Hurts — another feature most commonly misapplied.
Scale-Up vs Scale-Out: Why Every Language Wins Somewhere — the meta-question behind every language-feature debate.

Go Profiling in Anger: pprof, Escape Analysis, and Inlining Without Magic

Harrison Guo — Mon, 20 Apr 2026 16:29:23 +0000

Go's performance culture has a ritual quality. "Use sync.Pool." "Avoid interface boxing." "Preallocate slices." Copy-pasted from blog posts and applied without measurement. Sometimes helpful. Often hollow.

The honest answer is that Go performance work is mostly just profiling. Good profiling tells you what's actually slow. Bad profiling — or no profiling — leaves you guessing. The toolchain that Go ships with is genuinely excellent; more engineers should use it, and fewer should follow checklist optimizations they haven't measured.

This is a practical, end-to-end guide to pprof, escape analysis, and inlining — the three Go-specific tools that answer most performance questions.

tl;dr — Start every Go perf investigation with a CPU pprof of the hot path under realistic load. 80% of issues are obvious in the flame graph. For the remaining 20%, add a heap profile and look for allocation pressure driving GC. Only after you've localized the problem with real data should you reach for micro-optimizations: escape analysis via -gcflags='-m', inlining hints, and targeted benchmark-driven rewrites. Skip the profile step, and you are optimizing the wrong thing.

The Investigation Flow

CPU Profiling: The First Thing, Always

Every Go binary can expose a pprof HTTP endpoint in two lines:

import _ "net/http/pprof"
// later
go http.ListenAndServe("localhost:6060", nil)

Under load, grab a CPU profile:

$ go tool pprof -http=:9999 http://localhost:6060/debug/pprof/profile?seconds=30

This opens a flame graph in your browser. The wide blocks are where CPU time is spent. Usually the answer is immediate — "oh, JSON encoding is 40% of my CPU; let me switch to a faster encoder." Or "regex compilation is in the hot path because someone forgot to pre-compile."

A few things that look surprising on first profile but shouldn't:

runtime.mallocgc taking 10%+ is GC pressure. You're allocating a lot. Look at heap profile next.
runtime.schedule or runtime.findrunnable taking 5%+ means you have too many goroutines churning. Check if you're spawning per-request.
syscall.Syscall high means you're system-call-heavy — usually I/O. Either buffer/batch, or consider epoll-direct if it's in your hot path.
mutex.Lock visible means contention. Either shrink the lock hold time or shard the lock.

Don't guess your way through these. Click into each, read the stack, find the user code that caused it.

Heap Profiling: When CPU Points to GC

If runtime.mallocgc shows up in your CPU profile as a non-trivial chunk, heap profile tells you why:

$ go tool pprof -http=:9999 http://localhost:6060/debug/pprof/heap
$ go tool pprof -http=:9999 http://localhost:6060/debug/pprof/allocs

heap shows current memory usage. allocs shows cumulative allocations since program start — this is usually what you want to optimize.

In the flame graph, look for:

Specific allocation sites taking disproportionate share. A single line of code creating 50% of allocations is an obvious target.
Calls to makeslice, makemap, newobject with known-size inputs. If you know the size, preallocate.
Interface boxing in hot paths. Every time you pass a concrete type through an interface{} argument in a tight loop, the runtime may heap-allocate the boxed value.
String concatenation with +. This is the textbook preventable allocation — use strings.Builder.

The goal isn't "zero allocations" — that's usually not practical. The goal is "allocations per operation in a tight, repeated path are bounded and understood."

Escape Analysis: The Compiler's Story

Go's compiler decides at compile time whether a variable lives on the stack (free, garbage-collected with the function) or the heap (allocated, GC-tracked). This is called escape analysis.

To see the analysis for your code:

$ go build -gcflags='-m' ./...

Output looks like:

./foo.go:12:6: can inline hotFunction
./foo.go:15:10: &Thing{} escapes to heap
./foo.go:18:14: make([]int, 100) does not escape
./foo.go:22:6: parameter "x" escapes to heap

Key things to read for:

escapes to heap — this allocation is heap-allocated. If it's in a hot path, investigate.
does not escape — stack-allocated, free. You want most short-lived locals to do this.
parameter escapes to heap — the caller's passed value escapes because this function keeps a reference to it. Often fixable by taking a copy or not storing a reference.

The most common surprise: passing a value to a function that eventually hands it to interface{} causes the value to escape. A pattern like:

func log(msg string, args ...interface{}) {...}
func handleRequest(req *Request) {
    log("got request", req.ID) // req.ID boxes to interface{} and may escape
}

req.ID escapes because of the ...interface{} argument. In a tight path, this is measurable. Fix: use a typed logger that takes concrete types, or accept the cost because logging on the hot path is usually not the hot path.

Escape analysis is one of those things where reading the output a few times is worth it. You start seeing your code differently.

Inlining: When the Compiler Eliminates the Call

Go's compiler inlines small functions to avoid call overhead. Seeing what got inlined:

$ go build -gcflags='-m' ./... 2>&1 | grep -E 'can inline|cannot inline'

./foo.go:12:6: can inline hotFunction
./foo.go:18:6: cannot inline bigFunction: function too complex: cost 117 exceeds budget 80
./foo.go:22:6: cannot inline interfacingFunction: call to unknown method

Default budget is 80 AST nodes. Hard blockers:

Calls through interfaces. The compiler doesn't know what concrete method gets called. No inlining.
Calls to functions that contain loops with for range over a channel. Historically blocked, though the mid-stack inliner has improved this.
Recursive functions. Obvious.
Functions over the budget. Refactor smaller if the call is hot.

When to care:

Never in normal code. Go inlines what it can; your code runs.
Sometimes in tight hot loops where the call overhead is 10%+ of the total work. Benchmark shows it.
Occasionally when you control an interface boundary and can replace it with a concrete type on a hot path.

Don't structure your code around inlining. Code readability beats hypothetical call-overhead wins in nearly every case.

Benchmarks: The Ground Truth

Every perf claim should be backed by a benchmark. testing.B is the tool:

func BenchmarkEncodeResponse(b *testing.B) {
    resp := newResponse()
    b.ReportAllocs()
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        _, _ = encode(resp)
    }
}

Run:

$ go test -bench=BenchmarkEncode -benchmem -count=5

-count=5 runs each bench 5 times, so you can compare variance. Don't trust a single run. Hardware, OS scheduling, thermals — all add noise.

For comparing two implementations:

$ go test -bench=BenchmarkEncodeResponse -benchmem -count=10 ./... > old.txt
# (change code)
$ go test -bench=BenchmarkEncodeResponse -benchmem -count=10 ./... > new.txt
$ benchstat old.txt new.txt

benchstat (golang.org/x/perf/cmd/benchstat) gives you statistical significance. If the difference isn't statistically meaningful, you didn't actually improve anything — you just rolled the dice differently.

The 80/20 of Go Performance

After enough of this work, a few patterns dominate the real wins:

Query shape, not language. A slow endpoint is usually doing 10 DB queries when it could do 1. Go is almost never the bottleneck; the data layer is.
Network hop count. Every inter-service call adds latency. Merging two small services or co-locating tight integrations beats any language-level optimization.
Caching at the right layer. A well-placed LRU cache saves more than micro-optimizing the uncached path.
Preallocating known-size slices/maps. make([]int, 0, n) when you know n is almost free. The default make([]int, 0) reallocates as you append.
Avoiding interface boxing in loops. This is the one micro-optimization that regularly shows up in real profiles.

Everything else — sync.Pool, escape analysis hand-tuning, loop unrolling — is a long-tail optimization. Worth it when profiling tells you it is. Premature otherwise.

A Habit I Recommend

Before adding any optimization, do exactly three things:

Take a profile with the optimization off. Save it.
Apply the optimization.
Take a profile with the optimization on. Compare.

If the comparison doesn't show clear improvement on the metric you cared about, revert. Do not add complexity without evidence.

This sounds obvious. Almost nobody does it. Most perf work in Go codebases accumulates dead optimizations that add nothing or actively hurt — but nobody knows which, because nobody benchmarked.

The Habit That Compounds

Go's performance tooling is better than Go's performance culture gives it credit for. pprof, escape analysis, inlining diagnostics, and benchmarks are built in. They're precise. They tell you the truth.

The reason most Go code isn't as fast as it could be isn't that Go is slow (it isn't). It's that engineers copy-paste optimizations they haven't measured, call the work done, and move on. The few engineers who profile first and optimize second write code that's actually fast — and usually simpler than the ritual-heavy version.

Profile first. Everything else follows.

sync.Pool in Go: When It Actually Helps, and When It Quietly Hurts — the one Go optimization most likely to be misapplied.
Why Go Handles Millions of Connections: User-Space Context Switching, Explained — understanding the runtime is the prerequisite to understanding profiles.
Testing Real-World Go Backends Isn't What Many People Think — benchmarking is the last mile of testing.

sync.Pool in Go: When It Actually Helps, and When It Quietly Hurts

Harrison Guo — Mon, 20 Apr 2026 16:28:42 +0000

sync.Pool is one of those Go features that shows up prominently in "how to write fast Go" blog posts and then gets applied to everything. The result is a codebase sprinkled with pools that don't help and sometimes hurt. Most Go code I review does not need sync.Pool. The code that does need it often uses it wrong.

This is a working engineer's take on when pooling actually helps, when it's wasted effort, and the specific traps it creates.

tl;dr — sync.Pool is a GC-pressure reducer for workloads that allocate large-ish, short-lived objects at high frequency. It is not a general-purpose optimization. The cases where it clearly helps: per-request buffers in HTTP handlers, encoder/decoder instances, JSON buffers, protocol frame buffers. The cases where it hurts or is wasted: small objects, infrequent allocations, long-lived state, and any code that forgets to reset pooled items. Benchmark before and after — always.

What sync.Pool Actually Does

sync.Pool is a free-list for objects that the GC can clear. You Get() an object (fresh or recycled). You use it. You Put() it back. The runtime tries to give you a recycled one next time, but reserves the right to drop the whole pool on GC.

Key properties:

GC clears pools on every cycle. This is crucial. Pools are not a long-term cache — they're a hint to the runtime that "if you're going to collect these, wait a moment in case they get reused first."
Per-P (per-scheduler-thread) local storage. Most Get()/Put() calls hit a goroutine-local pool with no contention. Scaling across cores is nearly free.
No guarantees. A Get() might return a fresh object. A Put() might be discarded if the pool is full or the GC just fired.

This design is exactly right for "reusable scratch space." It's wrong for "cached resources I need to stay around" (use a real cache instead).

Should You Pool This?

Most paths in real code exit this flow long before hitting "use". That's correct.

When Pooling Helps: Per-Request Buffers

Canonical case. An HTTP handler serializes a response to a buffer, writes the buffer, moves on. The next request does the same thing. Without pooling, the GC collects the buffer every request. With pooling, the buffer is reused:

var bufferPool = sync.Pool{
    New: func() interface{} {
        return bytes.NewBuffer(make([]byte, 0, 4096))
    },
}

func handler(w http.ResponseWriter, r *http.Request) {
    buf := bufferPool.Get().(*bytes.Buffer)
    defer func() {
        buf.Reset()
        bufferPool.Put(buf)
    }()

    writeResponse(buf, r)
    w.Write(buf.Bytes())
}

Under realistic load (thousands of requests per second), this typically reduces allocation pressure by 20-40% and measurably lowers GC pause times. The exact number depends on your allocation pattern, but the principle holds: large, frequent, short-lived allocations are exactly what pooling is for.

What makes this the canonical case:

Buffers are big enough (4KB initial) that the allocation actually matters.
They're frequent — thousands per second.
Short-lived — used within one request.
Easy to reset — buf.Reset() clears it cleanly.
Same shape every time.

When you see a request-scoped buffer that fits all five, pooling almost always pays.

When Pooling Is Wasted Effort

Small objects. Pooling a 24-byte struct with three fields is almost never worth it. The pool's own overhead (per-P lookup, interface boxing) is larger than the allocation. Benchmark to confirm — you'll see allocs/op go down but ns/op stay the same or go up.

// Not worth it:
type Small struct { a, b, c int }
var smallPool = sync.Pool{New: func() interface{} { return &Small{} }}

// Just use new(Small) or &Small{}

Infrequent allocations. If your code path runs once an hour, pooling saves nothing meaningful. The GC handles a handful of allocations just fine.

Long-lived state. Connection objects, database handles, caches. These shouldn't be in sync.Pool — they should be in a proper cache or connection pool (like *sql.DB, which internally manages connections without sync.Pool).

Anything you can't reliably reset. If an object has state that needs to be "returned to zero," and you can forget to zero it, you're one typo away from data leaking between requests.

The Reset Trap

The single most dangerous mistake with sync.Pool: forgetting to reset the object before putting it back, or reusing it before clearing whatever was in it.

// Wrong:
buf := pool.Get().(*bytes.Buffer)
buf.Write(responseData) // might not start empty
w.Write(buf.Bytes())
pool.Put(buf) // buf still has data; next caller might see it

// Right:
buf := pool.Get().(*bytes.Buffer)
buf.Reset() // ← explicit
buf.Write(responseData)
w.Write(buf.Bytes())
buf.Reset()
pool.Put(buf)

This has caused real production incidents. Pooled buffers across request handlers have leaked bearer tokens, user PII, and password reset codes when a reset was missed. The runtime doesn't help — there's no "enforce reset" mechanism. You have to do it.

Habits that reduce the risk:

Always pair Get with a defer Reset+Put at the top of the function.
Reset at both ends (on Get and on Put) — paranoid but effective.
For byte slices, shrink before return: buf.Reset() on a bytes.Buffer resets length but keeps capacity — that's usually what you want. For a raw []byte, use buf[:0].
Make your New function return a pre-reset object. Don't assume it's always "fresh."

The Alloc Benchmark Methodology

The only honest way to know if pooling is helping is go test -bench -benchmem. Here's what a useful benchmark looks like:

func BenchmarkWithoutPool(b *testing.B) {
    b.ReportAllocs()
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        buf := bytes.NewBuffer(make([]byte, 0, 4096))
        writeResponse(buf, exampleRequest)
        _ = buf.Bytes()
    }
}

func BenchmarkWithPool(b *testing.B) {
    b.ReportAllocs()
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        buf := bufferPool.Get().(*bytes.Buffer)
        buf.Reset()
        writeResponse(buf, exampleRequest)
        _ = buf.Bytes()
        bufferPool.Put(buf)
    }
}

Run:

$ go test -bench=. -benchmem
BenchmarkWithoutPool-10    200000    8431 ns/op    4352 B/op    3 allocs/op
BenchmarkWithPool-10       500000    3214 ns/op     128 B/op    1 allocs/op

Look for two things:

allocs/op drops significantly (here: 3 → 1).
ns/op drops or stays flat (here: 8431 → 3214).

If allocs/op drops but ns/op goes up, pooling is adding overhead without saving enough GC pressure to justify itself. That's the "wasted effort" signal.

The benchmark alone isn't enough, though — you also need production evidence. pprof heap profiles before and after deployment should show reduced allocation. If the prod numbers don't match the benchmark, you're measuring the wrong thing.

A Pattern That Actually Works: Scoped Pools

One pattern I've found useful: scope the pool to the type of work it serves. Don't have one giant pool that everything pulls from.

// JSON response buffer pool
var jsonBufPool = sync.Pool{
    New: func() interface{} { return bytes.NewBuffer(make([]byte, 0, 4096)) },
}

// Protocol frame buffer pool (different typical size)
var frameBufPool = sync.Pool{
    New: func() interface{} { return bytes.NewBuffer(make([]byte, 0, 64*1024)) },
}

Why separate pools matter: if you have one shared pool, you might Get() a 64KB buffer when you needed a 4KB one and waste memory. Or worse, you might Get() a 4KB one for a 64KB job and grow it (defeating pooling's purpose).

Separate pools stay close to their intended sizes. Each pool's items are homogeneous. The New function's initial capacity reflects the typical workload.

The Big Thing `sync.Pool` Isn't

sync.Pool is not a replacement for bounded resource pools (database connections, HTTP clients, goroutine worker pools). Those need explicit lifecycle management, health checks, and non-discardable state. Use a real pool library for them.

sync.Pool is also not a cache. A cache holds items you want to find again. sync.Pool holds items you might reuse if one's convenient, and discards them otherwise. Different primitive for a different problem.

What Actually Matters

Most Go code is fast enough without pooling. Before adding sync.Pool to your hot path, ask:

Have I actually benchmarked this with -benchmem?
Are the objects I'd pool both large and frequent?
Can I reliably reset them?
Is GC pressure in pprof profiles actually a problem?

If any answer is no, skip the pool. The simpler code is almost always the better code.

The cases where pooling pays are real but narrower than internet wisdom suggests. Per-request buffers, protocol frame buffers, encoder/decoder state, crypto scratch space. Beyond that, the pool usually adds more lines of code than it saves nanoseconds — and each of those lines is one more place where a missing Reset() can leak bytes between requests.

Measure. Then decide.

Go's Concurrency Is About Structure, Not Speed — the bigger principle: Go optimizes for correct structure, not raw speed.
Testing Real-World Go Backends Isn't What Many People Think — how to actually benchmark and prove a pool helps.

IronSys: A Production Blueprint for Modern Concurrency

Harrison Guo — Mon, 20 Apr 2026 16:28:01 +0000

In the last post I walked through the four concurrency pillars — shared memory + locks, CSP, actors, STM — and argued that real systems mix them on purpose. Someone reasonably asked: okay, but what does that actually look like? Fair question. Abstract taxonomy is less useful than a worked example.

IronSys is that worked example. It's a composite blueprint — not a real service, but representative of a class of services I've designed, helped design, or debugged in production. Let's say it's a mid-sized backend system: public API, stateful user sessions, streaming data in, aggregation and reporting out. The kind of thing that appears in the middle of any serious platform.

The interesting part isn't the features. It's which concurrency primitive shows up where, and why.

tl;dr — IronSys is a composite production blueprint: a multi-service Go backend with stateful user sessions, streaming ingest, and usage aggregation. It uses CSP channels for pipelines and coordination, a goroutine-per-entity actor pattern for stateful sessions, mutexes and atomics for hot shared counters, and durable queues for cross-service handoff. Each primitive is picked for a specific failure mode. The pattern is not "mix for variety"; it's "match the primitive to the work."

The System Shape

Before deciding on concurrency primitives, sketch the work shapes. IronSys has four:

Public API — request/response, modest concurrency, latency-sensitive. The classic HTTP backend.
Live sessions — stateful, long-lived per-user entities. Think multiplayer game server, collaborative editor, real-time dashboard.
Streaming ingest — high-throughput events arriving over Kafka/NATS, fanned out to workers for processing.
Batch aggregation — periodic rollup jobs that read from storage, compute, write back.

Four shapes, four concurrency patterns. The wrong design would apply the same primitive to all four. The right design picks each separately.

stateless · request/response"]"/>

The API Handlers

Nothing fancy. Stock Go HTTP server. Each request is its own goroutine (Go's runtime does this automatically). Shared state — rate limiters, cache, config — is protected by mutexes or atomics:

type RateLimiter struct {
    mu      sync.Mutex
    buckets map[string]*bucket
}

func (r *RateLimiter) Allow(key string) bool {
    r.mu.Lock()
    defer r.mu.Unlock()
    b, ok := r.buckets[key]
    if !ok {
        b = newBucket()
        r.buckets[key] = b
    }
    return b.allow()
}

Obvious choice. The contention is bounded by request rate, the state is small, a mutex is the simplest possible tool. Over-engineering here — sharded maps, lock-free data structures — buys nothing.

What IronSys does here that many teams miss: every handler is context-aware from request entry:

func (s *Server) HandleFoo(w http.ResponseWriter, r *http.Request) {
    ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
    defer cancel()

    result, err := s.service.Foo(ctx, parseReq(r))
    writeResponse(w, result, err)
}

Context flows everywhere downstream. The handler layer is boring; that's the point.

The Live Sessions — Actor Pattern in Go

Each active user session is a long-lived goroutine with an inbox channel. I call this the goroutine-per-entity pattern — it's Erlang actors without the runtime, built from Go primitives.

type Session struct {
    id       SessionID
    mailbox  chan SessionCmd  // the "actor" inbox
    shutdown chan struct{}
    state    sessionState      // private to this goroutine
}

type SessionCmd struct {
    op     string
    args   interface{}
    reply  chan<- SessionReply // optional reply channel
}

func runSession(ctx context.Context, s *Session) {
    defer close(s.mailbox)
    for {
        select {
        case cmd := <-s.mailbox:
            s.handle(cmd)
        case <-s.shutdown:
            s.flush() // persist final state
            return
        case <-ctx.Done():
            return
        }
    }
}

Why this pattern, not "session is a struct with a mutex"?

State is private to one goroutine. No sharing, no locks, no lock-ordering bugs. The session state is accessed by exactly one execution context.
Serial message processing. Commands process one at a time, in FIFO order. Business invariants hold naturally.
Natural location for cross-session coordination. Each session is a message destination. Broadcasting to all sessions, or routing a command to a specific session, is just "send on its inbox."
Clean lifecycle. The goroutine runs until shutdown or ctx.Done. State is flushed once, on exit. No race between "is this session still alive" and "did we finish writing its state."

The manager that creates and routes to sessions looks like:

type SessionManager struct {
    mu       sync.RWMutex
    sessions map[SessionID]*Session
}

func (m *SessionManager) Get(id SessionID) (*Session, bool) {
    m.mu.RLock()
    defer m.mu.RUnlock()
    s, ok := m.sessions[id]
    return s, ok
}

func (m *SessionManager) Start(ctx context.Context, id SessionID) *Session {
    m.mu.Lock()
    defer m.mu.Unlock()

    s, ok := m.sessions[id]
    if ok { return s }

    s = newSession(id)
    m.sessions[id] = s
    go runSession(ctx, s) // supervisor goroutine
    return s
}

Note the mixing: the manager uses a mutex-protected map (shared state with a clear owner), individual sessions use the actor pattern (isolated state, message-passing). Two primitives, picked per-job.

This pattern scales to millions of sessions because goroutines are cheap. I've seen this exact pattern serve 400K concurrent sessions on a single pod.

The Streaming Ingest — Bounded Worker Pool (CSP)

Kafka consumer feeding a worker pool. Canonical CSP territory:

func runConsumer(ctx context.Context, cons *kafka.Consumer) error {
    jobs := make(chan Event, 256)
    var wg sync.WaitGroup

    // Fixed worker pool
    for i := 0; i < workerCount; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for {
                select {
                case job, ok := <-jobs:
                    if !ok { return }
                    if err := process(ctx, job); err != nil {
                        log.Error(err)
                    }
                case <-ctx.Done():
                    return
                }
            }
        }()
    }

    // Producer
    go func() {
        defer close(jobs)
        for {
            msg, err := cons.ReadMessage(ctx)
            if err != nil { return }
            select {
            case jobs <- msg:
            case <-ctx.Done():
                return
            }
        }
    }()

    <-ctx.Done()
    wg.Wait()
    return ctx.Err()
}

The bounded channel is the concurrency clamp. Kafka can push as fast as it wants; the worker pool consumes at its own pace; backpressure propagates back to Kafka's consumer offset naturally.

Why not actors here? Because the work items are stateless — you're processing events, not maintaining per-entity state. The overhead of an actor (mailbox, dispatch, ownership) is unjustified. CSP is the right fit.

Why not mutex + a worker loop? You could, but the channel primitive is exactly the right shape — bounded capacity + safe cross-goroutine handoff + graceful shutdown — without needing to build those three features yourself.

The Batch Aggregation — Pipelines + errgroup

Nightly rollup: read from storage, compute per-account aggregates, write back.

func runRollup(ctx context.Context, input <-chan Event) error {
    g, gctx := errgroup.WithContext(ctx)

    // Stage 1: parse
    parsed := make(chan ParsedEvent, 64)
    g.Go(func() error {
        defer close(parsed)
        return parseStage(gctx, input, parsed)
    })

    // Stage 2: aggregate (keyed by account)
    agged := make(chan Aggregate, 64)
    g.Go(func() error {
        defer close(agged)
        return aggregateStage(gctx, parsed, agged)
    })

    // Stage 3: persist
    g.Go(func() error {
        return persistStage(gctx, agged)
    })

    return g.Wait()
}

Three stages in a pipeline. Each stage is a goroutine, connected by bounded channels. errgroup ties them together: first error cancels the whole pipeline.

The aggregation stage internally uses a map protected by a mutex, because it's a single goroutine reading the map — no contention at all, but still safe if a future change introduces more readers.

This is textbook CSP: the topology of channels is the architecture. Read the code and the shape of the computation is obvious.

The Cross-Service Handoff — Durable Queues

IronSys talks to two other services: a billing service (async, eventually consistent) and an auth service (sync, immediate).

For billing: a dedicated NATS JetStream subject with at-least-once delivery. Usage events go in one end; the billing service reads them. The emission codepath has a local write-ahead log so that if NATS is briefly down, events buffer on disk and replay when the connection recovers.

For auth: gRPC with tight timeouts. Caller owns completion. If auth is slow, the API handler's deadline fires and the request fails fast.

Two different ownership models for two different shapes of work. See: RPC vs NATS: Who Owns Completion.

How the Primitives Map

Summarizing which primitive serves which job in IronSys:

Work shape	Primitive	Why
HTTP request handling	Stock `net/http` + goroutine per request	Language default, right for stateless
Hot shared state (rate limiter, cache)	Mutex / atomic	Simplest primitive that works
Stateful user sessions	Goroutine-per-entity (actor-like)	Isolated state, message-passing, serial processing
Session directory	RWMutex-protected map	Shared lookup, read-heavy
Streaming event processing	Bounded channel + worker pool (CSP)	Backpressure, parallelism, graceful shutdown
Multi-stage data pipeline	CSP pipeline + errgroup	Stage topology = architecture; first-error cancels all
Async cross-service handoff	Durable queue (NATS JetStream / Kafka)	Receiver owns completion, at-least-once delivery
Sync cross-service call	gRPC with ctx timeout	Caller owns completion, fast failure

Notice: all four concurrency pillars show up. Mutexes in the rate limiter. CSP in the event pipeline. Actors (in pattern) in the session runtime. (STM is missing; it would show up if I were doing this in Clojure or Haskell.)

What This Architecture Gets Wrong

Every architecture has weaknesses. IronSys's are real:

The actor pattern isn't real actors. Without Erlang-style supervision, if a session goroutine panics, Go's default behavior is to kill the process. Adding panic recovery per-session is easy but not free. In practice, most teams hit this 6 months in, add a recovery wrapper, and move on.
Bounded channels can mask slow downstream. If a channel fills up and the producer blocks, that's backpressure — great. But if the channel is buffered too large, you can buffer a lot of work into memory before realizing downstream is slow. Tune buffer sizes with measurements, not guesses.
Goroutine-per-entity has a per-session baseline cost. Cheap but not free. A million sessions is ~2.5GB of goroutine stacks. For services where most entities are inactive, a lazy pattern (spin up on activity, suspend to disk on idle) is better.
Mixing paradigms cognitively. New engineers have to learn four patterns instead of one. The productivity hit is real for the first two weeks; the payoff is in the next two years.

What This Blueprint Is Really Selling

A system with four work shapes should have four concurrency patterns, not one stretched to cover everything. The four pillars aren't theoretical; they map to real design decisions, and production Go services that use them deliberately are easier to reason about than those that don't.

What IronSys is really selling is intentional heterogeneity. Every primitive is there for a reason. Every reason is traceable to a specific failure mode you want to prevent. The architecture should be legible — a new engineer reading the code should understand why a channel is there instead of a mutex, why a session has its own goroutine instead of being a struct in a shared map, why billing goes through a durable queue instead of a gRPC call.

If you can't answer "why this primitive here," the code isn't finished. It's just working, for now.

Blueprints are useful precisely because they're generic. The specifics of your system will be different. But the decision framework — what's the work shape, what's the failure mode, what's the right primitive — is the same every time.

From Locks to Actors: The Four Pillars of Modern Concurrency — the taxonomy behind the choices in IronSys.
Go's Concurrency Is About Structure, Not Speed — chan and context as the glue across all of these.
RPC vs NATS: It's Not About Sync vs Async — It's About Who Owns Completion — the cross-service handoff choices.
Testing Real-World Go Backends Isn't What Many People Think — how you verify a system like this actually holds up.

Docker Kubernetes: What They Really Changed (It's Not What You Think)

Harrison Guo — Mon, 20 Apr 2026 07:13:19 +0000

"A Docker container is basically a lightweight VM, right?" No. That sentence alone causes more architectural misunderstandings than any other in modern backend engineering. A VM virtualizes hardware. A container is a set of Linux kernel features — namespaces, cgroups, overlay filesystems — wrapped in a nicer CLI. Same host kernel, same memory space, same attack surface if the kernel has a bug. The marketing that says otherwise has cost teams real money in misconfigured production.

Kubernetes gets the same treatment. "It's a tool for running containers." Also not really. Kubernetes is a distributed scheduler, service mesh, declarative control plane, and reconciliation engine. Containers are one of the things it happens to run. Treating Kubernetes as "container orchestration" produces systems that break in predictable, frustrating ways — because the team never learned that the reconciliation loop, not the container, is the thing that actually matters.

This is a working engineer's re-read of what Docker and Kubernetes actually changed. Not the marketing story. The underneath-the-hood story that tells you when to reach for them and when they're overkill.

tl;dr — Docker didn't invent Linux namespaces, cgroups, or filesystem layering; it packaged them into a developer-friendly workflow. That workflow is what changed. Kubernetes didn't invent distributed scheduling, service discovery, or rolling deployments; it standardized the declarative, reconciliation-loop pattern for all of them. That pattern is what changed. Understanding these primitives (namespaces + cgroups + reconciliation loops) tells you when to reach for the tools and when the tools are overkill.

What Docker Actually Is

Docker is a set of Linux kernel features wrapped in a nice CLI and an image format. The features existed before Docker; they just weren't accessible.

Linux namespaces — process, mount, network, IPC, UTS, user, cgroup. Each namespace gives a process its own view of that resource. When your container thinks it has PID 1, it really thinks so; inside its PID namespace, the host's init is invisible.
cgroups (v1/v2) — resource accounting and limits. How much CPU, memory, I/O bandwidth a group of processes can use. This is why a misconfigured container can eat a host's memory and take everything else down.
Union / overlay filesystems — the thing that lets you stack "base image" + "layer 1" + "layer 2" without copying. OverlayFS on modern kernels.
Image format (OCI) — a standard way to package a root filesystem plus metadata into something reproducible.

Docker's innovation was not inventing any of this. It was making them accessible. docker run -p 8080:80 nginx hides a beautiful horror of namespace creation, iptables rules, virtual ethernet pairs, overlay mounts, and cgroup assignment. Before Docker, you'd have spent a week reading unshare(2) and ip netns add to reproduce this. After Docker, you did it in a workshop afternoon.

What actually changed: deployments became reproducible. The image you built on your laptop contained everything needed to run — OS libraries, Python version, environment. "Works on my machine" stopped being a coping mechanism and started being a legitimate development artifact. That's the Docker revolution. Not containers. Reproducible, portable environments.

The thing that is not true, despite the marketing: Docker containers are not VMs. They share the host kernel. A kernel exploit in one container can reach the host and other containers. Containers are a soft isolation — good enough for most production multi-tenant workloads, not good enough for hostile tenants.

What Kubernetes Actually Is

Kubernetes is a declarative control plane built on the reconciliation loop pattern. This is the single most important idea to internalize.

You write a manifest describing the desired state: "three replicas of this deployment, exposed through this service, attached to this config."
You hand the manifest to the control plane: "make it so."
Kubernetes runs an unending loop: observe the current state, compare to desired, take actions to close the gap.

Everything Kubernetes does follows this pattern:

Deployment controllers watch the pod count, scale up if low, scale down if high.
ReplicaSet controllers ensure N identical pods exist.
Service controllers maintain the iptables / IPVS / eBPF rules that route virtual IPs.
Ingress controllers watch Ingress resources and configure the edge proxy.
The scheduler watches for unscheduled pods and binds them to nodes.
Node controller watches node health and evicts pods from unhealthy nodes.

Your application is just the data in the reconciliation loop. The loops run forever, closing gaps. That's Kubernetes.

Every Kubernetes feature — Deployments, Services, Ingresses, HPAs, CronJobs, StatefulSets — is some controller running this exact pattern. Once you see it, the platform stops being magic.

What actually changed because of this: the operational model became shared across companies. Before Kubernetes, every engineering team had a bespoke orchestration system: a collection of Chef/Puppet/Ansible recipes, some custom scripts, a deploy button, and a few senior engineers who knew which knobs to turn during incidents. Different at every company. Opaque to new hires. Sensitive to key-person risk.

Kubernetes is many things, but the single biggest thing it did was replace a hundred bespoke orchestration glues with one standard. It's not the best tool for every problem — Nomad is simpler, ECS is more managed, Cloud Run hides the thing entirely — but it's the standard, and "it's the standard" has real value: hires know it, vendors build for it, books exist, the job market is liquid.

The Mental Model Most People Miss

Once you see "reconciliation loop," you stop asking questions Kubernetes doesn't answer.

"How do I deploy?" You don't. You update a manifest. A controller observes the change and reconciles.

"How do I roll back?" You don't. You update the manifest back. A controller observes the change and reconciles in the other direction.

"Why did my pod get killed?" Because a controller decided the current state (this pod is here, on this node) didn't match the desired state (node is draining, or pod is over its memory limit, or a replica count decreased). It closed the gap.

"Why can't I SSH in and hand-edit things?" Because the next reconcile loop will undo your edit. The manifest is the source of truth. If you want to change behavior, change the manifest.

This is a shift from imperative ops ("run these commands to deploy") to declarative ops ("the system should look like this; make it so"). Git becomes the history of what your infrastructure should be. Time travel works. Change review works. Disaster recovery becomes "re-apply the manifests to a new cluster." When it clicks, you stop fighting the platform.

Until it clicks, the platform feels maddening. "I just want to run a container" — yes, but the platform doesn't care what you want to do once. It cares about the continuous state. Every action through kubectl apply is a statement of desired state, not an imperative command.

What Changed in Practice

Concretely, what looks different on a team that's moved from "SSH into the box and systemctl restart" to a reconciled-state model:

Deployment became a git push

Before: log into the bastion, pull the latest build, restart the service, watch the log.
After: merge to main, CI pushes image to registry, ArgoCD/Flux observes the manifest change, the Deployment controller updates the ReplicaSet, pods roll gradually.

Benefits: change review, audit trail, rollback by git revert, consistent deploys across teams.
Costs: debugging a broken deploy requires understanding the CD pipeline, the manifest, and the controller that's reconciling. The failure mode surface is wider.

Scaling became a number in a file

Before: write a script that watches metrics, calls the cloud API, hopes for the best.
After: replicas: 10 in a manifest, or an HPA (Horizontal Pod Autoscaler) that watches metrics and adjusts the Deployment.

Benefits: declarative, versioned, reproducible.
Costs: HPA behavior is subtle — wrong thresholds cause thrashing, wrong metrics cause over/underscaling. Many teams never invest in tuning.

Service discovery became DNS

Before: register in Consul, read from Consul, have a catalog. Or hardcode IPs. Or service registry.
After: my-service.my-namespace.svc.cluster.local resolves to a stable virtual IP. Kube-proxy or CNI load-balances to healthy pods.

Benefits: services don't need to know how other services run. Standard DNS.
Costs: the DNS / networking layer is one of the hardest parts of Kubernetes to debug. When service discovery breaks, you're reading iptables or eBPF maps, not a Consul dashboard.

Configuration became a manifest

Before: environment variables, .env files, maybe Consul KV.
After: ConfigMaps and Secrets, mounted as env vars or volumes.

Benefits: versioned, reviewed, separate from code.
Costs: changing a ConfigMap doesn't automatically restart pods. You have to annotate the Deployment or use something like reloader. New users get bitten by this constantly.

When Kubernetes Is Overkill

I'll say it directly: most teams adopting Kubernetes for the first time don't need it.

Rules of thumb:

Two or three services, one team: you don't need Kubernetes. ECS, Nomad, Cloud Run, or even systemd + Ansible will do. The operational overhead of Kubernetes exceeds its benefit at this scale.
Ten to twenty services, small team: Kubernetes starts breaking even if you pick a managed service (EKS, GKE, AKS). Don't run your own control plane.
Fifty+ services, multiple teams, serious release engineering needs: Kubernetes is probably the right call. The cost of complexity is amortized over the benefits of a shared declarative platform.

The dangerous zone is 5-15 services on a small team. At that scale, Kubernetes often wins the resume-driven-development vote and loses the actual-outcomes vote. Pick a simpler tool.

When Kubernetes Is the Right Answer

The jobs where Kubernetes genuinely shines:

Multi-service, multi-team engineering orgs where consistency matters more than per-service optimality.
Scale-out workloads with heterogeneous shapes — web apps, job runners, ML batch jobs, stateful databases, all on one platform.
Teams that want declarative infrastructure — GitOps via ArgoCD/Flux, infra PRs reviewed like code.
Workloads with nontrivial scheduling — affinity rules, taints, GPU allocation, spot instances.
Operators ecosystem — Kubernetes operators (Prometheus operator, cert-manager, etc.) let you extend the same reconciliation model to application-specific concerns.

Notice the pattern: Kubernetes wins when you want the platform's primitives — declarative state, reconciliation, operators — beyond just container scheduling. If you only want "run my container," you're buying a jumbo jet to fly to the next town.

What I'd Tell a Team Starting Fresh

Two concrete takeaways I'd hand to engineers thinking about Docker and Kubernetes.

For Docker: the image isn't the point. Reproducibility is. An image built on your laptop that runs unchanged in CI and production — that's the contract you got. Break it (say, by mutating state inside the running container) and you lose the value. The container is a delivery mechanism for a reproducible environment.

For Kubernetes: the manifest is the source of truth. Every piece of your infrastructure — deployments, services, secrets, ingresses, policies — lives in git. Every change is a git change. Every rollback is a git revert. If you find yourself running kubectl edit on production, something is wrong with your workflow, not with Kubernetes.

Both tools won because they codified patterns that were already emerging in sophisticated shops. They didn't invent the patterns. They made them accessible, portable, and standard. That's the fifteen-year revolution. Not containers. Not YAML. The standardization of patterns that used to require a senior infrastructure team to implement from scratch at every company.

When you work with the grain of the pattern — reproducible environments for Docker, reconciled declarative state for Kubernetes — both tools get out of the way. When you fight the grain, they fight back.

Why Go Handles Millions of Connections — Linux primitives that Docker is built on, seen from the language side.
Observability and Cost Attribution: Why One Pipeline Isn't Enough — what happens to operational complexity when you have dozens of services.
Scale-Up vs Scale-Out: Why Every Language Wins Somewhere — the architectural decision that drives whether you need Kubernetes at all.

Observability and Cost Attribution: Why One Pipeline Isn't Enough

Harrison Guo — Mon, 20 Apr 2026 07:13:17 +0000

A team I worked with tried to build their billing system on top of their tracing pipeline. The idea was clean: every operation already generates a span; spans already have duration and attributes; adding user_id and billable_units to each span lets finance query the trace store to compute invoices. One pipeline, less infrastructure. Beautiful.

Six weeks before the first billing cycle, the wheels came off. The tracing system was sampling at 10% because full-capture was too expensive. The sampler was head-based, meaning whether a trace got kept was decided at request entry, long before the code knew whether the request was billable. Some users got charged for 10% of their actual usage; others got free service. Nobody's invoice agreed with the other team's report.

The workaround — "don't sample billable traces" — sounded reasonable, broke the tracing pipeline's cost model immediately, and created a dozen new edge cases around which requests counted as "billable." Within a month the team was reluctantly building a second pipeline for billing. They still had the first one for traces. Now they had two pipelines that disagreed with each other.

The postmortem landed on a single sentence: observability and cost attribution aren't the same problem, and pretending they are is expensive twice.

tl;dr — Tracing and metrics optimize for signal-to-noise — you want the interesting outliers, sampling is OK, dropping data is tolerable. Billing optimizes for completeness and auditability — every event must be captured and durably recorded, end of story. The two pipelines have opposite trade-offs on sampling, retention, schema evolution, and cost. Building them as one pipeline forces one of the two to lose. Build them as two, share primitives where possible, let each specialize where it must.

Why They Look Alike

Observability pipelines and billing pipelines do look eerily similar from a distance:

Both capture events from production systems.
Both attach metadata to those events.
Both aggregate events over time windows.
Both export to a query layer.

It's tempting — especially to engineers who like clean architecture — to say these are the same problem and build one system. The similarity is surface. The constraints are opposite.

Axis	Observability	Billing
Loss tolerance	High (sampling is fine)	Zero
Latency tolerance	Seconds to minutes	Minutes to hours is fine
Retention	Days to weeks	Years
Schema evolution	Fast, frequent	Slow, with audit trail
Cardinality profile	Low cardinality on hot dims	Arbitrary (per user, per resource)
Consumers	SRE, engineering, on-call	Finance, legal, customer
Failure mode	Blind spot in a dashboard	Wrong invoice, legal exposure

The one that really matters: loss tolerance. Everything else follows from it.

A tracing pipeline that drops 10% of spans is fine. You still see the outliers. You still find the slow paths. The system does its job.

A billing pipeline that drops 10% of events is a disaster. Some users underpay. Some users overpay. Finance reconciliation fails. You end up manually auditing transactions for weeks.

The moment one pipeline has to satisfy zero-loss and the other can tolerate 90% sampling, you have two different systems whether you wanted one or two.

The Dual-Path Architecture

The design I keep reaching back to is straightforward: two pipelines, shared ingest, separate durability and query paths.

Two emission paths from the application. Two pipelines behind them. Each tuned for its job.

The tracing path

Stays conventional. OpenTelemetry SDK emits spans. Collector applies head-based or tail-based sampling. Hot store (Tempo, Jaeger, Grafana Cloud) gets 10-20% of the volume. Retention a few days to a few weeks. Query layer is for engineers debugging incidents.

What I optimize for here:

Cost per span — you're keeping billions; every byte matters.
Query latency — on-call wants answers in seconds.
Auto-instrumentation coverage — the fewer things you have to manually instrument, the better.

What I don't care about:

Full capture. Sampling is fine.
Long retention. You're debugging last Tuesday, not last fiscal year.
Per-user accuracy. If a single user's trace got dropped, nobody cares.

The usage-event path

The dedicated billing pipeline. Every billable operation emits a usage event — a small, structured record with everything finance needs and nothing it doesn't.

{
  "event_id": "ue_01HFNGR...",
  "occurred_at": "2026-02-14T18:22:30.145Z",
  "account_id": "acc_12345",
  "resource_id": "res_6789",
  "operation": "api.request",
  "dimensions": {
    "region": "us-east-1",
    "tier": "standard"
  },
  "units": {
    "requests": 1,
    "cpu_ms": 147,
    "egress_bytes": 8342
  },
  "idempotency_key": "req_abc_20260214182230"
}

The rules on this path:

Unsampled. Every billable operation emits exactly one event. No head sampling. No tail sampling. No "approximate."
Durable writes. Emitter has a local write-ahead log or durable queue. If the downstream is down, events buffer locally until delivery. No dropped events under partial failure.
Idempotency keys. Every event has a unique ID (or composite key) so downstream dedup is trivial. This lets you retry safely.
Schema versioned and immutable. Once an event shape is shipped, it doesn't mutate. New fields add a new version. Old versions keep working until you intentionally deprecate.
Long retention. Years, usually. Auditors ask for 2023's data in 2027.

The downstream infrastructure matches: Kafka or NATS JetStream with high replication factor for ingest, columnar warehouse (BigQuery, Snowflake, ClickHouse) for aggregation and query, separate auth and access control from engineering-facing tools.

What the two paths share

Not nothing. They share:

The trace/request ID. Usage events include the trace ID of the request that generated them. This is the one cross-pipeline link that matters — when finance escalates "this user says they were charged for X requests but they swear they only made Y," you want to be able to find the traces of those Y requests.
OpenTelemetry as the emission library. OTel can emit both spans and custom events. Using it for both keeps the instrumentation codepaths uniform. But the pipelines behind the emitter are different.
The application's definition of an "operation." Both pipelines have opinions about what counts as one operation. Keep that definition single-source.

Why Head-Sampling Kills Billing

Worth dwelling on the specific thing that breaks when you try to unify.

Head-based sampling decides whether to record a trace at entry, based on trace ID. It's O(1), stateless, and fair across traffic shapes — the standard default.

The failure: at entry time, the system has no idea whether this request will be billable. The sampler doesn't know if the user is on a paid plan, if the request will succeed, if it will hit a billable feature. It just picks randomly.

Tail-based sampling fixes part of this — you decide after the fact, based on span attributes. Now you can keep all errors, all slow requests, all requests from paid users. Better, but still subject to buffering limits. Heavy tail-based samplers sit in front of your trace ingest pipeline and drop spans when buffers fill, which still gives you lossy billing during traffic bursts.

The only sampler that's correct for billing is "capture everything." And "capture everything" is what the tracing pipeline tries to avoid, because that's what makes it expensive.

You can do "capture everything for billable operations, sample everything else" in one pipeline. It works. It also ends up being the most complex sampler you've ever written, with an exception branch that duplicates the decision logic from your actual billing code. The dedicated usage-event path is simpler.

Cardinality and the Per-User Problem

A related anti-pattern: attaching user ID as a Prometheus label.

Prometheus (and most metrics systems) store one time series per label combination. Add a user_id label to a metric that ten thousand users hit, and you just created ten thousand time series. Add a request_type label alongside, and that's ten thousand × request-type-count. Cardinality explodes. Your metrics storage bill goes with it.

The instinct is fine — "I want to track per-user throughput" — the mechanism is wrong. Metrics with high-cardinality labels are the square peg. Usage events are the round hole. Emit a usage event with account_id as a dimension, aggregate per-user in the warehouse at query time.

Rule I use: metrics for engineering-facing dashboards, events for business-facing attribution. If the label cardinality could exceed ~1,000 distinct values, it belongs in an event, not a label.

The Boring Operational Details

Where the two pipelines actually differ in day-to-day ops:

Retention. Tracing a few weeks, maybe. Billing store, years. Warehouse partitioning by date and account_id makes multi-year queries practical. Archive older partitions to object storage.

Access control. Traces: engineers. Billing events: accounting + support + an audit-only read path for legal. Not the same principals, not the same ACL model.

Schema governance. Traces: OTel semantic conventions, loose. Billing events: your own schema with a proto or Avro definition, version bumps tracked in a migration log, additive only.

Reconciliation. Billing needs to agree with itself. Daily reconciliation job that asserts "yesterday's event count per user equals the sum of the per-hour counts" catches silent drops early. No equivalent makes sense for tracing.

Replay. When a billing bug is discovered, you need to replay historical events through a fixed pipeline. Kafka's offset model makes this natural; NATS JetStream has it too. The tracing pipeline rarely needs replay — if the last two weeks of traces have a bug, you shrug and fix forward.

When You Can Get Away With One

Small workloads with no audit requirement, usage-based pricing below ~$1/user, and a team of three — one pipeline is fine. Add user attributes to spans, store them all, build a nightly aggregation job, call it billing. It works.

The threshold where it stops working is somewhere around:

Revenue per customer exceeds the cost of a mistake. At $10k/month per customer, a dropped event is a $10k issue.
The first auditor asks "show me exactly what this customer used in March 2024." Unsampled, durable, retrievable, signed — that's the table stakes for audit-grade billing, and sampled traces can't meet any of those.
Engineering starts wanting cheaper traces. When the tracing pipeline outgrows your budget and someone proposes "let's sample more aggressively," you're about to break billing.

When any of those lights up, separate the pipelines. The cheapest time to separate is before you've built tools on top of the unified one.

When to Invest in Splitting the Pipelines

Observability and cost attribution are adjacent problems that optimize for opposite things. A tracing pipeline that compromises on completeness becomes a bad billing pipeline. A billing pipeline that compromises on cardinality and retention becomes a bad tracing pipeline. Building one system that satisfies both usually produces two systems that satisfy neither.

The dual-path design isn't more complex. It's just honest about the constraints. Same emission library, same operation definition, two paths behind the emitter, each tuned for its job.

If you're about to launch usage-based pricing and you're planning to compute invoices from your trace store, rethink it now. The sooner you split, the cheaper the split.

NATS vs Kafka vs MQTT: Same Category, Very Different Jobs — why the durability choice on the billing path matters so much.
RPC vs NATS: It's Not About Sync vs Async — It's About Who Owns Completion — completion ownership applies to the emit path, too.

The 90% Problem: Why Most AI Agents Are Still Broken

Harrison Guo — Sat, 18 Apr 2026 23:02:07 +0000

Your Agent Works Great. Until It Doesn't.

You built an AI agent over the weekend. It calls tools, remembers context, follows instructions. You demo it to your team. Everyone's impressed.

Monday morning, a user types "rename Ember to Infernia." Your agent loops 15 times, burns through your API budget, and returns a response that doesn't contain the word "Infernia." A rename. One entity. One operation.

I've been there. I ran an eval suite on a production agent — 5 test cases, 5 runs each. Pass rate: 40%. Not on hard tasks. On things like "update the right character out of six" and "rename one entity." The model was GPT-4 class. Plenty capable. The problem was everything around the model.

This is the 90% problem:

Building the core loop (perceive → reason → act):  10% of the work
Making it not break in production:                  90% of the work

It took me a while to see where the problem actually was. The real gap wasn't missing features. It was open loops: verification that doesn't retry, memory that doesn't decay, compression that doesn't circuit-break.

Here's what I found — from analyzing Claude Code's leaked source, where a 1,729-line query.ts file contains a 1,421-line while(true) loop inside a roughly 512,000-line codebase, and from fixing a production agent's pass rate with code changes alone. No model upgrade. No prompt magic. Just engineering.

The Five Pillars — And Where Agents Actually Fail

Every production agent needs five things. Most only build two of them well.

Pillar	What It Does	Who Does the Work	Most Agents' Status
Context Management	What the LLM sees	Code orchestrates; LLM helps compress	Dump everything, hope for the best
Memory Management	What persists across sessions	Code orchestrates; LLM helps recall	Basic store/retrieve, no lifecycle
Reflection	Agent checks its own output	Code triggers; LLM judges	Not implemented or logs-only
Planning	Agent thinks before acting	LLM (decompose tasks, sequence steps)	Decent — LLMs are good at this
Tool Use	Agent interacts with the world	LLM selects, Code executes	Decent — most mature pillar

Planning and Tool Use work reasonably well because they ride on model improvements. GPT-3.5 struggled with tool calling; Claude Opus 4.6 is reliable. You get these improvements for free with model upgrades.

Context and Memory are where agents fail because they're engineering problems, not model problems. Reflection sits in the middle: the LLM can judge quality, but code still has to trigger that check, route the result, and do something with it. No model upgrade will fix a context pipeline that dumps 10 irrelevant entities into the prompt.

The LLM vs Code Divide

This is the most important insight for anyone building agents:

HIGH LLM dependence (improves with better models):
  Planning        → LLM generates the plan
  Reflection      → LLM evaluates quality
  Tool Selection  → LLM picks the right tool

LOW LLM dependence (never improves from model upgrades):
  Context Management  → Code sorts, filters, compresses
  Memory Management   → Code stores, retrieves, scores, decays
  Error Handling      → Code classifies errors, retries, circuit-breaks
  Tool Execution      → Code runs tools, parallelizes, batches
  State Management    → Code tracks progress, checkpoints

But low LLM dependence does not mean zero model calls. It means the failure mode is mostly in the orchestration. Even code-dominated pillars still use models in three very different ways:

1. Direct LLM call (sideQuery) — Code asks a narrow question, takes the answer, and moves on. Example: Claude Code's memory recall uses a single Sonnet side-query to choose 5 relevant memories from roughly 200 files.

2. Forked sub-agent — Code delegates a bounded task to a child agent with its own context, tools, and loop. Example: Claude Code's autocompact hands summarization to a child agent instead of forcing the main loop to do it inline.

3. Tool-use loop — The LLM decides which tool to call, the program executes it, and the result flows back into the next turn. This is the main agent loop.

Simple question (which memories are relevant?)  → Direct call
Complex but bounded task (summarize this)       → Forked sub-agent
Open-ended execution (build this feature)       → Tool-use loop

This choice is not academic. It changes latency, token cost, and failure modes. In Claude Code's memory system, a side-query is cheap. A forked summarizer is much heavier. Using the wrong pattern wastes budget or hurts reliability.

The trap: Teams chase model upgrades ("let's switch to Claude Opus") instead of fixing their context pipeline. Better models help — but in my experience, fixing the context pipeline delivers a larger improvement per dollar than upgrading the model.

In one production system, fixing context management alone — without changing the model — moved quality from 40% to 60%. Seven out of eight fixes were pure code, zero LLM cost. The model was always capable. The context was holding it back.

What 90% Actually Looks Like — From Claude Code's Source

Claude Code's leaked source is roughly 512,000 lines. Here's the useful way to think about that split:

query.ts orchestrator file:             1,729 lines    (~0.3%)
Core while(true) loop inside it:        1,421 lines
Everything else:                     ~510,000 lines    (~99.7%)

That "everything else" is the 90%:

Context Management (3,960 lines in src/services/compact/):

5-level progressive compression pipeline
Microcompact with dual code paths based on cache state
Token estimation without API calls (<5% error)
Post-compression recovery (restore last 5 files, skills, agent state)
Circuit breaker: 3 consecutive failures → stop (after 250K API calls/day were wasted without it)

Memory System (1,736 lines in src/memdir/):

4-type closed taxonomy with structured frontmatter
Sonnet side-query for semantic retrieval (250ms, async prefetch)
Background extraction agent with mutual exclusion
Trust verification (eval went 0/2 → 3/3 with this addition)

Error Handling (spread across entire codebase):

Message normalization: fix orphan tool_use/tool_result pairs from crashes
Prompt-Too-Long recovery: reactive compression as last resort
Tool failure classification: timeout vs permission vs not-found
Max output token escalation: 8K default → 64K on truncation

Permission System (multi-layer):

Tool-level risk classification
User confirmation for dangerous operations
Sandbox isolation for high-risk tools
Context injection scanning

None of this is intellectually exciting. It's plumbing. But without it, the "exciting" part — the agent loop — crashes on every non-trivial conversation.

OpenClaw and Hermes Surface the Same Pattern

Two open-source agents worth watching right now — OpenClaw and Hermes Agent — illustrate the same architectural lesson.

OpenClaw:

Context management: still basic; I don't see the kind of progressive compression Claude Code built
Memory: Markdown + SQLite (more sophisticated than Claude Code's storage layer)
Reflection: limited; I don't yet see a strong closed verification loop
Security: public reports in early 2026 highlighted exposed instances and malicious marketplace skills; openclaw security audit exists, but tools alone don't close the operational loop

Hermes Agent:

Context management: still basic
Memory: SQLite + full-text search + MEMORY.md dual-layer
Reflection: self-evolving skills generated from completed tasks
Error handling: layered on paper, but still early
Maturity: promising, but I haven't seen evidence yet that the self-iteration loop holds up at scale

Both can complete tasks. My point is not that they don't work. It's that the hardest production loops — compression, failure accounting, verification retries, and memory hygiene — still look only partially closed. The features exist; the loops aren't closed.

"Features Exist" vs "Loops Are Closed"

This is the most overlooked distinction in agent architecture:

Open loop:   Build a verification step → log issues → done
Closed loop: Build a verification step → log issues → retry with feedback → verify again

Open loop:   Score memory relevance → store the score → done  
Closed loop: Score memory relevance → reinforce high-scoring memories → decay low-scoring → improve retrieval over time

Open loop:   Detect compression failure → log it → continue
Closed loop: Detect compression failure → count consecutive failures → circuit-break after 3

In every open-source agent I've analyzed so far, some of the critical loops are still open. The infrastructure is there. The wiring is only partially connected.

Here's the test: look at your agent's verification step. Does it log a failure and move on? That's an open loop. Does it log, retry with the failure as feedback, and verify again? That's closed. The difference is one if statement and a retry call — but it's the difference between "we have quality checks" and "we actually catch errors before users see them."

This is the hardest 10% of the 90%. Not building the infrastructure — connecting it.

The Proof: 40% to 60% With Code Alone

I ran A/B evals on a production agent. Same model, same test cases, different code. Result: 40% → 60% pass rate.

The breakdown: 8 fixes total. 7 were pure code — zero LLM cost. Context prioritization, structured error classification, round limits, conclusion preservation during truncation, circuit breakers. The only fix that used an LLM call was a pre-loop planning step at $0.003 per request.

The model was always capable. The context was holding it back.

(Full case study with per-test breakdown: How I Improved an AI Agent from 40% to 60%)

What This Means for Builders

If you're building an AI agent:

Don't start with the model. Start with context management. Clean, prioritized, bounded input is the highest-leverage investment you can make.
Close your loops. If you built a verification step, make it retry. If you built memory scoring, wire the reinforcement. Half-built infrastructure is worse than none — it gives false confidence.
Measure before you upgrade. Before switching to a more expensive model, run an eval suite on your current one. The bottleneck is probably context, not capability.
Budget 90% of your time for the 90%. The agent loop is a weekend project. Error handling, compression, memory lifecycle, permission systems — that's the real work. Plan accordingly.

The model is a commodity. The engineering around it is the product.

Ask yourself: what percentage of your agent's codebase is the core loop, and what percentage is everything else? If you don't know the answer, that's where to start.

Diagrams from this essay packaged as a single-file reference: Engineering Reliable Agents (PDF).

Part of the AI Agent Architecture series.

Deep dives into the 90%:

Claude Code Part 3: The 5-Level Compression Pipeline — how Anthropic solved context management
Claude Code Part 4: Why Markdown Instead of Vector DBs — first-principles memory tradeoffs
40% to 60% With A/B Data — the full case study behind the numbers in this article

Testing Real-World Go Backends Isn't What Many People Think

Harrison Guo — Sat, 18 Apr 2026 00:18:33 +0000

I've reviewed enough Go backend test suites to notice a pattern. The services with the most unit tests are often the ones with the most production incidents. Not because unit tests cause incidents — because the teams writing unit tests and calling it a day weren't testing the things that actually broke.

Production bugs in distributed Go backends don't usually look like "function computed wrong value." They look like:

"The context deadline didn't propagate into the background goroutine, so under load it leaked."
"Two services agreed on the happy path, but the error-shape contract diverged six months ago, and now one returns status.Code(codes.Unavailable) where the other expects codes.ResourceExhausted."
"The retry logic is race-y. With test-scale traffic it works; at 10x production it double-charges."
"The database migration works on SQLite (our test DB) but not Postgres 15's stricter planner."

No unit test catches those. A different set of test shapes does.

tl;dr — Stop framing tests as "unit vs integration." That's a level-of-isolation axis, and it's the least interesting one. The axes that matter for production Go: deterministic behavior (controlled clocks, seeded randomness), concurrency correctness (race detector, stress tests), contract fidelity (shared schemas, real downstreams), and environment fidelity (real DBs, real networks). Design your test suite around those; coverage follows.

The Wrong Taxonomy

"Unit tests test one function. Integration tests test several. E2E tests test the whole system."

That framing is a starting point for junior engineers. It stops being useful the moment you're debugging why your Go service silently dropped a message in production. The level of isolation isn't the interesting axis. What is:

Deterministic vs non-deterministic behavior. Do the same inputs produce the same outputs every time?
Concurrency correctness. Do the race conditions stay caught?
Contract fidelity. Do your assumptions about downstreams match what they actually do?
Environment fidelity. Does your test environment reproduce the production runtime closely enough to catch real bugs?

A test can be "unit" on the isolation axis but score on two or three of these. A test can be "integration" and miss all four.

Deterministic Behavior: The One Thing Every Test Should Have

If you can't run your test a thousand times and get the same result, you have a flaky test, and flaky tests are worse than no tests — they train the team to ignore failures.

The three sources of non-determinism in Go test suites, in order of prevalence:

1. Time

Any test that calls time.Now(), time.After(), time.Sleep(), or depends on wall-clock intervals is a landmine. It works on the developer's laptop and fails in a slow CI runner where GC decided to kick in.

Fix: inject a clock. A minimal clock interface:

type Clock interface {
    Now() time.Time
    Sleep(d time.Duration)
    After(d time.Duration) <-chan time.Time
}

type realClock struct{}
func (realClock) Now() time.Time            { return time.Now() }
func (realClock) Sleep(d time.Duration)     { time.Sleep(d) }
func (realClock) After(d time.Duration) <-chan time.Time { return time.After(d) }

In production, realClock. In tests, a FakeClock that advances manually. Libraries like github.com/benbjohnson/clock give you this for free.

Payoff: a test that verifies "retries happen every 500ms for 3 attempts" becomes deterministic — advance the fake clock 500ms, observe a retry, advance another 500ms, observe again. No sleeping in the test.

2. Randomness

Anything that shuffles, samples, picks a random ID, or generates random test data needs a seeded random source. math/rand.Intn with default source is a machine-global shared state; two tests running in parallel can interfere.

func New(seed int64) *Service {
    return &Service{rng: rand.New(rand.NewSource(seed))}
}

In tests, pass a known seed. In production, rand.NewSource(time.Now().UnixNano()).

3. Concurrency ordering

The nasty one. A test that creates goroutines and checks a result has to either (a) synchronize on a deterministic completion signal (a channel, a WaitGroup) or (b) poll with a timeout — which is back to non-determinism.

The best habit: design for deterministic completion. If you're testing "five goroutines should all complete and total the result," use sync.WaitGroup.Wait() or close a channel. Don't sleep. Don't poll.

Concurrency Correctness: The Race Detector Is Not Optional

Go ships with a race detector. Running go test -race is one flag and it catches an entire category of bugs that will otherwise show up as "works on my machine." In my experience, any production Go service will, on first -race run, surface at least one real data race that had been silently ignored.

The race detector adds ~5-10x runtime overhead, so people skip it on every-save tests. Fine. Run it in CI. Run it on nightly integration tests. Run it on anything touching shared state. Some configurations I've seen work:

Every PR: run unit tests with -race.
Nightly: run full integration suite with -race and a longer timeout.
Pre-release: run stress tests with -race against a production-sized dataset.

The cost of running with -race is engineering discipline. The payoff is not debugging a data race at 2 AM.

Beyond the race detector, stress tests are undervalued. A test that runs your concurrent path 1,000 times with different goroutine interleavings catches bugs that a single-iteration test never will.

func TestConcurrentWorkers_Stress(t *testing.T) {
    if testing.Short() {
        t.Skip("stress test")
    }
    for i := 0; i < 1000; i++ {
        t.Run(fmt.Sprintf("iter%d", i), func(t *testing.T) {
            t.Parallel()
            // ... actual test body ...
        })
    }
}

t.Parallel() + 1,000 iterations + -race finds race conditions that a single deterministic run happily misses.

Contract Fidelity: The Bug Class Everyone Misses

Say your service calls a downstream gRPC service for payments. You write a mock that returns a successful response. Your tests pass. The downstream team changes their error code vocabulary. Your service now misinterprets their new error. Production finds out first.

Contract testing addresses this. Two approaches work in practice:

Shared schema, shared types

If the downstream service publishes a protobuf file (they should), your service imports it directly. Your tests use types generated from the real contract. If the downstream bumps the proto, your next build fails — loudly, at compile time.

This is the simplest and often best answer for Go services with gRPC downstreams. The contract is literally the shared protobuf.

Consumer-driven contract tests

Each consumer writes tests that capture their expectations of the downstream. Those tests run against the real downstream (or a contract-test server like Pact). When the downstream changes, the contract tests catch it before the contract-as-written reality diverges.

This helps for REST APIs where there's no single source of truth schema. It's more ceremony. For most gRPC Go services, shared protobufs cover it.

The "mock everything" antipattern

If your test suite consists of mocks that return whatever your test needs, you're not testing integration. You're testing that your code calls your mocks correctly. That's a tautology. Real integration bugs live in the gap between your mock's behavior and the downstream's actual behavior.

Have at least one test per integration point that hits the real downstream — either in a staging environment or via Testcontainers. Keep the mocks for fast feedback, but don't pretend they're the only tests you need.

Environment Fidelity: Use Real Infra Where It Matters

The sharpest line in my test taxonomy is between "close to production runtime" and "not close."

Things that matter and are worth running on real infrastructure in tests:

Databases. SQLite is not Postgres is not MySQL. Query planner, isolation levels, and error shapes differ. Test with the DB you ship with.
Message brokers. Kafka's ordering and offset semantics cannot be faked well. Use a real Kafka (or Redpanda) in tests that exercise ordering or replay.
Caches. Redis has specific failover and eviction semantics. A fake in-memory map doesn't reproduce them.
Time-sensitive downstream APIs. Anything with rate limits or TTLs.

Things that rarely matter and are fine with fakes:

Object storage. A local file-system backend usually reproduces S3 well enough.
Metrics / tracing exporters. Tests don't need a real Prometheus.
Email / SMS. A mock recording calls is plenty.

The pattern: test with real infra for anything where semantic difference is possible. Testcontainers (github.com/testcontainers/testcontainers-go) makes this painless:

func setupPostgres(t *testing.T) string {
    ctx := context.Background()
    c, err := postgres.RunContainer(ctx,
        testcontainers.WithImage("postgres:15-alpine"),
        postgres.WithDatabase("testdb"),
        postgres.WithUsername("testuser"),
        postgres.WithPassword("testpass"),
    )
    require.NoError(t, err)
    t.Cleanup(func() { c.Terminate(ctx) })

    dsn, err := c.ConnectionString(ctx, "sslmode=disable")
    require.NoError(t, err)
    return dsn
}

Slow? Yes — each container takes a few seconds to start. But you can run them once per test package with a TestMain, and the bugs they catch are the ones most worth catching.

A Real Taxonomy

pure functions · algorithms"]"/>

Here's the taxonomy I actually use when designing a test suite for a Go backend:

Fast tests (seconds for the whole file): pure functions, algorithms, small state machines. Run on every save.
Concurrency tests (seconds to a minute): anything with goroutines. Run with -race. Run in PR.
Deterministic integration tests (single-digit seconds per test): one module + fakes + fake clock. Fast enough to keep in the main test run.
Real-infra integration tests (seconds per test): one module + real DB / Kafka / Redis via Testcontainers. Run in PR, longer timeout.
Contract tests (milliseconds): verify shared schemas with downstreams. Run on every schema change.
Stress tests (minutes): high-iteration, high-concurrency, with -race. Run nightly or on schedule.
End-to-end tests (minutes): real services, real network, against a staging environment. Run pre-release.

What you'll notice: "unit" and "integration" don't appear as categories. That's on purpose. The level of isolation is implementation detail. The purpose of the test is the taxonomy.

Small Habits That Pay Off

Use t.Cleanup over defer. Cleanups run in LIFO order, can be added anywhere in the test, and survive test panics better.
Prefer table-driven tests. Twenty tests as rows in a slice beats twenty nearly-identical test functions.
Fail tests with t.Fatalf, not t.Errorf, for setup failures. A broken setup should abort; a broken assertion might allow the test to continue collecting more failures.
Golden files for complex outputs. If you're verifying a generated SQL query, a serialized event, or a JSON response, a golden file comparison is more readable than a long string literal.
Separate _test.go files for slow tests with a build tag. //go:build integration lets you run them explicitly.

The Shift That Changed My Testing

Coverage numbers lie. The question is not "what percent of lines are executed by tests" — it's "what percent of the risky behaviors are covered by tests that will actually fail when those behaviors break."

A codebase with 95% line coverage and zero race tests, zero real-DB tests, and mock-heavy integration tests is brittle. A codebase with 60% line coverage, go test -race in CI, Testcontainers for the DB, and a stress test for every hot concurrent path is not.

The single biggest shift I recommend: stop thinking about tests in terms of isolation level, and start thinking about them in terms of the production failure modes you're actually afraid of. Map each failure mode to a test shape. If you don't have a test shape for a failure mode, you don't really have that failure mode covered — you just hope it doesn't happen.

Production has opinions about what you hope.

Go's Concurrency Is About Structure, Not Speed — the concurrency patterns that make production-shape Go possible.
Go Context in Distributed Systems: What Actually Works in Production — the single most common test gap in Go services I review.
Why Your "Fail-Fast" Strategy is Killing Your Distributed System — a production failure mode that's hard to test unless you design the test for it.

Scale-Up vs Scale-Out: Why Every Language Wins Somewhere

Harrison Guo — Sat, 18 Apr 2026 00:18:32 +0000

I worked with a team that rewrote a critical service from Go to Rust because "performance." Six months later, the service was 30% faster, the team was miserable, and feature velocity had dropped to a crawl. Meanwhile the competitor team, still on Go, had shipped four new features.

We did the postmortem eventually. The service handled maybe 2,000 requests per second on a 4-core machine. CPU utilization sat around 20%. Rust's extra speed bought us exactly nothing — the bottleneck was downstream database latency. What it cost us was every feature we didn't ship while writing unsafe, fighting the borrow checker, and nursing the team through the learning curve.

That incident taught me the question I wish I'd learned earlier: what are you actually scaling, and does the language buy you the right kind of scale?

tl;dr — Language benchmarks optimize for one axis: per-request performance. Real systems have multiple axes — throughput, latency, concurrency, developer velocity, operational complexity, memory efficiency. Rust, Go, Java, Python aren't competing to be "fastest." They're different answers to different bets about what you're going to scale. Pick by fit, not by leaderboard.

The Two Kinds of Scale

At the top level, two strategies dominate:

Scale-up: make one machine do more. Vertical scaling. Faster CPUs, more RAM, specialized hardware, lower per-operation cost.
Scale-out: add more machines. Horizontal scaling. Cheaper commodity hardware, more concurrency, lots of work running in parallel.

These aren't just infrastructure decisions. They're reflected in the language and ecosystem you pick. A language optimized for scale-up (Rust, C++) has different priorities than one optimized for scale-out (Go, Elixir) or one optimized for neither but for developer leverage (Python, Ruby).

The big confusion comes from mixing axes. "Rust is faster than Go" is true on per-op microbenchmarks and irrelevant if your workload is I/O-bound service-to-service traffic. "Python is slow" is true in a compute-bound loop and irrelevant for a 500-QPS API that spends 95% of its time waiting on PostgreSQL.

Where Each Language Actually Wins

Rough positioning — not a benchmark, a fit map. The language you pick should live near the kind of scaling your system actually demands.

Rust / C++ / Zig — Scale-up champions

These languages dominate when per-machine throughput is the bottleneck and you can afford the engineering cost. That's a narrower set of problems than Twitter would have you believe, but the problems that exist are real:

High-frequency trading engines — microseconds matter, GC pauses are unacceptable, every cache line counts.
Inference engines — llm.cpp, vllm, mistral.rs. Memory layout, SIMD, custom kernels.
Databases and storage engines — ScyllaDB, TiKV, Foundation internals. State machines that live forever and must not leak.
Network data planes — Cloudflare's Pingora, proxies at the edge.
Game engines, audio/video encoding, embedded.

The pattern: one box, pushed hard, for years. Memory safety matters because bugs compound over time. Performance matters because throughput per core is the product.

The cost: every commit is slower. Refactoring is expensive. Onboarding is measured in months, not weeks. The compile times are what they are. You pay this cost every day the service exists.

Go — Scale-out champion

Go hits a specific sweet spot: cheap concurrency, predictable performance, fast-to-ship code, and easy to hire for. It's a scale-out language.

Thousands of goroutines per core, 2KB stacks, user-space context switching. The "cost of one more waiter" is nearly zero.
Standard library is enough for 80% of backend work — HTTP server, JSON, SQL, crypto.
Compilation is fast enough to stay in flow. Iteration loop feels similar to a dynamic language.
Minimalism is aggressive. One person can read the whole language in a weekend. New hires are productive in days.

Where it loses: per-op performance. Go's GC is fine but not invisible. Zero-copy generic code is harder to write than in Rust. The type system doesn't prevent the entire class of bugs Rust's does.

Go's bet: the problem you're most likely to have is "I need to handle 10x the concurrent work with 2x the code." Not "I need this loop to be 5% faster." For most backend services, that bet is right.

Java / Kotlin — Mature scale-out with runtime depth

The JVM is what you want when the workload is scale-out but you need runtime flexibility Go doesn't give you:

A mature JIT that optimizes hot paths beyond what AOT can.
Rich profiling and monitoring (JFR, async-profiler, flight recorder) that makes post-deploy tuning feasible.
A library ecosystem that, after 25 years, has a mature library for basically anything.
Kotlin on top gives you modern syntax and coroutines without leaving the ecosystem.

Where it loses: startup time, memory overhead, operational complexity (GC tuning is a real job), the occasional "it works on my JDK 11 but the prod JDK 17 changed something." Also: hiring is harder than Go now, at least in my corner of the industry.

Java's bet: "you'll still be running this service in ten years, and you want to be able to tune its runtime when that day comes." For large enterprises with deep infrastructure, that bet pays off. For a startup shipping its first three services, the overhead is not worth it.

Python / Ruby — Developer-velocity champions

The forgotten-but-dominant answer: languages that optimize neither scale-up nor scale-out, but scale-the-team.

Fast to write, fast to read, fast to debug.
Massive libraries for data, ML, scripting, DSLs.
Easy to onboard anyone — CS students, data scientists, analysts.
Prototype-to-production path is shorter than anywhere else.

Where they lose: per-core throughput, concurrency (the GIL is real), memory. Python and Ruby are not your language for a 100K QPS service.

But a lot of real companies don't need a 100K QPS service. They need to get a thing working, put it in front of users, and iterate. If your current problem is "we need to ship the next feature this week," Python might be the right answer even if a Rust version would technically run faster.

Python's bet: throughput isn't the constraint yet. Time-to-shipped-feature is. For most companies most of the time, that's correct.

The Axes Nobody Talks About

Beyond scale-up/scale-out, a few axes decide more projects than raw performance.

Developer-velocity per week

"I can ship a feature and have it in production by Friday" beats "this service is 2x faster" most of the time. Measure it. If your current stack requires a two-day ceremony to deploy a one-line change, throughput is not your problem. Velocity is.

Operational complexity

Scale-up is operationally cheaper than scale-out. One machine, one process, one log. Scale-out gives you better redundancy but also distributed-systems problems — consistency, ordering, partial failure, chaos engineering. If your team is three people, the operational complexity of a 20-node scale-out cluster may eat more time than the language choice saves.

Memory efficiency per dollar

At cloud scale, memory is expensive. A Rust service that fits in 2GB where a Java service needs 8GB is a 4x savings on every instance. Multiply by thousands of instances and "per-op performance" stops being the interesting number — per-GB cost starts to matter.

Hiring pool

The language with the deepest talent pool in your market is usually the right answer for a new system, all else equal. A marginal technical improvement isn't worth a six-month hiring pipeline.

Learning curve shape

Some languages have shallow onboarding (Go, Python) and a long tail of depth. Others have steep onboarding (Rust, Haskell) and you're productive only after the ramp. For a senior team on a long-lived system, steep is fine. For a fast-moving team, steep is expensive.

The Pattern I See Repeated

A company starts small, picks Python or Ruby, builds the thing, ships to production. Ten employees. One codebase. Life is fast.

They grow to fifty engineers. The monolith cracks. Some services get rewritten in Go for concurrency and operational simplicity. A few performance-critical ones get written in Rust. Data infra sits on the JVM (Kafka, Spark, Flink). A few internal tools stay in Python because the team knows it and it works.

Five years in, the stack is polyglot. Nobody regrets it. What they regret is the six months they spent trying to make a single-language stack work past its comfort zone — the Python team pushing for "just async more things," or the Rust team fighting the borrow checker on code that could have been Go, or the Java team explaining to a new hire why the stack trace is 400 lines long.

The pattern: pick the language that fits the service, not the service that fits the language.

How I Ask the Question Now

When someone proposes "let's build this new thing in X," I ask:

What's the expected traffic profile, and what's the per-request work shape?
Is this scale-up limited (per-machine throughput) or scale-out limited (concurrent work)?
Who's going to write this, and how fast do we need them productive?
Who's going to operate this, and what's their tooling comfort?
Does this interact with an existing ecosystem (JVM data platform, Rust security infra)?
How long does it have to live?

The answer to those five questions usually lands me on one of three languages for 80% of systems I see: Go, Rust, or (for data-adjacent work) Kotlin on the JVM. Python still shows up for tools and glue. Everything else is contextual.

The benchmarks don't help. Per-op microbenchmarks answer questions nobody is actually asking. The right question is which axes matter for this system, and which language's bet lines up with those axes.

The Argument I've Stopped Having

I still see engineers argue about whether Rust or Go is "better." Both are good languages. Both are bad choices for problems they weren't designed for. The meaningful question is which kind of scale you're paying for — and the honest answer is almost always a mix, evolving over time.

The Rust rewrite I opened with wasn't a bad decision because Rust is a bad language. It was a bad decision because we weren't scale-up limited. We were downstream-database limited. No language could help with that.

Know which scale you're buying, and buy it on purpose.

Why Go Handles Millions of Connections: User-Space Context Switching, Explained — the design decision behind Go's scale-out bet.
Go's Concurrency Is About Structure, Not Speed — what you actually get with Go, and what you don't.
NATS vs Kafka vs MQTT: Same Category, Very Different Jobs — applying the same fit-vs-benchmark thinking to messaging.

From Locks to Actors: The Four Pillars of Modern Concurrency

Harrison Guo — Fri, 17 Apr 2026 05:50:27 +0000

Most working engineers have spent ninety percent of their concurrent-programming life in one model: shared memory protected by locks. Threads that all see the same variables. Mutexes around the critical sections. Hope and care. It's the model every OS textbook teaches, every mainstream language supports, and every senior engineer has a horror story about.

It's also not the only option. Or even the best one, for many of the problems it gets used for. Three other models — CSP, actors, and software transactional memory — have been around for decades, mature enough for production, and each solves a class of problems that lock-based designs handle poorly.

This is a map of all four, from a working backend engineer who uses each of them for different jobs, and a take on when each is the right answer.

tl;dr — Concurrency has four viable pillars: shared memory + locks (threads, mutexes), CSP (channels, Go), actors (mailboxes, Erlang), and STM (transactional memory, Clojure). None is universally better. Each solves a different problem and has a different failure mode. Senior designs often mix three of them in one system. Mutex-for-everything works until it doesn't — usually at exactly the scale you promised you'd never reach.

Pillar 1: Shared Memory + Locks

The default. Threads, mutexes, atomics, condition variables. Every mainstream language has them.

How it works: multiple threads of execution share the same address space. They read and write the same data. Mutexes make sure only one thread touches a critical section at a time. Atomics do the same for single-word operations without a full lock.

Where it shines:

Simple shared counters and caches. atomic.AddInt64, sync.Map, LRU caches. The right tool.
Tight single-process coordination where the code is small enough for one person to hold in their head.
Performance-critical paths where the overhead of channel sends or actor dispatches is too much.

Failure modes:

Deadlocks. Two threads acquire locks in opposite order. Happens.
Priority inversion. Low-priority thread holds the lock, high-priority thread waits, work piles up.
Lock ordering bugs at scale. When N components each take M locks, the reasoning gets exponential.
Memory-model weirdness. What one thread writes, another may not immediately see. You start caring about happens-before, acquire/release semantics, and why volatile in Java is not what you thought.
Invisible races. The worst kind. Tests pass; production fails weirdly twice a month.

Use mutexes for small, localized shared state. Once the shared state has three collaborators or more, or a nontrivial invariant across fields, reach for one of the other models.

Pillar 2: CSP (Communicating Sequential Processes)

Tony Hoare's 1978 paper, popularized by Occam and now Go. The model Rob Pike and Ken Thompson picked for Go's concurrency.

How it works: processes don't share memory; they send messages on named channels. Senders and receivers rendezvous on the channel. Ownership of data moves with the message. "Do not communicate by sharing memory; share memory by communicating."

Where it shines:

Pipelines. Data flows through stages, each a goroutine, connected by channels. Clean to read.
Fan-out / fan-in. One producer, many workers, one aggregator. The channel topology is the architecture.
Backpressure. A bounded channel blocks the producer when full. No extra flow control needed.
Cancellation coordination. select with <-ctx.Done() is a clean primitive.
Lifecycle control. Closing a channel is a broadcast to every listener.

Failure modes:

Deadlocks remain possible. Two goroutines each waiting on the other's channel. Cycles in the channel graph are lethal.
Memory leaks via unclosed channels. A goroutine blocked on a send that will never be received lives forever.
Awkward request/reply. You end up passing a reply channel with each request, which works but feels verbose.
Order isn't free. Channel ordering is only per-channel. If you fan out and fan in, the aggregation is unordered unless you sort.

Use CSP for coordination-heavy designs. When the structure of "who's alive, who sends to whom, when do things stop" is the architecture, channels make that visible in the code.

Go is the obvious exemplar, but CSP-style is also available in Rust (crossbeam-channel, tokio::sync::mpsc), Kotlin (coroutines with channels), Python (asyncio.Queue), and C# (System.Threading.Channels).

Pillar 3: Actors

Carl Hewitt's 1973 paper. Made practical by Erlang (1986) and later Akka (Scala/Java). The model behind WhatsApp, a decade of telecom, and most fault-tolerant messaging infrastructure.

How it works: an actor is a named entity with private state and a mailbox. Other actors send messages to its address. Messages are processed one at a time from the mailbox. No shared memory. Parent actors supervise children; when a child crashes, the parent decides to restart, escalate, or ignore. Crashes are normal.

Where it shines:

Fault isolation at scale. One actor crashing is expected; it doesn't take down the system. Supervision hierarchies make "let it crash" a sensible engineering strategy.
Stateful services. Each actor holds its own state. Conceptually clean: no shared global state, no locks around it.
Location transparency. An actor can live in the same process, another process, or another machine. The sender doesn't know. This is where actors shine in distributed systems — the model scales across the network boundary natively.
Massive concurrency with stateful semantics. Erlang routinely runs millions of actors per node. Each is cheap.

Failure modes:

Mailbox unboundedness. If a producer sends faster than the actor can process, the mailbox grows without bound. Bounded mailboxes exist; use them.
Message-ordering assumptions break across the network. Within one node, delivery order is preserved per sender. Across nodes, all bets are off without explicit sequencing.
Testing is harder. Actors make their own state opaque; you test behavior through message exchange. Good frameworks help, but the habits needed are different from testing normal code.
Conceptual mismatch in CRUD-style backends. If your business logic is "select some rows, transform them, insert result," actors are overkill. They shine on long-lived stateful entities (a game character, a connected device, a user session), not on stateless request handlers.

Erlang and Elixir are the canonical runtimes. Akka brings actors to the JVM. Pony is a rare actor-first typed language. In Go, you can simulate actors with a goroutine + channel-as-mailbox pattern, but you lose Erlang's supervision and "let it crash" semantics unless you build them yourself.

Use actors when you have long-lived stateful entities with fault requirements. Telecom, messaging, multiplayer game servers, IoT device shadows, any system where "this particular entity has its own state machine, and we really care when it crashes" is the shape.

Pillar 4: Software Transactional Memory (STM)

Imagine database transactions, but for in-memory data. That's STM.

How it works: critical sections are wrapped in transactions. The runtime tracks reads and writes optimistically. On commit, if any data touched was modified by another transaction, the current one rolls back and retries. No explicit locks. Composability — two transactions can be combined into a larger one without redesigning the locking order.

Where it shines:

Composable concurrent code. Combining operations that were individually correct usually stays correct under STM. Lock-based code famously does not.
Read-mostly workloads. STM with multi-version concurrency control scales reads without blocking.
Avoiding the lock-ordering bug class. No locks, no deadlocks. The failure mode is retry storms, which are easier to reason about.

Failure modes:

I/O inside transactions is awful. Transactions may retry. If you did I/O, you may have done it multiple times. Either separate I/O from transactional state, or the runtime has to forbid I/O inside transactions (Haskell's STM monad does this at the type level).
Retry storms under contention. Heavy write contention on the same data means constant retries. In the worst case, throughput can be worse than locks.
Limited language support. Clojure (built-in), Haskell (STM), Scala (scala-stm), Rust (experimental stm crates). Not a mainstream feature of Go/Java/C#.

Clojure is the canonical "STM as a first-class citizen" language — its refs and transactions are idiomatic. Haskell's STM monad is arguably the cleanest realization. In other ecosystems, STM exists as libraries but hasn't displaced mutexes.

Use STM when the concurrent state is small-to-medium, the access pattern is read-heavy with occasional writes, and you want the composability. For the rare problems that fit, STM is strictly simpler to reason about than locks. For problems that don't fit (I/O-heavy, write-contention-heavy), STM is worse.

How Real Systems Mix Them

The surprise for engineers who've only used one model: mature systems mix three of them in one codebase.

A typical backend service I'd build today:

Mutexes / atomics for the inner loops — counters, caches, rate-limiter state, anything performance-critical with one clear owner.
Channels (CSP) for coordination — worker pools, pipelines, cancellation, shutdown signaling, bounded queues.
Actors (in a sense) for long-lived stateful entities — each connected client session, each in-flight request, each background job. In Go I'd model this as "one goroutine per entity, communicating via channels," which isn't formal actors but inherits the useful semantics: isolated state, message-passing, crash-isolation.

And I wouldn't use STM in that stack. Not because it's bad, but because the language runtime doesn't make it first-class. If I were writing Clojure, STM would be a natural fit for the in-memory state machines that would otherwise be locked maps.

The old "pick one concurrency model" debate was always a false choice. The real decision is per-problem: what shape is the concurrent work, what's the state-sharing pattern, and what failure semantics do I want.

Decision Guide

Quick map:

I have a counter that multiple goroutines read and update. → atomic or mutex.
I have a pipeline of work that flows through stages. → channels (CSP).
I have a fleet of long-lived sessions, each with its own state and lifetime. → actor pattern (goroutine + mailbox channel, or real actor framework).
I have a fleet of connected devices each with a state machine that must survive crashes. → actor framework with supervision (Erlang, Akka, or Go with explicit crash/restart logic).
I have complex shared state with nontrivial invariants across fields, and updates are occasional but important to compose. → STM if your language supports it; otherwise, lots of careful mutex discipline.
I have a request/response flow with fan-out to downstreams. → CSP with errgroup.WithContext.
I have no idea what I have. → Start with mutexes, switch when it hurts. Don't over-engineer the first version.

The Real Lesson

Most people who get bitten by concurrency bugs got bitten because they used the wrong model, not because they used it wrong. A mutex-heavy design for a workload that's really a pipeline is fragile. A channels-for-everything design when there's a shared counter underneath ends up with awkward rendezvous. An actors-everywhere design when the business is CRUD requests reads like over-engineering.

The four pillars aren't competing theories of concurrency. They're four tools, each good at specific jobs. Senior engineers know all four and reach for the right one. Junior engineers reach for the only one they know and force-fit it.

If your career so far has been mostly mutexes, spend a weekend reading the other three. Write a toy pipeline in Go channels. Read Erlang's supervision documentation. Play with Clojure refs. The investment pays back every time you sit in a design review and someone proposes locking their way out of a structural problem.

Go's Concurrency Is About Structure, Not Speed — CSP applied concretely in Go.
Why Go Handles Millions of Connections — the runtime characteristics that make CSP cheap in Go.
Scale-Up vs Scale-Out: Why Every Language Wins Somewhere — the language-level view of the same question.

Forem: Harrison Guo

Node Turns Waiting Into Events. Go Moves Context Switching Into User Space.

The Wrong Question

Node's Answer: Turn Waiting Into an Event

The Visible Side Effect: Function Color

The Hard Limit

Go's Answer: Move Context Switching Into User Space

What "User-Space" Actually Buys You

The Unit of Scheduling

Where the Boundaries Diverge: CPU-Bound Work

Two Answers to the Same Question

The Closing Line

Appendix: Reproduce the Benchmark

gRPC Interceptors in Production: Design Patterns That Survive Real Load

The Four Interceptor Types

Chaining and Order

Keeping Interceptors Focused

Common Interceptor Recipes

Tracing (OpenTelemetry)

Metrics

Auth

Rate limiting

Retry (client-side)

The Stream Interceptor Trap

Testing Interceptor Chains

Patterns That Save Time

The Discipline That Makes This Work

Related

Go Generics, One Year In: Which Promises Held, Which Didn't

What Generics Actually Are

Where Generics Clearly Win

Standard-library style container and utility functions

Concurrency helpers

Typed collections

Numerical / algorithmic code

Where Generics Don't Help, Or Hurt

"Generic services" and similar framework-y code

Over-constrained helpers

Behavioral polymorphism

Performance: Usually a Wash

Idioms That Emerged

Prefer any to interface{}

Single-letter type parameters for simple cases, descriptive for complex

Put constraints in a dedicated package

Use ~T approximations for flexibility

Never overload generic helpers to do too much

The Standard Library Won

What I Do Now

Where Generics Actually Sit Now

Related

Go Profiling in Anger: pprof, Escape Analysis, and Inlining Without Magic

The Investigation Flow

CPU Profiling: The First Thing, Always

Heap Profiling: When CPU Points to GC

Escape Analysis: The Compiler's Story

Inlining: When the Compiler Eliminates the Call

Benchmarks: The Ground Truth

The 80/20 of Go Performance

A Habit I Recommend

The Habit That Compounds

Related

sync.Pool in Go: When It Actually Helps, and When It Quietly Hurts

What sync.Pool Actually Does

Should You Pool This?

When Pooling Helps: Per-Request Buffers

When Pooling Is Wasted Effort

The Reset Trap

The Alloc Benchmark Methodology

A Pattern That Actually Works: Scoped Pools

The Big Thing sync.Pool Isn't

What Actually Matters

Related

IronSys: A Production Blueprint for Modern Concurrency

The System Shape

The API Handlers

The Live Sessions — Actor Pattern in Go

The Streaming Ingest — Bounded Worker Pool (CSP)

The Batch Aggregation — Pipelines + errgroup

The Cross-Service Handoff — Durable Queues

How the Primitives Map

Prefer `any` to `interface{}`

Use `~T` approximations for flexibility

The Big Thing `sync.Pool` Isn't