Forem: Trung Duong

😴 Go Code You Can Trust: Sleep Well After You Commit

Trung Duong — Thu, 13 Mar 2025 03:11:52 +0000

It's Friday afternoon, 4:45 PM. My teammates are already discussing weekend plans in Slack. I'm finishing up a critical piece of our payment processing service. The code is done, tests pass, and I'm about to commit it before heading into the weekend. My finger hovers over the enter key for a moment...

But there's no hesitation, no worry. I commit the code, close my laptop, and join the weekend conversation without a second thought about my code failing in production.

This wasn't always the case. Ten years ago, I'd be checking my phone all weekend, worried about 3 AM calls from our on-call engineer.

What changed? Let me tell you my story of how I learned to write Go code I could trust completely.

Image source: Medium

The Cost of Uncertain Code

My journey with Go started in 2021 when I was working at a rapidly growing startup. We were moving from a monolithic Java application to microservices, and Go seemed like the perfect fit: fast, simple, and with built-in concurrency.

But my early Go code looked like this:

func ProcessPayment(paymentID string) error {
    // Get payment details
    payment, err := db.GetPayment(paymentID)
    if err != nil {
        return err // Which error? What happened?
    }

    // Process the payment
    err = paymentGateway.Process(payment)
    if err != nil {
        return err // Again, what went wrong?
    }

    // Update payment status
    err = db.UpdateStatus(paymentID, "processed")
    if err != nil {
        return err // Did the payment go through? Is it in a bad state?
    }

    return nil
}

Functionally, this code worked. But it haunted me at night because:

Error handling was minimal and provided no context
There was no logging for debugging
No consideration for partial failures
No validation of inputs
No testing for edge cases

The result? Support tickets on Saturday mornings. Mysterious payment failures. Hours spent on debugging sessions. And the constant anxiety that something might be failing silently.

The Path to Trustworthy Code

Everything changed when I joined a team at WF. My mentor there had a simple philosophy: "Write every line of code as if you'll be on vacation when it runs in production."

This mindset shift transformed how I approached Go development. Here's what I learned, and what now lets me sleep well after committing code:

1. Errors Are Your Friends, Not Exceptions

Go's error handling is verbose but powerful when used correctly. The secret is to treat errors as valuable information carriers, not just failure signals:

func ProcessPayment(paymentID string) error {
    // Get payment details
    payment, err := db.GetPayment(paymentID)
    if err != nil {
        return fmt.Errorf("failed to retrieve payment %s: %w", paymentID, err)
    }

    // Validate before proceeding
    if err := validatePayment(payment); err != nil {
        return fmt.Errorf("payment validation failed for %s: %w", paymentID, err)
    }

    // Process the payment with context
    err = paymentGateway.Process(payment)
    if err != nil {
        // Log additional details for debugging
        log.WithFields(log.Fields{
            "payment_id": paymentID,
            "amount":     payment.Amount,
            "currency":   payment.Currency,
        }).Error("Payment processing failed")

        return fmt.Errorf("gateway failed to process payment %s: %w", paymentID, err)
    }

    // Success path is clearly logged too
    log.WithField("payment_id", paymentID).Info("Payment processed successfully")
    return nil
}

The difference? When something goes wrong, I have rich context. The error messages tell a story, making debugging faster and easier.

2. Tests That Give You Confidence

My earlier self wrote tests that proved the code worked in the happy path. My current self writes tests that prove the code won't break in production:

func TestProcessPayment(t *testing.T) {
    // Happy path test
    t.Run("successful payment processing", func(t *testing.T) {
        // Setup and assertions
    })

    // What happens when things go wrong
    t.Run("database unavailable", func(t *testing.T) {
        // Simulate DB failure
    })

    t.Run("payment gateway timeout", func(t *testing.T) {
        // Simulate slow payment gateway
    })

    t.Run("invalid payment data", func(t *testing.T) {
        // Test validation logic
    })

    t.Run("partial failure - payment processed but status update fails", func(t *testing.T) {
        // Test recovery mechanisms
    })
}

A comprehensive test suite is like insurance. It doesn't prevent all problems, but it significantly reduces the likelihood of common failures reaching production.

3. Graceful Degradation

One key insight: not all failures are equal. The best Go code doesn't just handle errors; it gracefully degrades functionality:

func GetUserRecommendations(userID string) ([]Recommendation, error) {
    // Try to get personalized recommendations
    recommendations, err := recommendationService.GetPersonalized(userID)
    if err != nil {
        // Log the error
        log.WithError(err).Warn("Failed to get personalized recommendations, falling back to popular items")

        // Fall back to popular recommendations
        return recommendationService.GetPopular()
    }

    return recommendations, nil
}

This pattern means your code tries its best to provide value, even when some components fail.

4. Monitoring and Observability Built-In

Code I can trust doesn't just work well; it tells me how it's working:

func ProcessOrder(ctx context.Context, order Order) error {
    // Start timing the operation
    start := time.Now()

    // Use defer to ensure metrics are always recorded
    defer func() {
        metrics.ObserveOrderProcessingTime(time.Since(start))
        metrics.IncrementOrdersProcessed()
    }()

    // Trace this operation for distributed tracing
    span, ctx := tracer.StartSpanFromContext(ctx, "process_order")
    defer span.Finish()

    // Add helpful information to the trace
    span.SetTag("order_id", order.ID)
    span.SetTag("customer_id", order.CustomerID)

    // Process the order with the traced context
    if err := orderProcessor.Process(ctx, order); err != nil {
        // Record error in metrics and trace
        metrics.IncrementOrderErrors()
        span.SetTag("error", true)
        span.LogKV("error.message", err.Error())

        return fmt.Errorf("processing order %s failed: %w", order.ID, err)
    }

    return nil
}

When your code has built-in monitoring, you don't need to wonder if it's working correctly in production. You know.

Real-Life Example: The 3 AM Bug That Never Called

Last year, we deployed a new feature to our payment system right before a long holiday weekend. The old me would have been anxious the entire time, but I wasn't worried at all.

Sure enough, something unexpected happened: a third-party API we depended on changed their response format. But instead of a production outage, here's what happened:

Our validation caught the changed format and returned a clear error
The circuit breaker we implemented prevented cascading failures
The system fell back to a secondary processing method
Our monitoring alerted the on-call engineer with the exact issue
Detailed logs showed exactly where and how the format had changed

The on-call engineer fixed it with a simple config change—no emergency, no all-hands debugging session.

The best part? I only found out about this when I read the incident report on Tuesday morning.

How to Write Go Code You Can Trust

After a decade of writing Go, here's my checklist for code I can trust enough to disconnect completely from work:

Handle errors with context: Wrap errors, add information, make debugging easy
Test failure modes: Don't just test success cases; test what happens when things fail
Build in graceful degradation: Design systems that bend rather than break
Make it observable: Logging, metrics, and tracing are not afterthoughts
Validate early and thoroughly: Catch bad inputs before they cause damage
Document assumptions: Clear documentation helps future you and your teammates

Conclusion

The difference between Go code that keeps you up at night and Go code you can trust isn't about clever algorithms or advanced features. It's about care, attention to detail, and a mindset that prepares for the unexpected.

When I interview Go developers now, I don't just look at how well they can write code that works. I look at how they handle the edge cases, how they think about failures, and whether their code would let them enjoy their weekends without worry.

Because in the end, the best code isn't just functionally correct—it's trustworthy enough that you can commit it on Friday afternoon and genuinely disconnect until Monday morning.

And for me, that peace of mind is worth every extra line of error handling, every additional test case, and every minute spent making my code more robust.

So next time you're writing Go code, ask yourself: "Would I sleep well tonight if this ran in production after I left?" If the answer isn't a confident "yes," you have more work to do.

Your future self—possibly on a beach somewhere without laptop access—will thank you.

ByteDance/Sonic: A Lightning-Fast JSON Library for Go

Trung Duong — Tue, 04 Feb 2025 04:20:16 +0000

"In the world of microservices, every millisecond counts. See how TikTok's engineering team revolutionized JSON processing in Go."

Have you ever deployed a Go service and watched your CPU spike when processing thousands of JSON requests? You're not alone. TikTok's engineers faced this exact challenge at a massive scale - billions of requests per day. Their solution? They created Sonic, a JSON library that's changing the game for Go developers.

The JSON Problem Every Go Developer Knows

If you're a Go developer, you've probably used the standard encoding/json package. It works, but let's be real - it's not the fastest kid on the block. If you're building modern web services or APIs, you're working with JSON every day - from REST APIs to configuration files. When TikTok's engineers found their services processing millions of JSON requests per second, even small performance improvements in JSON handling could make a huge difference in server costs and user experience.

Let's look at a common scenario:

// Your typical JSON processing with standard library
import "encoding/json"

type User struct {
    Name  string `json:"name"`
    Posts []Post `json:"posts"`
}

// Processing this for millions of requests...
json.Unmarshal(data, &user)

At small scale, this works fine. But when you're handling TikTok-scale traffic, those milliseconds add up to significant server costs and latency. That's exactly why ByteDance's team decided to tackle this challenge head-on.

The Speed Champion: Meet Sonic

Let's look at some real numbers first. When working with a medium-sized JSON file (about 13KB):

// Processing user profile with posts and metadata
Standard JSON: 106,322 ns/op (nanoseconds per operation)
Sonic:         32,393 ns/op
Memory Usage:
- Standard:    49,136 bytes with 789 allocations
- Sonic:       11,965 bytes with just 4 allocations

Small (400B, 11 keys, 3 layers)
Large (635KB, 10000+ key, 6 layers)

See bench.sh for benchmark codes.

The Secret Sauce: Four Simple Tricks

1. Just-In-Time Compilation (JIT)

Imagine you're a chef. Instead of following a generic recipe every time, you create a special, optimized recipe for each specific dish you make frequently. That's what JIT does!

Regular Go JSON library: Uses the same generic code for all JSON
Sonic: Creates specialized code paths for your specific JSON structures

2. SIMD: Doing More Work at Once

Think of SIMD like having multiple hands to do a task:

Regular way: Sort cards one by one
SIMD way: Sort multiple cards at once

Sonic uses these "multiple hands" (SIMD instructions) to process JSON data in parallel, making everything faster.

3. Smart Memory Usage

Here's a clever trick Sonic uses: When it finds a string in your JSON that doesn't have any special characters, it doesn't make a copy. Instead, it just points to the original string. It's like giving directions instead of drawing a new map!

// Example JSON
{"name": "John", "city": "New York"}

// Standard library: Makes copies of "John" and "New York"
// Sonic: Just references these strings if they're simple

4. Optional Features = Better Speed

Sonic makes some smart choices about what features to make optional:

Doesn't sort map keys by default (saves 10% processing time)
Doesn't escape HTML by default (saves 15% processing time)

You can still turn these features on if you need them, but by making them optional, Sonic stays fast for most common uses.

Design

The design is easy to implement:

Aiming at the function-call overhead cost by the codec dynamic-assembly, JIT tech is used to assemble opcodes (asm) corresponding to the schema at runtime, which is finally cached into the off-heap memory in the form of Golang functions.
For practical scenarios where big data and small data coexist, we use pre-conditional judgment (string size, floating precision, etc.) to combine SIMD with scalar instructions to achieve the best adaptation.
As for insufficiency in compiling optimization of go language, we decided to use C/Clang to write and compile core computational functions, and developed a set of asm2asm tools to translate the fully optimized x86 assembly into plan9 and finally load it into Golang runtime.
Giving the big speed gap between parsing and skipping, the lazy-load mechanism is certainly used in our AST parser, but in a more adaptive and efficient way to reduce the overhead of multiple-key queries.

In detail, we conducted some further optimization:

Since the native-asm functions cannot be inlined in Golang, we found that its cost even exceeded the improvement brought by the optimization of the C compiler. So we reimplemented a set of lightweight function-calls in JIT:
- Global-function-table + static offset for calling instruction
- Pass parameters using registers
Sync.Map was used to cache the codecs at first, but for our quasi-static (read far more than write), fewer elements (usually no more than a few dozen) scenarios, its performance is not optimal, so we reimplement a high-performance and concurrent-safe cache with open-addressing-hash + RCU tech.

From Sonic Design

Real-World Benefits

Let's talk numbers that matter in the real world:

Memory Usage (for medium JSON):
- Standard Go: Uses about 49KB of memory with 789 allocations
- Sonic: Uses only about 12KB with just 4 allocations
- That's huge when you're processing millions of requests!
Practical Impact:
- Faster API responses
- Lower server costs
- Better user experience
- More requests handled per server

Should You Switch to Sonic?

Sonic might be great for you if:

You process lots of JSON data
Performance is important for your application
You're working on AMD64 or ARM64 processors

But remember:

It requires Go version 1.17 or higher
It works on Linux, MacOS, and Windows
Some features (like HTML escaping) need to be explicitly enabled if you need them

Quick Start Example

Here's how simple it is to use Sonic:

import "github.com/bytedance/sonic"

// Encoding
data := map[string]string{"hello": "world"}
bytes, err := sonic.Marshal(data)

// Decoding
var result map[string]string
err = sonic.Unmarshal(bytes, &result)

Wrapping Up

Sonic shows us that even with something as common as JSON processing, there's still room for impressive improvements. By using modern CPU features (SIMD), smart compilation (JIT), and thoughtful design choices, it achieves remarkable performance gains over the standard library.

Remember: In software development, it's not just about making things work - sometimes, making them work faster can open up new possibilities for what your applications can achieve!