Forem: Tuna

Why Memory Alignment Matters in Go: Making Your Structs Lean and Fast

Tuna — Thu, 03 Apr 2025 11:34:14 +0000

When writing Go code, it’s easy to forget what’s happening under the hood—especially when it comes to memory layout. But did you know that how you organize fields in a struct can actually bloat memory and even affect performance?

Let’s take a fun but technical dive into how memory alignment works in Go, and why struct layout matters more than you might think.

🧠 What Is Memory Alignment, Exactly?

Memory alignment is a concept rooted in how CPUs access memory. Most modern CPUs are optimized to access memory at aligned addresses—that is, addresses that are multiples of the data’s size.

For example:

An int64 (8 bytes) should ideally start at a memory address that’s a multiple of 8.
An int32 (4 bytes) should start at a multiple of 4.

If a variable is misaligned, the CPU might need to perform multiple memory reads just to get the full data. This slows things down. On top of that, if a variable spans across two cache lines, you’ll suffer a performance penalty because the CPU has to load both cache lines.

Here’s a simple analogy: imagine reading a sentence split across two pages in a book. You flip once, then again, just to get the whole message. Alignment keeps your "sentence" on the same page.

TL;DR:

Aligned data = fast memory access
Misaligned data = slow, possibly multiple reads

🛠️ How Go Handles Memory Alignment

Go takes care of alignment automatically. Each data type has an alignment requirement, and Go inserts padding bytes between struct fields to ensure proper alignment.

Let’s look at this struct:

type PoorlyAligned struct {
    a byte   // 1 byte
    b int64  // 8 bytes
    c byte   // 1 byte
}

Although the fields themselves total 9 bytes, the compiler inserts padding to align each field properly. This results in:

📆 Total size: 24 bytes

Why?

Field	Offset	Size	Notes
a (byte)	0	1
padding	1–7	7	To align `int64` on 8-byte boundary
b (int64)	8–15	8	Starts at offset 8.
c (byte)	16	1	Starts at offset 16.
padding	17–23	7	To round struct size to 8-byte multiple

✅ Well-Aligned Layout = Happy Memory

Now let’s rearrange the fields:

type WellAligned struct {
    b int64  // 8 bytes
    a byte   // 1 byte
    c byte   // 1 byte
}

Result:

📆 Total size: 16 bytes

Field	Offset	Size
b (int64)	0–7	8
a (byte)	8	1
c (byte)	9	1
padding	10–15	6

💡 By simply reordering the fields, we reduced the struct's size from 24 bytes down to 16 bytes—saving 8 bytes for every single instance of this struct.

🚀 Real-World Impact: Memory and Performance

Why does this matter?

Less memory per struct = lower overall memory usage
Smaller structs fit better into CPU cache lines
Better cache usage = fewer cache misses = faster processing
Less memory used = less work for the garbage collector

🔬 Benchmarking Time!

Let’s benchmark two slices: one using the poorly-aligned struct, one using the optimized version.

package main

import (
    "testing"
)

type PoorlyAligned struct {
    a byte   // 1 byte
    b int64  // 8 bytes
    c byte   // 1 byte
}

type WellAligned struct {
    b int64  // 8 bytes
    a byte   // 1 byte
    c byte   // 1 byte
}

var poorlySlice = make([]PoorlyAligned, 1_000_000)
var wellSlice = make([]WellAligned, 1_000_000)

func BenchmarkPoorlyAligned(b *testing.B) {
    var sum int64
    for n := 0; n < b.N; n++ {
        for i := range poorlySlice {
            sum += poorlySlice[i].c
        }
    }
}

func BenchmarkWellAligned(b *testing.B) {
    var sum int64
    for n := 0; n < b.N; n++ {
        for i := range wellSlice {
            sum += wellSlice[i].c
        }
    }
}

📊 Typical Results:

goos: darwin
goarch: arm64
pkg: metaleap/pkg/tunatest
cpu: Apple M1
BenchmarkPoorlyAligned-8            3609            323200 ns/op
BenchmarkWellAligned-8              3759            316617 ns/op
PASS

✅ Result: On my Apple M1 chip, optimizing the struct layout resulted in a ~2% performance improvement.

🛠️ Tools: Check Alignment with `go vet`

You don’t need to do this manually. Go provides a vetting tool to help:

go vet -fieldalignment ./...

It will suggest better ordering for your structs when applicable, like:

struct with 24 bytes could be 16 bytes

✅ Best Practices for Struct Layout in Go

Order fields from largest to smallest alignment.
Group fields with the same size together.
Consider memory layout when defining high-volume or performance-critical structs.
Use go vet -fieldalignment for automatic suggestions.

📝 Final Thoughts

Memory alignment is one of those “under-the-hood” details that can have outsized effects in real-world programs—especially those dealing with millions of objects or high-performance data processing.

With just a bit of attention to field ordering, you can:

Save memory
Speed up your programs
Make your data more cache-friendly

Go’s compiler does the heavy lifting to ensure safety and correctness. Your job is to be mindful of layout when performance or memory use matters.

📚 References

Go 1.24 and the Swiss Table Revolution: A New Era for Maps

Tuna — Mon, 03 Feb 2025 11:36:27 +0000

Introduction

With the release of Go 1.24, the language takes a significant leap forward in performance optimization—this time, targeting one of its most fundamental data structures: the map. Inspired by Google’s high-performance Swiss Table (popularized in C++’s Abseil library), the Go team has re-engineered the internals of its map implementation to deliver faster lookups, reduced memory overhead, and better cache utilization. This change marks a pivotal shift in how Go handles hash tables, addressing long-standing inefficiencies while embracing modern hash table design principles. Let’s dive into how this transformation works and why it matters.

Part 1: The Old Map (A Quick Refresher)

Before Go 1.24, maps worked like this:

Buckets: Your keys were stored in buckets (groups of 8 entries).
Overflow Chains: If a bucket filled up, new entries went into overflow buckets, linked like a chain.
Tophash: Each entry had a tiny tophash (8 bits of the key’s hash) to skip non-matching keys quickly.

It worked, but it had some rough edges:

Slow Lookups: Scanning overflow chains meant jumping around in memory (hello, cache misses!).
Memory Bloat: Overflow buckets ate up extra space.
Resizing Woes: Growing the map meant rehashing everything. Ouch.

👉 Want to dive deeper into the old design? Check out this great explainer.

Part 2: Swiss Table Deep Dive—How It Handles Collisions Like a Pro

Let’s break down the Swiss Table’s secret sauce step-by-step, using a concrete example. Spoiler: It’s all about smart metadata and cache-friendly groups.

Step 1: Hash Splitting (H1 vs. H2)

Imagine you insert the key price into a Go map. The hash function generates a 64-bit value, say:
H = 0x5F4A3B2C1D (simplified for clarity).

The Swiss Table splits this hash into two parts:

H2 (7 bits): The first 7 bits of the hash. Think of this as a "mini fingerprint" of the key.
H1 (57 bits): The remaining bits. This determines which group the key belongs to.

Example:

H2 = First 7 bits of 0x5F4A3B2C1D → 0x5F (binary: 01011111).
H1 = The rest → Used to calculate the group index.

Step 2: Metadata—The Magic Cheat Sheet

Every entry in the map has a 1-byte metadata field. This byte stores:

The 7-bit H2 value.
A 1-bit status flag:
- 0: Empty.
- 1: Occupied.
- 0x80 (binary 10000000): Tombstone (deleted).

Example:
When you insert price, Go stores 0x5F in the metadata (since its H2 is 0x5F) and marks the status as 1 (occupied). The full metadata byte becomes 0x5F | 0x01 = 0x5F1 (simplified).

Step 3: Group Hunting with H1

The map is divided into groups of 16 entries. To find where "price" belongs:

Use H1 to calculate the group index:

group_index = H1 % number_of_groups

Check the metadata of all 16 entries in that group.

Example:
If H1 points to Group 3, Go scans the 16 metadata bytes in Group 3.

Step 4: Handling Collisions (No Linked Lists!)

If Group 3 is full, here’s what happens:

Linear Probing: Go checks the next group (Group 4, then Group 5, etc.) until it finds an empty slot.
Tombstones: If a slot is marked as deleted (tombstone), it’s treated as empty for insertion but preserved for lookups.

Why This Rocks:

No more chasing linked lists—everything stays in contiguous memory.
CPU caches love this (fewer cache misses!).

Example Walkthrough:

Insert `price`

Hash: price → H = 0x5F4A3B2C1D.
Split:
- H2 = 0x5F → stored in metadata.
- H1 → calculates group index (e.g., Group 3).
Check Group 3:
- If Slot 5 in Group 3 is empty → insert "price" there.
- If Slot 5 is occupied, check Slot 6, then Slot 7, etc.
Collision? If Group 3 is full, move to Group 4.

Lookup `price`

Compute H2 = 0x5F.
Go to the group calculated by H1.
Scan the group’s metadata for 0x5F.
- Match found: Compare the actual key (to handle hash collisions).
- No match: Skip to the next group.

This skips 99% of non-matching keys without even looking at the keys!

Why This Beats the Old Map

Old Map	Swiss Table Map
Used linked lists for collisions	Uses linear probing (no pointers!)
Scanned keys one-by-one	Metadata filters out mismatches instantly
Overflow buckets wasted memory	Tombstones reuse space efficiently

Final Takeaway

The Swiss Table’s genius lies in:

Metadata: A tiny cheat sheet to skip useless work.
Groups: Cache-friendly memory layout.
Linear Probing: Simpler, faster collision handling.

Go 1.24’s new map isn’t just faster—it’s smarter. And you get all this without changing a single line of your code! 🎉

👉 For benchmarks and nitty-gritty details, check out this post.

Still have questions? Drop them in the comments! 👇

MongoDB Indexes: The Superpower to Speed Up Your Queries ⚡

Tuna — Mon, 11 Nov 2024 05:00:26 +0000

Imagine walking into a massive library to find a single book. Without an index or catalog, you’d have to go shelf by shelf, scanning every title—sounds exhausting, right? MongoDB indexes are like a magical catalog that speeds up your searches, so MongoDB can find what it needs in a flash.

Just like library catalogs, indexes make data retrieval way faster. MongoDB uses a clever little structure called a B-tree to keep everything organized. B-trees allow MongoDB to skip over data that’s irrelevant to your search.

Want to know more about B-trees? Check out this awesome blog on B-trees.

Types of MongoDB Indexes

MongoDB offers several types of indexes, each with its own quirks and superpowers. Let’s take a look! 🚀

Single Field Index

This is your classic, no-frills index, created on a single field. If you have a collection of users and often query by username, creating a single field index on username helps MongoDB find username quickly instead of skimming through every document.

db.users.createIndex({ username: 1 })

Here:

username is the field we’re indexing.
1 means ascending order (you could also use -1 for descending order).

Compound Indexes

Need to search by multiple fields? A compound index includes more than one field, making multi-field searches lightning fast! Imagine a shopping app where you often query category and price. A compound index lets MongoDB efficiently find items in that category and price range.

db.products.createIndex({ category: 1, price: -1 })

MongoDB can use this index to search by both category and price, or just category (since it’s the first field in the index). However, searches only on price won’t use this index.

Tip: If you have both a compound index { a: 1, b: 1 } and a single index { a: 1 }, consider removing the single index. MongoDB will happily use the compound index for both!

Multikey Indexes

Multikey indexes are MongoDB’s answer for indexing arrays. If you have a tags field storing an array (like ["electronics", "computing", "portable"]), creating a multikey index lets MongoDB search each element as if it were a standalone field.

db.products.createIndex({ tags: 1 })

Example query:

{
  "name": "Laptop",
  "tags": ["electronics", "computing", "portable"]
}

MongoDB uses the multikey index to find all products tagged with computing without digging through each tag manually.

Note: Compound indexes can have only one array field. Also, multikey indexes can get hefty for large arrays, so be mindful.

Index Properties

Index properties affect how the query planner uses an index and how indexed documents are stored. You can specify index properties as optional parameters when you create an index.

Case-Insensitive Indexes

Need case-insensitive searches? Use collation options when creating indexes.

db.fruit.createIndex(
   { type: 1 },
   { collation: { locale: 'en', strength: 2 } }
)

This will allow you to find all variations of apple (like Apple or APPLE). Just remember that your queries also need to specify the same collation.

Partial Indexes

Want an index that only includes specific documents? A partial index lets you add a filter, so only documents that meet the condition are indexed.

db.restaurants.createIndex(
   { cuisine: 1 },
   { partialFilterExpression: { rating: { $gt: 5 } } }
)

Only restaurants with a rating greater than 5 will be indexed—less storage, more efficiency!

Sparse Indexes

A sparse index skips any documents that don’t have the indexed field, unlike other indexes which include every document, even if the field is missing or null.

db.scores.createIndex( { score: 1 } , { sparse: true } )

This helps MongoDB return only documents that actually have a score field when you query based on that field.

TTL Indexes

TTL indexes automatically remove documents after a set time. Great for logs, session data, or anything that only needs to live temporarily.

The TTL index expireAfterSeconds value must be within 0 and 2147483647 inclusive.

Here, documents expire after 1 hour (3600 seconds) from their lastModifiedDate:

db.eventlog.createIndex(
   { lastModifiedDate: 1 },
   { expireAfterSeconds: 3600 }
)

You can create a TTL index with a Partial Index to expire documents with specific filter expressions.

db.eventlog.createIndex(
   { "lastModifiedDate": 1 },
   {
      name: "Partial-TTL-Index",
      partialFilterExpression: { canDelete : true },
      expireAfterSeconds: 10
   }
)

Unique Indexes

A unique index ensures no duplicate values for the indexed field. MongoDB automatically makes the _id field unique, but you can add unique constraints to other fields, too.

db.members.createIndex( { "user_id": 1 }, { unique: true } )

Need a unique constraint on a combination of fields? No problem! With compound unique indexes, MongoDB enforces uniqueness across the combination of the index key values, so only a unique pair of values for each indexed field can exist.

db.users.createIndex({ firstName: 1, lastName: 1 }, { unique: true })

Sometimes, you don’t want a unique constraint across your whole collection—just a specific subset of documents. With partialFilterExpression, you can apply a unique constraint only to documents that match certain criteria, while the rest of the documents ignore it.

db.events.createIndex(
   { userId: 1, status: 1 },
   {
      unique: true,
      partialFilterExpression: { status: "active" }
   }
)

Note: The unique constraint only applies to documents that match the filter expression. MongoDB won’t prevent you from inserting documents that don’t meet the unique constraint if they don’t satisfy the filter criteria.

Wrap-Up 🎉

Indexes are your friends! They make MongoDB queries way faster, helping you find exactly what you need without MongoDB combing through the entire collection. Whether you’re indexing single fields, arrays, or setting up TTL for auto-cleanup, each index type has its own special trick to speed things up. Just remember, indexes do require some storage, so don’t go overboard.

Happy querying, and may the MongoDB index force be with you! 🖖

Understanding Iterators in Go: A Fun Dive!

Tuna — Thu, 24 Oct 2024 11:08:28 +0000

If you're a Go programmer, you've probably heard about iterators many times in Go 1.22, and especially in Go 1.23. But maybe you're still scratching your head, wondering why they're useful or when you should use them. Well, you're in the right place! Let's start by looking at how iterators work in Go and why they can be so useful.

A Simple Transformation: No Iterator Yet

Imagine we have a list of numbers, and we want to double each number. We could do this using a straightforward function like the one below:

package main

import (
    "fmt"
)

func NormalTransform[T1, T2 any](list []T1, transform func(T1) T2) []T2 {
    transformed := make([]T2, len(list))

    for i, t := range list {
        transformed[i] = transform(t)
    }

    return transformed
}

func main() {
    list := []int{1, 2, 3, 4, 5}
    doubleFunc := func(i int) int { return i * 2 }

    for i, num := range NormalTransform(list, doubleFunc) {
        fmt.Println(i, num)
    }
}

Here’s what happens when you run this code:

Pretty simple, right? This is a basic generic Go function that takes a list of any type T1, applies a transformation function to each element, and returns a new list with the transformed list of any type T2. Easy to understand if you know Go generics!

But what if I told you there’s another way to handle this—using an iterator?

Enter the Iterator!

Now, let’s take a look at how you might use an iterator for the same transformation:

package main

import (
    "fmt"
)

func IteratorTransform[T1, T2 any](list []T1, transform func(T1) T2) iter.Seq2[int, T2] {
    return func(yield func(int, T2) bool) {
        for i, t := range list {
            if !yield(i, transform(t)) {
                return
            }
        }
    }
}

func main() {
    list := []int{1, 2, 3, 4, 5}
    doubleFunc := func(i int) int { return i * 2 }

    for i, num := range NormalTransform(list, doubleFunc) {
        fmt.Println(i, num)
    }
}

Before running it, you must make sure your Go version is 1.23. The output is exactly the same:

But wait, why would we need an iterator here? Isn't that more complicated? Let’s dig into the differences.

Why Use an Iterator?

At first glance, iterators seem a bit over-engineered for something as simple as transforming a list. But when you run benchmarks, you start to see why they’re worth considering!

Let’s benchmark both methods and see how they perform:

package main

import (
    "testing"
)

var (
    transform = func(i int) int { return i * 2 }
    list      = []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
)

func BenchmarkNormalTransform(b *testing.B) {
    for i := 0; i < b.N; i++ {
        NormalTransform(list, transform)
    }
}

func BenchmarkIteratorTransform(b *testing.B) {
    for i := 0; i < b.N; i++ {
        IteratorTransform(list, transform)
    }
}

Here’s the initial benchmark result:

BenchmarkNormalTransform-8      41292933                29.49 ns/op
BenchmarkIteratorTransform-8    1000000000               0.3135 ns/op

Whoa! That’s a huge difference! But hold on—there’s a bit of unfairness here. The NormalTransform function returns a fully transformed list, while the IteratorTransform function only sets up the iterator without transforming the list yet.

Let’s make it fair by fully looping through the iterator:

func BenchmarkIteratorTransform(b *testing.B) {
    for i := 0; i < b.N; i++ {
        for range IteratorTransform(list, transform) {
        }
    }
}

Now the results are more reasonable:

BenchmarkNormalTransform-8      40758822                29.16 ns/op
BenchmarkIteratorTransform-8    53967146                22.39 ns/op

Okay, the iterator is a bit faster. Why? Because NormalTransform creates an entire transformed list in memory (on the heap) before returning it, while the iterator does the transformation as you loop through it, saving time and memory.

Read more about Stack and Heap here

The iterator's real magic happens when you don’t need to process the whole list. Let’s benchmark a scenario where we only want to find the number 4 after transforming the list:

func BenchmarkNormalTransform(b *testing.B) {
    for i := 0; i < b.N; i++ {
        for _, num := range NormalTransform(list, transform) {
            if num == 4 {
                break
            }
        }
    }
}

func BenchmarkIteratorTransform(b *testing.B) {
    for i := 0; i < b.N; i++ {
        for _, num := range IteratorTransform(list, transform) {
            if num == 4 {
                break
            }
        }
    }
}

The results speak for themselves:

BenchmarkNormalTransform-8      39599744                30.03 ns/op
BenchmarkIteratorTransform-8    164853723                7.288 ns/op

In this case, the iterator is much faster! Why? Because the iterator doesn’t transform the entire list—it stops as soon as it finds the result you’re looking for. On the other hand, NormalTransform still transforms the entire list, even if we only care about one item.

Conclusion: When to Use Iterators?

So, why use an iterator in Go?

Efficiency: Iterators can save both time and memory by not processing the entire list if you don’t need it.
Flexibility: They allow you to handle large datasets efficiently, especially when working with streams of data or when you need to stop early. But keep in mind, iterators can be a bit trickier to understand and implement. Use them when you need that extra performance boost, especially in scenarios where you don’t need to work with an entire list upfront.

Iterators: They're fast, flexible, and fun—once you get the hang of them!

B-tree: The Secret Recipe Behind Efficient Data Handling 🍰

Tuna — Sat, 19 Oct 2024 16:58:28 +0000

Have you ever wondered how databases keep your data sorted and quickly accessible? It’s all thanks to B-trees! These powerful data structures are behind the scenes in many database systems, file storage systems, and your computer’s operating system. Let’s break down B-trees in an easy way.

What is a B-tree? 🌳

B-tree is a self-balancing data structure that keeps data sorted and supports efficient searches, sequential access, insertions, and deletions in logarithmic time. It extends the concept of a binary search tree by permitting nodes to have more than two children.

A B-tree is a self-balancing search tree defined by the following properties:

Maximum Children: Every node can have up to m children.
Minimum Children: Every internal node (except the root and leaves) has at least ⌈m/2⌉ children.
Root Node: The root has at least two children unless it's a leaf.
Leaf Level Consistency: All leaves exist on the same level.
Keys and Separation: A non-leaf node with k children has k-1 keys, which serve as separation values dividing the node’s children.

How B-trees Work: An Easy Breakdown ⚙️

Searching a Key 🔍

In a B-tree, each node contains multiple keys, and a search starts from the root, comparing the key against all keys in the node. If the key isn’t found, the appropriate child node is followed. Each node reduces the search range considerably because B-trees remain balanced, with multiple keys per node. This ensures a worst-case time complexity of O(logₘ n), where m is the order of the B-tree (maximum number of children) and n is the number of elements.

Inserting a Key 🎯

Inserting data into a B-tree is like putting a book into a sorted bookshelf. The B-tree checks where it belongs finds the right place, and slides it in while keeping everything neat. If a shelf (node) gets too full, it splits into two smaller shelves, keeping the tree balanced.

Sounds easy, right? Let’s dive into an example to see how it works:

We have a B-tree where each node can have up to 4 children. Our task is to add key 10.

First, we find the right spot and place it in the correct order.

Uh-oh! The node now has 5 children, which exceeds the maximum allowed for a single node. We must split it in half and push the middle key to the parent node.

But wait—the top node is full! So, we repeat the process until the tree satisfies all its rules. Now, we get the balanced B-tree shown below!

Deleting a Key 🗑️

Deleting a key from a B-tree is a bit more complex. The tree must ensure that after deletion, the structure remains balanced:

Deleting a Key from a Leaf Node:

Simplest Case: If the key is in a leaf node, it can be directly removed.
Rebalancing: After deletion, if the node still has enough keys (≥⌈m/2⌉), no further action is needed. If it falls below this minimum, the node needs to borrow a key from a sibling or merge with a sibling, ensuring the B-tree properties are maintained.

Deleting a Key from an Internal (Middle) Node:

This case is trickier because internal nodes serve as separators for their subtrees. There are two approaches:

Replace with Predecessor/Successor: Replace the key with its predecessor (the largest key from the left subtree) or successor (the smallest key from the right subtree), then delete the predecessor/successor from its leaf node. This may trigger a recursive deletion in the leaf.
Merge or Borrow: If the node becomes underfilled after deletion (fewer than ⌈m/2⌉ keys), borrowing a key from an adjacent sibling or merging with a sibling is required. This ensures that the tree remains balanced. If merging happens, a key from the parent node must move down, potentially requiring further rebalancing up the tree.

Let’s say a B-tree of order 4 has the following structure:

In the example shown in the image, when we delete the root key 15 (an internal node), we follow these steps:

Replace Key: We replace 15 with the largest key from the left subtree, which is 10 (we could also use the smallest key from the right subtree, 20).
Remove Key: After replacement, we remove 10 from the leaf node.
Merge: Since the left leaf node now has fewer keys than required (below the minimum), we merge it with its sibling to maintain the B-tree properties.

This keeps the B-tree balanced.

Why B-trees Dominate Databases

Let’s consider a database with 1 million records stored on a disk, where finding one record using binary search would take 20 comparisons ( log2 (1,000,000) ≈ 20). Each comparison requires a disk read, and since each disk read takes around 10 milliseconds, searching would take 0.2 seconds.

However, databases improve this process using B-trees. By grouping data into disk blocks (e.g., 100 records per block) and using an index structure, B-trees dramatically reduce the number of disk reads. A B-tree with 100 entries per node reduces the number of reads to about 3 disk reads (because log100 (1,000,000) = 3), taking just 30 milliseconds.

In practice, B-trees ensure efficient searching because they minimize disk reads. This is critical in databases, where disk access is slow compared to in-memory comparisons. With fewer disk reads, B-trees make searches faster by using logarithmic time based on the number of children per node (logₘ n), where m can be large, making search times drastically shorter than in a binary search tree (BST).

In Conclusion

B-trees are like the librarians of data structures, ensuring everything is neatly organized, quickly accessible, and always balanced. Whether you're dealing with small data sets or massive databases, B-trees keep your data sorted and efficient behind the scenes! 🌳

For more in-depth information on B-trees, you can check out this detailed explanation on Wikipedia.

Understanding Stack and Heap in Go: A Simple Guide

Tuna — Sun, 13 Oct 2024 15:59:45 +0000

When you start learning Go, or any programming language for that matter, you will often hear about stack and heap memory. These two areas of memory are super important for understanding how your program runs and manages data behind the scenes. But don’t worry—today, we’ll explain them in an easy-to-understand way with a fun twist.

What is the Stack?

Imagine the stack as a neat pile of trays in a cafeteria. Every time someone needs a tray, they grab one from the top. And when they return a tray, they put it back on top of the pile. The stack in Go works similarly!

The stack is a small, super-fast memory area.
It stores things like function calls and local variables (like integers or small structs).
Every time a function is called, Go adds a “tray” (frame) to the top of the stack. When the function finishes, it removes that tray from the top.

So, the stack follows a LIFO (Last In, First Out) system, just like how you’d take and return trays.

Example: Stack in Action
Let’s say we have this simple Go code:

func main() {
    greet("John")
}

func greet(name string) {
    message := "Hello, " + name
    fmt.Println(message)
}

Here’s what happens step-by-step:

Go puts a tray on the stack for the main function.
main calls greet(), so another tray (stack frame) is added for greet().
The greet function creates a local variable message, and Go places that in the tray.
Once greet() is done, that tray is removed (popped) from the stack.
Finally, when main is done, the last tray is removed.

Neat and organized, right? The stack is perfect for handling things that are temporary and go away quickly—like local variables inside functions.

What is the Heap?

Now, let’s imagine the heap as a large playground. Unlike the stack, where you can only add or remove things from the top, the heap is more like a big open area where you can put things anywhere.

The heap is a much larger memory space.
It is used to store data that needs to stick around for a while, even after a function finishes.
Things stored on the heap have no specific order, and Go has to keep track of them using something called pointers.

While the heap is big and can store more data, it’s slower to access than the stack because Go has to figure out where things are and clean up after itself. Go has a garbage collector that automatically tidies up unused heap memory, just like someone sweeping up the playground.

Example: Heap in Action
Take a look at this Go code:

func main() {
    user := newUser("Alice")
    fmt.Println(user.name)
}

func newUser(name string) *User {
    user := &User{name: name}
    return user
}

type User struct {
    name string
}

Here’s how the heap comes into play:

main calls newUser().
Inside newUser, we create a new User struct with the name field.
Go decides to store this struct on the heap, not on the stack, because it needs to stick around after newUser returns.
Go gives us a pointer (like a map to where the struct lives in memory) and returns it to main.
Now, even after newUser is done, the User struct stays in memory (on the heap), and main can still access it using the pointer.

The heap is useful when you need to store data that outlives the function it was created in, but it’s a little slower and needs careful management by Go’s garbage collector.

Stack vs. Heap: What’s the Difference?

Stack is like a tray stack: small, fast, and temporary. Perfect for local variables inside functions.
Heap is like a playground: big, more flexible, but slower. Used for things that need to live longer (like objects that need to be shared across functions).

Conclusion

Understanding the difference between the stack and heap is key to writing efficient Go programs. The stack is fast and easy to manage, great for temporary data. The heap is bigger but slower, used when you need something to stick around.

Go handles much of the complexity for you with automatic memory management, but knowing these concepts will help you write more optimized and efficient code.

Happy coding!

Introduction to SOLID Principles: The Heroic Saga of Code

Tuna — Sun, 13 Oct 2024 08:22:05 +0000

Welcome, fellow adventurer, to the realm of software design, where dragons of spaghetti code threaten the kingdom of Clean Code! Fear not, for today we arm ourselves with the legendary SOLID principles. These principles are not just any boring rules; they’re your enchanted shield and sword in the battle for maintainable, scalable, and bug-free code.

But first, what is SOLID in the name of code refactoring? It stands for five knightly virtues—Single Responsibility, Open/Closed, Liskov Substitution, Interface Segregation, and Dependency Inversion.

Grab your mouse and keyboard. Let’s embark on this journey together, full of twists, turns, and maybe even a few bad jokes.

S: Single Responsibility Principle (SRP) - The Anti-Monolithic Spell

Definition:
A class should have one, and only one, reason to change. Don’t give your class an existential crisis!

In Action:
Imagine a medieval blacksmith who makes swords, bakes bread, and does your taxes. He’s trying to sharpen your sword while thinking about tax deductions, and there’s flour in the forge. Messy, right? That’s what happens when your class tries to do too much!

Instead, keep your classes focused. Have one class for sword-making, another for baking, and a third for dealing with the tax trolls.

Example:

// Bad SRP example
type Blacksmith interface {
    MakeSword()
    BakeBread()
    FileTaxes()
}
// Better SRP example
type Blacksmith interface {
    MakeSword()
}

type Baker interface {
    BakeBread()
}

type TaxFiler interface {
    FileTaxes()
}

A knight with a single responsibility sleeps better at night. Trust me.

O: Open/Closed Principle (OCP) - The Door is Closed, but Feel Free to Paint it!

Definition:
Your classes should be open for extension but closed for modification. It’s like having a fancy door that you can decorate but never remove.

In Action:
You’ve built a loyal, trustworthy squire. But now you need him to do more—fetch water, sharpen your sword, and polish your armor. Don’t rewrite his DNA (code)! Just add to his training.

Instead of modifying the Squire class every time, create extensions. Keep the door closed, but hang a stylish sword on it!

Example:

// Without OCP
type Squire struct {
    FetchWater()
    SharpenSword()
}

// With OCP: Add without modifying
type Task interface {
    Perform()
}

type FetchWaterTask struct {}

func (f FetchWaterTask) Perform() {
    // Fetch water
}

type SharpenSwordTask struct {}

func (s SharpenSwordTask) Perform() {
    // Sharpen sword
}

Now your code can evolve as majestically as a knight’s skill set without rewriting the core.

L: Liskov Substitution Principle (LSP) - The “Imposter Syndrome” Cure

Definition:
If a class inherits from another, it should be able to replace the parent class without causing mayhem (or, in our case, runtime errors).

In Action:
Imagine you own a stable. Horses gallop, and unicorns… well, they gallop too, but with glitter. Now, if you swapped your horse for a unicorn, you wouldn’t expect the unicorn to explode or refuse to gallop, right? That’s LSP in action.

Your subclasses (like the unicorn) should behave like the parent (the horse), without any unexpected behavior (no glitter explosions, please).

Example:

// LSP violation
type Horse struct {
    Gallop()
}

type Unicorn struct {
    Gallop() {
        // Throws glitter instead of galloping!
    }
}

Following LSP is like having a well-behaved unicorn. It looks magical but doesn’t break the barn.

I: Interface Segregation Principle (ISP) - The Curse of Unnecessary Contracts

Definition:
Clients should not be forced to depend on interfaces they don’t use. Consider it breaking down contracts so you only sign what’s necessary.
In Action:
Let’s say you’re a knight with a sword. Your knight’s guild gives you a contract that also requires you to know archery, jousting, and dragon taming. But you just want to swing your sword! Don’t make a sword-wielding knight learn dragon-taming unless they want to.

Example:

// Bad ISP Example
type Warrior interface {
    FightWithSword()
    RideDragon()  // Not every warrior rides dragons!
}

// Better ISP Example
type Swordsman interface {
    FightWithSword()
}

type DragonRider interface {
    RideDragon()
}

This way, you can focus on what you do best—sword-swinging—and leave the dragon-taming to the professionals.

D: Dependency Inversion Principle (DIP) - No More Royal Puppeteers

Definition:
High-level modules should not depend on low-level modules. Both should depend on abstractions. Let the peasants (low-level modules) and kings (high-level modules) follow the same laws (interfaces).

In Action:
Think of a knight who needs a weapon. Instead of demanding a specific sword from the blacksmith, they ask for any weapon that adheres to the Weapon interface. This way, they can use a sword, a bow, or even a banana (if someone implements it). The knight isn’t tied to one weapon, just the idea of a weapon.

Example:

// Without DIP
type Knight struct {
    sword Sword
}

// With DIP
type Weapon interface {
    Use()
}

type Knight struct {
    weapon Weapon
}

func (k *Knight) Fight() {
    k.weapon.Use()
}

Now the knight is flexible and can fight with any weapon, even a legendary potato cannon (if coded properly).

Conclusion: The Hero’s Journey to SOLID Code

The journey to mastering SOLID principles is full of challenges, much like slaying dragons. But with Single Responsibility, Open/Closed, Liskov Substitution, Interface Segregation, and Dependency Inversion by your side, your code will be as smooth as a bard’s ballad.

You now wield the mighty sword of maintainable design. Go forth and refactor with confidence, brave coder!

Let me know if you’d like to see any specific areas covered more deeply!

Clustered index vs Non-clustered and how they look like in MongoDB

Tuna — Thu, 10 Oct 2024 10:19:15 +0000

When working with databases, performance optimization is often a key concern. One of the most effective ways to optimize queries is through indexing. In relational databases like SQL Server or MySQL, clustered and non-clustered indexes are important concepts for efficiently organizing and retrieving data. However, if you're moving to or working with NoSQL databases like MongoDB, you might wonder how these concepts apply. Let’s explore both and look at how MongoDB handles indexing.

What is an Index?

Before diving into clustered and non-clustered indexes, let's clarify what an index is. An index is a special data structure that improves the speed of data retrieval operations on a database table at the cost of additional storage space. Without an index, the database engine would have to perform a full scan of the entire collection or table to find the necessary records.

Clustered vs Non-Clustered Index

1. Clustered Index

Definition: A clustered index is a type of index where the data rows are stored physically on the disk in the same order as the index. In other words, the clustered index defines the order in which data is stored in a table.
Key Features:
- A table can have only one clustered index because the data can only be sorted in one way.
- The actual data of the table is part of the clustered index.
- Clustered indexes are generally created on a primary key.
- Faster data retrieval for range queries, as data is stored in sorted order.

2. Non-Clustered Index

Definition: A non-clustered index stores the data separately from the actual table. The index contains pointers to the physical location of the data rather than storing the data in the index itself.
Key Features:
- A table can have multiple non-clustered indexes.
- Non-clustered indexes are generally used to speed up lookups for columns that are not sorted physically.
- It creates a logical ordering of data that points to the physical data blocks.
- They take up additional space because they are separate from the data table.

3. When to Use Each Type

Choose a clustered index when:

You frequently run range queries
The column has high cardinality (many unique values)
You need fast data retrieval for a specific column

Opt for non-clustered indexes when:

You need multiple indexes on a table
You frequently search for specific values
You want to index columns used in JOIN, WHERE, and ORDER BY clauses

Indexing in MongoDB

MongoDB, being a NoSQL database, does not use the exact same terminology as SQL databases when it comes to indexes, but the underlying concepts of how data is indexed are similar. MongoDB uses B-tree indexes to store and retrieve data efficiently, and indexes can drastically improve query performance.

Clustered Indexes in MongoDB

MongoDB traditionally didn’t have a direct equivalent to SQL’s clustered indexes. However, as of MongoDB 5.3, MongoDB introduced clustered collections, which provide functionality similar to a clustered index in relational databases.

What are Clustered Collections in MongoDB?

A clustered collection in MongoDB organizes documents based on a specified field and stores data in the same order as the index on that field. The clustered index is embedded directly into the collection, which eliminates the need for a separate B-tree structure that MongoDB typically uses for secondary indexes.

Key Features:
- Documents are stored in the same order as the specified clustered index field.
- It is primarily useful for time-series data or use cases where data is naturally ordered by a specific field (e.g., timestamps, log data, etc.).
- MongoDB ensures that the clustered index field must be unique, so there can be only one clustered index per collection, much like SQL’s clustered index limitation.

How Clustered Collections Work in MongoDB

When you create a clustered collection, MongoDB physically organizes the documents based on the clustered index field, making range queries or searches on that field more efficient.
Unlike traditional secondary indexes, MongoDB does not need to maintain a separate index structure for clustered collections, as the documents are inherently organized according to the clustered index field.

Benchmarks

package main

import (
    "context"
    "fmt"
    "log"
    "testing"
    "time"

    "go.mongodb.org/mongo-driver/bson"
    "go.mongodb.org/mongo-driver/bson/primitive"
    "go.mongodb.org/mongo-driver/mongo"
    "go.mongodb.org/mongo-driver/mongo/options"
)

var (
    uri                  = "mongodb://localhost:27017"
    dbName               = "performance_test"
    clusteredCollName    = "clustered_logs"
    nonClusteredCollName = "non_clustered_logs"
    client               *mongo.Client
    db                   *mongo.Database
    numDocuments         = 1_000_000
)

func setup() {
    if client == nil {
        var err error
        client, err = mongo.Connect(context.TODO(), options.Client().ApplyURI(uri))
        if err != nil {
            log.Fatal(err)
        }
        db = client.Database(dbName)
    }
}

func BenchmarkQueryClusteredCollection(b *testing.B) {
    setup()

    // Drop collection before the benchmark to ensure fresh test data
    db.Collection(clusteredCollName).Drop(context.TODO())

    // // Create Clustered Collection
    opts := options.CreateCollection().
        SetClusteredIndex(bson.D{{Key: "key", Value: bson.D{{Key: "_id", Value: 1}}}, {Key: "unique", Value: true}})

    err := db.CreateCollection(context.TODO(), clusteredCollName, opts)
    if err != nil {
        log.Fatal("Error creating clustered collection: ", err)
    }

    coll := db.Collection(clusteredCollName)
    startTime := time.Now()
    insertTestData(coll)
    log.Printf("Clustered Collection: Inserted %d documents in %s\n", numDocuments, time.Since(startTime))

    startID := primitive.NewObjectIDFromTimestamp(startTime)
    endID := primitive.NewObjectIDFromTimestamp(time.Now())

    b.ResetTimer() // Reset the timer for the actual query benchmarking

    for i := 0; i < b.N; i++ {
        filter := bson.D{
            {Key: "_id", Value: bson.D{
                {Key: "$gte", Value: startID},
                {Key: "$lt", Value: endID},
            }},
        }

        _, err := coll.Find(context.TODO(), filter)
        if err != nil {
            b.Fatal(err)
        }
    }
}

func BenchmarkQueryNonClusteredCollection(b *testing.B) {
    setup()

    // Drop collection before the benchmark to ensure fresh test data
    db.Collection(nonClusteredCollName).Drop(context.TODO())

    // Create Non-Clustered Collection and Index
    err := db.CreateCollection(context.TODO(), nonClusteredCollName)
    if err != nil {
        log.Fatal("Error creating non-clustered collection: ", err)
    }

    coll := db.Collection(nonClusteredCollName)
    startTime := time.Now()
    insertTestData(coll)
    log.Printf("Non-Clustered Collection: Inserted %d documents in %s\n", numDocuments, time.Since(startTime))

    startID := primitive.NewObjectIDFromTimestamp(startTime)
    endID := primitive.NewObjectIDFromTimestamp(time.Now())

    b.ResetTimer() // Reset the timer for the actual query benchmarking

    for i := 0; i < b.N; i++ {
        filter := bson.D{
            {Key: "_id", Value: bson.D{
                {Key: "$gte", Value: startID},
                {Key: "$lt", Value: endID},
            }},
        }

        _, err := coll.Find(context.TODO(), filter)
        if err != nil {
            b.Fatal(err)
        }
    }
}

func insertTestData(coll *mongo.Collection) {
    for n := 0; n < numDocuments/100; n++ {
        docs := make([]interface{}, 100)
        for i := 0; i < 100; i++ {
            docs[i] = bson.D{
                {Key: "log", Value: fmt.Sprintf("Log entry %d", n*100+i)},
            }
        }
        _, err := coll.InsertMany(context.TODO(), docs)
        if err != nil {
            log.Fatal(err)
        }
    }
}

go test -bench=.

BenchmarkQueryClusteredCollection-8
Clustered Collection: Inserted 1000000 documents in 2.032176167s
    9693            113248 ns/op
BenchmarkQueryNonClusteredCollection-8
Non-Clustered Collection: Inserted 1000000 documents in 2.738731583s
    8781            137176 ns/op
PASS

Interpretation:

Insertion Time: The clustered collection is faster at inserting documents. This is expected because, in a clustered collection, the documents are already stored in the order of the _id field, which optimizes the insertion process. In contrast, non-clustered collections require managing the separate index for _id, adding overhead.
Query Performance: The clustered collection is faster for queries, which is consistent with MongoDB's behavior. Since the data in a clustered collection is stored according to the _id field, range queries can be executed more efficiently. The non-clustered collection requires additional lookups on the _id index, which adds overhead and makes the query slightly slower.

Key Differences between SQL Clustered Indexes and MongoDB Clustered Collections

Physical Data Organization: In both SQL and MongoDB, clustered indexes determine the physical order of data. However, MongoDB only recently introduced clustered collections in version 5.3, whereas SQL has long had clustered indexes.
Default Behavior: In SQL, a primary key often defaults to a clustered index, whereas in MongoDB, the _id field is indexed by default, but not as a clustered index. You need to explicitly create a clustered collection if you want MongoDB to organize documents based on a specific field.
Use Cases: Clustered collections in MongoDB are especially beneficial for time-series data and logs where the data is naturally sequential. In SQL, clustered indexes can be applied to various types of data and use cases.

Conclusion

With the introduction of clustered collections, MongoDB now offers a way to physically organize documents in a collection based on a specified field, similar to a clustered index in relational databases. This feature is especially useful for use cases like time-series data, where the natural ordering of data can improve query efficiency.

For MongoDB users, understanding and applying clustered collections to appropriate workloads can significantly enhance query performance, particularly when dealing with sequential or range-based data.

For more details, you can refer to MongoDB’s official documentation on clustered collections.

Forem: Tuna

Why Memory Alignment Matters in Go: Making Your Structs Lean and Fast

🧠 What Is Memory Alignment, Exactly?

🛠️ How Go Handles Memory Alignment

Let’s look at this struct:

✅ Well-Aligned Layout = Happy Memory

🚀 Real-World Impact: Memory and Performance

🔬 Benchmarking Time!

📊 Typical Results:

🛠️ Tools: Check Alignment with go vet

✅ Best Practices for Struct Layout in Go

📝 Final Thoughts

📚 References

Go 1.24 and the Swiss Table Revolution: A New Era for Maps

Introduction

Part 1: The Old Map (A Quick Refresher)

Part 2: Swiss Table Deep Dive—How It Handles Collisions Like a Pro

Step 1: Hash Splitting (H1 vs. H2)

Step 2: Metadata—The Magic Cheat Sheet

Step 3: Group Hunting with H1

Step 4: Handling Collisions (No Linked Lists!)

Example Walkthrough:

Insert price

Lookup price

Why This Beats the Old Map

Final Takeaway

MongoDB Indexes: The Superpower to Speed Up Your Queries ⚡

Types of MongoDB Indexes

Single Field Index

Compound Indexes

Multikey Indexes

Index Properties

Case-Insensitive Indexes

Partial Indexes

Sparse Indexes

TTL Indexes

Unique Indexes

Wrap-Up 🎉

Understanding Iterators in Go: A Fun Dive!

A Simple Transformation: No Iterator Yet

Enter the Iterator!

Why Use an Iterator?

Conclusion: When to Use Iterators?

B-tree: The Secret Recipe Behind Efficient Data Handling 🍰

What is a B-tree? 🌳

How B-trees Work: An Easy Breakdown ⚙️

Searching a Key 🔍

Inserting a Key 🎯

Deleting a Key 🗑️

Deleting a Key from a Leaf Node:

Deleting a Key from an Internal (Middle) Node:

Why B-trees Dominate Databases

In Conclusion

Understanding Stack and Heap in Go: A Simple Guide

What is the Stack?

What is the Heap?

Stack vs. Heap: What’s the Difference?

Conclusion

Introduction to SOLID Principles: The Heroic Saga of Code

S: Single Responsibility Principle (SRP) - The Anti-Monolithic Spell

O: Open/Closed Principle (OCP) - The Door is Closed, but Feel Free to Paint it!

L: Liskov Substitution Principle (LSP) - The “Imposter Syndrome” Cure

I: Interface Segregation Principle (ISP) - The Curse of Unnecessary Contracts

D: Dependency Inversion Principle (DIP) - No More Royal Puppeteers

Conclusion: The Hero’s Journey to SOLID Code

Clustered index vs Non-clustered and how they look like in MongoDB

What is an Index?

Clustered vs Non-Clustered Index

1. Clustered Index

2. Non-Clustered Index

3. When to Use Each Type

Indexing in MongoDB

Clustered Indexes in MongoDB

What are Clustered Collections in MongoDB?

How Clustered Collections Work in MongoDB

Benchmarks

Interpretation:

Key Differences between SQL Clustered Indexes and MongoDB Clustered Collections

Conclusion

🛠️ Tools: Check Alignment with `go vet`

Insert `price`

Lookup `price`