Forem: Satyajit Roy

[Golang] Garbage Collection in General

Satyajit Roy — Tue, 13 Jan 2026 04:42:37 +0000

As we all know the golang is a garbage collected language like other languages like java, python , C# etc. Golang is a statically typed garbage collected language.

What is Garbage Collection and Why it is needed

So many articles has written about this subject. So I am going to keep that small and try to get some deeper insight about the concept.

In SML programs (and in most other programming languages), it is possible to create garbage : allocated space that is no longer usable by the program.

When software is executed on your computer, there are two important memory location parts that it uses: Stack and Heap.

Above code has a RandomBox type and GenerateRandomBox() is a function which returns RandomBox type. The ref has been assigned in stack and struct data is in heap. Golang uses Escape Analysis to to determine that.

When the reference goes away, we end up with Garbage. This is how we get Garbage, which requires to be clean up time to time.

When the program memory footprint reaches a certain threshold, the whole application will be suspended, the Garbage Collector scans all the objects assigned memory space and recycling are no longer used, after the end of this process, the user program can continue, the language also use this strategy implement garbage collection in the early days, but today’s implementation is much complicated

Garbage collection can be initiated manually or automatically depending on the programming language you are using. Every program compiler or interpreter uses a specific algorithm to perform Garbage collection. Garbage collection in a compiled language works the same way as in an interpreted language.

Golang use Tracing garbage collectors even though their code is usually compiled to machine code ahead-of-time. Go uses a concurrent mark and sweep garbage collector algorithm.

What happens in Garbage Collection Process

Go’s garbage collector is called concurrent because it can safely run in parallel with the main program. When compiler decides that this is the time to run garbage collection based on some condition (discussed below), this is what it follows

Mark Setup

which means compiler try to stops everything, literally called stop the world step. Nothing gets done in your application at this time.
First it does is to enable write barrier which mean nothing get written in memory when this barrier is on (Stop all go-routines). Compiler has to perform this to make sure that your application is not losing any data.

Once the STW is performed and the write barrier is on, collector moves on to next phase.

Marking

In this phase the following happens

Inspect stacks to find the root pointer in heap
Traverse the heap and see if they are still in use

Collector also uses go-routine like us and it takes 25% of your available go-routines and assign them to itself. Means based on our previous example, 1 thread will be dedicated to collector.

Now, if the collect finds out that it might go out of memory while performing this task, because some other go-routine is allocating more then it can mark. So it will choose that go-routine and ask it to help with marking. This process is called Mark Assist

Mark Termination

Here collector will again perform the STW, turn the write barrier off, perform some clean-up and calculate the next Garbage collection schedule

Note: The goal is to keep the STW down or within the 100ms on every collection it needs to perform

One the Mark Termination process is complete (STW and Write Barrier is Off), application start working again with all the OS Threads available.

This is what happens every time the collection happens, you may ask that yes, it did the marking by identifying the dangling values, however, it didn’t clean up them. That part is called Sweeping

Myth Buster: Sweeping is not part of Garbage Collection, it happens outside of collection process.

Sweeping

This is process where we claim the non-marked memory allocations. We need to get them back, thats the whole purpose of Garbage Collection.

Claiming the unused locations happens when a new allocation happens. So, technically latency for sweeping is not added to garbage collection.

Garbage Collection Triggers

The first metric the garbage collector will watch is the growth of the heap. By default, it will run when the heap doubles its size. (code)
The second metric the garbage collector is watching is the delay between two garbage collectors. If it has not been triggered for more than two minutes, one cycle will be forced. (code)
Application memory can also trigger garbage collection is the runtime.mallocgc function, which at runtime divides objects on the heap into micro objects, small objects, and large objects by size. The creation of each of these three objects can trigger a new garbage collection cycle.
Manually triggering it by calling gc().

GC Collector knobs

Unlike java it was decided that a golang developer shouldn’t have to tune their Garbage collector whenever they move to different hardware. So they only provided one tuning config called SetGCPercentage or GOGC environment variable. Default for GOGC is 100%

GC Pacer

There is pacer algorithm trying to figure out when to start new collection. As we know that the calculation happens when Mark Termination phase of Garbage collection. The Pacer algorithm tries to calculate this and if it find that it can get more advantage of starting the collection even before the condition applies, it will do that. So take away is that the collection can even start before then the calculated time if pacer think that it can get more benefit.

Hope this blog was able to provide little more in-depth inside about the Garbage Collection process and how it is implemented in Golang

Happy Learning!!

Git Selective Ignore: Because Sometimes You Need to Keep Secrets from Git (But Not From Yourself)

Satyajit Roy — Tue, 13 Jan 2026 04:05:45 +0000

Git Selective Ignore: Because Sometimes You Need to Keep Secrets from Git (But Not From Yourself)

Or: How I Learned to Stop Worrying and Love Committing Without Fear

Picture this: It's 2 AM, you're deep in the coding zone, your coffee has gone cold (again), and you've just figured out why your API integration wasn't working. You quickly hardcode that API key to test it out—just temporarily, of course. The fix works! You're ecstatic. You commit your changes with a triumphant message: "Fixed the thing that was broken!"

Fast forward to 9 AM the next day. Your security team is having what can only be described as a "spirited discussion" about API keys appearing in your Git history. Your coffee is cold again, but this time it's because your blood has turned to ice.

Sound familiar? Welcome to the club. We have t-shirts, but ironically, we can't put the design in our Git repo because it has our logo's secret color codes in it.

The Git Philosophy: Everything Is Sacred (Even Your Mistakes)

Git operates on a beautiful but sometimes inconvenient principle: everything matters. When Linus Torvalds created Git in 2005 (in just 10 days, because apparently some people are just built different), he designed it around the idea that every character in your codebase tells a story. Git treats files as atomic units—either the whole file is tracked, or it isn't.

This philosophy has served us well. Git's immutable history and complete file tracking have prevented countless disasters. But it also means that when you accidentally commit API_KEY = "sk_live_definitely_not_my_real_key_12345", Git faithfully preserves that mistake for all eternity, like a digital time capsule of your poor judgment.

The traditional .gitignore file is fantastic for what it does—ignoring entire files or directories. Need to keep your node_modules out of the repo? Perfect. Don't want your IDE settings cluttering up the project? Easy. But try telling .gitignore to ignore just line 42 of config.py while keeping the rest of the file, and it'll give you the digital equivalent of a blank stare.

Enter the Real World: Where Perfect Theory Meets Messy Reality

Here's the thing about development: we live in a world of temporary hacks that become permanent features, debug statements that somehow make it to production, and test configurations that work so well locally that we forget they contain production credentials.

Consider these common scenarios:

The Debug Block: Those console.log statements that helped you figure out why the async function was returning undefined, but now clutter your clean, professional codebase.
The Local Config: Database connection strings, API endpoints, and feature flags that need to be different for local development but shouldn't make their way into the shared repository.
The Temporary Experiment: That experimental algorithm you're testing, complete with hardcoded values and performance logging, sitting right in the middle of your otherwise pristine production code.
The Security Nightmare: API keys, tokens, passwords, and other credentials that somehow always seem to sneak into codebases despite our best intentions.

Traditional Solutions: A Comedy of Errors

Let's look at how we typically handle these situations:

Option 1: The Paranoid Approach

You meticulously remove every sensitive line before committing, then carefully add them back after. This works great until you forget to add something back, spend three hours debugging why your app won't start, and realize you removed the line that sets up the database connection.

Option 2: The Environment Variable Dance

You move everything to environment variables. This is actually good practice, but now your local setup requires a .env file with 47 different variables, and new developers need a PhD in configuration management just to run the project locally.

Option 3: The Separate Config Files

You keep sensitive configs in separate files and gitignore those files entirely. This works until you need to share the structure of the config with your team, or until someone accidentally deletes the local config file and has no template to recreate it.

Option 4: The Git Surgery

You realize you've committed something sensitive and spend the next hour learning about git filter-branch and BFG Repo-Cleaner. By the time you're done, you've rewritten half your Git history, broken everyone else's local repos, and you're pretty sure you've violated several laws of physics.

Git Selective Ignore: A Surgical Solution

This is where git-selective-ignore comes in. Think of it as a precision instrument in a world of sledgehammers.

Instead of treating files as atomic units, this tool lets you specify exactly which parts of a file should be ignored during commits. It’s like having a conversation with Git:

<span>You</span>
<span>"Hey Git, commit this file, but ignore lines 13-16, and also any line that contains 'API_KEY', and oh, while you're at it, skip that debug block between the comments."</span>


<span>Git</span>
<span>"That's not how I—"</span>


<span>Tool</span>
<span>"I got this. <em>Git</em>, just commit what they want you to commit. Trust me."</span>


<span>Git</span>
<span>"...okay, but this feels weird."</span>

How It Actually Works (The Magic Behind the Curtain)

The tool operates using Git hooks—specifically pre-commit and post-commit hooks. Here's the elegant dance it performs:

Pre-commit: Before Git commits your changes, the tool scans your staged files, creates temporary clean versions with the specified content removed, and stages these sanitized versions.
Commit: Git commits the clean versions to history.
Post-commit: The tool restores your original files, so your working directory contains all your local configs, debug statements, and temporary code.

It's like having a butler who tidies up your room before guests arrive, then puts everything back exactly where you left it after they leave.

Real-World Example: The API Key Tango

Let's say you have this in your main.rs and lib.rs:

fn main() {
    println!("Starting application...");

    // DEBUG BLOCK START
    println!("Debug: Application started in debug mode");
    // DEBUG BLOCK END

    let API_KEY = "sk_live_1234567890abcdef";
    println!("Using API key: {}", API_KEY);

    /* Temporary lines for testing - remove before prod */
    let temp_feature = "experimental_feature_xyz";
    println!("Testing temporary feature: {}", temp_feature);
    /* End temporary section */

    let SECRET = "Some secret value";
    println!("SECRET configured");

    println!("Application completed successfully");
}

use std::env;

fn main() {
    println!("Another Test");

    let GITHUB_TOKEN = "github_fake_token_093790841-831-8lncdlwnelkqix12=-1x;xm;m"

    println!("{} <- My GitHub Token", GITHUB_TOKEN);

    let API_KEY = env::var('API_KEY');

    match env::var('API_KEY') {
        Ok(value) => {
            println!("The value of APP_KEY is: {}", API_KEY);
        }
        Err(e) => {
            eprintln!("Error getting environment variable {}: {}", 'API_KEY', e);
        }
    }
}

With traditional Git, you have two choices:

Commit everything (including secrets) 😬😬😬
Manually clean it up before every commit 🫩🫩🫩

With git-selective-ignore, you set up your patterns once 🕺🏽 💃🏻:

# Ignore any lines containing these sensitive terms
>> git-selective-ignore add all API_KEY --pattern-type line-regex
>> git-selective-ignore add all SECRET --pattern-type line-regex

# Ignore debug blocks
>> git-selective-ignore add all "// DEBUG BLOCK START ||| // DEBUG BLOCK END" --pattern-type block-start-end

# Ignore specific line ranges for temporary code
>> git-selective-ignore add src/main.rs 13-16 --pattern-type line-range

Now when you commit, Git's history will only contain:

fn main() {
    println!("Starting application...");

    println!("Using API key: {}", API_KEY);

    println!("SECRET configured");

    println!("Application completed successfully");
}

But your local file remains unchanged, so you can keep working with all your debug statements and test configurations intact.

Why This Approach Works Better

Compared to Manual Cleanup

No more forgetting: You don't have to remember what to remove and add back
No broken local environments: Your working directory always contains what you need
Consistent team experience: Everyone gets the same sanitized commits

Compared to Environment Variables

Faster iteration: No need to restart processes when changing test values
Better debugging: You can see actual values in your code while developing
Simpler onboarding: New developers can run the project with sensible defaults

Compared to Separate Config Files

No structural divergence: The team sees the shape of your configuration
No lost configs: Your local setup is preserved in your working directory
Version control benefits: You can still track changes to the configuration structure

Advanced Patterns: Getting Surgical

The tool supports several pattern types for different use cases:

Line Regex (for scattered sensitive data)

Perfect for API keys, passwords, or debug statements scattered throughout your codebase:

>> git-selective-ignore add all "console\.log.*debug" --pattern-type line-regex

Line Ranges (for temporary code blocks)

When you know exactly which lines contain temporary code:

>> git-selective-ignore add src/config.py 45-52 --pattern-type line-range

Block Start/End (for structured temporary sections)

When you use comments to mark temporary code:

>> git-selective-ignore add all "// TODO: REMOVE ||| // END TODO" --pattern-type block-start-end

The Philosophy Shift: From Binary to Granular

git-selective-ignore represents a philosophical shift in how we think about version control. Instead of the binary choice of "track this file or don't," we get granular control over what parts of our development process should be preserved in history.

This isn't about hiding poor practices—it's about recognizing that the code we need to develop effectively isn't always the same as the code we want to preserve historically. It's the difference between your private workshop (messy, full of tools and work-in-progress) and the finished product you show to the world.

Getting Started: Your First Steps into Selective Ignoring

Installation is straightforward. You can use their setup-devbox tool, download from releases, or build from source:

# Build from source
>> git clone https://github.com/kodelint/git-selective-ignore.git
>> cd git-selective-ignore
>> cargo install --path .

Then in any repository:

# Initialize selective ignore
git-selective-ignore init

# Install the Git hooks
git-selective-ignore install-hooks

# Add your first pattern
git-selective-ignore add all "API_KEY" --pattern-type line-regex

# Check what would be ignored
git-selective-ignore status

The tool stores its configuration in .git/selective-ignore.toml, so each repository can have its own rules without affecting others.

You can also see the violations in the code base with status

>> git-selective-ignore status
✓ Configuration is valid.
   ├─ Line Range Pattern '13-16': 4 line(s) matched
   │  └─ Lines 13-16
   ├─ Regex Pattern 'API_KEY': 1 line(s) matched
   │  └─ Line 10
   ├─ Regex Pattern 'GITHUB_TOKEN': 1 line(s) matched
   │  └─ Line 20
   ├─ Block Pattern '// DEBUG BLOCK START ||| // DEBUG BLOCK END': 3 line(s) matched
   │  └─ Lines 6-8
   ├─ Regex Pattern 'SECRET': 1 line(s) matched
   │  └─ Line 18
   └─ Summary: 10 line(s) ignored, 17 line(s) remaining (of 27 total)
   ├─ Regex Pattern 'GITHUB_TOKEN': 1 line(s) matched
   │  └─ Line 7
   └─ Summary: 1 line(s) ignored, 18 line(s) remaining (of 19 total)
📊 Git Selective Ignore Status Report
=====================================
🎯 Specifically Configured Files:
🟡 src/main.rs (8 patterns, 10/27 lines ignored, 37.0%)

🌐 Files Affected by Global 'ALL' Patterns:
🟡 src/lib.rs (6 patterns, 1/19 lines ignored, 5.3%)

📈 Summary:
  Total files: 2
  Total patterns: 8
  Total ignored lines: 11
  Files with issues: 2

📋 Breakdown:
  Specifically configured files: 1
  Files affected by 'ALL' patterns only: 1

Use list to see what Ignore Patterns are installed

>> git-selective-ignore list
✓ Configuration is valid.

📁 File: all
  🔍 ID: 78ed02f4-db7c-4921-b565-5e8986f19705 | Type: LineRegex | Pattern: API_KEY
  🔍 ID: 7fb165d1-bab6-4c79-a13b-51f2f29a88e9 | Type: LineRegex | Pattern: APP_KEY
  🔍 ID: 02b17597-bb85-428c-be56-3d0cd4a3c44b | Type: LineRegex | Pattern: GITHUB_TOKEN
  🔍 ID: 76447f06-dd03-4c3b-b27a-b611579e9cb8 | Type: BlockStartEnd | Pattern: // DEBUG BLOCK START ||| // DEBUG BLOCK END
  🔍 ID: 48f984d1-dd90-4984-99d6-ae6c63c591d6 | Type: LineRegex | Pattern: SECRET
  🔍 ID: b9a54bc2-048d-4fa0-b6ff-dc66aff6e706 | Type: LineRegex | Pattern: password

📁 File: src/main.rs
  🔍 ID: 31ca2ff0-90d8-47ea-90db-413cedf09bcf | Type: LineRange | Pattern: 13-16
  🔍 ID: a941d428-87ed-4378-898d-d5156723dfd0 | Type: BlockStartEnd | Pattern: /* TEMP_CODE_START */ ||| /* TEMP_CODE_END */

A Word of Caution: With Great Power...

Like any powerful tool, git-selective-ignore should be used thoughtfully. It's not a license to be sloppy with sensitive data, it's a safety net for the inevitable times when we are human.

Remember:

Security is still your responsibility: This tool helps prevent accidental commits, but you should still follow security best practices
Team coordination matters: Make sure your team knows which patterns are in place
Test your patterns: Use the status command to verify your patterns work as expected

The Future of Granular Version Control

git-selective-ignore points toward a future where version control systems understand that not everything in our working directory needs to be preserved for posterity. It's a tool that acknowledges the messy reality of software development while maintaining the clean history that makes Git so valuable.

In a world where we're constantly told to "shift left" on security and best practices, tools like this help us maintain good hygiene without sacrificing development velocity. We can iterate quickly, debug effectively, and experiment freely, all while ensuring our Git history remains clean and secure.

Conclusion: Sleep Better at Night

The next time you're coding at 2 AM and need to hardcode an API key for testing, you can do so without the nagging worry that you'll forget to clean it up. The next time you want to add verbose debug logging to figure out a tricky bug, you can do it without cluttering your team's commit history.

git-selective-ignore gives you the freedom to develop the way you need to while maintaining the professional, secure codebase your team depends on. It's not just a tool, it's peace of mind.

And your security team? They'll sleep better too, knowing that those 2 AM coding sessions are far less likely to result in morning meetings about exposed credentials.

Now if you'll excuse me, I need to go commit some code. Don't worry, I've got selective ignore set up for my embarrassing variable names.

Ready to try git-selective-ignore? Check it out on GitHub and start committing with confidence.

Git Selective Ignore: Because Sometimes You Need to Keep Secrets from Git (But Not From Yourself)

Satyajit Roy — Tue, 13 Jan 2026 03:53:19 +0000

Git Selective Ignore: Because Sometimes You Need to Keep Secrets from Git (But Not From Yourself)

Or: How I Learned to Stop Worrying and Love Committing Without Fear

Sound familiar? Welcome to the club. We have t-shirts, but ironically, we can't put the design in our Git repo because it has our logo's secret color codes in it.

The Git Philosophy: Everything Is Sacred (Even Your Mistakes)

Enter the Real World: Where Perfect Theory Meets Messy Reality

Consider these common scenarios:

The Debug Block: Those console.log statements that helped you figure out why the async function was returning undefined, but now clutter your clean, professional codebase.
The Local Config: Database connection strings, API endpoints, and feature flags that need to be different for local development but shouldn't make their way into the shared repository.
The Temporary Experiment: That experimental algorithm you're testing, complete with hardcoded values and performance logging, sitting right in the middle of your otherwise pristine production code.
The Security Nightmare: API keys, tokens, passwords, and other credentials that somehow always seem to sneak into codebases despite our best intentions.

Traditional Solutions: A Comedy of Errors

Let's look at how we typically handle these situations:

Option 1: The Paranoid Approach

Option 2: The Environment Variable Dance

Option 3: The Separate Config Files

Option 4: The Git Surgery

Git Selective Ignore: A Surgical Solution

This is where git-selective-ignore comes in. Think of it as a precision instrument in a world of sledgehammers.

Instead of treating files as atomic units, this tool lets you specify exactly which parts of a file should be ignored during commits. It's like having a conversation with Git:

You: "Hey Git, commit this file, but ignore lines 13-16, and also any line that contains 'API_KEY', and oh, while you're at it, skip that debug block between the comments."

Git: "That's not how I—"

git-selective-ignore: "I got this. Git, just commit what they want you to commit. Trust me."

Git: "...okay, but this feels weird."

How It Actually Works (The Magic Behind the Curtain)

The tool operates using Git hooks—specifically pre-commit and post-commit hooks. Here's the elegant dance it performs:

Pre-commit: Before Git commits your changes, the tool scans your staged files, creates temporary clean versions with the specified content removed, and stages these sanitized versions.
Commit: Git commits the clean versions to history.
Post-commit: The tool restores your original files, so your working directory contains all your local configs, debug statements, and temporary code.

It's like having a butler who tidies up your room before guests arrive, then puts everything back exactly where you left it after they leave.

Real-World Example: The API Key Tango

Let's say you have this in your main.rs and lib.rs:

fn main() {
    println!("Starting application...");

    // DEBUG BLOCK START
    println!("Debug: Application started in debug mode");
    // DEBUG BLOCK END

    let API_KEY = "sk_live_1234567890abcdef";
    println!("Using API key: {}", API_KEY);

    /* Temporary lines for testing - remove before prod */
    let temp_feature = "experimental_feature_xyz";
    println!("Testing temporary feature: {}", temp_feature);
    /* End temporary section */

    let SECRET = "Some secret value";
    println!("SECRET configured");

    println!("Application completed successfully");
}

use std::env;

fn main() {
    println!("Another Test");

    let GITHUB_TOKEN = "github_fake_token_093790841-831-8lncdlwnelkqix12=-1x;xm;m"

    println!("{} <- My GitHub Token", GITHUB_TOKEN);

    let API_KEY = env::var('API_KEY');

    match env::var('API_KEY') {
        Ok(value) => {
            println!("The value of APP_KEY is: {}", API_KEY);
        }
        Err(e) => {
            eprintln!("Error getting environment variable {}: {}", 'API_KEY', e);
        }
    }
}

With traditional Git, you have two choices:

Commit everything (including secrets) 😬😬😬
Manually clean it up before every commit 🫩🫩🫩

With git-selective-ignore, you set up your patterns once 🕺🏽 💃🏻:

# Ignore any lines containing these sensitive terms
>> git-selective-ignore add all API_KEY --pattern-type line-regex
>> git-selective-ignore add all SECRET --pattern-type line-regex

# Ignore debug blocks
>> git-selective-ignore add all "// DEBUG BLOCK START ||| // DEBUG BLOCK END" --pattern-type block-start-end

# Ignore specific line ranges for temporary code
>> git-selective-ignore add src/main.rs 13-16 --pattern-type line-range

Now when you commit, Git's history will only contain:

fn main() {
    println!("Starting application...");

    println!("Using API key: {}", API_KEY);

    println!("SECRET configured");

    println!("Application completed successfully");
}

But your local file remains unchanged, so you can keep working with all your debug statements and test configurations intact.

Why This Approach Works Better

Compared to Manual Cleanup

No more forgetting: You don't have to remember what to remove and add back
No broken local environments: Your working directory always contains what you need
Consistent team experience: Everyone gets the same sanitized commits

Compared to Environment Variables

Faster iteration: No need to restart processes when changing test values
Better debugging: You can see actual values in your code while developing
Simpler onboarding: New developers can run the project with sensible defaults

Compared to Separate Config Files

No structural divergence: The team sees the shape of your configuration
No lost configs: Your local setup is preserved in your working directory
Version control benefits: You can still track changes to the configuration structure

Advanced Patterns: Getting Surgical

The tool supports several pattern types for different use cases:

Line Regex (for scattered sensitive data)

Perfect for API keys, passwords, or debug statements scattered throughout your codebase:

>> git-selective-ignore add all "console\.log.*debug" --pattern-type line-regex

Line Ranges (for temporary code blocks)

When you know exactly which lines contain temporary code:

>> git-selective-ignore add src/config.py 45-52 --pattern-type line-range

Block Start/End (for structured temporary sections)

When you use comments to mark temporary code:

>> git-selective-ignore add all "// TODO: REMOVE ||| // END TODO" --pattern-type block-start-end

The Philosophy Shift: From Binary to Granular

Getting Started: Your First Steps into Selective Ignoring

Installation is straightforward. You can use their setup-devbox tool, download from releases, or build from source:

# Build from source
>> git clone https://github.com/kodelint/git-selective-ignore.git
>> cd git-selective-ignore
>> cargo install --path .

Then in any repository:

# Initialize selective ignore
git-selective-ignore init

# Install the Git hooks
git-selective-ignore install-hooks

# Add your first pattern
git-selective-ignore add all "API_KEY" --pattern-type line-regex

# Check what would be ignored
git-selective-ignore status

The tool stores its configuration in .git/selective-ignore.toml, so each repository can have its own rules without affecting others.

You can also see the violations in the code base with status

>> git-selective-ignore status
✓ Configuration is valid.
   ├─ Line Range Pattern '13-16': 4 line(s) matched
   │  └─ Lines 13-16
   ├─ Regex Pattern 'API_KEY': 1 line(s) matched
   │  └─ Line 10
   ├─ Regex Pattern 'GITHUB_TOKEN': 1 line(s) matched
   │  └─ Line 20
   ├─ Block Pattern '// DEBUG BLOCK START ||| // DEBUG BLOCK END': 3 line(s) matched
   │  └─ Lines 6-8
   ├─ Regex Pattern 'SECRET': 1 line(s) matched
   │  └─ Line 18
   └─ Summary: 10 line(s) ignored, 17 line(s) remaining (of 27 total)
   ├─ Regex Pattern 'GITHUB_TOKEN': 1 line(s) matched
   │  └─ Line 7
   └─ Summary: 1 line(s) ignored, 18 line(s) remaining (of 19 total)
📊 Git Selective Ignore Status Report
=====================================
🎯 Specifically Configured Files:
🟡 src/main.rs (8 patterns, 10/27 lines ignored, 37.0%)

🌐 Files Affected by Global 'ALL' Patterns:
🟡 src/lib.rs (6 patterns, 1/19 lines ignored, 5.3%)

📈 Summary:
  Total files: 2
  Total patterns: 8
  Total ignored lines: 11
  Files with issues: 2

📋 Breakdown:
  Specifically configured files: 1
  Files affected by 'ALL' patterns only: 1

Use list to see what Ignore Patterns are installed

>> git-selective-ignore list
✓ Configuration is valid.

📁 File: all
  🔍 ID: 78ed02f4-db7c-4921-b565-5e8986f19705 | Type: LineRegex | Pattern: API_KEY
  🔍 ID: 7fb165d1-bab6-4c79-a13b-51f2f29a88e9 | Type: LineRegex | Pattern: APP_KEY
  🔍 ID: 02b17597-bb85-428c-be56-3d0cd4a3c44b | Type: LineRegex | Pattern: GITHUB_TOKEN
  🔍 ID: 76447f06-dd03-4c3b-b27a-b611579e9cb8 | Type: BlockStartEnd | Pattern: // DEBUG BLOCK START ||| // DEBUG BLOCK END
  🔍 ID: 48f984d1-dd90-4984-99d6-ae6c63c591d6 | Type: LineRegex | Pattern: SECRET
  🔍 ID: b9a54bc2-048d-4fa0-b6ff-dc66aff6e706 | Type: LineRegex | Pattern: password

📁 File: src/main.rs
  🔍 ID: 31ca2ff0-90d8-47ea-90db-413cedf09bcf | Type: LineRange | Pattern: 13-16
  🔍 ID: a941d428-87ed-4378-898d-d5156723dfd0 | Type: BlockStartEnd | Pattern: /* TEMP_CODE_START */ ||| /* TEMP_CODE_END */

A Word of Caution: With Great Power...

Like any powerful tool, git-selective-ignore should be used thoughtfully. It's not a license to be sloppy with sensitive data, it's a safety net for the inevitable times when we are human.

Remember:

Security is still your responsibility: This tool helps prevent accidental commits, but you should still follow security best practices
Team coordination matters: Make sure your team knows which patterns are in place
Test your patterns: Use the status command to verify your patterns work as expected

The Future of Granular Version Control

Conclusion: Sleep Better at Night

git-selective-ignore gives you the freedom to develop the way you need to while maintaining the professional, secure codebase your team depends on. It's not just a tool, it's peace of mind.

And your security team? They'll sleep better too, knowing that those 2 AM coding sessions are far less likely to result in morning meetings about exposed credentials.

Now if you'll excuse me, I need to go commit some code. Don't worry, I've got selective ignore set up for my embarrassing variable names.

Ready to try git-selective-ignore? Check it out on GitHub and start committing with confidence.

Git Selective Ignore: Because Sometimes You Need to Keep Secrets from Git (But Not From Yourself)

Satyajit Roy — Tue, 13 Jan 2026 02:55:54 +0000

Git Selective Ignore: Because Sometimes You Need to Keep Secrets from Git (But Not From Yourself)

Or: How I Learned to Stop Worrying and Love Committing Without Fear

Picture this: It's 2 AM, you're deep in the coding zone, your coffee has gone cold (again), and you've just figured out why your API integration wasn't working. You quickly hardcode that API key to test it out—just temporarily, of course. The fix works! You're ecstatic. You commit your changes with a triumphant message: "Fixed the thing that was broken!"

Fast forward to 9 AM the next day. Your security team is having what can only be described as a "spirited discussion" about API keys appearing in your Git history. Your coffee is cold again, but this time it's because your blood has turned to ice.

Sound familiar? Welcome to the club. We have t-shirts, but ironically, we can't put the design in our Git repo because it has our logo's secret color codes in it.

The Git Philosophy: Everything Is Sacred (Even Your Mistakes)

Enter the Real World: Where Perfect Theory Meets Messy Reality

Consider these common scenarios:

The Debug Block: Those console.log statements that helped you figure out why the async function was returning undefined, but now clutter your clean, professional codebase.
The Local Config: Database connection strings, API endpoints, and feature flags that need to be different for local development but shouldn't make their way into the shared repository.
The Temporary Experiment: That experimental algorithm you're testing, complete with hardcoded values and performance logging, sitting right in the middle of your otherwise pristine production code.
The Security Nightmare: API keys, tokens, passwords, and other credentials that somehow always seem to sneak into codebases despite our best intentions.

Traditional Solutions: A Comedy of Errors

Let's look at how we typically handle these situations:

Option 1: The Paranoid Approach

Option 2: The Environment Variable Dance

Option 3: The Separate Config Files

Option 4: The Git Surgery

Git Selective Ignore: A Surgical Solution

This is where git-selective-ignore comes in. Think of it as a precision instrument in a world of sledgehammers.

Instead of treating files as atomic units, this tool lets you specify exactly which parts of a file should be ignored during commits. It's like having a conversation with Git:

You: "Hey Git, commit this file, but ignore lines 13-16, and also any line that contains 'API_KEY', and oh, while you're at it, skip that debug block between the comments."

Git: "That's not how I—"

git-selective-ignore: "I got this. Git, just commit what they want you to commit. Trust me."

Git: "...okay, but this feels weird."

How It Actually Works (The Magic Behind the Curtain)

The tool operates using Git hooks—specifically pre-commit and post-commit hooks. Here's the elegant dance it performs:

Pre-commit: Before Git commits your changes, the tool scans your staged files, creates temporary clean versions with the specified content removed, and stages these sanitized versions.
Commit: Git commits the clean versions to history.
Post-commit: The tool restores your original files, so your working directory contains all your local configs, debug statements, and temporary code.

It's like having a butler who tidies up your room before guests arrive, then puts everything back exactly where you left it after they leave.

Real-World Example: The API Key Tango

Let's say you have this in your main.rs and lib.rs:

fn main() {
    println!("Starting application...");

    // DEBUG BLOCK START
    println!("Debug: Application started in debug mode");
    // DEBUG BLOCK END

    let API_KEY = "sk_live_1234567890abcdef";
    println!("Using API key: {}", API_KEY);

    /* Temporary lines for testing - remove before prod */
    let temp_feature = "experimental_feature_xyz";
    println!("Testing temporary feature: {}", temp_feature);
    /* End temporary section */

    let SECRET = "Some secret value";
    println!("SECRET configured");

    println!("Application completed successfully");
}

use std::env;

fn main() {
    println!("Another Test");

    let GITHUB_TOKEN = "github_fake_token_093790841-831-8lncdlwnelkqix12=-1x;xm;m"

    println!("{} <- My GitHub Token", GITHUB_TOKEN);

    let API_KEY = env::var('API_KEY');

    match env::var('API_KEY') {
        Ok(value) => {
            println!("The value of APP_KEY is: {}", API_KEY);
        }
        Err(e) => {
            eprintln!("Error getting environment variable {}: {}", 'API_KEY', e);
        }
    }
}

With traditional Git, you have two choices:

Commit everything (including secrets) 😬😬😬
Manually clean it up before every commit 🫩🫩🫩

With git-selective-ignore, you set up your patterns once 🕺🏽 💃🏻:

# Ignore any lines containing these sensitive terms
>> git-selective-ignore add all API_KEY --pattern-type line-regex
>> git-selective-ignore add all SECRET --pattern-type line-regex

# Ignore debug blocks
>> git-selective-ignore add all "// DEBUG BLOCK START ||| // DEBUG BLOCK END" --pattern-type block-start-end

# Ignore specific line ranges for temporary code
>> git-selective-ignore add src/main.rs 13-16 --pattern-type line-range

Now when you commit, Git's history will only contain:

fn main() {
    println!("Starting application...");

    println!("Using API key: {}", API_KEY);

    println!("SECRET configured");

    println!("Application completed successfully");
}

But your local file remains unchanged, so you can keep working with all your debug statements and test configurations intact.

Why This Approach Works Better

Compared to Manual Cleanup

No more forgetting: You don't have to remember what to remove and add back
No broken local environments: Your working directory always contains what you need
Consistent team experience: Everyone gets the same sanitized commits

Compared to Environment Variables

Faster iteration: No need to restart processes when changing test values
Better debugging: You can see actual values in your code while developing
Simpler onboarding: New developers can run the project with sensible defaults

Compared to Separate Config Files

No structural divergence: The team sees the shape of your configuration
No lost configs: Your local setup is preserved in your working directory
Version control benefits: You can still track changes to the configuration structure

Advanced Patterns: Getting Surgical

The tool supports several pattern types for different use cases:

Line Regex (for scattered sensitive data)

Perfect for API keys, passwords, or debug statements scattered throughout your codebase:

>> git-selective-ignore add all "console\.log.*debug" --pattern-type line-regex

Line Ranges (for temporary code blocks)

When you know exactly which lines contain temporary code:

>> git-selective-ignore add src/config.py 45-52 --pattern-type line-range

Block Start/End (for structured temporary sections)

When you use comments to mark temporary code:

>> git-selective-ignore add all "// TODO: REMOVE ||| // END TODO" --pattern-type block-start-end

The Philosophy Shift: From Binary to Granular

Getting Started: Your First Steps into Selective Ignoring

Installation is straightforward. You can use their setup-devbox tool, download from releases, or build from source:

# Build from source
>> git clone https://github.com/kodelint/git-selective-ignore.git
>> cd git-selective-ignore
>> cargo install --path .

Then in any repository:

# Initialize selective ignore
git-selective-ignore init

# Install the Git hooks
git-selective-ignore install-hooks

# Add your first pattern
git-selective-ignore add all "API_KEY" --pattern-type line-regex

# Check what would be ignored
git-selective-ignore status

The tool stores its configuration in .git/selective-ignore.toml, so each repository can have its own rules without affecting others.

You can also see the violations in the code base with status

>> git-selective-ignore status
✓ Configuration is valid.
   ├─ Line Range Pattern '13-16': 4 line(s) matched
   │  └─ Lines 13-16
   ├─ Regex Pattern 'API_KEY': 1 line(s) matched
   │  └─ Line 10
   ├─ Regex Pattern 'GITHUB_TOKEN': 1 line(s) matched
   │  └─ Line 20
   ├─ Block Pattern '// DEBUG BLOCK START ||| // DEBUG BLOCK END': 3 line(s) matched
   │  └─ Lines 6-8
   ├─ Regex Pattern 'SECRET': 1 line(s) matched
   │  └─ Line 18
   └─ Summary: 10 line(s) ignored, 17 line(s) remaining (of 27 total)
   ├─ Regex Pattern 'GITHUB_TOKEN': 1 line(s) matched
   │  └─ Line 7
   └─ Summary: 1 line(s) ignored, 18 line(s) remaining (of 19 total)
📊 Git Selective Ignore Status Report
=====================================
🎯 Specifically Configured Files:
🟡 src/main.rs (8 patterns, 10/27 lines ignored, 37.0%)

🌐 Files Affected by Global 'ALL' Patterns:
🟡 src/lib.rs (6 patterns, 1/19 lines ignored, 5.3%)

📈 Summary:
  Total files: 2
  Total patterns: 8
  Total ignored lines: 11
  Files with issues: 2

📋 Breakdown:
  Specifically configured files: 1
  Files affected by 'ALL' patterns only: 1

Use list to see what Ignore Patterns are installed

>> git-selective-ignore list
✓ Configuration is valid.

📁 File: all
  🔍 ID: 78ed02f4-db7c-4921-b565-5e8986f19705 | Type: LineRegex | Pattern: API_KEY
  🔍 ID: 7fb165d1-bab6-4c79-a13b-51f2f29a88e9 | Type: LineRegex | Pattern: APP_KEY
  🔍 ID: 02b17597-bb85-428c-be56-3d0cd4a3c44b | Type: LineRegex | Pattern: GITHUB_TOKEN
  🔍 ID: 76447f06-dd03-4c3b-b27a-b611579e9cb8 | Type: BlockStartEnd | Pattern: // DEBUG BLOCK START ||| // DEBUG BLOCK END
  🔍 ID: 48f984d1-dd90-4984-99d6-ae6c63c591d6 | Type: LineRegex | Pattern: SECRET
  🔍 ID: b9a54bc2-048d-4fa0-b6ff-dc66aff6e706 | Type: LineRegex | Pattern: password

📁 File: src/main.rs
  🔍 ID: 31ca2ff0-90d8-47ea-90db-413cedf09bcf | Type: LineRange | Pattern: 13-16
  🔍 ID: a941d428-87ed-4378-898d-d5156723dfd0 | Type: BlockStartEnd | Pattern: /* TEMP_CODE_START */ ||| /* TEMP_CODE_END */

A Word of Caution: With Great Power...

Like any powerful tool, git-selective-ignore should be used thoughtfully. It's not a license to be sloppy with sensitive data, it's a safety net for the inevitable times when we are human.

Remember:

Security is still your responsibility: This tool helps prevent accidental commits, but you should still follow security best practices
Team coordination matters: Make sure your team knows which patterns are in place
Test your patterns: Use the status command to verify your patterns work as expected

The Future of Granular Version Control

Conclusion: Sleep Better at Night

git-selective-ignore gives you the freedom to develop the way you need to while maintaining the professional, secure codebase your team depends on. It's not just a tool, it's peace of mind.

And your security team? They'll sleep better too, knowing that those 2 AM coding sessions are far less likely to result in morning meetings about exposed credentials.

Now if you'll excuse me, I need to go commit some code. Don't worry, I've got selective ignore set up for my embarrassing variable names.

Ready to try git-selective-ignore? Check it out on GitHub and start committing with confidence.

Rust - Struct, Generics

Satyajit Roy — Tue, 13 Jan 2026 02:51:41 +0000

Let’s talk about the some custom data types in Rust like struct and emuns . We all know why need custom data type, like any other language regular data type may not suffice the need and hence we have custom data types.

Structs, How to use them in `Rust` ?

Structure, struct in short, are very similar to tuple . Tuple are used to store related items with mixed data type in Order. Now if the use case is to have large number of elements and the Order is not obvious then it could be difficult to identify them out of Tuples

So you can see that if number of elements are larger than normal then it becomes very difficult to keep track of the order and that’s when we use struct.

fn main() {
    let mut a_tuple = (1, "Rust");
    println!("the first element of tuple [a_tuple] is {}", a_tuple.0);
    println!("the last element of tuple [a_tuple] is {}", a_tuple.1);

    a_tuple.0 = 6;
    a_tuple.1 = "Rustic";

    println!("the first element of tuple [a_tuple] after modification is {}", a_tuple.0);
    println!("the last element of tuple [a_tuple] after modification is {}", a_tuple.1);
}

💡 Memory Insight

Struct data are usually stored in stack given it contain stack only data types like numbers. To store Struct in heap you have specifically mentioned that. Also, if your Struct contain heap data types like String then it will be stored in heap and reference, associated data of the heap data will be stored in stack. So, when your Struct instance goes out of scope the associated data in heap will be automatically dropped.

the first element of tuple [a_tuple] is 1
the last element of tuple [a_tuple] is Rust
the first element of tuple [a_tuple] after modification is 6
the last element of tuple [a_tuple] after modification is Rustic

Defining a struct in Rust is similar to Golang. Golang require type keyword along side with struct keyword

Rust

struct Car {
    name: String,
    model: String,
    year: u32,
    price: u32,
}

Golang

type Car struct {
    Name  string
    Model string
    Year  uint32
    Price uint32
}

A struct is like a tuple, which allows you to package together related items with mixed data type, however you don’t need to use the index to access the elements, instead you use the field name to access them.

Struct also allow us to update an instance from another instance and the syntax is pretty simple called update syntax, which basically tells complier if there are missing field in instance should have the same field from previous instance.

let second_car = Car {
  name : String::*from*("Tesla"),
  model : String::from("Model 3"),
  ..new_car,
};

Any update to the first instance after the second instance initialization will not reflect in second instance.

If you see then the missing fields are int datatype means they live in stack, therefore they gets implicitly copied. However, if there were any String datatype involved, it would have error time as it violates the ownership rule of rust . To make that work we need to use explicit clone for copying the data from first instance, something like this ..new_car.clone(). To know more about ownership and borrowing. Also we have to add trait as the Car datatype doesn’t have the trait to Clone() data, so we need to derived that at the struct definition, something like this #[derive(Clone)].

Struct has methods

In Rust we can call subroutines, which are method for the struct. They are pretty much like a function and defined using fn key word. The difference is between methods and function is that method are always within the context of the struct and the first input parameter is alway the struct itself.

To define the method we need to impl key word (short for implementation) followed by the struct name

impl Car {
    // Method to update the price
    // &mut self is short for self: &mut Self
    fn update_price(&mut self, new_price: u32) {
        self.price = new_price;
        println!("Price of {} {} has increased to ${}/-", self.name, self.model, self.price);
    }
}

Struct has function too

Rust also allow us to create associated functions, they are pretty much like method, however they don’t take &self as argument. They are mostly used to create initialize the new instance of custom datatype, like a constructors in other object oriented languages.

impl Car {
    // Associated Function to create new instance
    // It is like a constructor in other languages
    fn new_car(name: String, model: String) -> Car {
        Car {
            name,
            model,
            year: 2022,
            price: 70000,
        }
    }
}

There is something call Struct Tuple, it is a combination of Struct and Tuple. In Rust Struct Tuple’s are defined similarly as Struct but they don’t have any named field. It is usually deployed to make a custom type with mixed primary datatypes, however don’t need field to be named. Something like struct CarFeatures(4, "electric", "falcon doors")

So this is how the whole code looks like

// Struct Tuple
struct CarFeatures(i32, String, String);

fn main() {
    let mut new_car = Car::new_car(String::from("Runner"), String::from("Tesla Model Y"));
    // Accessing fields using dot notation
    println!("Car Name: {}, Model: {}, Year: {}, Price: ${}/-", new_car.name, new_car.model, new_car.year, new_car.price);

    // Calling the method using dot notation
    new_car.update_price(75000);

    // Using update syntax
    let mut second_car = Car {
        name: String::from("Beast"),
        model: String::from("Tesla Plaid"),
        price: 110000,
        ..new_car.clone() // Need clone because Strings are heap allocated (move semantics)
    };

    println!("Car Name: {}, Model: {}, Year: {}, Price: ${}/-", second_car.name, second_car.model, second_car.year, second_car.price);
    second_car.update_price(135000);
}

Car Name: Runner, Model: Tesla Model Y, Year: 2022, Price: $70000/-
Price of Tesla Model Y has increased to $75000/-
Car Name: Beast, Model: Tesla Plaid, Year: 2022, Price: $110000/-
Price of Tesla Plaid has increased to $135000/-

Generic Types … Yay!!

Rust is statically type language so a defined struct, function or method can only be used for it own defined variable data types. Which mean you might end up maintaining same code body for struct, function or method with different data types. What if we can define struct, function or method in such a way that we can use any data type with it. Enters…..Generic Type!!

use std::cmp::PartialOrd;
use std::ops::AddAssign;

// Struct with Generic Types
struct Car<S, I, T> {
    name: S,
    model: S,
    year: I,
    price: T,
}

impl<S, I, T> Car<S, I, T>
where T: AddAssign + Copy + std::fmt::Display
{
    // Method with Generic Types
    fn update_price(&mut self, increment: T) {
        self.price += increment;
        println!("Price Updated: {}", self.price);
    }
}

// Function with Generic Types
fn choose<T: PartialOrd>(electric_per_unit: T, gas_per_gallon: T) -> &'static str {
    if electric_per_unit < gas_per_gallon {
        "Electric"
    } else {
        "Gas"
    }
}

Above we have struct Car with generic type (denoted using <..>) <S,I,T> which feeds the type for struct fields. Similarly, a impl block to define new_car and update_price methods. They are also using generic types. Lastly, a function choose with generics, used to get confirm for the right choice based on electric_per_unit and gas_per_gallon price constraints.

Don’t worry about the traits like std::cmp::PartialOrd and std::ops::AddAssign for now. I will have separate blog explaining them. For now, we need them because rust complier doesn’t know what kind of data types generics will have to perform comparison and addition.

Box Datatype

One more thing I want touch is the Box datatype in rust . Box datatype are usually used to put the data in heap instead of stack. In simple words stack are usually small in size and when you are storing data which can be large in size like a trait or struct (combination of different datatypes and sizes), you might want to store them in heap and have reference of that stored in stack

Note: If you are making the data to be boxed, means moving it from stack to heap using box datatype. It performs the move operation, not copy, so previous location in stack gets de-allocated.

fn main() {
    let car = Car {
        name: String::from("Model X"),
        price: 120000,
        // ... other fields
    };

    println!("I bought Model X in 2021 for $120000, it is an electric Vehicle");
    println!("Market prediction is that Model X in 2022 will be for $140000");

    println!("BEFORE BOXING, Car struct data size is {} bytes in Stack", std::mem::size_of_val(&car));

    // ... logic for price change ...

    println!("BOXING the data.....");
    let boxed_car = Box::new(car);

    println!("AFTER BOXING, Car struct data size is {} bytes in Stack", std::mem::size_of_val(&boxed_car));
    println!("Car struct data size is {} bytes in Heap, because we are using de-referencing operator `*` to access the data", std::mem::size_of_val(&*boxed_car));
}

I bought Model X in 2021 for $120000, it is an electric Vehicle
Market prediction is that Model X in 2022 will be for $140000
BEFORE BOXING, Car struct data size is 56 bytes in Stack
Price Changed: in Inventory aka Stack, Now Price is $140000
BOXING the data.....
AFTER BOXING, Car struct data size is 8 bytes in Stack
Car struct data size is 56 bytes in Heap, because we are using de-referencing operator `*` to access the data
DATA AFTER BOXING: Model X Price: $140000 Fuel Type: electric

As you can see that when I used the boxed_car variable as box type data got moved from stack to heap. Now, stack only contains the reference of the data (hence 8 bytes) and actual data is in heap (hence 56 bytes)
and we are using & for the reference for the data in stack and de-referencing operator * to fetch the size of the data in heap

Hope this explains some of the internal and usage of structs and generics in Rust. I will have more write-ups coming for other important concepts of Rust

Happy Programming!!

The Matryoshka Dolls of Modern Networking: A Technical Evolution

Satyajit Roy — Tue, 06 Jan 2026 16:58:20 +0000

The Matryoshka Dolls of Modern Networking: A Technical Evolution

Introduction: The Internet is a Lie

If you think the internet is a flat, happy place where your computer talks directly to a server, I have bad news for you: The internet is a lie.

Modern networking is fundamentally a history of deception. We want the logical freedom to move things around without the physical reality of moving cables. To achieve this, we turned to the "Matryoshka Doll" strategy (Russian nesting dolls). We take a packet, wrap it inside another packet, and sometimes when we are feeling particularly chaotic wrap that inside a third packet.

It’s an illusion of infinite connectivity built on top of a physical reality that is rigid, hierarchical, and frankly, a pain in the ass to wire.

Today, we are going to look at how we went from manual spreadsheets to the hyper-programmable, kernel-bypassing insanity that runs the cloud today.

Part 1: A Brief History of "Why We Did This"

To understand why we complicate things with overlays, you have to understand the nightmare we were waking up from.

The Villain: The Number 4,096

In the old days (pre-2010), if you wanted to separate two departments (say, HR and Engineering), you used a VLAN (Virtual Local Area Network).The IEEE 802.1Q standard gave us a 12-bit identifier for VLANs. If you do the math (2^12), that gives you 4,096 possible segments.For a university campus? Fine. For a cloud provider trying to host 1,000,000 customers? Catastrophic.

# The mathematical impossibility of early cloud networking
max_vlans = 4094
cloud_tenants_needed = 1000000 
coverage = (max_vlans / cloud_tenants_needed) * 100
print(f"We can only support {coverage:.2f}% of our customers.") 
# Result: The CEO fires everyone.

The $500,000 Rack Problem

Before overlays, an IP address wasn't just a number; it was a physical location. IP 10.1.1.5 meant "Row 4, Rack 2, Switch Port 12."

If you wanted to move a Virtual Machine to a different server with a different subnet, you had to change its IP address. This broke the application. The alternative? Network engineers physically rewiring switches to extend Layer 2 domains across the data center. This resulted in "Spanning Tree" loops that would occasionally take down the entire banking sector for an afternoon.

We needed a way to let the logical network (the VM) float above the physical network (the cables). Enter the Overlay.

Part 2: The Modern Era (What We Actually Use)

We stopped trying to fix the physical network and decided to just hide it. Today, almost every packet you send in the cloud is encapsulated. Let's look at the two heavyweights running the world right now.

1. EVPN-VXLAN: The Enterprise Workhorse

Used by: 80% of Fortune 500 Data Centers, Banks, Telecoms.

VXLAN (Virtual Extensible LAN) solved the "4,096 problem" by giving us a 24-bit ID. That’s 16 million segments. Enough for everyone to have their own private party.

But VXLAN had a dumb control plane. It used "Flood and Learn" essentially shouting, "WHO HAS MAC ADDRESS AA:BB:CC?" to every switch in the building.

The Fix: We married VXLAN to BGP EVPN. Instead of shouting, switches now whisper to each other using BGP (the protocol that runs the internet).

Switch A: "Hey, just so you know, I am hosting the HR Web Server now."
Switch B: "Cool, I'll update my routing table. Thanks."

This eliminated the flooding and made the network stable enough for JP Morgan to trade billions of dollars on it.

2. Geneve + eBPF: The Cool Cloud Kids

Used by: Kubernetes (Cilium), AWS (Nitro), Service Meshes.

VXLAN is great, but its header is fixed. You can't add notes to it. Modern cloud apps need to say things like, "This packet is from the Frontend, it has a Security Clearance of Level 5, and it prefers to avoid the slow router."

The Fix: Enter Geneve (Generic Network Virtualization Encapsulation). It has a variable-length header, meaning we can shove metadata into the packet itself. Combined with eBPF (extended Berkeley Packet Filter), we can now program the Linux kernel to read these headers and make decisions before the packet even hits the heavy parts of the Operating System.

3. AWS Nitro (The Hardware Way)

Used by: AWS (obviously), and copied by everyone else (IPUs/DPUs).

While Spotify and Kubernetes developers were busy optimizing software with eBPF to save CPU cycles, Amazon took a different approach: They just built better hardware. The AWS Nitro System solves the "Matryoshka Tax" by cheating.

How It Works:

Amazon realized that if the main CPU has to spend 20% of its time wrapping packets in Geneve headers, the customer gets angry because they are paying for a 100% CPU and getting 80%. So, they physically ripped the virtualization logic off the motherboard and put it on a separate card (the Nitro Card).

The Main CPU: Does nothing but run your code
The Nitro Card: Handles the VPC networking, encryption, storage, and security groups.
The Impact: You get "Bare Metal" performance on a virtual machine. The encapsulation still happens—the Matryoshka doll is still there but it’s being assembled by a dedicated robot (ASIC) on the side, not by your main CPU.
Why AWS Loves Geneve: AWS switched from VXLAN to Geneve specifically for this hardware. The flexible "Type-Length-Value" (TLV) headers allow the Nitro card to inject metadata that VXLAN couldn't handle, such as:
Flow Cookies: To ensure your session doesn't break when a load balancer scales.
VPC IDs: Ensuring absolute isolation at the hardware level.

4. Disaggregated Core: The "IKEA" Router

Used by: AT&T (DriveNets), Major ISPs, Hyperscalers.

For decades, if you wanted to be an ISP like AT&T, you had to buy a "Router Chassis." This was essentially a refrigerator-sized metal box that cost more than a house, sold by a vendor who locked you into their proprietary ecosystem forever. If you needed more ports? You had to buy a new fridge.

The Revolution:

The "Lego" Approach (DDC) AT&T looked at these giant metal fridges and said, "No thanks." They moved to Distributed Disaggregated Chassis (DDC) architecture using DriveNets.

Instead of one giant proprietary machine, they built a "router" out of standard, cheap white-box switches connected by cables.

The Architecture (Deconstructed): They broke the router into three generic pieces, like a kit of parts:

The Brain (Control Plane): Standard x86 servers running the routing software in containers.
The Ports (NCPs): "Pizza Box" switches (using Broadcom Jericho chips) that plug into the outside world.
The Fabric (NCFs): "Pizza Box" switches (using Broadcom Ramon chips) that just glue the other boxes together.

Why It Wins:

It acts exactly like Kubernetes for Networking.

Need more bandwidth? Don't buy a new chassis. Just plug in another "Pizza Box" (NCP).
Need more throughput? Plug in another Fabric box (NCF).
Scale: This system scales to 192 Tbps. That is a "distributed cloud of white boxes" pretending to be a router.

Part 3: The Anatomy of a Modern Packet

(Or: Why your MTU settings are ruining your life)

This is where the "Matryoshka Doll" analogy gets real. If you capture a packet flying through a Kubernetes cluster or AWS VPC, you are looking at a Russian nesting doll of headers.

Let's dissect a Geneve packet byte-by-byte.

The "Doll" Breakdown

Layer 1: The Delivery Truck (The Underlay)

This gets the packet from Physical Server A to Physical Server B. The switches only look at this.

Outer Ethernet (14 Bytes): Source and Destination MAC of the physical servers.
Outer IP (20 Bytes): Source and Destination IP of the physical servers.
Outer UDP (8 Bytes):
- SRC Port: A calculated hash (this creates entropy so we can use multiple cables at once).
- DST Port: 6081 ( The port for Geneve).

Layer 2: The Magic Wrapper (The Overlay)

This is where the software-defined magic happens.

Geneve Base Header (8 Bytes):
- VNI (24-bit): The Tenant ID. This ensures Coke's data doesn't leak into Pepsi's network.
Geneve Options (Variable, ~12-20 Bytes):
- This is the killer feature. eBPF inserts Metadata here.
- Example: SecurityID=45912. The destination firewall doesn't check IPs anymore; it just checks this ID. It's O(1) complexity (aka: blazing fast).

The Math: Why Fragmentation Kills Performance

If your application sends a standard max-size packet (1500 bytes), look what happens when we wrap it:

Here’s the encapsulation overhead breakdown in a clean table format:

Layer	Size Added	Running Total
Your App Payload	1500 bytes	1500
Geneve Base	+8 bytes	1508
Geneve TLV Options	+12 bytes (average)	1520
Outer UDP	+8 bytes	1528
Outer IP	+20 bytes	1548
Outer Ethernet	+14 bytes	1562 bytes

The Problem: The standard physical internet cable has an MTU (Maximum Transmission Unit) of 1500 bytes.

The Result: Your 1562-byte packet hits the switch and gets dropped or fragmented (chopped into two). Fragmentation destroys CPU performance.

The Solution: You must enable Jumbo Frames (MTU 9000) on your physical switches. If you build a cloud without Jumbo Frames, you are trying to fit an elephant into a Mini Cooper.

Part 4: The Future - Where Are We Going?

We are moving away from "Smart Software on Dumb Hardware" to "Genius Hardware."

1. Identity over IP

For 40 years, we allowed traffic based on Location (Source IP). "Allow 10.0.0.5 to talk to 10.0.0.6." This is dumb because IPs change. The future is Metadata. The packet carries its own passport in the Geneve header. "Allow 'Frontend-Service' to talk to 'Backend-Database'." The network enforces this policy at line rate. We don't care what the IP is anymore.

2. The Rise of the DPU (SmartNICs)

The "Tax" of the Matryoshka doll is CPU usage. Wrapping and unwrapping packets takes cycles away from your app. We are now moving this logic to DPUs (Data Processing Units) like NVIDIA BlueField. The server's CPU doesn't even know encapsulation is happening. The Network Card (NIC) handles the encryption, the Geneve wrapping, and the firewalling. It is a "free lunch" for application performance.

3. Quantum-Safe Overlays

Quantum computers will eventually break our current encryption. The beauty of flexible overlays like Geneve is that we don't need to rip out cables to fix this. We just add a new "Option TLV" to the header containing Quantum-Resistant keys. We can upgrade the security of the entire internet just by changing the packet wrapper.

# Conceptualizing Hybrid Security in Geneve
class QuantumSafeOverlay:
    def encrypt_packet(self, packet):
        # We carry TWO locks on the packet:
        return GenevePacket(
            vni=packet.vni,
            options=[
                # 1. The lock we trust today (Fast, Standard)
                TraditionalEncryption(RSA_encrypt(packet)),

                # 2. The lock we need for tomorrow (Quantum-Resistant)
                QuantumEncryption(Kyber_encrypt(packet))
            ],
            inner_packet=packet
        )

Conclusion: The Evolution Validated

The "Matryoshka Dolls" of modern networking aren't just a neat trick; they are the only reason the internet hasn't collapsed under its own weight. We have spent 25 years moving from physical cables to virtual lies, and we have the receipts to prove it.

If you scrolled to the bottom just for the summary, here is a quarter century of network engineering pain condensed into a grid:

Era	Archetype	Representative	The Tech	The Result
1995–2010	Physical (Single Doll)	AOL	VLANs	"Rack Rewire" nightmares; engineers crying in server rooms.
2010–2015	Virtualization (Dual Doll)	eBay	VXLAN / OpenStack	16M segments; we stopped caring about cables.
2015–2018	Scale Crisis (Heavy Doll)	Spotify	Kubernetes / CNI	40% CPU overhead because we got greedy with encapsulation.
2018–2022	Optimization (Smart Doll)	Facebook	eBPF / XDP	10× performance; we taught the kernel to skip the boring parts.
2022–Present	Hardware Offload	AWS	Nitro / Geneve	"Bare metal" speeds; the hardware finally caught up to our software.

The Bottom Line

For decades, we had to choose between Flexibility (Overlays) and Performance (Bare Metal).

If you wanted speed, you dealt with rigid physical switches.
If you wanted agility, you paid a "CPU Tax" to wrap packets in software.

That era is over. With eBPF optimization and hardware offloads like AWS Nitro, we have successfully paid down the technical debt. We can now wrap packets in three layers of metadata, route them based on identity rather than IP, and encrypt them against quantum computers all without the application feeling a thing.

The network has successfully abstracted itself. It is no longer a set of pipes; it is a programmable substrate that is just as fluid as the code running on top of it.

See the Matrix Yourself

Don't take my word for it. If you have a Kubernetes cluster running Cilium, you can watch the dolls being assembled in real-time. Run this on any node:


bash
# 1. See the eBPF programs hijacking your traffic
bpftool prog show

# 2. Watch the overlay traffic fly by (Standard Port 4789 or Geneve 6081)
tcpdump -i any udp port 4789 or udp port 6081 -vv

[Concept] Functions 101- Golang

Satyajit Roy — Sat, 03 Jan 2026 17:43:59 +0000

Functions are the building blocks of procedural programming. They help in creating modular code where a big job can be segmented into small pieces of code and might have been written by different people separated by both time and space. In fact, function is one kind of first-class citizen types in Go. In other words, we can use functions as values. Although Go is a static language, Go functions are very flexible. The feeling of using Go functions is much like using many dynamic languages. A function is a mapping of zero or more input parameters to zero or more output parameters.

The advantages of using functions are:

Reducing duplication of code
Decomposing complex problems into simpler pieces
Improving clarity of the code
Reuse of code
Information hiding

Special Functions

Golang has 2 special functions main() and init() . They have some special responsibilities assigned to them

The main() and init() function

The main package is a special package which is used with the programs that are executable and this package contains main() function. The main() function is a special type of function and it is the entry point of the executable programs. It does not take any argument nor return anything. Go automatically call main() function, so there is no need to call main() function explicitly and every executable program must contain single main package and main() function. The init() function is declared implicitly and we cannot reference it from anywhere in the code, but we are allowed to create multiple init() functions in the same program. The init() function can be incredibly powerful and compared to some other languages, is a lot easier to use within your Go programs. These init() functions can be used within a package block and regardless of how many times that package is imported, the init() function will only be called once.

Function with returns (none, single value or multiple values)

We declare a function using the func keyword. A function has a name, a list of comma-separated input parameters along with their types, the result type(s), and a body.

func function_name(Parameter-list)(Return_type){
    // function body.....
}

Let say we have function repeatWord() which return nothing, so main() just executes the repeatWord() which take a string and int as arguments and prints. Nothing to return

Now we change the function to return more than one values and also return if there are any errors.

You see if the condition below matches, function return error using fmt.Errorf else it send a nil in place of error.

if len(s) <= 0 {
    return "", len(s), fmt.Errorf("Length of string can't less than euql zero")
  }

Overall repeatWord() returns multiple value, along side with error.

Function as Parameter

A Go function can be passed to other functions as a parameter. Such a function is called a higher-order function.

Here we have simple action functions called repeatWord() and revertWord(). Also we have function called applyFunction() which take a function and string as parameters.

Function as custom type

Go allows to create reusable functions signatures with the type keyword. They have the same number of arguments with each argument is the same type. They have the same number of return values and each return value is of the same type

With the type keyword, we create a function type which accepts one string parameter and returns a string.

Function as closure aka anonymous function

It is possible to create functions inside of functions. Go supports anonymous functions, which can form closures. Anonymous functions are useful when you want to define a function inline without having to name it. Closure is a nested function that helps us access the outer function’s variables even after the outer function is closed

Function Higher Order

Higher order functions are functions that operate on other functions, either by taking them as arguments or by returning them.

Above you can see that concatStringUsingFunctionOfHigerOrder() returns another function, which again return another function. It is kind of Spaghetti Code in my opinion. However above is just an example, I would avoid writing code like this.

Hope this provide little bit more clarity on function and way they can be used in Golang program. Above examples can be found here: go-functions-101

Happy Coding!!

[Concept] And I thought I knew DNS

Satyajit Roy — Sat, 03 Jan 2026 17:39:42 +0000

Yeah, I know. We all know, use DNS and know how they work. There are numerous of articles and blog explaining how DNS works. However as I dig more about it I found more information which were, frankly speaking, new to me. So I thought I would write about it.

I know that there are so many excellent blogs and articles about DNS. However, I encourage you to give this a try, hopefully you might find something new in it.

History

Why DNS was created ? Believe it or not before DNS the mapping were use to copy over network from computer to computer for the name based communication. So, each site maintained a HOSTS.TXT file that provided a mapping between host names and network addresses in a set of simple text records that could be easily read by a person or program. It wasn’t long before people realized that keeping multiple copies of the hosts file was inefficient and error-prone. So it was decided on RFC 623 and RFC 625, Stanford Research Institute Network Information Center (NIC) would serve as the official source of the master hosts file.

Paul Mockapetris and his team had the mission to create a friendlier for use network, where people wouldn’t need to remember the IP address of every computer. He suggested that host names should include:

Name — for example, IBM
Categories/Purpose — for example, .com — for commercial purposes

After a year, the categories (or generic Top-Level Domains — gTLDs) were created. They included familiar extensions such as .com, .edu, .net, .org, .int, .gov and .mil. Before the end of 1985, there were six new names with .com. The first one ever registered, Symbolics.com, still exists today.

Trivia Detail: What is DNS Interceptors ? DNS Interceptors is used for optimizations, censorship, captive portal etc. Basically DNS is you gateway to internet and by controlling that we can control queries and requests
Trivia Detail: About Privacy. there is a (RFC 7816) called QNAME minimisation (Query Name Minimisation) to Improve Privacy, because DNS Resolver sends the whole query to the authoritative name server, the same outcome can be achieved with minimum information

OK, So how does DNS Work ?

So, DNS in nutshell is Global Distributed Identifier DB. It was created to replace the static file like /etc/hosts file. So understand the distributed model of DNS DB, let’s try to trace it backwards

Let say, we want to resolve www.google.com and as we already know

So, lets see the how the hierarchy works when we try to visit https://google.com

Browser will try to see if the answer is in it’s cache, it will replay
If not, then either the browser or operation system behave like a Stub Resolver (gethostaddr(), gethostbyname(), gethostent() are all Stub Resolver)and asks the question

The DNS Stub Resolver is a component of the DNS that is accessed by application programs when using the DNS for e.g. resolving domain names to IP addresses. The stub resolver simply serves as an intermediary between the application requiring DNS resolution, and a recursive DNS resolver.

Stub Resolver will ask the question to Recursive Resolver (in most cases) and the Recursive Resolver will do that bulk of work to find you the answer

Sometime between DNS Stub Resolver and Recursive Resolver one might have DNS Forwarders, Proxy Server which basically forward the query to Recursive Resolver. This is mostly done keep security and hide the internal details. There are multiple types of Resolvers like Stub, Recursive (mentioned above) then you have Forwarders, Validating, Pay Wall etc

Most ISPs run their own Recursive Resolvers. Pretty much all Recursive Resolvers keep their cache from their previous queries.
If the answer for the query is not found in cache then Recursive Resolvers try to identify the Authoritative Name Server for the record.
Recursive Resolvers might have to talk to N number of Authoritative Name Server to find the answer or sometime talk to other Recursive Resolvers to maintain the hierarchy of cache.

Caching, as mentioned all Recursive Resolvers maintain their own. The lifetime of that cache is handled by something called TTL (Time to live). The TTL is responsible to maintain the hierarchy of cache too. Say, the Recursive Resolvers answered the Stub Resolver and said TTL is 300 Seconds. Though the answer was received on 250th Second for Stub Resolver and as per Recursive Resolvers it can cache the answer for 300 Seconds. Which mean it will end up keeping the cache for 550 Seconds Stale. So, to honor the TTL, any Recursive Resolvers answering should also remove the time spent in TTL. So that the entity lower in the hierarchy get to cache the answer for valid TTL.
Trivia Detail: TTL are only suggestions for cache lifetime, you may or may not use it. Also there is something call Max TTL set by cacheing resolvers so if you set any value higher then that, will be ignored

Recursive Resolvers usually keeps Root Hint File (yes, they don’t user resolv.conf) and it can function if one of the IP from Root Hint File work it can pull the latest. All it queries is that what is data around (root domain). This query is call Priming Query. In nutshell it means initialing the DNS or Recursive Resolver

ICANN, Verisign and the Root Server Operators play significant roles in the management and process of the root zone

Query to Root Domain returns the Name Servers, (Which is 13 in count)

You can see the Root Domain . has 13 Name Servers and what you see underneath the ADDITIONAL SECTION are GLUE Records

GLUE Records are created to eliminate circular reference, in simple terms circular reference occurs when the Authoritative Name Server exists inside the Domain itself. You can’t reach to the Name Server because the Domain is still unknown. Glue records can only be created at the domain registrar as the registrar controls the DNS settings for a given domain’s delegation. Every Name Server on the internet has its own glue record created by the domain’s owner.
Glue Records are nothing but (direct and hardcoded) A Records for the Name Servers

Now Recursive Resolvers which is responsible for most of the work, finds the Authoritative Name Server, gets the answer for the query and the remaining TTL, hands it over to Stub Resolver and that’s how we resolve a Domain Name to an IP Address

So I have to summarize this whole fiasco then it would something like, fun intended pseudo code

  var cache = map[string]string

    func stubResolver(domain string) string {
      var (
        ipAddress string
        cacheHit bool
      )
      ipAddress, cacheHit = cache[domain]
      if !cache {
        ipAddress = recursiveResolver(domain)
      }
      return ipAddress
    }

    func recursiveResolver(domain string) string {
      ...
      // Will recursively keep trying to identify the
      **// Authoritative Name Server**, by getting the **Name Servers
      // of each elements. It may very well go to
      // the root zone which is .

      return ipAddress
    }

However there are lot more to it and to understand it properly we need to understand What is DNS Query ?

So, What is DNS Query and What does it contains ?

A DNS Query is a tuple, called Query Tuple, consisting of Query Name, TTL, Class, Query type and RDATA

So requester is asking for QTUPLE which includes everything, all the information mentioned above. It can’t just ask for something in isolation, it’s either all or nothing. That’s what is going on the raw level on the DNS Query. The wire format (RFC1035) for this is combination of half ascii, half binary and pretty old compression scheme call Label Compression.

Trivia Detail: DNS usually prefers UDP, however it actually works on both UDP and TCP. First it send the request on UDP, if it gets a response then good or If a DNS Response is larger than 512 bytes, or if a DNS server is managing tasks like zone transfers (transferring DNS records from primary to secondary DNS server), it will try TCP.
Trivia Detail: What is in-balliwick and out-of-balliwick servers? In-balliwick servers are within the zone itself and work with the help of glue records. Out-of-balliwick servers are external servers which is not part of that TLD.

How does DNS Packet looks like ?

Below is the picture of bits used to create a DNS Packet. Before and After basically shows how additional bits has been added later in time of DNS Evolution.

Mostly it was due to handle the in RCODE section. There was only 4 bits assigned for RCODE.

Later it was realized that we would need little more space in the packet and that when EDNS0’s “OPT” Record was introduced (RFC-8691). So the extended pseudo resource record call OPT added to additional section. Of-course the client needs to indicate that it supports it. Most software in these days supports this.

It reuses the resource record byte format, however changes many fields
Total RCODE size becomes 4 + 8 = 12 bits
Support additional protocol flags
Adds support for additional DNS extensions
Used for Extended errors (RFC 8914)
Used for Client Subnet DNS Queries (RFC 7871)

Trivia Detail: What is truncation in DNS ? When Clients receives bigger payload size than it can handle via OPT/UDP. Truncation bit is set on the DNS Packet. Some other method like removing unimportant items from packet to make it fit, or completely drop everything and ask the Client to connect using TCP.
Trivia Detail: Response Rate limiting (RRL) is actually used as DDoS Defense mechanism based on the frequency of the query.

Reverse DNS Queries

Basically find the name using IP Address, usually it is PTR record.

[www.google.com](http://www.google.com).  100 IN A 142.250.72.196

So, the IP Address of www.google.com is 142.250.72.196 . We just reverse it like this 196.72.250.142 and add in-ipaddr.arpa and resolve it using PTR record

Trivia Detail: What is apex zone, Terminal and lame delegations? Apex is basically the root for your domain and Terminal is the last node of your zone. Lame delegation mean parent still holding to NS Record which Child doesn’t uses anymore. (DNS Terminology)

What is Happy Eyeballs ?

Morden Clients usually send 2 queries in parallel one for A Record and one for AAAA Record and its called Happy Eyeballs(RFC 8305). Browsers or Application should use whichever comes first. So there are 2 type non-existence in DNS, domain doesn’t exist and you get NXDOMAIN Response or type doesn’t exist then you get NOERROR Response. Find more detail here

Trivia Detail: .com is actually dependent on .net because all the .com NameServers are using .net NameSever
Trivia Detail: CNAME and DNAME, what are difference ? CNAMEs is aliases for the other elements in the tree (same or different zone doesn’t matter) and it points to all other records A, MX, TXT etc. However, DNAMEs are the aliases for zone themselves.
Trivia Detail: Can you have something like “foo*.some-domain.com” ? No More detail about Wildcard is (RFC 4592)

Hope this blog was able to provide some more interesting fact and details about DNS.

Happy Learning!!

Solve Dynamic backend problem with Golang

Satyajit Roy — Mon, 25 Aug 2025 18:11:54 +0000

As mentioned in a previous blog post, Terraform can store its state file (.tfstate) in a remote location. However, the backend block requires initialization with terraform init and does not support interpolation. This means you have to manually populate the backend configuration before running terraform init or use the -backend-config switch.

terraform {
   backend "s3" {
      bucket = "mybucket"
      key    = "path/to/my/key"
      region = "us-east-1"
   }
}

This manual process of changing the key for every new resource or combination of resources is manageable for small deployments but can become a major headache for larger ones.

To solve this, we can dynamically generate the backend block before running terraform init (explained here my previous blog) .

Goal

The goal is to create a tool that automatically generates the backend block and establishes a 1:1 relationship between the tfvars file and the tfstate file. The tool should adhere to the following rules:

It should accept environment variables for S3, DynamoDB, and Region.
It should take a tfvars file as an argument.
It should generate the backend configuration using either the environment variables or the absolute path of the tfvars file.
It should be able to run terraform init, terraform plan, terraform apply, and terraform destroy.
It should run terraform init, terraform plan, and terraform apply if the filename ends with *.tfvars.
It should run terraform plan -destroy and terraform destroy if the filename ends with *.destroy.

Explanation

Let's call our tool autotf. It will have two sub-commands: verify and deploy.

Flow Control and Decision Making

Actions are performed based on the file extension.

Building Resources

1. Verify Command

When you run the following command, autotf will automatically generate the backend configuration, initialize Terraform, and then run terraform plan or terraform plan -destroy based on the file extension.

export S3Bucket="some-s3-bucket"
export Region="us-east-1"
export DynamoDB="some-dynamodb-table"
./autotf verify stage/s3/autotf-testing01.tfvars

autotf should automatically generate the backend configuration and initialize terraform using it by running terraform init and then run terraform plan or terraform plan -destroy depending on the file name extension _.tfvars or _.destroy sequentially.

Output

INFO 2022-03-28 17:50:29 will run terraform init, plan on [autotf-testing01.tfvars]

+------------------+----------------+-------------------------+-----------------------------------+
| Resource Name    | Backend Bucket | TFVars Name             | Backend Key                       |
+------------------+----------------+-------------------------+-----------------------------------+
| autotf-testing01 | autotf-testing | autotf-testing01.tfvars | stage/s3/autotf-testing01.tfstate |
+------------------+----------------+-------------------------+-----------------------------------+

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/aws v4.8.0...
- Installed hashicorp/aws v4.8.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Acquiring state lock. This may take a few moments...

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_s3_bucket.bucket will be created
  + resource "aws_s3_bucket" "bucket" {
      + acceleration_status                  = (known after apply)
      + acl                                  = (known after apply)
      + arn                                  = (known after apply)
      + bucket                               = "autotf-testing01"
      + bucket_domain_name                   = (known after apply)
      + bucket_regional_domain_name          = (known after apply)
      + cors_rule                            = (known after apply)
      + force_destroy                        = false
      + grant                                = (known after apply)
      + hosted_zone_id                       = (known after apply)
      + id                                   = (known after apply)
      + lifecycle_rule                       = (known after apply)
      + logging                              = (known after apply)
      + object_lock_enabled                  = (known after apply)
      + policy                               = (known after apply)
      + region                               = (known after apply)
      + replication_configuration            = (known after apply)
      + request_payer                        = (known after apply)
      + server_side_encryption_configuration = (known after apply)
      + tags                                 = {
          + "Environment" = "Dev"
          + "Name"        = "autotf-testing01"
        }
      + tags_all                             = {
          + "Environment" = "Dev"
          + "Name"        = "autotf-testing01"
        }
      + versioning                           = (known after apply)
      + website                              = (known after apply)
      + website_domain                       = (known after apply)
      + website_endpoint                     = (known after apply)

      + object_lock_configuration {
          + object_lock_enabled = (known after apply)
          + rule                = (known after apply)
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + bucket_name = (known after apply)

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.
Releasing state lock. This may take a few moments...

2. Deploy Command

Running the deploy command with a *.tfvars file will initiate the resource creation process.

export S3Bucket="some-s3-bucket"
export Region="us-east-1"
export DynamoDB="some-dynamodb-table"
./autotf deploy stage/s3/autotf-testing01.tfvars

Destroying Resources

Resource destruction follows the same flow control. If the file extension is *.destroy, the verify command will run terraform init and terraform plan -destroy, and the deploy command will run terraform init and terraform destroy.

# Change the file name from *.tfvars to *.destroy
mv stage/s3/autotf-testing01.tfvars stage/s3/autotf-testing01.destroy

# Run the `autotf` deploy command again with the new filename
./autotf deploy stage/s3/autotf-testing01.destroy

Output

INFO 2022-03-28 18:13:07 will run terraform init, plan -destory on [autotf-testing01.destroy]

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v4.8.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Acquiring state lock. This may take a few moments...
aws_s3_bucket.bucket: Refreshing state... [id=autotf-testing01]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # aws_s3_bucket.bucket will be destroyed
  - resource "aws_s3_bucket" "bucket" {
      - acl                                  = "private" -> null
      - arn                                  = "arn:aws:s3:::autotf-testing01" -> null
      - bucket                               = "autotf-testing01" -> null
      - bucket_domain_name                   = "autotf-testing01.s3.amazonaws.com" -> null
      - bucket_regional_domain_name          = "autotf-testing01.s3.amazonaws.com" -> null
      - cors_rule                            = [] -> null
      - force_destroy                        = false -> null
      - grant                                = [] -> null
      - hosted_zone_id                       = "Z3AQBSTGFYJSTF" -> null
      - id                                   = "autotf-testing01" -> null
      - lifecycle_rule                       = [] -> null
      - logging                              = [] -> null
      - object_lock_enabled                  = false -> null
      - region                               = "us-east-1" -> null
      - replication_configuration            = [] -> null
      - request_payer                        = "BucketOwner" -> null
      - server_side_encryption_configuration = [] -> null
      - tags                                 = {
          - "Environment" = "Dev"
          - "Name"        = "autotf-testing01"
        } -> null
      - tags_all                             = {
          - "Environment" = "Dev"
          - "Name"        = "autotf-testing01"
        } -> null
      - versioning                           = [
          - {
              - enabled    = false
              - mfa_delete = false
            },
        ] -> null
      - website                              = [] -> null
    }

Plan: 0 to add, 0 to change, 1 to destroy.

Changes to Outputs:
  - bucket_name = "autotf-testing01" -> null

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.
Releasing state lock. This may take a few moments...
Acquiring state lock. This may take a few moments...
aws_s3_bucket.bucket: Refreshing state... [id=autotf-testing01]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # aws_s3_bucket.bucket will be destroyed
  - resource "aws_s3_bucket" "bucket" {
      - acl                                  = "private" -> null
      - arn                                  = "arn:aws:s3:::autotf-testing01" -> null
      - bucket                               = "autotf-testing01" -> null
      - bucket_domain_name                   = "autotf-testing01.s3.amazonaws.com" -> null
      - bucket_regional_domain_name          = "autotf-testing01.s3.amazonaws.com" -> null
      - cors_rule                            = [] -> null
      - force_destroy                        = false -> null
      - grant                                = [] -> null
      - hosted_zone_id                       = "Z3AQBSTGFYJSTF" -> null
      - id                                   = "autotf-testing01" -> null
      - lifecycle_rule                       = [] -> null
      - logging                              = [] -> null
      - object_lock_enabled                  = false -> null
      - region                               = "us-east-1" -> null
      - replication_configuration            = [] -> null
      - request_payer                        = "BucketOwner" -> null
      - server_side_encryption_configuration = [] -> null
      - tags                                 = {
          - "Environment" = "Dev"
          - "Name"        = "autotf-testing01"
        } -> null
      - tags_all                             = {
          - "Environment" = "Dev"
          - "Name"        = "autotf-testing01"
        } -> null
      - versioning                           = [
          - {
              - enabled    = false
              - mfa_delete = false
            },
        ] -> null
      - website                              = [] -> null
    }

Plan: 0 to add, 0 to change, 1 to destroy.

Changes to Outputs:
  - bucket_name = "autotf-testing01" -> null
aws_s3_bucket.bucket: Destroying... [id=autotf-testing01]
aws_s3_bucket.bucket: Destruction complete after 0s
Releasing state lock. This may take a few moments...

Destroy complete! Resources: 1 destroyed.

The `autotf` Tool

The autotf tool is available on GitHub. It's a demo to showcase how to solve this common Terraform problem. The control flow is straightforward: for each verify and deploy command, the backend configuration is dynamically generated from the tfvars file, ensuring it's always consistent.

Here is an example of the Terraform code for creating an S3 bucket.

Terraform Code

terraform {
  backend "s3" {}
}

provider "aws" {
  region = var.region
}

resource "aws_s3_bucket" "bucket" {
  bucket = var.name

  tags = {
    Name        = var.name
    Environment = "Dev"
  }
}

variable "region" {
  type = string
}
variable "name" {
  type = string
}

output "bucket_name" {
  value = aws_s3_bucket.bucket.id
}

Command Run

./autotf verify stage/s3/autotf-testing01.tfvars

Using an external tool provides the ability to manage Terraform without manually editing the backend configuration. Generating the backend configuration based on a predefined logic is a much more efficient approach.

Above flow-control explains how the sequence of events will occur for verify and deploy sub-commands respectively. Because the backend configuration is getting generated dynamically with each tfvars file so we don’t have to persist it and it will always be the same as long as we used the same tfvars file.

Here is example for verify step for creating a S3 Bucket.

Terraform Code

terraform {
  backend "s3" {}
}

provider "aws" {
  region = var.region
}

resource "aws_s3_bucket" "bucket" {
  bucket = var.name

  tags = {
    Name        = var.name
    Environment = "Dev"
  }
}

variable "region" {
  type = string
}
variable "name" {
  type = string
}

output "bucket_name" {
  value = aws_s3_bucket.bucket.id
}

Command I Ran

>>> ./autotf verify stage/s3/autotf-testing01.tfvars

Output

INFO 2022-03-28 17:50:29 will run terraform init, plan on [autotf-testing01.tfvars]

+------------------+----------------+-------------------------+-----------------------------------+
| Resource Name    | Backend Bucket | TFVars Name             | Backend Key                       |
+------------------+----------------+-------------------------+-----------------------------------+
| autotf-testing01 | autotf-testing | autotf-testing01.tfvars | stage/s3/autotf-testing01.tfstate |
+------------------+----------------+-------------------------+-----------------------------------+

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/aws v4.8.0...
- Installed hashicorp/aws v4.8.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Acquiring state lock. This may take a few moments...

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_s3_bucket.bucket will be created
  + resource "aws_s3_bucket" "bucket" {
      + acceleration_status                  = (known after apply)
      + acl                                  = (known after apply)
      + arn                                  = (known after apply)
      + bucket                               = "autotf-testing01"
      + bucket_domain_name                   = (known after apply)
      + bucket_regional_domain_name          = (known after apply)
      + cors_rule                            = (known after apply)
      + force_destroy                        = false
      + grant                                = (known after apply)
      + hosted_zone_id                       = (known after apply)
      + id                                   = (known after apply)
      + lifecycle_rule                       = (known after apply)
      + logging                              = (known after apply)
      + object_lock_enabled                  = (known after apply)
      + policy                               = (known after apply)
      + region                               = (known after apply)
      + replication_configuration            = (known after apply)
      + request_payer                        = (known after apply)
      + server_side_encryption_configuration = (known after apply)
      + tags                                 = {
          + "Environment" = "Dev"
          + "Name"        = "autotf-testing01"
        }
      + tags_all                             = {
          + "Environment" = "Dev"
          + "Name"        = "autotf-testing01"
        }
      + versioning                           = (known after apply)
      + website                              = (known after apply)
      + website_domain                       = (known after apply)
      + website_endpoint                     = (known after apply)

      + object_lock_configuration {
          + object_lock_enabled = (known after apply)
          + rule                = (known after apply)
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + bucket_name = (known after apply)

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.
Releasing state lock. This may take a few moments...

As I mentioned before in my previous blog with the help of some external tool to achieve the dynamic backend configuration generation and we did it. We can even run this tool in docker container (How ? Explained here) in our CI System.

Personal Best Practices

Here are some rules I personally follow for better Terraform management:

Ensure consistency: The resource name, tfvars filename, and tfstate filename should be the same to maintain a 1:1 relationship.
Derive names automatically: For multiple resources forming a single entity, names are automatically derived from the primary resource's name.
Keep tfvars short: Use defaults to minimize the size of tfvars files.
Use ternary operators: This helps manage both computed and provided values effectively.
Make modules self-sufficient: Design modules to be as nimble and independent as possible.
Write validators: Implement validators to support a "fail-fast" approach.
Avoid depends_on: Try to use it as little as possible.

Happy Coding and Terraforming! 💻

I Got Tired of Typoing in My Git Commits, So I Wrote a Spellchecker

Satyajit Roy — Sat, 23 Aug 2025 09:38:26 +0000

I Got Tired of Typoing in My Git Commits, So I Wrote a Spellchecker

The Problem: A Constant, Tiny Annoyance

I’m a decent developer, but a terrible speller. It’s a combination that leads to a specific, low-grade, yet constant form of humiliation: the typo-ridden Git commit message.

You know the ones:

git commit -m "Fix spec for user controlelr"
git commit -m "Add new feture for exporting data"
git commit -m "Corect a stupid tpyo in a stupid comment"

Every time I’d push a branch and open a pull request, I’d see my own mistakes staring back at me from the beautifully crafted UI of GitHub or GitLab. It was a tiny, public testament to my carelessness. Linters protect my code, but nothing was protecting my prose.

I was tired of it. It was time to automate the solution.

The Solution: Git Hooks to the Rescue

The natural place to solve this is with a Git hook, specifically the commit-msg hook. This hook runs after the default commit message file is created but before the commit is finalized. It’s the perfect place to inspect the message and, if it doesn’t meet your standards, reject the commit entirely.

The plan was simple:

Write a script that takes the proposed commit message file as an argument.
Extract the message from the file.
Run it through a spellchecker.
If there are potential errors, print them and exit with a non-zero status (telling Git to abort the commit).
If it’s clean, exit successfully and let the commit proceed.

The Implementation: A Bit of Bash and `aspell`

I’m a big fan of using existing command-line tools, and for spellchecking, the classic aspell is perfect for this job.

Here is the core of the commit-msg hook I wrote:

#!/bin/bash

# commit-msg hook to spell check the commit message

# Check if aspell is installed
if ! command -v aspell &> /dev/null
then
    echo "Error: aspell is not installed. Please install it to use the spellcheck hook."
    exit 1
fi

# Get the commit message file path from Git
MSG_FILE="$1"

# Read the commit message
MESSAGE=$(cat "$MSG_FILE")

# Check if the message is empty or just comments (for verbose editing)
if [[ -z "$MESSAGE" || "$MESSAGE" =~ ^# ]]; then
    # If empty or all comments, this is likely an interactive edit, so we skip checking.
    exit 0
fi

# Use aspell to check the message.
# - list: just list the misspelled words
# --mode=email: better for checking prose, not code
# --ignore-case: self-explanatory
# --ignore=3: ignore words under 3 characters (lets 'git' through)
ERRORS=$(echo "$MESSAGE" | aspell --mode=email --ignore-case --ignore=3 list)

if [[ -n "$ERRORS" ]]; then
    echo "---------------------------------------------------------"
    echo "OH NO! Your commit message has spelling mistakes!"
    echo "---------------------------------------------------------"
    echo "The following words are not recognized:"
    echo "$ERRORS" | sort | uniq
    echo ""
    echo "Please fix your commit message and try again."
    echo "To bypass this check, use 'git commit --no-verify'."
    echo "---------------------------------------------------------"
    exit 1
fi

# If we get here, the spell check passed.
exit 0

How to Install It

Make the script executable.
```
chmod +x commit-msg
```

Place it in your Git hooks directory. For a single repository, that’s .git/hooks/. To use it globally for all your repos (highly recommended!), you can set a global core.hooksPath or copy it to your global template directory.

# For a single project
cp commit-msg /path/to/my/project/.git/hooks/

# To install it globally for all new repos
# 1. Create a global templates directory (if you haven't already)
mkdir -p ~/.git-templates/hooks
# 2. Copy your hook into it
cp commit-msg ~/.git-templates/hooks/
# 3. Tell Git to use this directory for all new 'git init' or clones
git config --global init.templatedir '~/.git-templates'
# (For existing repos, you'll need to manually copy the hook or re-run 'git init')

The Result: Beautiful, Typo-Free Commit History

The effect was immediate and glorious. Now, when I try to commit with a typo:

$ git commit -m "Fix the tpyo"
OH NO! Your commit message has spelling mistakes!
---------------------------------------------------------
The following words are not recognized:
tpyo
---------------------------------------------------------
Please fix your commit message and try again.
To bypass this check, use 'git commit --no-verify'.

It forces me to be better. It’s a small, automated nudge towards quality that has completely eliminated this petty annoyance from my workflow. My commit history is cleaner, and my pull requests no longer have that initial, embarrassing comment from me that just says "fixed a typo in the commit message."

It’s a tiny investment in tooling that has paid for itself in reduced frustration a hundred times over. If you’re also tired of typos, I highly recommend giving it a try.

Dev Setup on Autopilot

Satyajit Roy — Mon, 18 Aug 2025 18:46:59 +0000

Dev Setup on Autopilot

Every developer knows the ritual. A new laptop arrives shiny, fast, and brimming with potential. Or perhaps you’re upgrading your existing machine, or starting a new role with a fresh corporate setup. The excitement is palpable, but then reality hits: the painstaking process of recreating your perfect development environment.

For years, I’ve lived this saga. Like many of you, I have my carefully curated arsenal of tools, fonts, and configurations that make me productive. My terminal needs to look just right, my version managers must be in place, and a dozen specific CLI utilities are non-negotiable.

The Deja Vu of Re-Setup

The problem isn’t just the initial setup; it’s the memory part. I’d spend hours, sometimes days, painstakingly recalling, searching, and reinstalling. Let me give you some concrete examples of the frustration:

The Elusive brew Formula: You remember using brew to install that niche CLI tool for managing cloud resources, but was it brew install aws-cli or brew install awscli? Did it have a specific --with-zsh-completion flag? Or was it aws-cli@2 because the latest version broke a script? You spend 15 minutes in brew search and then another 10 digging through old documentation or ~/.zsh_history (if you even remembered to back it up).
The Go Tooling Conundrum: You need protoc-gen-go-grpc. Did you go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest? Or was it an older v1.2.3 because the team's build system isn't on the newest protobuf? Or, even worse, did you download a pre-compiled binary from a GitHub Release because it was faster, and now you have no idea where it came from?
The Perfect Font, Gone: Your beautiful coding font, say “Fira Code Nerd Font Mono” suddenly isn’t rendering correctly. Was it version v3.0.2 or v3.1.1 of Nerd Fonts? And which specific *.ttf file within the archive did you copy to ~/Library/Fonts? Now you're hunting through GitHub Releases, downloading multi-gigabyte archives just to find that one precise file.
Dotfiles’ Blind Spots: While a ~/.dotfiles repository is a lifesaver for shell aliases (alias gcm='git commit -m') or Vim plugins, they notoriously fall short when it comes to:
- Actual Software Installation: Your dotfiles won’t magically install Homebrew, or git itself, or Node.js, or the Rust toolchain. They assume these foundational tools are already there.
- Installation Methods: Your shellrc might expect pyenv to be in your $PATH, but it won't tell you if pyenv was installed via brew, git clone, or a custom script.
- System-Level Settings: Those macOS defaults write commands that perfect your trackpad speed, hide desktop icons, or ensure hidden files are always visible, these are often scattered notes or manual tweaks, not cleanly managed.

This constant “re-setup tax” was a major pain point, a recurring nightmare that chipped away at my valuable time and focus. I wanted a way to truly encapsulate my entire development environment, not just bits and pieces.

My Solution: Introducing `setup-devbox`

That frustration finally boiled over, and I decided to tackle it head-on. I focused on a piece of software that solves this exact problem: setup-devbox.

setup-devbox is my answer to the "new laptop dilemma." It’s a command-line tool that lets you define your entire development environment in declarative YAML files. This isn't just about listing tools; it's about stating the desired end-state of your machine, which is repeatable every single time, in a format that's both human-readable and machine-executable.

Why Declarative `YAML`?

The power of declarative YAML is that it acts as your single source of truth. It lives alongside your code in version control, meaning:

Readability: It’s easy to glance at tools.yaml and instantly know what's supposed to be installed.
Version Control: You can track changes to your environment setup over time, just like you track changes to your application code.
Shareability: Want to onboard a new team member to an identical setup? Share your setup-devbox config files.
No More Guesswork: No more trying to remember a specific brew flag or the exact go install command. The YAML specifies it all.

What `setup-devbox` Manages:

When you define your environment, setup-devbox steps up to handle the heavy lifting across several crucial domains:

Tools (and Their Installation Methods!): This is where it shines. You define tools like git, Node.js, kubectl, or even specific versions of gh (GitHub CLI). Crucially, you also tell setup-devbox how to install them: - brew: For standard packages on macOS and Linux. - cargo: For your favorite Rust CLI tools. - github: For tools distributed as GitHub Releases (e.g., specific versions of terraform or helm). - url: For direct downloads of binaries or installers that don't fit other categories (e.g., a specific Go installer .pkg).
It intelligently uses the correct installer and even handles rename_to for convenience (e.g., cli to gh).
macOS System Settings (defaults write): Tired of manually changing Finder settings or keyboard behaviors after a clean install? setup-devbox lets you declare those defaults write commands directly in your settings.yaml, ensuring consistency every time.
Shell Configuration (.zshrc, .bashrc etc.): Beyond just aliases, you can define raw lines of configuration to be injected into your shell startup files. This includes everything from $PATH modifications to eval "$(starship init zsh)" or complex fzf bindings.
Fonts: My coding experience isn’t complete without specific fonts, often Nerd Fonts. setup-devbox automates the download and installation of these, ensuring my terminal looks perfect on every machine.

At present it support GitHub, Brew, Cargo, Rustup, Go, Pip and Direct URL.

Here are examples of my tools.yaml

- name: git-spellcheck
  version: 0.0.1
  source: github
  repo: kodelint/git-spellcheck
  tag: v0.0.1
  rename_to: git-spellcheck
- name: git-pr
  version: 0.1.0
  source: github
  repo: kodelint/git-pr
  tag: v0.1.0
  rename_to: git-pr
- name: cli
  version: "2.74.0"
  source: github
  repo: cli/cli
  tag: v2.74.0
  rename_to: gh
- name: delta
  version: "0.18.2"
  source: github
  repo: dandavison/delta
  tag: 0.18.2
  rename_to: delta

Example of fonts.yaml

fonts:
- name: 0xProto
  version: 3.4.0
  source: github
  repo: ryanoasis/nerd-fonts
  tag: v3.4.0
  install_only: ["regular", "Mono"]
  Example of shellrc.yaml

Example of shellrc.yaml

shellrc:
  shell: zsh
  raw_configs:
   - export PATH=$HOME/bin:$PATH
   - export PATH=$HOME/.cargo/bin:$PATH
   - export PATH=$HOME/go/bin:$PATH
   - eval "$(starship init zsh)"
   - "[ -f ~/.fzf.zsh ] && source ~/.fzf.zsh"
   - zle -N fzf-history-widget - bindkey '^R' fzf-history-widget
   - 'export FZF_CTRL_R_OPTS="--no-preview --scheme=history --tiebreak=index   --bind=ctrl-r:toggle-sort"'
   - | export FZF_CTRL_T_OPTS="
      --preview 'bat --style=numbers --color=always {} || cat {}'
      --preview-window=right:80%
      --bind ctrl-b:preview-page-up,ctrl-f:preview-page-down"
aliases:
    - name: code
      value: cd $HOME/Documents/github/
    - name: gocode
      value: cd $HOME/go/src/
    - name: edconf
      value: zed $HOME/.zshrc
    - name: g
      value: git
    - name: gs
      value: g status

Example of settings.yaml

settings:
  macos:
    - domain: NSGlobalDomain
      key: AppleShowAllExtensions
      value: "true"
      type: bool
    - domain: com.apple.finder
      key: AppleShowAllFiles
      value: "true"
      type: bool
    - domain: com.apple.dock
      key: expose-animation-duration
      value: "0.1"
      type: float
    - domain: NSGlobalDomain
      key: KeyRepeat
      value: "1"
      type: int
    - domain: com.apple.finder
      key: ShowPathbar
      value: "true"
      type: bool

The Magic of State Management

The core idea is simple yet powerful: you run setup-devbox now, and it springs into action. But it's not a dumb script that reinstalls everything every time. It leverages an internal state.json file. This file acts as setup-devbox's memory, keeping track of what has already been installed and configured, and how.

This means that:

Idempotency: Running setup-devbox now multiple times won't break anything or perform redundant work. If a tool is already installed and matches your desired version, setup-devbox simply skips it.
Efficiency: Only missing or outdated components are installed or updated, saving you precious time and bandwidth.
Reliability: The state.json ensures that your environment progresses predictably towards the desired state you've defined.

This means that whether I’m upgrading my MacBook, spinning up a new virtual machine, or onboarding onto a client’s specific environment, I can replicate my ideal setup with minimal effort. The entire “re-setup tax” is almost eliminated, freeing me up to do what I love: develop.

Reclaim Your Setup Time, Forever

Building setup-devbox has transformed my personal development workflow from a recurring chore into a one-command automation. My development environment is no longer a collection of scattered memories and manual steps; it’s a declarative, version-controlled blueprint that setup-devbox brings to life with intelligent efficiency.

If you, like me, have suffered the “new laptop setup” headache too many times, I genuinely believe setup-devbox can be a game-changer for you. It's my personal solution to a universal developer problem, and I'm excited to share it with the community.

Head over to the GitHub Repository: setup-devbox, explore the configuration examples, and give setup-devbox a try. I'm actively developing it, and your feedback, ideas, or even contributions would be incredibly valuable.

Repository: `setup-devbox`

Happy Developing!!

Git Selective Ignore: Because Sometimes You Need to Keep Secrets from Git (But Not From Yourself)

Satyajit Roy — Wed, 17 Aug 2022 02:24:56 +0000

Git Selective Ignore: Because Sometimes You Need to Keep Secrets from Git (But Not From Yourself)

Or: How I Learned to Stop Worrying and Love Committing Without Fear

Sound familiar? Welcome to the club. We have t-shirts, but ironically, we can't put the design in our Git repo because it has our logo's secret color codes in it.

The Git Philosophy: Everything Is Sacred (Even Your Mistakes)

Enter the Real World: Where Perfect Theory Meets Messy Reality

Consider these common scenarios:

The Debug Block: Those console.log statements that helped you figure out why the async function was returning undefined, but now clutter your clean, professional codebase.
The Local Config: Database connection strings, API endpoints, and feature flags that need to be different for local development but shouldn't make their way into the shared repository.
The Temporary Experiment: That experimental algorithm you're testing, complete with hardcoded values and performance logging, sitting right in the middle of your otherwise pristine production code.
The Security Nightmare: API keys, tokens, passwords, and other credentials that somehow always seem to sneak into codebases despite our best intentions.

Traditional Solutions: A Comedy of Errors

Let's look at how we typically handle these situations:

Option 1: The Paranoid Approach

Option 2: The Environment Variable Dance

Option 3: The Separate Config Files

Option 4: The Git Surgery

Git Selective Ignore: A Surgical Solution

This is where git-selective-ignore comes in. Think of it as a precision instrument in a world of sledgehammers.

Instead of treating files as atomic units, this tool lets you specify exactly which parts of a file should be ignored during commits. It's like having a conversation with Git:

You: "Hey Git, commit this file, but ignore lines 13-16, and also any line that contains 'API_KEY', and oh, while you're at it, skip that debug block between the comments."

Git: "That's not how I—"

git-selective-ignore: "I got this. Git, just commit what they want you to commit. Trust me."

Git: "...okay, but this feels weird."

How It Actually Works (The Magic Behind the Curtain)

The tool operates using Git hooks—specifically pre-commit and post-commit hooks. Here's the elegant dance it performs:

Pre-commit: Before Git commits your changes, the tool scans your staged files, creates temporary clean versions with the specified content removed, and stages these sanitized versions.
Commit: Git commits the clean versions to history.
Post-commit: The tool restores your original files, so your working directory contains all your local configs, debug statements, and temporary code.

It's like having a butler who tidies up your room before guests arrive, then puts everything back exactly where you left it after they leave.

Real-World Example: The API Key Tango

Let's say you have this in your main.rs and lib.rs:

fn main() {
    println!("Starting application...");

    // DEBUG BLOCK START
    println!("Debug: Application started in debug mode");
    // DEBUG BLOCK END

    let API_KEY = "sk_live_1234567890abcdef";
    println!("Using API key: {}", API_KEY);

    /* Temporary lines for testing - remove before prod */
    let temp_feature = "experimental_feature_xyz";
    println!("Testing temporary feature: {}", temp_feature);
    /* End temporary section */

    let SECRET = "Some secret value";
    println!("SECRET configured");

    println!("Application completed successfully");
}

use std::env;

fn main() {
    println!("Another Test");

    let GITHUB_TOKEN = "github_fake_token_093790841-831-8lncdlwnelkqix12=-1x;xm;m"

    println!("{} <- My GitHub Token", GITHUB_TOKEN);

    let API_KEY = env::var('API_KEY');

    match env::var('API_KEY') {
        Ok(value) => {
            println!("The value of APP_KEY is: {}", API_KEY);
        }
        Err(e) => {
            eprintln!("Error getting environment variable {}: {}", 'API_KEY', e);
        }
    }
}

With traditional Git, you have two choices:

Commit everything (including secrets) 😬😬😬
Manually clean it up before every commit 🫩🫩🫩

With git-selective-ignore, you set up your patterns once 🕺🏽 💃🏻:

# Ignore any lines containing these sensitive terms
>> git-selective-ignore add all API_KEY --pattern-type line-regex
>> git-selective-ignore add all SECRET --pattern-type line-regex

# Ignore debug blocks
>> git-selective-ignore add all "// DEBUG BLOCK START ||| // DEBUG BLOCK END" --pattern-type block-start-end

# Ignore specific line ranges for temporary code
>> git-selective-ignore add src/main.rs 13-16 --pattern-type line-range

Now when you commit, Git's history will only contain:

fn main() {
    println!("Starting application...");

    println!("Using API key: {}", API_KEY);

    println!("SECRET configured");

    println!("Application completed successfully");
}

But your local file remains unchanged, so you can keep working with all your debug statements and test configurations intact.

Why This Approach Works Better

Compared to Manual Cleanup

No more forgetting: You don't have to remember what to remove and add back
No broken local environments: Your working directory always contains what you need
Consistent team experience: Everyone gets the same sanitized commits

Compared to Environment Variables

Faster iteration: No need to restart processes when changing test values
Better debugging: You can see actual values in your code while developing
Simpler onboarding: New developers can run the project with sensible defaults

Compared to Separate Config Files

No structural divergence: The team sees the shape of your configuration
No lost configs: Your local setup is preserved in your working directory
Version control benefits: You can still track changes to the configuration structure

Advanced Patterns: Getting Surgical

The tool supports several pattern types for different use cases:

Line Regex (for scattered sensitive data)

Perfect for API keys, passwords, or debug statements scattered throughout your codebase:

>> git-selective-ignore add all "console\.log.*debug" --pattern-type line-regex

Line Ranges (for temporary code blocks)

When you know exactly which lines contain temporary code:

>> git-selective-ignore add src/config.py 45-52 --pattern-type line-range

Block Start/End (for structured temporary sections)

When you use comments to mark temporary code:

>> git-selective-ignore add all "// TODO: REMOVE ||| // END TODO" --pattern-type block-start-end

The Philosophy Shift: From Binary to Granular

Getting Started: Your First Steps into Selective Ignoring

Installation is straightforward. You can use their setup-devbox tool, download from releases, or build from source:

# Build from source
>> git clone https://github.com/kodelint/git-selective-ignore.git
>> cd git-selective-ignore
>> cargo install --path .

Then in any repository:

# Initialize selective ignore
git-selective-ignore init

# Install the Git hooks
git-selective-ignore install-hooks

# Add your first pattern
git-selective-ignore add all "API_KEY" --pattern-type line-regex

# Check what would be ignored
git-selective-ignore status

The tool stores its configuration in .git/selective-ignore.toml, so each repository can have its own rules without affecting others.

You can also see the violations in the code base with status

>> git-selective-ignore status
✓ Configuration is valid.
   ├─ Line Range Pattern '13-16': 4 line(s) matched
   │  └─ Lines 13-16
   ├─ Regex Pattern 'API_KEY': 1 line(s) matched
   │  └─ Line 10
   ├─ Regex Pattern 'GITHUB_TOKEN': 1 line(s) matched
   │  └─ Line 20
   ├─ Block Pattern '// DEBUG BLOCK START ||| // DEBUG BLOCK END': 3 line(s) matched
   │  └─ Lines 6-8
   ├─ Regex Pattern 'SECRET': 1 line(s) matched
   │  └─ Line 18
   └─ Summary: 10 line(s) ignored, 17 line(s) remaining (of 27 total)
   ├─ Regex Pattern 'GITHUB_TOKEN': 1 line(s) matched
   │  └─ Line 7
   └─ Summary: 1 line(s) ignored, 18 line(s) remaining (of 19 total)
📊 Git Selective Ignore Status Report
=====================================
🎯 Specifically Configured Files:
🟡 src/main.rs (8 patterns, 10/27 lines ignored, 37.0%)

🌐 Files Affected by Global 'ALL' Patterns:
🟡 src/lib.rs (6 patterns, 1/19 lines ignored, 5.3%)

📈 Summary:
  Total files: 2
  Total patterns: 8
  Total ignored lines: 11
  Files with issues: 2

📋 Breakdown:
  Specifically configured files: 1
  Files affected by 'ALL' patterns only: 1

Use list to see what Ignore Patterns are installed

>> git-selective-ignore list
✓ Configuration is valid.

📁 File: all
  🔍 ID: 78ed02f4-db7c-4921-b565-5e8986f19705 | Type: LineRegex | Pattern: API_KEY
  🔍 ID: 7fb165d1-bab6-4c79-a13b-51f2f29a88e9 | Type: LineRegex | Pattern: APP_KEY
  🔍 ID: 02b17597-bb85-428c-be56-3d0cd4a3c44b | Type: LineRegex | Pattern: GITHUB_TOKEN
  🔍 ID: 76447f06-dd03-4c3b-b27a-b611579e9cb8 | Type: BlockStartEnd | Pattern: // DEBUG BLOCK START ||| // DEBUG BLOCK END
  🔍 ID: 48f984d1-dd90-4984-99d6-ae6c63c591d6 | Type: LineRegex | Pattern: SECRET
  🔍 ID: b9a54bc2-048d-4fa0-b6ff-dc66aff6e706 | Type: LineRegex | Pattern: password

📁 File: src/main.rs
  🔍 ID: 31ca2ff0-90d8-47ea-90db-413cedf09bcf | Type: LineRange | Pattern: 13-16
  🔍 ID: a941d428-87ed-4378-898d-d5156723dfd0 | Type: BlockStartEnd | Pattern: /* TEMP_CODE_START */ ||| /* TEMP_CODE_END */

A Word of Caution: With Great Power...

Like any powerful tool, git-selective-ignore should be used thoughtfully. It's not a license to be sloppy with sensitive data, it's a safety net for the inevitable times when we are human.

Remember:

Security is still your responsibility: This tool helps prevent accidental commits, but you should still follow security best practices
Team coordination matters: Make sure your team knows which patterns are in place
Test your patterns: Use the status command to verify your patterns work as expected

The Future of Granular Version Control

Conclusion: Sleep Better at Night

git-selective-ignore gives you the freedom to develop the way you need to while maintaining the professional, secure codebase your team depends on. It's not just a tool, it's peace of mind.

And your security team? They'll sleep better too, knowing that those 2 AM coding sessions are far less likely to result in morning meetings about exposed credentials.

Now if you'll excuse me, I need to go commit some code. Don't worry, I've got selective ignore set up for my embarrassing variable names.

Ready to try git-selective-ignore? Check it out on GitHub and start committing with confidence.