The git bisect run habit I should have learned ten years sooner

Matthew — Mon, 04 May 2026 06:18:18 +0000

Last Tuesday I lost about three hours to a regression in our checkout service. The cart total was off by a cent on certain promo combinations, and the only signal was a Slack ping from finance with a screenshot. No stack trace. No exception. Just wrong numbers.

I did what I always do first. I opened the diff for the last deploy, scrolled, squinted, and tried to feel my way to the bug. Forty minutes in, I had a mental shortlist of suspects and zero proof. The code I was sure caused it had been merged a week earlier and was already reverted in a different branch for unrelated reasons. I was, as my old tech lead used to say, debugging the file I wanted the bug to be in.

Then I remembered git bisect run.

I keep relearning this tool. It's been around forever. I've recommended it to juniors more times than I can count. And yet whenever I'm in the thick of a real outage, I forget it exists for the first hour, because my brain wants to read the diff like a detective novel instead of running an experiment.

Here's the workflow that finally stuck for me. I write a tiny script that exits 0 when the bug is absent and non-zero when it's present. For this regression, the script was twelve lines:

#!/usr/bin/env bash
set -e
go build ./cmd/checkout
./checkout-test --promo=BACK2SCHOOL --items=3 > /tmp/out
grep -q '"total":4297' /tmp/out

Then git bisect start HEAD v4.18.2, git bisect run ./check.sh, and walk to the kitchen for coffee. By the time I came back, git had narrowed it to a single commit. Not the one I'd been staring at. Not even close. It was a four-line change to how we round per-line tax that someone had landed quietly two weeks ago, with a benign-looking commit message about "consistency."

What surprised me wasn't the bug. The bug was small. What surprised me was how much faster the machine was at this than I was. I had been pattern-matching on author, recency, and which file looked spicy. Bisect doesn't care about any of that. It just keeps halving the search space until there's one commit left.

A few things I've learned about making this a habit instead of a heroic last resort:

The check script needs to be cheap. If your build is eight minutes, bisect across two hundred commits is a coffee break and a meeting and lunch. Mock the slow stuff. Build a smaller binary. I keep a bisect/ directory in some repos with pre-baked check scripts for known classes of bugs, so when something feels familiar I'm not writing the harness from scratch under pressure.

Skip commits that don't build. git bisect skip is your friend. You'll hit a few broken middle-of-refactor commits and that's fine; bisect handles them.

Don't trust your gut on the "good" commit. I once spent forty minutes bisecting before realizing my "known good" reference also had the bug, just less visibly. Now I always run the check script against the supposedly-good commit first, before bisect start. Two minutes of paranoia saves an hour of confusion.

Bisect across squash-merged PRs and you might land on a commit that touches forty files. That's still useful. The PR title alone usually points you somewhere. I've also started linking PR numbers in commit bodies more aggressively for exactly this reason.

The reason I keep forgetting this tool, I think, is that it feels like cheating. I want to understand the bug by reading the code, because that's the part that feels like real engineering. Bisect skips the understanding and goes straight to the answer. But the answer is what unblocks the team. Understanding can come after, in a calmer hour, with the offending commit already pinned to the wall.

The Tuesday bug ended up being a one-line fix. The git bisect run took eleven minutes including the coffee. The forty minutes I'd spent reading the diff before remembering the tool existed is the part I want back.

Next time I feel the urge to scroll through a deploy diff while a fire is burning, I'm going to write the check script first. Even if I'm sure I know where the bug is. Especially then.

20 years in, the agent stack finally feels like plumbing

Matthew — Fri, 01 May 2026 14:34:31 +0000

Twenty-something years ago I wrote my first SOAP client. The XML was nested four deep, the WSDL was wrong, the vendor's docs lied, and I still got paged at 2am when something a Java app I didn't write decided the namespace had moved. I think about that morning a lot whenever someone tries to sell me a new "agent framework."

A year ago I'd have rolled my eyes at MCP too. Every cycle has Its Protocol. Most don't outlive their conference talk. But MCP crossed 97 million installs in March, the Linux Foundation just took it under open governance, and OpenAI's newest Responses API speaks it natively. Around when that happened I noticed something a little embarrassing. I'd quietly stopped writing one-off integration scripts. They just sort of fell off the to-do list.

So what's it doing right that the last forty things didn't?

It's boring. That's the whole answer.

There's no clever DSL. No SDK lock-in. No "first-class agent abstractions" that turn out to mean three layers of decorator soup. A server exposes some tools, an agent calls them, the transport doesn't care which model you're paying this month. I run a Postgres MCP server, a Linear one, and a Playwright one. They have absolutely no idea about each other and they don't need to. It's the same dumb composability we got from pipes and from HTTP.

The interesting bit isn't the install count. It's that OpenAI ships compatibility, Google ships compatibility, and now LF owns the spec. We're past the line where any one company can yank it. That is rare. The last protocol I can remember crossing it was OAuth 2.0, and that took years longer than people remember.

A few things I'm still not sold on:

Auth gets reinvented per server and most of them get it wrong
Discovery is hand-wavy. We're going to need something better than "paste this URL"
Tool permissioning is closer to "the user clicks yes" than to anything you'd accept in production

None of that kills it. It just means the next two years probably look like 2010-era REST. Everyone agrees on the wire format and we slowly figure out the tooling around it. CORS will get reinvented. Someone will write a bad spec for tool versioning. We'll all live.

If you've been on the fence, this is a good week to ship a small server. Pick one annoying internal tool you wish your editor knew about. Wrap it. Plug it in. The leverage is genuinely silly once you stop fighting the model into pretending to know things it never could.

Curious what other graybeards on here are using MCP for day to day. The thing I'd most like to see: a battle-tested auth pattern that isn't "static bearer token in env."

Forem: Matthew

The git bisect run habit I should have learned ten years sooner

20 years in, the agent stack finally feels like plumbing