Forem: Nico Reyes

Git bisect saved me from a week of debugging. Should've used it sooner.

Nico Reyes — Mon, 06 Apr 2026 19:26:58 +0000

Git bisect saved me from a week of debugging. Should've used it sooner.

Two weeks ago, my coworker merged a feature branch on Friday afternoon. By Monday morning, half our test suite was red. Nobody knew which commit broke things. The branch had 47 commits.

I did what most devs do. Opened git log, stared at it, closed it. Forty-seven commits. No way I was reading all of them.

So I did what any reasonable person would do. Blamed everyone in the commit history until I found someone to be mad at. Just kidding. Mostly.

The manual approach

Git blame told me which lines changed but not why they broke. Git log showed me filenames but not the actual damage. I spent two days narrowing it down manually. Checked out commit one by one. Ran tests. Watched them fail. Checked out another. Ran tests. Watched them fail.

My approach was scientific. My time investment was not.

# What I was doing (spoiler: wrong)
git checkout commit-xyz
npm test  # 47 times

Forty-seven times. At 3 minutes per run, that's over two hours of pure waiting. Plus the checkout time. Plus the context switching.

Enter git bisect

A senior dev walked by, saw my screen, and asked why I wasn't using git bisect.

I didn't have a good answer.

Git bisect does binary search on your commit history. You tell it a known good commit and a known bad commit. It checkpoints you to the midpoint. You test. You mark good or bad. It narrows down. Repeat until it finds the culprit commit.

# Start the hunt
git bisect start

# Mark current HEAD as bad
git bisect bad

# Mark the last known good commit
git bisect good v2.4.0

# Git checkpoints you to midpoint
# Test. Then mark:
git bisect good  # tests pass
git bisect bad   # tests fail

That's it. That's the whole workflow.

Instead of 47 manual checkouts, I did 6. Six tests. Found the commit that broke everything. It was a configuration change someone made at 11 PM. Added a required field to user validation. Didn't update the test fixtures.

Total time: 20 minutes.

The binary search math is why it works so fast. N commits needs at most log2(N) tests. Forty-seven commits needed 6 tests instead of 47. That's the difference between an hour of waiting and five minutes.

Git bisect sounds intimidating because "binary search" sounds technical. It's not. You run two commands to start, then one command after each test. Comes with git. No plugins needed.

# Cleanup when done
git bisect reset

test

Nico Reyes — Mon, 06 Apr 2026 19:26:10 +0000

test body

Python multiprocessing crashed on my custom class. Took forever to figure out why.

Nico Reyes — Mon, 06 Apr 2026 19:18:36 +0000

So I had this script that processed a bunch of images. Image processing, right, perfect use case for multiprocessing. Split the work across 8 cores, should be way faster.

First attempt:

from multiprocessing import Pool

def process_image(image_path):
    # do stuff
    return processed

with Pool(8) as pool:
    results = pool.map(process_image, image_paths)

Looked fine. Ran it.

Crashed immediately. Pickle error.

Cannot pickle <class 'MyImageProcessor'>

My what now?

So here's the thing I forgot. Multiprocessing in Python uses pickle to send data between processes. Functions get pickled, arguments get pickled, results get pickled. Everything gets pickled.

And my image processor class? Not pickleable.

Tried a few things:

Made the class simpler
Tried with just a function
Switched to threading instead

Threading worked but was slow as hell because of the GIL. Image processing is CPU intensive, threading doesn't help there.

Finally figured out the fix. The class held some state that couldn't be pickled. Moved the unpickleable stuff to getstate and setstate methods, or just initialized it inside the worker function instead of passing the whole object.

Working version:

from multiprocessing import Pool

def init_worker():
    # Initialize any unpickleable objects here
    global processor
    processor = MyImageProcessor()

def process_image(image_path):
    global processor
    return processor.process(image_path)

if __name__ == "__main__":
    with Pool(8, initializer=init_worker) as pool:
        results = pool.map(process_image, image_paths)

Using the initializer pattern means each worker process sets up its own processor. No pickling needed, no crashes.

Alternatively, if your objects are simpler, just use functools.partial:

from functools import partial
from multiprocessing import Pool

def process_image(processor, image_path):
    return processor.process(image_path)

processor = MyImageProcessor()
bounded_func = partial(process_image, processor)

with Pool(8) as pool:
    results = pool.map(bounded_func, image_paths)

This only works if the processor itself is pickleable of course.

Still not sure why Python's multiprocessing defaults to pickle instead of something more forgiving. The error message is garbage too, just points at the class without explaining what's actually blocking serialization.

Made the script work in the end. Still annoyed about losing a few hours to this.

Got CAPTCHA'd on page 47. Every single time.

Nico Reyes — Mon, 06 Apr 2026 19:12:23 +0000

I thought I was being clever with my scraping setup. 10 requests per minute, rotating user agents, residential proxies. Page 47, CAPTCHA. Page 48, CAPTCHA. Page 49, CAPTCHA. Fun times.

The site decided I was a bot once I hit some threshold. Didn't matter that I was going slow. Didn't matter that I looked like a real browser. CAPTCHA wall, every time, starting at page 47.

First thing I tried: more delays. 30 seconds between requests. Still got CAPTCHA'd on page 47. Interesting.

Then I tried different proxy providers. Three of them. Same result, same page number. At this point I was convinced the threshold was tied to my IP somehow, but no, same thing happened with fresh IPs.

Turns out the site just really, really doesn't like automated scraping. And I was too stubborn to give up.

Ended up biting the bullet and paying for 2captcha. If you're not familiar, they solve CAPTCHAs using actual humans (or very good models, unclear). The integration looked like this:

import time
from twocaptcha import TwoCaptcha

solver = TwoCaptcha('YOUR_API_KEY')

def scrape_with_captcha_handling(url):
    response = requests.get(url, headers=headers)

    if 'captcha' in response.text:
        # Find the CAPTCHA image/sitekey
        captcha_element = response.text.find('data-sitekey')
        sitekey = extract_between(response.text, 'data-sitekey="', '"')

        # Send to 2captcha
        result = solver.turnstile(sitekey=sitekey, url=url)

        # Retry with token
        response = requests.post(
            url + '/verify',
            data={'captcha_token': result['code']}
        )

    return parse_page(response)

Not exactly elegant. But it worked.

The key insight that nobody wants to hear: if a site throws CAPTCHAs at you, maybe just don't scrape it. I know, I know, not helpful. But honestly? If a site is that aggressive about blocking bots, there's usually a reason. Either they're protecting something valuable, or they're worried about liability, or both.

That said, if you really need the data and the site doesn't offer an API, 2captcha is there as a last resort. Costs money, adds latency, but you get your data.

Still worth it in my case tho

npm install worked. CI passed. Production was empty.

Nico Reyes — Mon, 06 Apr 2026 18:59:07 +0000

npm install worked. CI passed. Production was empty.

Last month. Client says search is broken. I check. Search endpoint returns 404.

Weird thing is the code definitely existed. Search component imported SearchService. SearchService called GET /api/internal/search. Route just... wasn't registered anywhere in Express.

Tracing it down

I went through it line by line. Search imports SearchService. Service makes that call. Where does it go?

Nowhere. Wasn't in app.use anywhere.

Local worked because of some weird hot reload setup I had from debugging. That route existed because I'd imported it in a test file during a spike. Never made it to the actual production router.

Why CI didn't catch it

npm install. npm run build. Tests pass. Image pushes. Server pulls. Container restarts. All green.

Tests mocked the service layer directly. Never actually called the endpoint through Express. Mock said "here's what SearchService returns" and the test believed it.

So route registration was never validated. Ever.

The fix was literally one line:

app.use('/api', searchRouter)

That's it. Missing line. 404 on production.

Still annoyed about it

Took longer than I'd like to admit. Added that one line, worked immediately.

Now I have smoke tests that hit real endpoints. At least one test that validates route registration actually exists. Should've had that before.

I wasted a weekend building a link checker. Manual clicking would've been faster.

Nico Reyes — Mon, 06 Apr 2026 18:54:59 +0000

Found 800 URLs in my old blog export. Some were broken. Clicking manually was not happening.

Started simple: fetch each URL, check if it returns an error.

import requests

with open('urls.txt') as f:
    urls = [line.strip() for line in f]

for url in urls:
    r = requests.get(url, timeout=10)
    if r.status_code >= 400:
        print(f'broken: {url}')

Ran it. Got rate limited after 20 requests. Added a delay.

import time

for url in urls:
    r = requests.get(url, timeout=10)
    if r.status_code >= 400:
        print(f'broken: {url}')
    time.sleep(1)

Better. Site stopped blocking me. Script ran for a few hours and found 47 broken links.

Cool right? Done.

Except I went back to manually check a few of the "working" URLs and some were clearly dead. What.

Turns out the script was stopping at redirects. If URL A redirected to URL B, it only checked if A existed. URL B could be 404 and nobody would know.

So I rewrote it to follow the whole chain.

import requests

def check_url(url):
    current = url
    while True:
        r = requests.get(current, timeout=10, allow_redirects=False)
        if r.status_code >= 400:
            return False, current
        if not r.is_redirect:
            return True, None
        current = r.headers['Location']

for url in urls:
    works, failed_at = check_url(url)
    if not works:
        print(f'broken at {failed_at}: {url}')

This traces the full redirect chain. If anything in the chain returns an error, it reports which URL actually failed.

Found about 20 more broken links that the first version missed. These were URLs that technically "worked" but only because they redirected somewhere that didn't exist.

So yeah. 67 broken links total instead of 47. Fun times.

The script takes forever now. Hours for 800 URLs. But it's accurate, and I'll have it for the next time I need to check links.

Manual clicking would've taken maybe 2 hours. This took a full weekend.

Not worth it. But I have a script that works now so whatever, next time will be faster

I tried 3 ways to validate Python function arguments. 2 of them lied to me.

Nico Reyes — Mon, 06 Apr 2026 17:58:26 +0000

Every Python project hits this moment. You need a function that only accepts valid input. Easy right? Just add some checks at the top.

That's what I thought anyway.

Building an expense tracker. Function takes a list of transactions, each with amount and category. Categories had to be from a fixed set: food, transport, entertainment, utilities. Amounts had to be positive numbers.

Basic validation. Should take 10 minutes.

Took 4 hours.

First, I tried type checks:

def add_expenses(transactions):
    if not isinstance(transactions, list):
        raise TypeError("Transactions must be a list")
    for t in transactions:
        if not isinstance(t, dict):
            raise TypeError("Each transaction must be a dict")
        if 'amount' not in t or 'category' not in t:
            raise ValueError("Missing required fields")
        if not isinstance(t['amount'], (int, float)):
            raise TypeError("Amount must be number")
    return process(transactions)

Looked solid. Types are right, required fields exist.

Ran the tests. All green.

Shipped it.

User submitted {"amount": -50, "category": "food"}. Negative amount. Type check passed. Negative numbers are valid int and float after all.

My validation said "this is fine" while the data was completely wrong.

Okay, value validation time:

VALID_CATEGORIES = {'food', 'transport', 'entertainment', 'utilities'}

def add_expenses(transactions):
    if not isinstance(transactions, list):
        raise TypeError("Transactions must be a list")
    for t in transactions:
        if not isinstance(t, dict):
            raise TypeError("Each transaction must be a dict")
        amount = t.get('amount')
        category = t.get('category')
        if amount is None or category is None:
            raise ValueError("Missing required fields")
        if not isinstance(amount, (int, float)):
            raise TypeError("Amount must be number")
        if amount <= 0:
            raise ValueError("Amount must be positive")
        if category not in VALID_CATEGORIES:
            raise ValueError(f"Category must be one of {VALID_CATEGORIES}")
    return process(transactions)

Better. Now checks that amounts are actually positive. Now checks that categories are in the valid set.

Tests passed again.

User submitted {"amount": 0, "category": "food"}. Zero dollars. Is zero a valid expense? Depends on your business logic, but in this case, no. Zero means the transaction wasn't recorded properly. Our validation said "this is fine" again.

Wait, I wrote if amount <= 0. That's <=, not <. So zero would pass. Fixing that.

Except I also forgot that isinstance(50.0, int) returns False. Fifty dollars as a float. Fifty cents as 0.50. Both valid amounts. But my check said floats that aren't integers are fine, while actual integers that are zero are caught.

Third time's the charm right? Pydantic:

from pydantic import BaseModel, field_validator

class Transaction(BaseModel):
    amount: float
    category: str

    @field_validator('amount')
    @classmethod
    def amount_must_be_positive(cls, v):
        if v <= 0:
            raise ValueError('Amount must be greater than zero')
        return v

    @field_validator('category')
    @classmethod
    def category_must_be_valid(cls, v):
        if v not in VALID_CATEGORIES:
            raise ValueError(f'Category must be one of {VALID_CATEGORIES}')
        return v

class TransactionList(BaseModel):
    transactions: list[Transaction]

def add_expenses(data):
    validated = TransactionList(transactions=data)
    return process([t.model_dump() for t in validated.transactions])

This time the validation actually works at the value level. Not just checking types but checking that the types have the right values. Amount 0 or -50 or "fifty" (yes someone tried it) all fail before even reaching my code.

Tests passed. User tested it. Negative amounts rejected. Zero rejected. Invalid categories rejected. String amounts rejected.

Worked.

Honestly I should have just used Pydantic from the start. Spent 3 hours writing validation that accomplished less than what a library does for free.

Git made me resolve 47 commits worth of conflicts. Here's what happened.

Nico Reyes — Mon, 06 Apr 2026 17:54:34 +0000

Git made me resolve 47 commits worth of conflicts. Here's what happened.

Friday afternoon. Pushed some code to GitHub, opened a PR, feeling good about myself. Client wanted to see a demo Monday morning.

Monday I open the repo, switch to main, pull latest.

47 commits behind.

Cool cool cool.

So here's the setup. Multiple devs, feature branches, normal stuff. I was working on auth-redesign for about a week. Made changes, pushed, opened PR.

Except the PR had conflicts. I thought "no big deal, fix them after the demo, code works fine for what client needs."

This was a mistake.

I let the branch drift for three days. Just kept making commits while the team kept merging to main. Didn't touch main once during that time.

Monday morning my local main and the actual main were basically different universes. Someone had force-pushed while I was gone. When I tried to merge my feature back in, git saw 47 commits worth of differences.

I had two options.

Option A was to merge main into my branch, resolve conflicts, then merge back. This works but gives you ugly merge commits everywhere. Your git log looks like spaghetti.

Option B was to rebase:

git fetch origin
git checkout main
git reset --hard origin/main
git checkout auth-redesign
git rebase main

This replays my commits on top of current main. Clean history.

But rebase replays each commit one by one. And if any of those commits conflict with the new base, you resolve manually. For each one. Commit 1, resolve, commit 2, resolve, commit 3, resolve.

I had 47 commits. Some of them conflicted. Some of the same files conflicted across multiple commits because I had made similar changes in consecutive commits.

By commit 30 I was fixing the same three files over and over.

2 hours. Not exaggerating. 2 hours of git conflict resolution. At one point I had five files open with conflict markers and I couldn't remember what the original code was supposed to do.

The problem wasn't git. It was that I let my branch drift.

When you don't sync with main for days, the gap grows. A rebase that could be 3 commits becomes 20 becomes 47 because you kept making small commits without pulling in the latest changes.

Now every morning before I start anything:

git fetch origin
git rebase origin/main

Even when I'm not ready to merge. Even when I'm deep in feature work. Keeps everything current so when I am ready, it's a couple commits instead of 47.

Sounds simple. Would have saved me 2 hours.

Site changed their HTML at page 200. Scraping 500 pages took 3 attempts.

Nico Reyes — Mon, 06 Apr 2026 17:50:27 +0000

Last week I'm running this scrape job, right. Everything going smooth, page 200 hits and suddenly every single field extraction returns null.

First thought was my code broke. Checked the code, looked fine. Checked the URL, still valid. Checked if the site was down, it wasn't. Pulled up page 200 in my browser and the HTML looked completely different in the devtools.

Turns out the site had two different page templates. Most pages used Template A. Somewhere around page 180-200 they switched to Template B. Same URL structure, same CSS classes in the HTML, but the actual DOM hierarchy was totally different.

The annoying part was figuring out exactly which page changed. Can't just eyeball it when you have 500 pages to deal with. Wrote a quick script to flag any page where my extraction count dropped below a threshold.

from pathlib import Path
import re

# Find pages with suspiciously low extraction counts
pattern = r'page_(\d+)\.html'
extracted_counts = {}

for html_file in Path('scraped_pages').glob('*.html'):
    match = re.search(pattern, html_file.name)
    if match:
        page_num = int(match.group(1))
        content = html_file.read_text()
        # Count product containers
        count = content.count('class="product-card')
        extracted_counts[page_num] = count

# Flag anomalies
for page_num, count in sorted(extracted_counts.items()):
    if count < 5:  # Normal is 15-20 per page
        print(f'Suspicious: page {page_num} only extracted {count} items')

Found 47 pages with low extraction counts. The actual boundary was page 187. Not 200. Not 180. Page 187 exactly.

Why page 187? No idea honestly. Maybe they ran a split test and page 187 was when the new template shipped to 100% of users. Maybe someone pushed a template change without checking existing scraped pages. Could be aliens for all I know.

Fix was simple enough, two extraction functions and route based on page number.

def extract_products(html, page_num):
    if page_num < 187:
        return extract_template_a(html)
    else:
        return extract_template_b(html)

Three passes to get clean data. Still annoyed about it.

CSV quote marks broke my parser. 2,000 records silently corrupted.

Nico Reyes — Mon, 06 Apr 2026 17:40:48 +0000

Got a CSV from the marketing team last week. Simple request: clean it up and import into the database.

Opened it in Excel first (mistake #1). Looked fine. 2,000 rows, standard product data. SKU, name, price, quantity.

Wrote a quick parser in Python.

import csv

with open('products.csv', 'r') as f:
    reader = csv.DictReader(f)
    for row in reader:
        print(row)

Printed fine. Pushed to the database.

Two hours later, the frontend stopped rendering prices correctly.

Debugging time.

Turns out the CSV had quote marks inside text fields. The product name column had entries like:

"Ergonomic Desk Chair (with "ergonomic" mesh)"

My parser treated that inner quote as the field terminator. Everything after "with " got truncated. So instead of "Ergonomic Desk Chair (with ergonomic mesh)" we got "Ergonomic Desk Chair (with ".

The database had half product names. The frontend was trying to parse "$" and failing.

Excel had hidden this from me. It auto-escapes quotes in the display. The CSV looked clean but the actual file content was a mess.

Fixed it with:

import csv

with open('products.csv', 'r') as f:
    reader = csv.DictReader(f, quotechar='"', doublequote=True)
    for row in reader:
        print(row)

Wait, that still didn't work. The actual fix needed csv.QUOTE_ALL behavior or manual preprocessing.

Ended up with:

import csv

with open('products.csv', 'r') as f:
    content = f.read()
    # Escape unescaped quotes manually
    lines = []
    for line in content.split('\n'):
        # Handle cases where quotes aren't properly escaped
        fixed = line.replace('""', '<<QUOTE>>')
        fixed = fixed.replace('"', '""')
        fixed = fixed.replace('<<QUOTE>>', '""')
        lines.append(fixed)

    reader = csv.DictReader(lines, quotechar='"', doublequote=True)
    for row in reader:
        print(row)

Not pretty but it worked.

I'm just gonna validate CSV files from now on. Every single one. And marketing saying it's "simple" before sending it is basically a red flag at this point honestly.

Python threw 'TypeError'. Took 4 hours to realize what it actually meant.

Nico Reyes — Mon, 06 Apr 2026 16:58:11 +0000

Python threw 'TypeError'. Took 4 hours to realize what it actually meant.

Python exceptions look helpful until they lie to you.

Had a function that took two arguments. Called it with a dict and a list. Python said TypeError, expected something else. Simple enough, right?

Except it wasn't.

The setup

I'm building this small script to merge two data sources. One gives me a dict of user IDs to emails. The other gives me a list of tuples with (user_id, signup_date). I need to combine them into one clean structure.

def merge_users(user_data, signup_list):
    result = {}
    for user_id, email in user_data.items():
        result[user_id] = {
            'email': email,
            'signup_date': None
        }

    for user_id, date in signup_list:
        if user_id in result:
            result[user_id]['signup_date'] = date

    return result

Classic stuff. Straightforward.

Then I call it:

user_emails = {'u1': 'alice@example.com', 'u2': 'bob@example.com'}
signups = [('u1', '2026-01-15'), ('u2', '2026-02-20')]

merged = merge_users(user_emails, signups)

TypeError. Something about unhashable type 'list'.

What I tried

First thought: I probably passed something wrong. Checked my call. User_data is a dict, signup_list is a list of tuples. Should work.

Maybe the dict has nested lists as keys? Double-checked. Nope, just normal string keys.

Maybe Python version issue? I was on 3.11, tried 3.12. Same error.

Spent an hour looking at the function. Then I decided to simplify. Strip out everything except the essentials. Make it as barebones as possible.

Still failing.

That's when it hit me. The error wasn't coming from my function. It was coming from somewhere else in the call stack.
Turns out the actual call was wrapped in some validation helper I forgot I had. It was doing something like data.get('users', {}).items() but data was sometimes returning a list instead of a dict because of some legacy field naming.

Python's error message pointed at my merge function because that's where the exception propagated. But the actual problem was three stack frames earlier.

The fix

# Before (broken)
data = some_api_call()
users = data.get('users', {})
merged = merge_users(users, signups)

# After (working)
data = some_api_call()
users = data.get('users') or {}
# Also handle the case where users might be a list
if isinstance(users, list):
    users = {u['id']: u['email'] for u in users}
merged = merge_users(users, signups)

Finding that took forever because I trusted the error message too much. Still annoyed about it tbh.

Docker said 'healthy'. My app said 'connection refused'. 4 hours gone.

Nico Reyes — Mon, 06 Apr 2026 16:45:01 +0000

Docker said 'healthy'. My app said 'connection refused'. 4 hours gone.

Last week I deployed something that should've been straightforward. Docker Compose, two containers, one talks to the other. Classic setup. Everything looked fine in docker ps. Both containers "healthy". Except my API couldn't reach the database and had no idea why.

What I saw

docker ps showed both containers up. Postgres on port 5432, my API on 3000. STATUS said "healthy" for both.

My app starts up, tries to connect to postgres:5432, and immediately throws "connection refused".

Tried localhost:5432. Tried 127.0.0.1:5432. Tried the container IP I found via docker inspect. Nothing worked.

What I did wrong first

Restarted everything. Updated docker-compose. Pulled fresh images. Wrote a test script to verify the port was open. That script worked fine when run locally but my actual app still couldn't connect.

At this point I'm googling "docker connection refused postgres" and trying every solution. Most of them assume you haven't already verified the port is accessible.

Then I noticed something in docker network inspect. My containers were on different networks.

The actual problem

My docker-compose had something like:

services:
  api:
    build: ./api
    ports:
      - "3000:3000"
    networks:
      - frontend
  postgres:
    image: postgres:15
    ports:
      - "5432:5432"
    networks:
      - backend

Api on frontend network. Postgres on backend network. They couldn't talk to each other even though both were running on the same Docker host.

The health check passed because it was just checking if postgres process was alive, not whether other containers could reach it.

The fix

services:
  api:
    build: ./api
    ports:
      - "3000:3000"
    networks:
      - frontend
      - backend
    depends_on:
      - postgres
  postgres:
    image: postgres:15
    ports:
      - "5432:5432"
    networks:
      - backend

networks:
  frontend:
  backend:

Put both on the same network. Restarted. Worked immediately.

The kicker

Docker Compose doesn't automatically put all containers on the same network. Each service only gets a network if you explicitly assign it. If you add a service without specifying networks, you get whatever the default is. And if you have multiple services with no explicit assignment, they might not share a network depending on how your Compose file is structured.

Health checks lie sometimes too. A container can be "healthy" while the actual application inside it can't do its job. The health check is scoped to what you tell it to check, not real world connectivity.

"Connection refused" is technically accurate but not that helpful when you're debugging across multiple layers. Wrong port, wrong host, network isolation. All give you the same error.

Been explicit with networks ever since. Every service gets assigned, nothing is left to defaults. Learned this one the hard way.