Forem: GasPriceCheck

Google deindexed half of my Next.js site. Here's the four-phase recovery.

GasPriceCheck — Thu, 07 May 2026 05:00:00 +0000

I run a side project: a gas price finder that's mostly programmatic content. Roughly 33,620 ZIP code pages plus a few hundred state and city pages, all built on Next.js 15 with ISR. It runs on Vercel, fronted by Cloudflare for DNS.

For about eight months it was indexing fine. Traffic was small but growing. Then on April 11 I checked Google Search Console and saw something I'd never seen at this scale: 87 URLs flagged as "not found (404)," 61 flagged as "soft 404," and a chunk of others sitting in "crawled, currently not indexed."

I'm a data analyst. This is my side project, not my day job. So I had a weekend, a coffee subscription, and the GSC export. Here's the four-phase recovery, what each phase actually fixed, and the things I'd do differently.

What the damage actually was

GSC's URL Inspection tool is the only way to figure out what Google thinks of any specific URL. The 87 hard 404s broke down into two groups when I sampled them:

Stale city slugs. URLs like /austin or /dallas that I had in my sitemap historically but removed when I migrated to disambiguated slugs (/austin-tx, /dallas-tx). Google still had the old slugs cached and was hitting 404s when it tried to refresh.
Typo URLs in third-party referrers. A handful of blog posts I'd written had link typos to my own site. Google followed those, hit 404s, and now thought those URLs were canonical.

The 61 soft 404s were a different beast. These were real ZIP code pages that returned 200 OK but Google decided didn't have enough content to be "real" pages. Looking at the SSR output, I could see why: when a ZIP wasn't cached in Redis (which was most of the long tail), the page rendered a hero, a search box, and a thin "search results loading" placeholder. About 640 visible words in the SSR HTML. From Google's perspective: this is a glorified 404 wearing a 200 costume.

The "crawled, currently not indexed" pile was the soft-404 pile's pre-stage. Google had crawled them, decided they weren't worth indexing, but hadn't formally classified them as soft 404s yet.

That's the diagnosis. Now the fixes.

Phase 1: The boring redirect fix

This phase is unglamorous and important. For each of the 87 hard 404s, I had to decide: does this URL have a clear successor, and if so, where do I redirect it?

I did this in next.config.mjs:

const STALE_CITY_REDIRECTS = [
  { source: '/austin', destination: '/austin-tx', permanent: true },
  { source: '/dallas', destination: '/dallas-tx', permanent: true },
  // ...64 more
];

module.exports = {
  async redirects() {
    return STALE_CITY_REDIRECTS;
  },
};

permanent: true emits a 308. That tells Google "this URL has moved permanently, transfer signals to the destination." A 301 would also work; I went with 308 because Next.js's defaults align with that and I didn't want to fight the framework.

The unglamorous part: most of the 87 URLs didn't have an obvious destination. Some pointed to cities I no longer covered. Some pointed to ZIPs that didn't exist. For those, I redirected to the parent state page (e.g. /austin-foo → /texas) so the user lands somewhere useful, and the link equity isn't dropped on the floor.

Total output of Phase 1: 66 redirects, deployed in one commit. Pushed it. Watched GSC for a week.

Result: about half the hard 404s validated. The other half stuck. Why?

Phase 1.5: The Cloudflare redirect chain I didn't know I had

When I inspected one of the still-failing URLs in GSC, the URL Inspection tool said "page with redirect" and showed a chain. Google was hitting http://gas-price-check.com/austin, getting redirected to https://www.gas-price-check.com/austin, and then redirected again to https://www.gas-price-check.com/austin-tx.

Two hops. Google does not like two hops.

I had Cloudflare in front of Vercel as a DNS proxy. Cloudflare was handling the http → https and apex → www redirect. Vercel was handling the slug → slug redirect. Each layer worked. Together they made a chain.

The fix was a single Cloudflare Redirect Rule that does both transforms in one hop:

If: hostname matches "gas-price-check.com" AND scheme is http
Then: 308 to https://www.gas-price-check.com/$path

After that rule landed, the chain collapsed from 2 hops to 1. Google reprocessed and started clearing the rest of the hard 404s. Lesson I should have known: when you have multiple layers (CDN, edge, framework), each one defaulting to "I'll handle the redirect" stacks. Audit the hop count with curl -IL <url> and look for chains.

Phase 2: The real work (thin SSR content)

The 61 soft 404s couldn't be redirected away. These were real pages I wanted indexed. Google just thought they were thin.

The diagnosis was straightforward once I started reading my own SSR output. When a ZIP wasn't cached, the page rendered:

A hero with the ZIP, city, state
A search box
A "loading prices..." placeholder (waiting for client-side fetch)
A footer

That's it. Maybe 640 visible words, of which 400 were the footer and global nav. The actual page-specific content was a hero header and a placeholder.

The fix had three sub-components.

2a. Server-render the nearby ZIP grid

I had a helper called getNearbyZips(zip, radius) that returned ZIP codes within a given mile radius. I'd been using it on the client. I moved it to the server component so the SSR HTML included an actual grid of "nearby ZIP codes" with links.

This added about 80 words of unique-per-ZIP content (different neighbors for each ZIP). More importantly, it added 8-12 internal links per page, which gave Google more signal about the URL's place in the site graph.

// Before: client-side, invisible to Google
const nearby = useNearbyZips(zip);

// After: server-rendered, visible to Google
const nearby = await getNearbyZips(zip, 25);
return (
  <section>
    <h2>Nearby ZIP codes</h2>
    <ul>
      {nearby.map(z => <li key={z}><Link href={`/${z}`}>{z}</Link></li>)}
    </ul>
  </section>
);

2b. Unconditional save tips

I added a hand-written "How to save on gas in {city}" section that rendered regardless of cache state. About 120 words of static-but-locally-relevant content per city. This is templated, but with enough variable interpolation that no two pages have identical text.

2c. State backlink with name fallback

Every ZIP page already had a "Back to {state} state guide" link, but it relied on a state abbreviation lookup that returned null for some edge cases. So those pages were rendering "Back to undefined state guide" or worse, no link at all. Fixed it with a fallback:

const stateName = getStateByAbbr(state) ?? getStateByName(state) ?? state;

Small fix, but it meant every ZIP page now had a working internal link to its parent state, which closes a major site-graph gap.

After Phase 2, my SSR word count went from 640 to about 890 on previously-thin pages. That's the threshold I cared about. Google's "soft 404" verdict is based on relative content depth, not an absolute word count, but more depth is always better than less.

Phase 3: The geocoding gap I didn't know I had

While I was at it, I noticed something weird. Some ZIP pages were rendering with lat/lng of (0, 0). This made the distance calculations on the page nonsensical ("nearest gas station: 8,247 miles away"). It also meant the "nearby ZIPs" grid was showing up empty for those pages.

The cause: my ZIP-to-lat/lng resolver had a single source: zippopotam.us. It's free, fast, and most of the time correct. But for some valid US ZIPs (75072 in McKinney TX, for one), it returns a 404.

I rebuilt the resolver as a 4-tier fallback chain:

zipContent.json (a static file with 33,620 ZIPs and pre-resolved coords)
Redis cache (per-request resolved coords)
zippopotam.us API
Nominatim API (slower but covers the gaps)
Placeholder (0, 0) with degraded behavior

I'll write up the chain in detail in a separate post. The point for this post: when GSC flagged these pages as soft 404s, the broken geocoding was part of the picture even though it wasn't the headline issue.

Phase 7B: The second sweep

Three days after deploying Phases 1 through 3, I ran the GSC validation again. About 80% of the URLs had cleared. Some hadn't. So I wrote a script (find-redirect-candidates.js) that programmatically tested every plausible city slug variant against the live site:

const variants = [
  `/${city}`,
  `/${city}-${state}`,
  `/${city.replace(' ', '-')}`,
  `/cheap-gas-${city}`,
  // ...
];

for (const v of variants) {
  const res = await fetch(`https://www.gas-price-check.com${v}`);
  if (res.status === 404) console.log(v);
}

This caught 35 more 404s I hadn't found in the GSC export. Stale links in old blog posts I'd written, third-party links from a directory submission I'd forgotten about, typo URLs in my own social media posts. Each one got a redirect.

Phase 7B added 35 new redirects, bringing the total to 101. I deployed those, and the second GSC validation came back clean.

Phase 8: Per-state context (the depth fix that actually moved the needle)

After all the redirects and SSR enrichment, I still had some pages stuck in "crawled, currently not indexed." Word count was up. Internal links were up. But Google was still skeptical.

The thing I hadn't done: make the templated content actually different across pages. My "save tips" section was different by city, but my page-level content above the fold was nearly identical. A page about ZIPs in California and a page about ZIPs in Maine had no state-specific context.

I built a stateContext.ts module:

const STATE_CONTEXT = {
  CA: "California's gas prices are shaped by the state's unique CARB...",
  TX: "Texas typically has some of the lowest gas prices in the country...",
  // ...15 hand-written for top-traffic states
  // ...35 generated from a parameterized template for the rest
};

export function getStateContext(state: string): string {
  return STATE_CONTEXT[state] ?? defaultContext(state);
}

The 15 hand-written paragraphs are 80 to 120 words each. They explain the state's gas tax, refinery capacity, regulatory regime, and seasonal pricing patterns. These are the things you'd say to a friend if they asked "why are gas prices weird in California?"

The other 35 states get a templated paragraph with state-specific variables (avg price, neighboring states, gas tax rate). Templated, but with enough variation that each is unique.

After deploying Phase 8, the previously-stuck pages started getting indexed within 10 days. Not all of them. But enough that I stopped worrying about them.

Final SSR word count on a representative ZIP page (77386, Spring TX) when measured post-deploy: 810 visible words, up from 635 before Phase 8 alone. The whole journey took it from 640 to 810: a +170 word lift made of mostly per-state context plus the SSR enrichment.

The diagnostic that lied to me

One quick aside that became its own post. While diagnosing the soft 404s, I wrote a script to grep the SSR output for content markers ("does this page have an EIA average price rendered?"). The script reported zero matches across 14 URLs. I spent an hour debugging the data layer before realizing the script was broken.

The cause: React inserts HTML comments () between adjacent text expressions during SSR. My regex was using [^<]+ which fails immediately when the next character is <. The data was rendering correctly the whole time. My detector was the bug.

I wrote that one up separately. The summary: strip HTML comments before any content matching on Next.js SSR output.

The numbers

Before:

87 hard 404s in GSC
61 soft 404s in GSC
~640 SSR words on cached-miss ZIP pages
1 source of truth for ZIP geocoding (and gaps)
2-hop redirect chain from http://apex/* to https://www/*

After:

0 hard 404s (101 redirects in next.config.mjs, 1 Cloudflare Redirect Rule)
0 soft 404s after Phase 8 (per latest GSC validation)
~810 SSR words on the same pages
4-tier geocoding chain with 100% coverage of 33,620 ZIPs
1-hop redirect

Total commits: 7. Total deploy waves: 3 (April 17, April 23, April 27). Total weekend hours: I stopped counting somewhere around 14.

Things I'd do differently

If I were starting this side project again:

Set up GSC URL Inspection alerts before launch. I had GSC connected but wasn't watching it daily. The 404 problem accumulated for weeks before I noticed.
Add a curl -IL check to my deploy script. A redirect chain check would have caught the http+https+slug double-hop before it became a Google problem.
Server-render the unique parts of every templated page from day one. Anything that varies per page should be in the SSR HTML, not the client bundle. Loading states are the enemy of programmatic SEO.
Hand-write the top 10-20% of templated content. The remaining 80% can be generated, but the long-tail-of-the-long-tail is where Google will smell templating and dock you. Hand-writing the highest-traffic variants is high leverage.
Keep a diff log of redirect rules. Mine grew to 101 entries in next.config.mjs and I'd already lost track of which ones I added when. A separate JSON file with timestamps would have been smarter.

What I learned about Google's verdict mechanism

Three things I genuinely didn't know before this:

Soft 404 is sticky. Once Google decides a page is a soft 404, fixing the page doesn't immediately clear the verdict. You have to ask GSC to re-validate, wait 1-4 weeks for re-crawl, and accept that some of those URLs will not come back even after you fix them. The verdict has memory.

"Crawled, currently not indexed" is the bench. Google has a finite indexation budget per site. Pages they don't think are worth indexing go on the bench. You can move pages off the bench by improving them, but it's not automatic and it's not fast.

Internal redirect hops add up. Each hop is a small signal-loss for Google. If your CDN does one redirect and your framework does another, you're below capacity even if both are individually correct. Audit your hop count.

If you've been through a similar programmatic-SEO recovery, I'd love to hear what your phase breakdown looked like. Mine was four phases over two weekends. The longest phase by far wasn't the bulk redirects (that was a few hours). It was Phase 8: the part where I had to admit my templated content was actually pretty thin and rewrite the per-state context by hand.

Templating gets you to 33,620 pages fast. Earning the right to keep those pages indexed takes longer.

Why your `[^<]+` regex is silently breaking on React SSR output

GasPriceCheck — Thu, 30 Apr 2026 06:02:34 +0000

Picture this. You've shipped a programmatic SEO site, a few thousand pages of templated content. Google flags 14 URLs as soft 404s. You write a quick diagnostic: hit each URL, fetch the SSR HTML, check for a few content markers (a price string, a state average, a section header). Confirm what's really rendering, fix what's missing, move on.

That was the plan. Forty minutes in, my script told me 0 of 14 pages had any EIA price data rendered. None. I was about to dig into the data fetching layer when something nagged at me. I curled one of the URLs by hand. The price was right there in the HTML. Plain text, easy to spot.

The script was lying. The regex was lying.

It took me longer than I want to admit to figure out what was happening, so here's the writeup so you don't burn the same hour.

The regex that "worked"

The diagnostic was straightforward. For each ZIP code page, I wanted to detect whether the EIA state average price was rendered in the SSR HTML. The component looked roughly like this:

<p>
  Texas average: ${eiaAverage} per gallon as of {eiaDate}.
</p>

So I grepped the SSR output for the marker pattern:

const html = await fetch(url).then(r => r.text());
const match = html.match(/Texas average: \$([^<]+) per gallon/);

The capture group uses [^<]+ to pick up "everything until the next tag." Standard pattern. I've used variations of this in dozens of throwaway scrapers.

Across all 14 URLs, this regex returned no match. Not "match with empty capture." No match at all.

Meanwhile, when I curled the same URL and read the response, the rendered text was right there:

Texas average: $2.97 per gallon as of 4/22/2026.

How does a regex miss text that's literally in the string?

What React is actually putting in your HTML

Here's the thing nobody told me about server-rendered React. When you have multiple adjacent text expressions inside a single element, React inserts an HTML comment as a hydration boundary marker. The actual SSR output for that paragraph looked like:

<p>Texas average: $<!-- -->2.97<!-- --> per gallon as of <!-- -->4/22/2026<!-- -->.</p>

Each {expression} interpolation gets bracketed by . This is intentional. React uses these markers during hydration to know where one text node ends and the next one begins. Without them, React can't reconcile the SSR text with what its virtual DOM says should be there, because adjacent text nodes get merged in the browser.

So when my regex hits Texas average: $, the next character is < (start of the comment). The [^<]+ capture group requires at least one non-< character. It fails immediately. The regex moves on, finds no other "Texas average:" anchor in the page, and reports no match.

The price ($2.97) is in the HTML. It's just not adjacent to the anchor text. There's a comment node between them.

The fix

Strip the comments before any content matching. Two lines:

const html = await fetch(url).then(r => r.text());
const stripped = html.replace(/<!--[\s\S]*?-->/g, '');
const match = stripped.match(/Texas average: \$([^<]+) per gallon/);

Use [\s\S] rather than . because comments can span multiple lines and . in JavaScript doesn't match newlines without the s flag (and s flag support is patchy enough that I just default to [\s\S]).

Once I added that one line, the regex worked on all 14 URLs. The EIA data was rendering correctly the whole time. My "soft 404 root cause" was a bug in my diagnostic, not a real content gap.

Why this gotcha matters more than you think

The reason I'm writing this up: the failure mode is silent and high-impact.

If your regex returned an obvious garbage value, you'd debug it in five minutes. But mine returned no match at all. That's the same return value as "this content is genuinely missing." Which is the exact thing I was trying to detect. The diagnostic was indistinguishable from the bug it was looking for.

I trusted the output. I started forming hypotheses based on the output. "The EIA fetch must be failing on the server. Let me check my fallback chain. Let me check Redis." I burned an hour on those wrong paths because the symptom was confirmed by my (broken) detector.

Two takeaways that generalize beyond this exact case.

First, when you grep server-rendered React HTML, always strip comments first. Any regex with [^<]+, \w+, or word-boundary anchors will trip on the hydration markers. Some browser View Source viewers hide comments by default, which is part of why this gotcha is invisible. View the raw response with curl -s URL | head -200 and look for the  pattern. You'll see them everywhere, especially on text-heavy pages with lots of variable interpolation.

Second, validate your diagnostic before you trust its output. Run it against a known-good page. If your "missing content" detector reports content as missing on a page where you can visually confirm the content exists, your detector is broken, not the content. I should have done this sanity check before chasing fix hypotheses. I didn't, because the detector was "obviously simple."

This bit me on a Next.js 15 / React 18 project. I checked: the same hydration-comment behavior is documented in the Next.js docs as part of how React handles hydration. It's not going away. If you're parsing SSR HTML programmatically, assume comments are everywhere.

One last thing worth knowing

There's a sister gotcha. If you're using cheerio or another DOM parser instead of regex, you don't have this problem. The DOM parser walks the tree, and adjacent text nodes are joined when you call .text(). So $('p').text() returns "Texas average: $2.97 per gallon as of 4/22/2026." with no comment artifacts.

But if you're using regex (faster, simpler for one-off scripts), strip first, match second. Or write a one-line helper:

const stripComments = (html) => html.replace(/<!--[\s\S]*?-->/g, '');

Drop it at the top of every SSR-parsing script you write. Future-you will thank you.

That's the whole post. One bug, one fix, one hour I'm not getting back. Hope I saved you the same hour.

If you've hit a similar silent-failure-mode debugging story, drop it in the comments. I collect these.