Forem: Isaac Sunday

How I Built a Soccer Coach Contact Extractor for Messy Athletics Websites

Isaac Sunday — Tue, 24 Mar 2026 12:21:43 +0000

Most athletics websites look simple until you try to extract structured data from them at scale.

Coach pages are especially messy. One school gives you a clean staff directory with mailto: links. Another hides emails behind Cloudflare. Another puts names on the roster page and the actual contact info on a separate bio page. Another sends back an empty shell and expects JavaScript to do the rest.

That is what this project solves.

football-soccer-emails is a TypeScript-based extractor that pulls soccer and football coach contact information from athletics websites and turns it into structured records. It supports direct URLs, public Google Sheets, and an Apify workflow for batch runs.

The reason I built it this way is simple: I tried a version of this problem around 2017 or 2018 using heuristics only, and it was roughly 40% accurate. That was about as far as rules alone would take me. With LLMs and a multi-stage extraction flow, this same class of problem can now get into the 90%+ range.

The Core Idea

The project does not rely on one extraction method. It has three:

heuristic: free, pattern-based extraction for static sites
llm: AI-assisted extraction through OpenRouter
firecrawl: structured extraction through Firecrawl

That matters because athletics sites are not consistent enough for a one-size-fits-all scraper. Some pages are easy and should be handled with deterministic parsing. Some are messy but still understandable to an LLM. Some are JavaScript-heavy enough that you want a different extraction path entirely.

The main pipeline stays simple. It resolves input URLs, builds the selected extractor, fetches HTML when needed, and pushes normalized coach records into the output dataset. The interface is stable even when the extraction strategy changes underneath it.

What It Extracts

For each coach, the system tries to capture:

first name
last name
title or position
email
phone number
school
division or conference
source URL
profile URL
confidence levels for name, email, and phone

Those confidence levels are useful because not all matches are equal. A mailto: link is very different from an email pulled out of weak body text. I wanted the output to show that difference instead of hiding it.

In the repo, I used a 71-URL comparison batch to sanity-check cost and mode tradeoffs. The heuristic path was still useful, but it left too many messy pages unresolved. The LLM path closed most of that gap and was the difference between a pipeline that felt partly usable and one that could realistically clear 90%+ accuracy on this kind of data.

Why the Heuristic Layer Still Matters

It is easy to tell this story like the LLM replaced everything. That is not what happened.

The heuristic layer still matters because it is fast, free, and trustworthy on obvious patterns. If a page gives me a mailto: link, a tel: link, or a Cloudflare-protected email I can decode directly, I would rather use that than ask a model to interpret it.

The heuristic extractor is layered on purpose. It checks:

mailto: links
Cloudflare email protection
custom data attributes
JavaScript variable patterns
reversed text and simple obfuscation
plain body text
script tags
raw HTML

That list only exists because sports sites do weird things. The funny part is how often a page that looks like a soccer directory is also, somewhere in the HTML, a baseball page, a basketball page, and two hidden mobile layouts at the same time.

The Part That Actually Improved Accuracy

The biggest improvement was not just "use an LLM." It was combining multiple passes.

If the extractor finds a coach on a roster page but does not find an email, it can follow the coach’s profile URL and try again in profile mode. That matters because many athletics sites split their data across two layers:

roster page: names and titles
profile page: bio, phone, and email

Without that follow-up pass, you miss a lot of good data. With it, the extractor can keep the roster context and fill in the missing fields from the profile page.

There is also a guardrail I like here: if a model returns a profile URL on the wrong domain, the code rewrites it back onto the source site. That keeps the pipeline grounded in the actual athletics site instead of trusting a hallucinated cross-domain link.

Handling the Messy Cases

The project includes a few defensive pieces that came directly from running into bad pages over and over.

One is SPA detection. If the HTML looks like a React, Vue, or Angular shell instead of a real document, the heuristic extractor can fail explicitly instead of pretending there was just no data.

Another is hidden-content filtering. Athletics sites often include hidden sections for other sports, old layouts, or alternate mobile markup. If you do not filter that out, you can easily extract the wrong staff from the right page.

The fetch layer also uses TLS client fingerprinting instead of a plain default fetch stack. That helps with sites that behave differently when the client looks too synthetic.

None of that is glamorous, but it is the difference between a toy scraper and something you can run in bulk.

Built Around Spreadsheets, Not Just Code

I also wanted the workflow to be usable by someone who is not living in the codebase.

That sounds small, but it changes the project from "a scraper" into something an operator can actually use.

The project accepts input in three ways:

The project accepts:

direct URL lists
public Google Sheets
spreadsheet-driven workflows through Apify and Apps Script

That means the operating model can stay simple:

Put coach-page URLs into a sheet.
Run the extractor in the mode you want.
Get back structured coach records.

Why This Project Matters to Me

What I like about this build is that it settled a question I had back in 2017: could this problem ever be solved well without writing endless custom scraping rules?

Not in a vague "agentic" marketing sense. Not as magic. Just as a better way to handle messy, inconsistent data that used to force you into an endless loop of brittle rules.

The old version of this idea topped out around 40% because heuristics had to do everything. This version works because heuristics do the obvious work, LLMs handle ambiguity, and the pipeline follows up when the first pass is incomplete.

The next step is making that routing automatic so the system can decide, page by page, when cheap rules are enough and when a stronger extractor is worth using.

Never Miss a Reservation Again: Building an Automated Restaurant Booking Bot

Isaac Sunday — Sat, 22 Nov 2025 16:33:32 +0000

The Problem

We've all been there. That new restaurant everyone's talking about just opened reservations for next month, and within seconds, every slot is gone. You refresh frantically, check multiple times a day, and still end up empty-handed. The most sought-after restaurants in major cities can book up in literal seconds, making it nearly impossible to secure a table through traditional means.

What if you could automate the entire process?

Introducing the Reservation Bot

I built an automated reservation system that monitors restaurant availability across major booking platforms and instantly grabs reservations the moment they become available. Think of it as having a tireless assistant who checks for tables 24/7 and books them the instant they appear.

What It Does

The bot handles the entire reservation workflow:

Multi-Platform Support - Works with both OpenTable and Resy, the two dominant restaurant reservation platforms
Smart Monitoring - Continuously checks for availability based on your preferences (date, time, party size)
Instant Booking - Automatically secures reservations when spots open up, faster than any human could
Account Management - Securely stores your dining platform credentials for seamless bookings
Reservation Dashboard - View, manage, and cancel all your reservations from a single interface

How It Works

The system is surprisingly straightforward from a user perspective:

Connect Your Accounts - Add your OpenTable and/or Resy login credentials
Search for Venues - Browse and select restaurants you want to book
Set Up Inquiries - Specify your preferences (dates, times, party size)
Let It Run - The bot monitors availability and books automatically

Behind the scenes, the bot uses scheduled tasks that run continuously, checking for availability and executing reservations the moment a table becomes available.

Technical Architecture

For those interested in the technical details, here's what powers the system:

Backend Stack

Go (Golang) with the Fiber web framework for high-performance HTTP handling
PostgreSQL for reliable data persistence
Redis for caching and fast lookups
Automated scheduling via cron jobs for continuous monitoring

Key Features

Secure authentication with OAuth integration
Proxy support for account-level IP rotation
Concurrent processing to handle multiple inquiries simultaneously
Real-time logging to track booking attempts and successes

The architecture is designed for speed and reliability. When a reservation slot opens up, milliseconds matter. The system can detect and book faster than manual clicking could ever achieve.

Real-World Use Cases

This bot excels in scenarios where:

New restaurant launches - Be first in line when reservations open
Last-minute cancellations - Catch tables that become available unexpectedly
Hard-to-book venues - Secure spots at consistently sold-out restaurants
Special occasions - Never miss that anniversary dinner reservation
Flexible schedules - Let the bot find any available slot within your preferred range

A Note on Responsible Use

This tool is designed to help individuals secure reservations for personal dining experiences. It's important to:

Only book reservations you actually intend to use
Cancel promptly if your plans change
Respect restaurant booking policies
Use responsibly and ethically

The goal is to level the playing field for regular diners, not to enable scalping or commercial resale of reservations.

The Results

The system has successfully automated thousands of reservation bookings across OpenTable and Resy platforms. What used to require constant manual checking and lightning-fast reflexes now happens automatically in the background.

For popular restaurants that book up instantly, this can mean the difference between never getting a table and dining regularly at your favorite spots.

The combination of real-time monitoring, instant execution, and multi-platform support makes this one of the most comprehensive restaurant reservation automation tools available.

Tech Stack Summary: Go, Fiber, PostgreSQL, Redis, OAuth
Platforms Supported: OpenTable, Resy

Built for food enthusiasts who refuse to miss out on their next great dining experience.

Building a Fast Web Scraper Without Puppeteer: A Live Coding Challenge

Isaac Sunday — Mon, 17 Nov 2025 08:39:28 +0000

How I built a production-grade scraper for NC public notices using Cheerio and TLS fingerprinting instead of headless browsers

The Challenge

During a live coding interview, I was tasked with building a scraper for ncnotices.com, a North Carolina public notices database. The catch? I had limited time and needed to handle a complex ASP.NET application with state management, pagination, and CAPTCHA protection.

The interviewer mentioned their team used Puppeteer for their existing scraper. I chose a different path.

Why Not Puppeteer?

Don't get me wrong - Puppeteer is fantastic for many use cases. But for this task, it felt like bringing a tank to a knife fight:

Resource overhead: Running a headless Chrome instance uses 100-200MB of RAM per instance
Slower execution: Browser startup alone takes 1-2 seconds
Complexity: Managing browser lifecycles, waiting for selectors, handling browser crashes

Since ncnotices.com renders server-side (classic ASP.NET), I didn't need JavaScript execution. I just needed to:

Parse HTML efficiently
Manage cookies and sessions
Bypass basic bot detection

The Tech Stack

{
  "dependencies": {
    "cheerio": "^1.1.2",      // Fast HTML parsing
    "impit": "^0.7.0",        // TLS fingerprinting
    "tough-cookie": "^6.0.0"  // Cookie management
  }
}

Key Libraries

Cheerio: jQuery-like HTML parsing that's blazing fast. Perfect for server-rendered content.

Impit: The secret sauce. This library mimics real browser TLS fingerprints, making requests indistinguishable from actual Chrome/Firefox browsers. Much lighter than Puppeteer.

tough-cookie: Proper cookie jar implementation for session management.

The Technical Challenges

1. ASP.NET ViewState Management

ASP.NET uses __VIEWSTATE and __VIEWSTATEGENERATOR tokens for state management. These need to be extracted and sent with each request:

private getStateTokens($: CheerioAPI): {
  view_state: string;
  view_state_generator: string;
} {
  const configText = $.text().split("0|hiddenField").at(-1)?.split("|") || [];
  let view_state = $("#__VIEWSTATE").attr("value") || "";
  let view_state_generator = $("#__VIEWSTATEGENERATOR").attr("value") || "";

  if (!view_state) {
    view_state = configText[configText.findIndex((x) => x.trim() == "__VIEWSTATE") + 1];
  }

  if (!view_state_generator) {
    view_state_generator = configText[configText.findIndex((x) => x.trim() == "__VIEWSTATEGENERATOR") + 1];
  }

  return { view_state, view_state_generator };
}

The tokens can be in hidden fields OR embedded in the response text. I handle both cases.

2. Pagination Hell

Classic ASP.NET pagination requires simulating user interactions through POST requests with specific event targets:

const queryParams: Record<string, string> = {
  ctl00$ToolkitScriptManager1:
    params.page > 1
      ? "ctl00$ContentPlaceHolder1$WSExtendedGridNP1$updateWSGrid|ctl00$ContentPlaceHolder1$WSExtendedGridNP1$GridView1$ctl01$btnNext"
      : "ctl00$ContentPlaceHolder1$as1$upSearch|ctl00$ContentPlaceHolder1$as1$btnGo",
  __VIEWSTATE: params.config.view_state,
  __VIEWSTATEGENERATOR: params.config.view_state_generator,
  __ASYNCPOST: "true",
  // ... more params
};

Each page requires the previous page's state tokens, creating a chain of requests.

3. CAPTCHA Support

The site occasionally shows reCAPTCHA. I built an extensible solver interface:

export interface ICaptchaSolver {
  solveCaptcha(siteKey: string, pageUrl?: string): Promise<string>;
}

This allows plugging in any CAPTCHA solving service (2captcha, Anti-Captcha, etc.) without modifying the scraper logic.

4. Session Management

The scraper maintains cookies across requests using tough-cookie:

this.client = new Impit({
  browser: "chrome",
  proxyUrl: proxyUrl,
  ignoreTlsErrors: true,
  cookieJar: new CookieJar(),
});

The Complete Flow

async searchKeyword(query: string): Promise<NCSearchItem[]> {
  let hasNext = true;
  let setPreset = true; // Simulate form fill on first request
  let page = 1;
  const config = await this.getSearchConfig();
  const items: NCSearchItem[] = [];

  do {
    const { items: currentItems, view_state_generator, view_state, has_next }
      = await this.executeSearch(query, { page, date: config.date, config }, setPreset);

    items.push(...currentItems);
    page++;
    hasNext = has_next;
    config.view_state = view_state;
    config.view_state_generator = view_state_generator;
    setPreset = false;
  } while (hasNext);

  return items;
}

Get initial config and state tokens
Execute search with form simulation
Extract results and pagination state
Continue until no more pages
Return all items

Performance Comparison

Let's crunch the numbers for a realistic scenario: scraping 100 public notices across 10 paginated search result pages.

Puppeteer Approach

Browser startup:        1,500ms (one-time cost)
10 search page loads:  10 × 300ms = 3,000ms
100 detail page loads: 100 × 300ms = 30,000ms
───────────────────────────────────────────
Total time:            ~34.5 seconds
Memory usage:          ~150MB per instance

This Approach (Cheerio + Impit)

HTTP client init:      50ms (one-time cost)
10 search requests:    10 × 100ms = 1,000ms
100 detail requests:   100 × 100ms = 10,000ms
───────────────────────────────────────────
Total time:            ~11 seconds
Memory usage:          ~10MB per instance

Result: 3.1x faster execution time, 15x lower memory footprint

Scalability on 8GB RAM

Approach	Concurrent Instances	Theoretical Throughput
Puppeteer	~50 instances	~145 notices/sec
Cheerio + Impit	~800 instances	~7,270 notices/sec

50x better concurrency - though realistically limited by network bandwidth and rate limiting rather than memory.

What I Learned

Choose the right tool: Puppeteer is amazing, but not always necessary
Understand the target: Server-side rendering = no JavaScript needed
TLS fingerprinting matters: Impit makes requests look legitimate
State management is crucial: ASP.NET apps require careful token handling
Build for extensibility: The CAPTCHA solver interface allows easy integration

The Outcome

I completed this scraper 2 hours after the interview ended and sent it to the interviewer. While I didn't get the job (they chose another candidate), the interviewer specifically mentioned:

"You are clearly very smart and have a lot of expertise. I will put you at the top of my list for the future."

Sometimes the best outcome isn't getting the job - it's building something you're proud of and demonstrating genuine problem-solving skills.

Try It Yourself

The full source code demonstrates:

Clean TypeScript architecture
Comprehensive type definitions
Testable design with dependency injection
Production-ready error handling

const scraper = new NCNotices("http://your-proxy:8888");
const results = await scraper.searchKeyword("foreclosure");

for (const searchItem of results) {
  const detail = await scraper.getItem(searchItem);
  console.log(detail);
}

Final Thoughts

Web scraping isn't about using the most popular tools - it's about understanding your target, choosing the right approach, and building maintainable solutions. Sometimes a lightweight HTTP client with good HTML parsing beats a full browser automation framework.

What's your experience with web scraping? Have you faced similar challenges with state management or bot detection? Drop a comment below!

P.S. - The Plot Twist

So... did I get the job?

Spoiler alert: Nope! But hey, I had fun building this and learned a ton in the process. Sometimes the best outcome isn't getting hired - it's the skills you sharpen and the code you're proud to share.

Plus, I got one of the nicest rejection messages I've ever received:

Not bad for a live coding challenge I completed 2 hours after the interview ended. 😄

Interested in more deep dives on web scraping, TypeScript, and performance optimization? Follow me for more technical breakdowns!

Building a Full-Stack Product Monitoring System: A Technical Deep Dive

Isaac Sunday — Sun, 16 Nov 2025 22:18:17 +0000

Overview

This is a production-grade, full-stack application designed to monitor product availability on an e-commerce website. The system automatically tracks inventory changes, sends Discord notifications for new products and restocks, and provides a comprehensive admin dashboard for monitoring configuration.

This technical deep-dive explores the architecture, design decisions, and implementation strategies behind building a scalable monitoring system capable of handling thousands of products with real-time notifications.

System Architecture

High-Level Architecture

The application follows a modern hybrid architecture combining TypeScript and Go microservices:

┌─────────────────┐      ┌─────────────────┐      ┌──────────────────┐
│   React 19      │─────▶│   Fastify       │─────▶│  PostgreSQL 16   │
│   Frontend      │◀─────│   Backend (TS)  │◀─────│    Database      │
│   (Port 80)     │      │   (Port 4000)   │      │   (Port 5432)    │
└─────────────────┘      └─────────────────┘      └──────────────────┘
                               │         │
                               │         └────────▶┌──────────────────┐
                               │                   │  Go Checkout API │
                               │                   │    (Port 8080)   │
                               │                   │  ┌────────────┐  │
                               │                   │  │ TLS Client │  │
                               │                   │  │  Akamai    │  │
                               │                   │  └────────────┘  │
                               │                   └──────────────────┘
                               ▼
                         ┌──────────┐         ┌─────────────┐
                         │  Redis 7 │◀───────▶│   BullMQ    │
                         │(Port 6379)│        │ Job Queues  │
                         └──────────┘         └─────────────┘
                               │
                               ▼
                         ┌──────────┐
                         │  Discord │
                         │ Webhooks │
                         └──────────┘

Architecture Decisions

1. Go Microservice for Checkout (Critical Decision)

Why Go instead of TypeScript? The checkout flow requires bypassing sophisticated bot protection (Akamai)
TLS Fingerprinting: Modern websites fingerprint TLS handshakes to detect bots. Node.js's built-in HTTP client has a static TLS fingerprint that's easily detected and blocked
bogdanfinn/tls-client: Go library that allows custom TLS fingerprints mimicking real browsers (Chrome 133 profile)
Advanced HTTP/2 Features: Precise control over header ordering, pseudo-headers, and connection properties
Hyper Solutions SDK: Commercial Akamai bypass solution with sensor data generation
Result: Successfully bypasses bot detection while maintaining high performance and reliability

2. Fastify over Express

Fastify was chosen for its superior performance (up to 2x faster than Express in benchmarks)
Built-in schema validation with JSON Schema
Native TypeScript support and excellent plugin ecosystem
First-class async/await support without middleware complexity

3. BullMQ for Job Processing

Reliable job queue with Redis backing for distributed task processing
Built-in support for retries, delayed jobs, and repeatable patterns
Priority queues for handling different notification types
Persistent job storage prevents data loss during restarts

4. PostgreSQL with Zapatos

Type-safe database queries without ORM overhead
Direct SQL when needed with full TypeScript inference
Automatic type generation from database schema
Eliminates runtime type mismatches between DB and application

5. Redis for Caching and Queue Management

Low-latency caching for frequently accessed data
BullMQ job queue backing
Session storage for authentication
Pub/sub capabilities for real-time features

Backend Architecture

Application Structure

The backend follows a clean, modular architecture with separate TypeScript and Go services:

TypeScript Backend:

src/backend/
├── config/              # Environment and database configuration
├── controllers/         # Request handlers (thin layer)
├── services/           # Business logic layer
├── routes/             # API route definitions
├── middleware/         # Authentication and validation
├── jobs/               # Background job processing
│   ├── queues.ts      # BullMQ queue setup
│   └── workers.ts     # Job processors
└── server.ts          # Application entry point

Go Checkout Service:

cmd/api/
└── main.go             # Fiber HTTP server (Port 8080)

src/
├── checkout.go         # Main checkout flow orchestration
├── checkout_types.go   # Type definitions
├── akamai.go          # Akamai bot bypass implementation
├── headers.go         # Browser-like header generation
└── card.go            # Credit card validation

Dependency Injection Pattern

The application uses a centralized dependency management system to handle database and Redis connections:

// Simplified dependency management pattern
class AppDependencies {
  private static pool: Pool;
  private static redis: Redis;

  static async connect() {
    // Initialize connections
    await pool.query("SELECT 1");
    await redis.ping();
  }

  static get<T>(name: string): T {
    // Return requested dependency
  }
}

Benefits:

Single source of truth for shared resources
Easy testing with mock dependencies
Graceful connection management and cleanup
Type-safe dependency access

Authentication System

JWT-based authentication with bcrypt password hashing:

Key Features:

Stateless authentication using JWT tokens
10-round bcrypt hashing for password security
Cookie-based token storage with httpOnly flag
Token expiration and refresh patterns
Protected routes via middleware

Authentication Flow:

User submits credentials
Password verified against bcrypt hash
JWT token generated with user claims
Token stored in httpOnly cookie
Subsequent requests validated via middleware

Go Checkout Microservice

The checkout functionality is implemented as a separate Go microservice that communicates with the TypeScript backend via HTTP.

Architecture Pattern:

TypeScript backend receives checkout request from user
Backend validates task and assembles checkout payload
HTTP request sent to Go service at GO_API_URL
Go service performs checkout with TLS fingerprinting
Response returned to TypeScript backend
Backend updates database and sends notifications

Key Technologies:

Fiber Framework:

High-performance web framework for Go (built on Fastify)
Minimal API surface: /checkout, /checkout/:id, /status, /health
Context-based request cancellation support

TLS Client (bogdanfinn/tls-client):

Custom TLS fingerprinting to mimic Chrome 133
Randomized TLS extension ordering
Support for HTTP/2 with precise control
Cookie jar management with domain-specific cookies

Akamai Bypass Implementation:

The Akamai client implements a sophisticated multi-step bot detection bypass:

Script Extraction: Parse Akamai bot detection script from HTML
Sensor Generation: Generate sensor data using Hyper Solutions SDK
Cookie Management: Track _abck and bm_sz cookies
SBSD Challenge Handling: Detect and solve secondary challenges automatically
IP Consistency: Maintain consistent IP address for session validation

Checkout Flow (src/checkout.go:39-118):

1. Generate visitor/visit IDs from Oracle tracking system
2. Set session cookies (_abck, bm_sz, AGEVERIFY)
3. Add product to cart with Akamai sensor
4. Add shipping address information
5. Validate inventory availability
6. Update cart items with pricing
7. Tokenize credit card via CardConnect
8. Submit order with payment details

Error Handling:

Context cancellation for user-initiated aborts
Automatic retry logic for transient failures
Comprehensive error messages for debugging
Graceful fallback when challenges fail

Security Features:

Credit card tokenization via CardConnect (PCI compliance)
Proxy rotation support for IP diversity
No credential storage (stateless operation)
Request validation and sanitization

Integration Pattern:

The TypeScript backend integrates via a CheckoutClient class (src/checkout.ts):

const response = await this.client.post<CheckoutResponse>(
  `/checkout`,
  requestPayload,
  {
    headers: { 'X-Checkout-ID': taskId }
  }
);

Benefits of This Hybrid Approach:

TypeScript handles business logic, database, and orchestration
Go handles performance-critical bot bypass operations
Each service optimized for its specific use case
Independent scaling and deployment
Clear separation of concerns

API Design

RESTful API with clear resource-based routing:

Endpoints:

/api/auth/* - Authentication (login, register, me)
/api/filtered-products - Keyword-based monitoring
/api/restock-products - Product restock tracking
/api/webhooks - Discord webhook management
/api/proxies - Proxy configuration
/api/dashboard - Statistics and activity
/api/products - Product search and details
/api/user-tasks - User task management

API Patterns:

Consistent error handling with proper HTTP status codes
Request validation using JSON Schema
Paginated responses for large datasets
Filtering and search capabilities

Job Queue System

BullMQ Integration

The monitoring system uses BullMQ for handling asynchronous jobs with different priorities and schedules.

Job Types:

scrape-all (Scheduled)
- Runs every 30 minutes
- Fetches sitemap from target website
- Identifies new products
- Queues individual product scrape jobs
scrape-product (On-demand)
- Scrapes individual product details
- Updates product information in database
- Triggers notifications for new products
- Handles keyword matching for filtered alerts
monitor-restock (Periodic)
- Checks specific products for stock changes
- Compares current vs previous stock levels
- Triggers restock notifications when inventory increases
alert (Immediate)
- Sends Discord webhook notifications
- Handles different notification types
- Manages user-specific alerts
- Supports batch notification delivery

Worker Architecture

Concurrency: 100 concurrent jobs for high throughput

Error Handling:

Automatic retry with exponential backoff
Failed job logging with full context
Dead letter queue for persistent failures
Monitoring via Bull Board UI

Graceful Shutdown:

// Graceful shutdown pattern
process.on('SIGTERM', async () => {
  await worker.close(); // Finish current jobs
  await redis.disconnect();
  await pool.end();
});

Database Design

Schema Architecture

Core Tables:

users - User accounts and authentication
products - Product catalog with inventory tracking
product_meta - Extended product information
filtered_products - Keyword-based monitoring rules
restock_products - Product restock subscriptions
user_webhooks - Discord webhook configurations
user_proxies - Proxy pool management
user_tasks - Custom user automation tasks

Migration Strategy

Using dbmate for database migrations:

Benefits:

Version-controlled schema changes
Up/down migration support
Language-agnostic SQL migrations
Simple CLI interface

Workflow:

Create migration: dbmate new add_feature
Write up/down SQL
Apply: dbmate up
Generate types: npx zapatos

Type-Safe Queries with Zapatos

Zapatos generates TypeScript types directly from the database schema:

// Type-safe queries
const products = await select(
  'products',
  { in_stock: conditions.gt(0) },
  { columns: ['id', 'name', 'price'] }
).run(pool);

// Full type inference - no manual typing needed
// products: Array<{ id: string, name: string, price: number }>

Advantages:

Compile-time type checking
Autocomplete for table names and columns
Zero runtime overhead (generates SQL)
Catches schema mismatches before deployment

Web Scraping & Monitoring

Scraping Architecture

Strategy: Polite scraping with proxy rotation and rate limiting

Key Components:

Sitemap Parsing
- Fetches XML sitemap from target website
- Extracts product URLs
- Identifies new products not in database
Product Extraction
- Uses Cheerio for HTML parsing
- Extracts structured product data
- Handles dynamic content and variations
Proxy Management
- Random proxy selection per request
- Health checking and rotation
- User-specific proxy support
- Fallback to direct requests
Rate Limiting
- Configurable delays between requests
- Queue-based throttling
- Respectful of target server resources

Notification System

Discord Integration:

Rich embeds with product images
Multiple notification types (new products, restocks, filtered)
User-specific webhook routing
Batch notification support

Notification Types:

New Products - All newly discovered products
Filtered Products - Keyword-matched products for specific users
Restocks - Inventory increase alerts
Whiskey Release - Special release notifications

Frontend Architecture

React 19 with TypeScript

Technology Stack:

React 19 with latest features
TypeScript for type safety
Vite for fast builds and HMR
React Router for client-side routing

Component Structure

src/frontend/
├── components/         # Reusable UI components
│   └── Card.tsx       # Generic card component
├── pages/             # Route-based page components
│   ├── SignIn.tsx
│   ├── SignUp.tsx
│   ├── Dashboard.tsx
│   ├── NewFilteredProducts.tsx
│   ├── Restocks.tsx
│   ├── Webhooks.tsx
│   ├── Proxies.tsx
│   └── UserTasks.tsx
├── App.tsx            # Root component with routing
└── main.tsx           # Entry point

State Management

Approach: Local state with fetch-on-mount pattern

Rationale:

Simple dashboard application without complex state
Direct API calls from components
Minimal state synchronization needs
Reduced bundle size (no Redux/MobX)

Authentication Flow

Login form submission
API call with credentials
JWT token stored in httpOnly cookie
Protected routes check token validity
Automatic redirect to login if unauthorized

DevOps & Deployment

Docker Architecture

Multi-Container Setup:

Frontend container (Nginx-served static files)
Backend container (Node.js/TypeScript application)
Go API container (Checkout microservice)
Redis container (cache and job queue)
PostgreSQL external (managed database)

Docker Compose Configuration:

Service orchestration
Health checks for dependencies
Volume mounting for development
Environment-based configuration

Development Workflow

Local Development:

# Terminal 1 - Backend with hot reload
yarn dev:backend

# Terminal 2 - Frontend with Vite HMR
yarn dev:frontend

Docker Development:

# Start all services
docker compose up -d

# View logs
docker compose logs -f backend

# Rebuild after changes
docker compose up -d --build

Security Considerations

Application Security:

JWT tokens with expiration
Bcrypt password hashing (10 rounds)
SQL injection prevention via parameterized queries
CORS configuration for API protection
Webhook URL validation
Input sanitization and validation

Infrastructure Security:

Environment-based secrets (no hardcoded credentials)
TLS/SSL for database connections
Redis password authentication
Docker network isolation
Read-only volume mounts where appropriate

Performance Optimizations

Backend Optimizations

Connection Pooling
- PostgreSQL connection pool for efficient DB access
- Redis connection reuse across requests
Job Queue Concurrency
- 100 concurrent workers for parallel processing
- Bulk job enqueueing for efficiency
Database Indexing
- Indexed foreign keys
- Composite indexes for common queries
- Unique constraints for fast lookups
Caching Strategy
- Redis caching for frequent queries
- LRU eviction policy
- Configurable TTLs per data type

Frontend Optimizations

Build Optimization
- Vite's optimized bundling
- Tree shaking for smaller bundles
- Code splitting for route-based loading
Asset Optimization
- Lazy loading for images
- Minified CSS and JavaScript
- Gzip compression in production

Testing Strategy

Test Infrastructure

Framework: Jest with TypeScript support

Configuration:

Separate test database
Environment-based test configuration
Long timeout for integration tests
VM modules for ESM compatibility

Test Types

Unit Tests
- Service layer business logic
- Utility functions
- Data transformations
Integration Tests
- API endpoint testing
- Database operations
- Job queue processing
E2E Tests (Future)
- User flows
- Authentication scenarios
- Critical paths

Monitoring & Observability

Logging

Structured Logging:

JSON-formatted logs
Contextual metadata (module, timestamp, error traces)
Different log levels (debug, info, warn, error)
Production vs development logging strategies

Job Queue Monitoring

Bull Board Integration:

Web UI for queue visualization
Real-time job status
Failed job inspection
Job retry management
Performance metrics

Health Checks

Endpoint: /health

Database connectivity check
Redis availability check
System timestamp
Quick response for load balancers

Lessons Learned & Best Practices

What Worked Well

Type Safety Everywhere
- Zapatos for database types
- TypeScript across frontend and backend
- Fewer runtime errors, better developer experience
Dependency Injection Pattern
- Easy testing with mocked dependencies
- Clean resource management
- Single source of truth for connections
Job Queue Architecture
- Decoupled processing from API requests
- Reliable with built-in retries
- Scalable horizontal processing
Docker for Development
- Consistent environment across team
- Easy onboarding for new developers
- Production-like local setup

Challenges & Solutions

Challenge 1: Bot Detection and TLS Fingerprinting

Problem: Node.js HTTP clients getting blocked by Akamai bot detection
Root Cause: Static TLS fingerprints and predictable HTTP/2 characteristics
Solution: Implemented Go microservice with bogdanfinn/tls-client for dynamic TLS fingerprinting
Outcome: Successfully bypassed Akamai protection with Chrome 133 browser profile

Challenge 2: Akamai Sensor Data Generation

Problem: Complex JavaScript-based bot detection requiring precise sensor payload
Solution: Integrated Hyper Solutions SDK for commercial-grade Akamai bypass
Implementation: Real-time sensor generation with context preservation across requests
Result: Consistent checkout success rate with minimal detection

Challenge 3: Multi-Language Architecture

Problem: Coordinating TypeScript and Go services with different paradigms
Solution: Clean HTTP API contract with JSON payloads and typed interfaces
Benefits: Each language handles what it does best (TS for orchestration, Go for performance)

Challenge 4: Rate Limiting

Problem: Getting blocked by target website
Solution: Proxy rotation and intelligent delays

Challenge 5: Database Type Sync

Problem: Manual type definitions going stale
Solution: Automated type generation with Zapatos

Challenge 6: Worker Failures

Problem: Jobs failing silently
Solution: Comprehensive error logging and Bull Board monitoring

Challenge 7: Docker Networking

Problem: Services unable to communicate
Solution: Proper network configuration and service discovery

Future Enhancements

Planned Features

Advanced Analytics
- Price history tracking
- Stock pattern analysis
- Sell-out time predictions
User Preferences
- Customizable notification frequency
- Product category filters
- Price threshold alerts
API Rate Limiting
- Per-user rate limits
- API key management
- Usage analytics
Enhanced Testing
- Comprehensive E2E test coverage
- Load testing for scalability
- Automated security scanning

Scaling Considerations

Horizontal Scaling:

Stateless backend for easy replication
Shared Redis for distributed locking
Database read replicas for queries

Performance Improvements:

GraphQL for efficient data fetching
Server-side rendering for faster initial load
CDN for static asset delivery

Conclusion

Building this monitoring system provided valuable insights into creating a production-grade application. The combination of modern technologies (Fastify, React 19, BullMQ, Zapatos, Go) with solid architectural patterns resulted in a maintainable, scalable application.

Key takeaways:

Polyglot architecture works: Using TypeScript for business logic and Go for performance-critical operations proved highly effective
TLS fingerprinting matters: Modern bot detection requires sophisticated bypass techniques that standard HTTP libraries can't provide
Type safety reduces bugs: Both TypeScript and Go's type systems caught errors at compile time
Job queues are essential: Asynchronous processing decouples operations and improves reliability
Docker simplifies deployment: Multi-service orchestration becomes manageable with proper containerization
Commercial SDKs have value: Hyper Solutions SDK saved weeks of reverse engineering Akamai protection
Monitoring and observability are critical: Bull Board and structured logging enabled rapid debugging

The system successfully monitors thousands of products, delivers real-time notifications, performs automated checkouts while bypassing sophisticated bot detection, and provides a robust platform for future enhancements.

The hybrid TypeScript/Go architecture demonstrates that choosing the right tool for each job—even within the same application—leads to better outcomes than forcing a single technology to handle all requirements.

Tech Stack Summary

Frontend:

React 19
TypeScript
Vite
React Router

Backend (TypeScript):

Fastify
TypeScript
Node.js
Zapatos (type-safe DB queries)

Backend (Go):

Fiber (web framework)
Go 1.24.5
bogdanfinn/tls-client (TLS fingerprinting)
Hyper Solutions SDK (Akamai bypass)
PuerkitoBio/goquery (HTML parsing)

Database & Cache:

PostgreSQL 16
Redis 7
Zapatos (type-safe queries)

Job Processing:

BullMQ
Bull Board (monitoring)

Authentication & Security:

JWT (jsonwebtoken)
Bcrypt
CardConnect (payment tokenization)

Infrastructure:

Docker & Docker Compose
Nginx (frontend serving)
Multi-service orchestration

Development Tools:

Jest (testing)
dbmate (migrations)
tsx (TypeScript execution)
Go modules (dependency management)

Author: Isaac Sunday
Last Updated: November 2025