Forem: Yash Dubey

[Boost]

Yash Dubey — Thu, 05 Feb 2026 12:58:05 +0000

Markdown Is Not The Future of LLM Data Infrastructure

Yash Dubey ・ Feb 5

#ai #llm #webscraping #dataengineering

Markdown Is Not The Future of LLM Data Infrastructure

Yash Dubey — Thu, 05 Feb 2026 12:52:23 +0000

Markdown Is Not The Future of LLM Data Infrastructure

There's a fundamental mismatch in how we're feeding data to AI systems.

OpenAI, Anthropic, Google, every major AI lab spends billions training models to reason, plan, and take actions. Then we feed them web data converted to markdown and hope for the best.

Markdown was designed for humans to write formatted documents. It was never meant to be a data interchange format for machine intelligence. Yet somehow it became the default output of every "LLM-ready" web scraping API.

This is a problem. And it's getting worse as AI systems move from chat interfaces to autonomous agents that need to act on data, not just summarize it.

What "LLM-Ready" Markdown Actually Looks Like

I scraped the Wikipedia article for "Association football" using a popular markdown-based scraping API. Here's what came back (abbreviated):

[Jump to content](https://en.wikipedia.org/wiki/Association_football#bodyContent)

Main menu

Main menu

move to sidebarhide

Navigation

- [Main page](https://en.wikipedia.org/wiki/Main_Page "Visit the main page [alt-z]")
- [Contents](https://en.wikipedia.org/wiki/Wikipedia:Contents)
- [Current events](https://en.wikipedia.org/wiki/Portal:Current_events)
- [Random article](https://en.wikipedia.org/wiki/Special:Random)

Contribute

- [Help](https://en.wikipedia.org/wiki/Help:Contents)
- [Learn to edit](https://en.wikipedia.org/wiki/Help:Introduction)
- [Community portal](https://en.wikipedia.org/wiki/Wikipedia:Community_portal)

## Contents

move to sidebarhide

- [(Top)](https://en.wikipedia.org/wiki/Association_football#)
- [1Name](https://en.wikipedia.org/wiki/Association_football#Name)
- [2History](https://en.wikipedia.org/wiki/Association_football#History)Toggle History subsection

# Association football

246 languages

- [Acèh](https://ace.wikipedia.org/wiki/Sipak_bhan "Sipak bhan – Acehnese")
- [Afrikaans](https://af.wikipedia.org/wiki/Sokker "Sokker – Afrikaans")
- [አማርኛ](https://am.wikipedia.org/wiki/እግር_ኳስ "እግር ኳስ – Amharic")
- [العربية](https://ar.wikipedia.org/wiki/كرة_القدم "كرة القدم – Arabic")
... (242 more language links)

[Edit links](https://www.wikidata.org/wiki/Special:EntityPage/Q2736)

- [Article](https://en.wikipedia.org/wiki/Association_football)
- [Talk](https://en.wikipedia.org/wiki/Talk:Association_football)

English

- [Read](https://en.wikipedia.org/wiki/Association_football)
- [View source](https://en.wikipedia.org/w/index.php?title=Association_football&action=edit)

Tools

Actions

- [Read](https://en.wikipedia.org/wiki/Association_football)
- [View source](https://en.wikipedia.org/w/index.php?title=Association_football&action=edit)

General

- [What links here](https://en.wikipedia.org/wiki/Special:WhatLinksHere/Association_football)
- [Related changes](https://en.wikipedia.org/wiki/Special:RecentChangesLinked/Association_football)

Appearance

move to sidebarhide

Text

- Small
- Standard
- Large

Width

- Standard
- Wide

Color (beta)

- Automatic
- Light
- Dark

From Wikipedia, the free encyclopedia

Team sport played with a ball

**Association football**, commonly shortened as **football**...

That's 373KB of output. The actual article content doesn't start until line 465.

Before that? Navigation menus. Table of contents. 246 language selector links. UI toggle instructions. Appearance settings. Action buttons. Metadata panels.

This is what gets fed to LLMs when you use markdown-based scraping.

The Same Page Through Structured Extraction

Same URL. AlterLab's output:

{
  "@context": "https://alterlab.io/context/combined",
  "@type": "Article",
  "headline": "Association football",
  "author": {
    "@type": "Organization",
    "name": "Contributors to Wikimedia projects"
  },
  "datePublished": "2001-11-16T00:52:59Z",
  "dateModified": "2026-01-29T21:03:22Z",
  "image": "https://upload.wikimedia.org/wikipedia/commons/4/42/Football_in_Bloomington%2C_Indiana%2C_1995.jpg",
  "publisher": {
    "@type": "Organization",
    "name": "Wikimedia Foundation, Inc.",
    "logo": {
      "@type": "ImageObject",
      "url": "https://www.wikimedia.org/static/images/wmf-hor-googpub.png"
    }
  },
  "url": "https://en.wikipedia.org/wiki/Association_football",
  "_meta": {
    "extractionMethod": "schema_org",
    "extractionTimestamp": "2026-02-05T12:23:30.036979Z",
    "sourceUrl": "https://en.wikipedia.org/wiki/Association_football",
    "language": "en"
  },
  "content": {
    "paragraphs": [
      "Association football, commonly shortened as football and named soccer in some locations, is a team sport played between two teams of 11 players who almost exclusively use their feet to propel a ball around a rectangular field called a pitch.",
      "The objective of the game is to score more goals than the opposing team by moving the ball beyond the goal line into a rectangular-framed goal defended by the opponent...",
      "Association football is played in accordance with the Laws of the Game, a set of rules that has been in effect since 1863..."
    ],
    "headings": [
      {"level": 2, "text": "Name"},
      {"level": 2, "text": "History"},
      {"level": 3, "text": "Ancient precursors"},
      {"level": 3, "text": "Medieval precursors"},
      {"level": 2, "text": "Gameplay"},
      {"level": 2, "text": "Laws"}
    ],
    "links": [
      {"text": "team sport", "url": "https://en.wikipedia.org/wiki/Team_sport"},
      {"text": "FIFA", "url": "https://en.wikipedia.org/wiki/FIFA"},
      {"text": "Laws of the Game", "url": "https://en.wikipedia.org/wiki/Laws_of_the_Game_(association_football)"}
    ],
    "images": [
      {
        "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/4/42/Football_in_Bloomington%2C_Indiana%2C_1995.jpg/330px-Football_in_Bloomington%2C_Indiana%2C_1995.jpg",
        "alt": "The attacking player attempts to kick the ball into the net"
      }
    ]
  }
}

No navigation. No language selectors. No UI chrome. No "move to sidebarhide."

Just the article. Structured. Typed. Ready to use.

Why This Matters for AI Infrastructure

GPT-5.2 can reason across complex domains. Claude Opus 4.5 handles nuanced multi-step analysis. Gemini 3 processes multimodal inputs natively.

But none of that matters if you're feeding them 373KB of garbage to find 15KB of content.

Context Windows Are Expensive

Every token in a context window costs money and attention. When 95% of your input is navigation menus and language selectors, you're:

Paying for tokens that add zero value
Diluting the signal with noise
Forcing the model to separate content from chrome

A 128K context window sounds huge until you realize you're filling 120K of it with Wikipedia's UI.

Agents Need Clean Data for Actions

An LLM agent trying to answer "When was association football first played?" has to:

With markdown:

Parse through navigation menus
Skip 246 language links
Find the infobox (which is mangled into markdown)
Hope "First played | December, 1863" survived the conversion
Extract "December, 1863" from the surrounding noise

With structured extraction:

Access article.datePublished or find "First played" in the structured metadata
Done

Error probability compounds with complexity. When your agent needs to cross-reference five Wikipedia articles, markdown parsing errors multiply. Structured data stays clean.

Training Data Quality Determines Model Quality

If you're building training datasets from web content, the extraction format determines downstream quality.

Markdown-converted training data contains:

Navigation patterns the model shouldn't learn
UI text that pollutes language understanding
Structural artifacts that create noise

Structured training data contains:

Actual content
Proper semantic relationships
Clean boundaries between data types

Models learn what they see. Show them navigation menus, they learn navigation menus.

What Data Infrastructure Consumers Actually Need

OpenAI's browsing, Anthropic's computer use, Google's grounding all need web data. Here's what would make that data actually useful:

1. Content Separation

The article is not the page. A page contains navigation, ads, related content, footers, cookie banners. The article is the thing you came for.

Structured extraction separates these. Markdown mixes them.

2. Semantic Typing

"December, 1863" as a string is ambiguous. Is it a date? A title? A caption?

"datePublished": "1863-12-01T00:00:00Z" is unambiguous. It's a date. It's typed. It can be compared, sorted, filtered.

3. Hierarchical Structure

Article content has structure: headings, paragraphs, lists, tables. This hierarchy carries meaning.

Markdown flattens hierarchy into text with formatting hints. JSON preserves it:

{
  "headings": [
    {"level": 2, "text": "History"},
    {"level": 3, "text": "Ancient precursors"},
    {"level": 3, "text": "Medieval precursors"}
  ]
}

An agent can now navigate the document programmatically: "Find the section about medieval history" becomes a data operation, not a text search.

4. Relationship Preservation

Links aren't just URLs. They're relationships. "Association football is governed by FIFA" contains a relationship: football → governed_by → FIFA.

Structured extraction can preserve these:

{
  "links": [
    {"text": "FIFA", "url": "https://en.wikipedia.org/wiki/FIFA", "context": "governing body"}
  ]
}

Markdown gives you [FIFA](https://en.wikipedia.org/wiki/FIFA). The relationship is implied, not stated.

The Economics of Clean Data

Here's the math that should matter to anyone building AI infrastructure:

Markdown approach:

373KB input per article
~93K tokens at ~4 chars/token
At $0.01/1K tokens (input), that's $0.93 per article just to read it
95% of that cost is noise

Structured approach:

~15KB input per article
~3.7K tokens
$0.037 per article
Near 100% signal

Scale to millions of pages and the difference is millions of dollars.

But cost isn't even the main issue. The main issue is that noisy inputs produce noisy outputs. Garbage in, garbage out scales with your ambitions.

AlterLab's Approach

We built AlterLab around a principle: extract what matters, in the format that matters.

Schema.org extraction: When pages have structured data, we use it. The Wikipedia example above was extracted from embedded schema.org markup, giving us typed fields automatically.

Intelligent content separation: We identify and isolate the main content from navigation, ads, and chrome. The article is the article, not the page.

Hierarchical preservation: Document structure survives extraction. Headings, paragraphs, lists, tables retain their relationships.

Flexible output: Need markdown for a specific use case? We provide it. But we also provide the structured data so you can choose based on your needs, not our assumptions.

For AI Labs and Data Infrastructure Teams

If you're building systems that consume web data at scale:

For training pipelines: Structured extraction produces cleaner datasets. Less noise in training means less noise in outputs.

For RAG systems: Structured chunks with metadata outperform markdown blobs. Retrieval improves when content boundaries are clear.

For agents: Actions need typed data. When your agent calls a function, it needs parameters, not markdown to parse.

For evaluations: Test your models on clean inputs. Separate parsing ability from reasoning ability.

The Future Is Typed, Not Formatted

Markdown was a reasonable v1 approach when LLMs were primarily chat interfaces summarizing text. We're past that.

Agents need typed data for tool use. Training pipelines need clean structured inputs. Evaluation needs isolated, verifiable data points.

The scraping APIs that survive will be the ones that understand: AI systems don't need documents. They need data.

Try It

AlterLab is a web scraping API built for AI applications. Structured JSON output, schema-based extraction, typed fields by default.

No credit card required. Free tier available.

alterlab.io

If you're at an AI company evaluating data infrastructure, reach out. We'd love to show you what structured web extraction looks like at scale.

The markdown-to-LLM pipeline was a reasonable first attempt. It's time for v2.

Yash Dubey
CEO, RapierCraft

How AlterLab Cortex Is Changing the Web Scraping Game

Yash Dubey — Fri, 26 Dec 2025 02:57:28 +0000

Your Web Scraper Shouldn't Need Babysitting

There's a dirty secret in web scraping that nobody talks about: the scraper you built last month is probably broken right now.

A site changed their HTML. A new anti-bot system appeared. That selector that worked perfectly? It's returning empty strings. And you won't know until someone complains.

This is the lifecycle of every web scraper ever built:

Build scraper
Works great
Something changes
Scraper breaks silently
Bad data pollutes your pipeline
Someone notices days later
Emergency fix
Return to step 2

We call this the "scraper maintenance treadmill." After watching our users (and ourselves) run on it for years, we decided to build something that gets off it entirely.

We call it Cortex.

The Problem Is Systemic

Here's what happens when you build scraping infrastructure the traditional way:

You set up a scraper for an e-commerce site. It works. You move on. Three weeks later, the site updates their product page layout. Your scraper still returns 200 OK, but the extracted data is garbage. Your pricing algorithm now thinks everything costs $0.

Or maybe the site deployed Cloudflare. Your success rate dropped from 95% to 12%. But your logs show "requests completed" because challenge pages count as responses.

The fundamental issue: traditional scrapers don't know when they're failing. They execute instructions, not outcomes.

Cortex: Self-Healing Scraping Infrastructure

Cortex is AlterLab's approach to this problem. Instead of building scrapers that execute static instructions, we built infrastructure that optimizes for outcomes.

Here's what that means in practice:

Continuous Quality Monitoring

Cortex doesn't just track success/failure status codes. It analyzes what actually comes back:

Is this a real product page or a challenge page?
Does this response contain the data we expected?
Are there structural patterns indicating the site changed?
Did extraction quality degrade compared to yesterday?

When issues appear, they're caught in minutes, not days.

Automatic Adaptation

When Cortex detects degradation, it doesn't just alert. It investigates.

Is this a selector issue? A new anti-bot system? A site structure change? Cortex identifies the root cause and tests potential solutions before a human ever needs to get involved.

We've seen domains where traditional scrapers would require weekly manual fixes run for months without human intervention.

Domain Intelligence

Every request teaches the system something. Cortex builds profiles for each domain:

What extraction strategies work best
Which anti-bot protections are present
Optimal timing and request patterns
Historical success patterns

This intelligence compounds. The system doesn't just fix current problems. It gets better at preventing future ones.

What This Means For You

If you're using AlterLab's scraping API, Cortex works automatically behind the scenes. You don't configure it. You don't manage it. You just get reliable data.

For e-commerce teams: Price monitoring that doesn't silently fail when competitors update their sites.

For data teams: Pipelines that stay healthy without dedicating engineering time to scraper maintenance.

For AI applications: Training data collection that maintains quality over time, not just volume.

For research: Long-running studies that don't break mid-collection.

The Scraper Maintenance Paradox

Here's the counterintuitive thing about web scraping infrastructure: the more successful you are, the worse the problem gets.

Scrape 10 sites? Manageable.
Scrape 100 sites? Dedicated maintenance team.
Scrape 1,000 sites? Good luck.

Every additional domain adds maintenance burden. Every site redesign requires attention. The work scales with your success.

Cortex inverts this. More domains mean more learning. More requests mean more intelligence. Scale makes the system better, not more fragile.

What Cortex Doesn't Replace

Let's be clear about what this isn't:

Cortex is not a no-code scraper builder. If you need custom extraction logic for your specific use case, you still define that. Cortex handles the infrastructure (anti-bot, reliability, adaptation) not your business logic.

Cortex is not magic. Some sites require authentication, user sessions, or other context that only you can provide. Cortex optimizes what can be automated; it doesn't eliminate the need to understand your target domains.

Cortex is not a compliance shield. You're still responsible for scraping ethically and legally. Cortex makes scraping more reliable, not more aggressive.

The Real Cost of Scraper Babysitting

Every hour spent debugging why a scraper stopped working is an hour not spent on actual analysis. Not building features. Not serving customers.

We've talked to teams spending 20+ hours per week on scraper maintenance. That's half a full-time engineer just keeping the data flowing. Not improving anything, just preventing regressions.

Cortex isn't really about scraping. It's about giving those hours back.

Try It

AlterLab is a managed web scraping API built by RapierCraft. One endpoint, any website, reliable data extraction.

Cortex runs automatically for all customers. No configuration required.

Up to 1000 free scrapes. No credit card.

alterlab.io

What's your worst scraper maintenance horror story? Drop it in the comments. Misery loves company.

The Information Blind Spot: How a 2-Hour News Delay Can Cost You Millions

Yash Dubey — Sat, 13 Sep 2025 18:40:08 +0000

The Multi-Million Dollar Delay

Imagine we're back in 2021, a container ship, the Ever Given of its route, suddenly veers off course and blocks a critical strait. For the next two hours, the world operates as if nothing has happened. Global markets continue trading, supply chain managers plan their logistics, and political analysts brief their governments—all based on information that is now dangerously out of date.

When the news finally breaks, it's chaos. The price of oil spikes. Shipping stocks plummet. Businesses scramble to reroute cargo, but the damage is done. The first movers, those who knew what was happening in that two-hour information blind spot, have already acted. They've hedged their positions, secured alternative routes, and briefed their stakeholders. They didn't just avoid a crisis; they capitalized on it.

This isn't a hypothetical scenario; it's the new reality of our hyper-connected world. The speed of events has outpaced the speed of traditional reportage, and that delay, whether it's two hours or twenty minutes, represents one of the single greatest sources of risk and opportunity in the modern world.

The High Cost of Latency

At RapierCraft, while building UltraNews, we became obsessed with this information-action gap. We analyzed its impact across various sectors:

Finance: A hedge fund that gets an alert about a factory fire at a key semiconductor plant 90 minutes before the story hits Bloomberg can liquidate its position in the company's stock before it tumbles.
Supply Chain: A logistics company that detects a wildcat strike at a major port hours before it's officially reported can reroute its fleet, saving millions in demurrage fees and contractual penalties.
Geopolitics: A government that receives near-instant alerts on unexpected military movements or localized protests can react proactively, shaping its diplomatic response instead of being caught on the back foot.
Disaster Response: An NGO that gets real-time data on the true scale of a natural disaster, sourced from on-the-ground social media and satellite imagery, can deploy resources more effectively, saving lives.

The common thread is that legacy news systems, even digital ones, are inherently reactive. They report on what has happened. An AI-first system, however, can detect what is happening, right now.

Closing the Gap with AI

How do you sift through millions of data points in real-time? Humanly, it's impossible. The only solution is an AI-first system designed for this specific purpose. This is the core principle behind UltraNews.

Instead of waiting for a journalist to verify a story and an editor to publish it, our system ingests a firehose of information from hundreds of thousands of sources: local news outlets, social media, maritime and flight trackers, satellite imagery feeds, corporate filings, and more.

An AI orchestrator then works to connect the dots. Here’s a simplified look at how an event detection pipeline might function:

class RealTimeEventDetector:
    def __init__(self):
        self.data_ingestors = [
            SocialMediaIngestor(keywords=["protest", "blockade"]),
            MaritimeAISIngestor(region="Suez Canal"),
            LocalNewsIngestor(language="ar-EG"),
            SatelliteImageryIngestor(area_of_interest="Suez Canal")
        ]
        self.event_correlator = AIEventCorrelator()
        self.alert_system = InstantAlerter()

    async def monitor_for_events(self):
        # Ingest data streams in parallel
        raw_signals = await asyncio.gather(
            *[ingestor.stream() for ingestor in self.data_ingestors]
        )

        # The AI brain connecting the dots
        potential_event = await self.event_correlator.analyze(raw_signals)

        if potential_event and potential_event.confidence > 0.95:
            # Confidence is high enough to issue an alert
            # before any single source has the full story.
            await self.alert_system.dispatch(
                title="Potential Blockade Detected: Suez Canal",
                details=potential_event.summary,
                confidence=potential_event.confidence,
                impact_assessment=potential_event.impact,
                subscribers=["finance", "supply-chain", "geopolitics"]
            )

# Output of such a system:
#
# [ALERT] 06:15 UTC: High-confidence event detected.
# Type: Supply Chain Disruption
# Location: Suez Canal, Egypt
# Summary: Multiple signals indicate a major shipping obstruction.
#          - Maritime data shows a large container ship stationary and sideways.
#          - Social media chatter from Port Said reports traffic halting.
#          - Local news mentions "unconfirmed incident" in the canal.
# Impact Score: 9.8/10 (Global Supply Chain)
#
# This alert is generated 1 hour and 45 minutes before the first major
# news outlet confirms the story.

This is the "unfair advantage." It's not about predicting the future; it's about having a more accurate and faster-calibrated picture of the present.

The UltraNews Edge: Your Phone is Now the Situation Room

What makes this technology truly transformative is its accessibility. In the past, this level of intelligence was the exclusive domain of three-letter agencies and the world's most expensive trading desks.

UltraNews is designed to change that. It delivers this power directly to your phone through beautifully illustrated, accessible alerts. You don't need to be a data scientist to understand the impact of a geopolitical event or a supply chain snag. You get a clear, concise, and visually intuitive briefing that tells you what you need to know, right now.

The Consumer Market Paradox: Why Isn't This Everywhere?

A valid question is: if this is so powerful, why haven't the big players in news or data rolled this out for consumers? I reckon it's because it would fundamentally disrupt their primary business models.

For major news organizations, their brand is built on trust and verification. An AI-first model that prioritizes speed and confidence scores over editorial sign-off is a cultural anathema. It's a different business entirely.
For financial data providers, their core business is selling expensive terminals and proprietary data feeds to institutions. A low-cost, mobile-first consumer product would cannibalize their high-margin enterprise offerings.

This creates a unique opening for focused, AI-first companies like RapierCraft to build products like UltraNews from the ground up, without the baggage of a legacy business model.

The Future is Instantaneous

We are moving from an era of periodic information to one of continuous, real-time intelligence. The advantage will no longer go to those with the most data, but to those who can extract the most meaningful signal from the noise, the fastest.

The ability to see around the corner—even if only by an hour or two—is no longer a luxury. It's becoming a fundamental requirement for navigating an increasingly volatile world. The only question is whether you'll be the one reacting to the headlines, or the one acting on the intelligence before the headlines are even written.

At RapierCraft, we're obsessed with building the systems that will define this new era of real-time intelligence. What are your thoughts on the information-action gap? Have you ever been burned by a news delay? Let's discuss in the comments.

Building Semantic Search That Actually Works: Beyond Basic Vector Similarity

Yash Dubey — Tue, 09 Sep 2025 07:43:48 +0000

Most semantic search implementations are just fancy keyword matching. Here's how to build search that actually understands meaning and context.

The $10K Mistake Everyone Makes

I spent $10,000 and three months building what I thought was "semantic search." Users typed queries, got embedding vectors, found similar vectors, returned results. Technically correct, practically useless.

The problem? Semantic similarity ≠ Search relevance.

When users searched for "Tesla stock analysis," my system returned articles about:

Tesla car reviews (similar topic)
General stock market trends (similar words)
Elon Musk interviews (related entity)

But it completely missed the actual Tesla financial analysis articles because they used different vocabulary.

This is the semantic search trap that costs companies millions in wasted development time.

What Real Semantic Search Looks Like

After rebuilding our system from scratch at RapierCraft for UltraNews (now processing 15,000+ articles daily with 94% search satisfaction), here's what I learned:

Real semantic search needs three layers:

Intent Understanding - What is the user actually trying to find?
Context Awareness - What domain/timeframe/perspective matters?
Relevance Scoring - How well does each result match the complete query context?

Layer 1: Intent Understanding

Traditional approach:

# Basic embedding similarity - what everyone does wrong
def search(query: str):
    query_embedding = embed(query)
    results = vector_db.similarity_search(query_embedding, top_k=10)
    return results

Smart approach:

class IntentAnalyzer:
    def __init__(self):
        self.intent_classifier = self.load_intent_model()
        self.entity_extractor = EntityExtractor()
        self.temporal_analyzer = TemporalAnalyzer()

    async def analyze_query(self, query: str) -> QueryIntent:
        # Extract structured intent
        intent_type = await self.intent_classifier.classify(query)
        entities = await self.entity_extractor.extract(query)
        temporal_context = await self.temporal_analyzer.analyze(query)

        return QueryIntent(
            type=intent_type,  # analysis, news, comparison, summary
            entities=entities,  # companies, people, locations
            temporal_context=temporal_context,  # recent, historical, trending
            specificity_score=self.calculate_specificity(query),
            domain_hints=self.extract_domain_hints(query)
        )

Layer 2: Context-Aware Embedding

Instead of generic embeddings, we create context-specific ones:

class ContextualEmbedding:
    def __init__(self):
        self.domain_models = {
            'finance': FinanceDomainModel(),
            'technology': TechDomainModel(),
            'politics': PoliticsDomainModel()
        }
        self.temporal_weights = TemporalWeighting()

    async def embed_with_context(self, text: str, context: QueryIntent) -> np.ndarray:
        # Select domain-specific model
        domain_model = self.domain_models.get(
            context.primary_domain, 
            self.default_model
        )

        # Generate base embedding
        base_embedding = await domain_model.embed(text)

        # Apply temporal weighting
        if context.temporal_context.is_time_sensitive:
            temporal_weight = self.temporal_weights.calculate_weight(
                text_timestamp=self.extract_timestamp(text),
                query_time_preference=context.temporal_context.preference
            )
            base_embedding = base_embedding * temporal_weight

        # Entity boost - increase relevance for matching entities
        entity_boost = self.calculate_entity_boost(text, context.entities)

        return base_embedding + entity_boost

Layer 3: Multi-Signal Relevance Scoring

This is where most systems fail. They rely purely on vector similarity:

class RelevanceScorer:
    def __init__(self):
        self.signals = [
            SemanticSimilaritySignal(),
            EntityMatchingSignal(), 
            TemporalRelevanceSignal(),
            PopularitySignal(),
            QualitySignal(),
            UserContextSignal()
        ]

    async def score_result(self, document: Document, query_intent: QueryIntent, user_context: UserContext) -> float:
        signal_scores = {}

        for signal in self.signals:
            try:
                score = await signal.calculate_score(document, query_intent, user_context)
                weight = self.get_signal_weight(signal.name, query_intent.type)
                signal_scores[signal.name] = score * weight
            except Exception as e:
                # Graceful degradation - skip failed signals
                signal_scores[signal.name] = 0.0

        # Combine signals intelligently
        final_score = self.combine_signals(signal_scores, query_intent)

        return final_score

    def combine_signals(self, scores: Dict[str, float], intent: QueryIntent) -> float:
        # Dynamic weighting based on query type
        if intent.type == "factual_lookup":
            return (scores['entity_matching'] * 0.4 + 
                   scores['semantic_similarity'] * 0.3 + 
                   scores['quality'] * 0.3)

        elif intent.type == "trend_analysis":
            return (scores['temporal_relevance'] * 0.4 + 
                   scores['popularity'] * 0.3 + 
                   scores['semantic_similarity'] * 0.3)

        else:
            # Balanced approach for general queries
            return np.average(list(scores.values()))

The Real-World Implementation

Here's our complete search pipeline:

class SemanticSearchEngine:
    def __init__(self):
        self.intent_analyzer = IntentAnalyzer()
        self.contextual_embedding = ContextualEmbedding()
        self.relevance_scorer = RelevanceScorer()
        self.result_diversifier = ResultDiversifier()

    async def search(self, query: str, user_context: UserContext) -> SearchResults:
        # Phase 1: Understand what user wants
        query_intent = await self.intent_analyzer.analyze_query(query)

        # Phase 2: Get contextually relevant candidates  
        query_embedding = await self.contextual_embedding.embed_with_context(
            query, query_intent
        )

        # Cast a wide net initially
        candidates = await self.vector_db.similarity_search(
            query_embedding, 
            top_k=100  # Get more candidates than needed
        )

        # Phase 3: Score each candidate with multiple signals
        scored_results = []
        for candidate in candidates:
            relevance_score = await self.relevance_scorer.score_result(
                candidate, query_intent, user_context
            )
            scored_results.append((candidate, relevance_score))

        # Phase 4: Re-rank and diversify
        ranked_results = sorted(scored_results, key=lambda x: x[1], reverse=True)
        diversified_results = await self.result_diversifier.diversify(
            ranked_results, query_intent
        )

        return SearchResults(
            results=diversified_results[:10],
            total_found=len(candidates),
            query_understanding=query_intent,
            search_time_ms=self.timer.elapsed()
        )

Performance Results

Before (basic vector similarity):

User satisfaction: 67%
Click-through rate: 34%
Average session length: 2.3 minutes
Zero-result queries: 18%

After (multi-layer semantic search):

User satisfaction: 94%
Click-through rate: 78%
Average session length: 7.8 minutes
Zero-result queries: 3%

The Hidden Complexity: Entity Relationships

One breakthrough was understanding that search isn't just about documents—it's about entity relationships:

class EntityRelationshipGraph:
    def __init__(self):
        self.graph = nx.DiGraph()
        self.relationship_types = [
            'works_for', 'competes_with', 'supplies_to', 
            'influences', 'reports_on', 'similar_to'
        ]

    async def expand_query_entities(self, entities: List[Entity]) -> List[Entity]:
        expanded_entities = entities.copy()

        for entity in entities:
            # Find related entities within 2 degrees
            related = self.graph.neighbors(entity.id, depth=2)

            for related_entity in related:
                relationship_strength = self.calculate_relationship_strength(
                    entity, related_entity
                )

                if relationship_strength > 0.7:  # Strong relationship
                    expanded_entities.append(related_entity)

        return expanded_entities

When someone searches for "Apple earnings," our system automatically includes results about:

Tim Cook (CEO relationship)
iPhone sales (product relationship)
Samsung (competitor relationship)
AAPL stock (ticker relationship)

Lessons Learned: What Not To Do

1. Don't Use Generic Embeddings for Everything

OpenAI's text-embedding-ada-002 is great, but it's not optimized for your specific domain. Train or fine-tune embeddings on your data.

2. Don't Ignore Temporal Context

News from 2020 about "remote work trends" has different relevance than news from 2024. Build time-awareness into your scoring.

3. Don't Forget About User Intent

"Python" could mean the programming language or the snake. Context matters more than similarity.

4. Don't Skip Result Diversification

Returning 10 very similar articles about the same event isn't helpful. Diversify by perspective, source, and sub-topic.

The Business Impact

This approach enabled UltraNews to:

Process 15,000+ articles daily with intelligent categorization
Maintain 94% user satisfaction with search results
Reduce support tickets about "can't find relevant content" by 89%
Enable complex queries like "show me contrarian views on Tesla's recent earnings"

What's Next: The AI Search Evolution

We're experimenting with:

Conversational search: Multi-turn queries that build context
Proactive suggestions: Anticipating what users want to search next
Cross-domain reasoning: Connecting insights across different topics
Real-time intent adaptation: Learning from user behavior within the session

Your Turn

How are you implementing semantic search? Are you stuck in the basic vector similarity trap? Share your challenges in the comments—I'd love to help troubleshoot.

Building intelligent search systems at scale is just one of many challenges we've tackled at **RapierCraft. If you're working on similar data-intensive problems, I'd love to connect and share more war stories.

Follow for more deep dives into building AI-powered systems that actually work in production.

Tags: #semanticsearch #ai #machinelearning #embeddings #search #nlp #python #vectordatabase #informationretrieval #datascience

Breaking Changes: Why Your API Monitoring is Failing You (And How We Fixed It)

Yash Dubey — Sun, 07 Sep 2025 07:16:09 +0000

Ever had your integration break because a third-party changed their API without warning? Yeah, me too. Here's how we built a system that catches breaking changes before they break your production.

The $2M Problem Nobody Talks About

Last month, a major news API provider silently changed their response format. Within hours, over 200 applications broke. Financial trading bots stopped working. News aggregators went dark. Research platforms crashed.

The worst part? Nobody saw it coming.

This isn't a rare occurrence. According to recent surveys:

67% of developers have experienced unexpected API breaking changes
Average downtime cost: $5,600 per minute for enterprise applications
89% of teams have inadequate monitoring for third-party API changes

We learned this the hard way while building UltraNews, a platform that processes 15,000+ news articles daily from hundreds of different sources. When sources change their structure, layout, or API—our entire pipeline can break.

The Traditional Monitoring Trap

Most developers rely on basic uptime monitoring:

# Traditional approach - checking if endpoint is alive
def check_api_health():
    response = requests.get("https://api.example.com/health")
    return response.status_code == 200

This tells you if the API is up, but not if it's working correctly. It's like checking if your car starts without verifying if the steering wheel is connected.

Real-World Breaking Changes We've Seen

1. The Silent Schema Evolution

A major news API changed their article object from:

{
  "title": "Breaking News",
  "content": "Full article text"
}

To:

{
  "headline": "Breaking News",
  "body": {
    "text": "Full article text",
    "html": "<p>Full article text</p>"
  }
}

Zero documentation updates. Zero deprecation notices.

2. The Rate Limit Surprise

An API provider changed their rate limits from 1000 requests/hour to 100 requests/hour. Overnight. Our automated systems started failing, and we had no idea why.

3. The Authentication Shuffle

A source switched from API keys to OAuth2 without warning. All integrations broke instantly.

How We Solved It: Intelligent Change Detection

After getting burned too many times, we built a proactive change detection system that monitors not just availability, but behavior consistency.

1. Response Structure Monitoring

class SchemaValidator:
    def __init__(self, endpoint_url: str):
        self.endpoint_url = endpoint_url
        self.baseline_schema = self.establish_baseline()
        self.tolerance_config = ToleranceConfig()

    async def detect_schema_changes(self):
        current_response = await self.fetch_sample_response()
        current_schema = self.extract_schema(current_response)

        differences = self.compare_schemas(
            self.baseline_schema, 
            current_schema
        )

        critical_changes = [
            diff for diff in differences 
            if diff.severity >= self.tolerance_config.alert_threshold
        ]

        if critical_changes:
            await self.trigger_alert(critical_changes)

        return {
            'changes_detected': len(differences) > 0,
            'critical_changes': critical_changes,
            'compatibility_score': self.calculate_compatibility_score(differences)
        }

2. Behavioral Pattern Analysis

We don't just check responses—we analyze patterns:

class BehaviorAnalyzer:
    def __init__(self):
        self.pattern_history = PatternHistory()
        self.ml_predictor = AnomalyPredictor()

    async def analyze_endpoint_behavior(self, endpoint: str):
        # Collect multiple data points
        responses = await self.collect_sample_responses(endpoint, count=50)

        patterns = {
            'response_times': [r.elapsed_time for r in responses],
            'data_consistency': self.check_data_consistency(responses),
            'error_rates': self.calculate_error_distribution(responses),
            'field_presence': self.analyze_field_presence(responses)
        }

        # Compare against historical patterns
        anomalies = await self.ml_predictor.detect_anomalies(
            current=patterns,
            historical=self.pattern_history.get_patterns(endpoint)
        )

        return {
            'behavior_score': self.calculate_behavior_score(patterns),
            'anomalies_detected': anomalies,
            'trend_analysis': self.analyze_trends(patterns)
        }

3. Multi-Layered Validation Strategy

class IntegrationHealthMonitor:
    def __init__(self):
        self.validators = [
            ConnectivityValidator(),    # Basic uptime
            SchemaValidator(),          # Response structure
            BehaviorAnalyzer(),         # Pattern analysis
            SemanticValidator(),        # Content meaning
            PerformanceValidator()      # Speed/reliability
        ]

    async def comprehensive_health_check(self, integration: Integration):
        results = {}

        for validator in self.validators:
            try:
                result = await validator.validate(integration)
                results[validator.name] = result

                # Early exit on critical failures
                if result.severity == Severity.CRITICAL:
                    await self.emergency_notification(integration, result)

            except Exception as e:
                results[validator.name] = ValidationError(str(e))

        # Generate comprehensive health report
        return IntegrationHealthReport(
            integration_id=integration.id,
            overall_health=self.calculate_overall_health(results),
            individual_results=results,
            recommendations=self.generate_recommendations(results)
        )

The Results: Proactive vs Reactive

Before implementing intelligent monitoring:

Average detection time for breaking changes: 4-6 hours
False positive rate: 23%
Production incidents: 12 per month
Mean time to resolution: 45 minutes

After implementing intelligent monitoring:

Average detection time: 8-12 minutes
False positive rate: 3%
Production incidents: 1-2 per month
Mean time to resolution: 8 minutes

Lessons Learned: Building Resilient Integrations

1. Monitor Behavior, Not Just Availability

Uptime checks are table stakes. Your monitoring needs to understand what "working correctly" means for each integration.

2. Embrace Graceful Degradation

class ResilientIntegration:
    async def fetch_data(self):
        try:
            return await self.primary_source.get_data()
        except SchemaChangeDetected as e:
            # Automatic adaptation attempt
            adapted_parser = await self.schema_adapter.adapt_to_changes(e.changes)
            return await adapted_parser.parse(self.primary_source.get_raw_data())
        except CriticalFailure:
            # Fallback to secondary sources
            return await self.fallback_chain.execute()

3. Build Learning Systems

Your monitoring should get smarter over time, learning what "normal" looks like for each integration and adjusting sensitivity accordingly.

The Business Impact

This approach has saved us approximately $47,000 in potential downtime costs and countless hours of debugging. More importantly, it's enabled us to:

Process news from 500+ diverse sources reliably
Maintain 99.9% uptime despite external API instability
Scale to 15,000+ articles daily without breaking
Automatically adapt to source changes in real-time

Open Source Opportunity

We're considering open-sourcing parts of our monitoring infrastructure. Would the dev community find value in a tool that provides intelligent API change detection out of the box?

Your Experience?

How do you handle third-party API monitoring? Have you been burned by silent breaking changes? Share your war stories in the comments—let's learn from each other's pain points.

P.S. If you're dealing with high-volume data processing challenges, check out how we architected UltraNews to handle real-time intelligence at scale. Happy to share more technical details if there's interest.

Follow me for more posts on building resilient systems and scaling data infrastructure.

Tags: #api #monitoring #reliability #devops #integration #microservices #scalability #automation

[Boost]

Yash Dubey — Fri, 05 Sep 2025 05:37:58 +0000

Yash Dubey

Sep 5 '25

Architecting UltraNews: Building a Real-Time News Intelligence Platform That Scales

#webdev #programming #javascript #ai

Comments

6 min read

Architecting UltraNews: Building a Real-Time News Intelligence Platform That Scales

Yash Dubey — Fri, 05 Sep 2025 05:37:00 +0000

At RapierCraft, we faced a unique challenge: how do you build a system that can intelligently process thousands of news articles daily while maintaining sub-second response times and 99.9% uptime? The answer led us to create UltraNews, a platform that processes over 15,000 stories daily with 98.7% AI accuracy.

This article breaks down the architectural decisions, challenges, and innovations that power UltraNews—from our autonomous AI discovery systems to our multi-LLM orchestration approach.

The Core Architecture Challenge

When we started building UltraNews, we knew traditional news aggregation approaches wouldn't work. We needed a system that could:

Process content from thousands of diverse sources with different structures
Adapt automatically when websites change their layouts
Maintain consistent performance under varying loads
Orchestrate multiple AI providers for optimal results
Scale horizontally without losing intelligence

The solution was a three-tier architecture that separates concerns while enabling seamless communication between intelligent components.

Tier 1: The AI Intelligence Engine (Backend)

Our backend is built on FastAPI with Python 3.11+, chosen for its async capabilities and automatic API documentation. But the real innovation lies in how we've structured the intelligent components.

Autonomous Discovery Architecture

# Simplified view of our discovery coordination system
class DiscoveryCoordinator:
    def __init__(self):
        self.strategies = [
            StructuredDiscovery(),  # Sitemaps, RSS feeds
            IntelligentCrawling(),  # List page detection
            AIExploration()         # LLM-guided discovery
        ]

    async def discover_content(self, source: Source):
        for strategy in self.strategies:
            try:
                results = await strategy.execute(source)
                if self.validate_results(results):
                    return results
            except Exception as e:
                # Automatic fallback to next strategy
                continue

        # If all strategies fail, mark for human review
        await self.queue_for_review(source)

The key architectural decision here was escalation-based discovery. Instead of trying one approach and failing, our system automatically escalates through three phases:

Structured Discovery: Fast, efficient parsing of sitemaps and feeds
Intelligent Crawling: Pattern-based content detection and extraction
AI Exploration: LLM-guided discovery when other methods fail

Multi-LLM Orchestration System

One of our biggest architectural innovations is the Multi-LLM Orchestration system. Rather than being locked into a single AI provider, we built an abstraction layer that can work with any LLM:

class LLMOrchestrator:
    def __init__(self):
        self.providers = {
            'groq': GroqProvider(),
            'openrouter': OpenRouterProvider(), 
            'gemini': GeminiProvider(),
            'nvidia': NVIDIAProvider(),
            'local': OllamaProvider()
        }
        self.selector = IntelligentProviderSelector()

    async def process_content(self, content: str, task_type: str):
        # Select optimal provider based on task complexity and cost
        provider = await self.selector.select_provider(content, task_type)

        try:
            result = await provider.process(content)
            await self.log_success(provider, task_type)
            return result
        except Exception as e:
            # Intelligent fallback with context preservation
            fallback_provider = await self.selector.get_fallback(provider)
            return await fallback_provider.process(content)

This architecture gives us several advantages:

Cost Optimization: Automatically route simple tasks to cheaper models
Performance Optimization: Use the fastest model for time-critical tasks
Reliability: Seamless fallback when providers have issues
Quality Optimization: Route complex tasks to the most capable models

Per-Source Orchestration

Traditional systems treat all content sources equally. We implemented per-source orchestration, where each news source gets its own dedicated processing pipeline:

class PerSourceOrchestrator:
    def __init__(self):
        self.source_managers = {}
        self.adaptive_controller = AdaptiveController()

    async def process_source(self, source_id: int):
        if source_id not in self.source_managers:
            self.source_managers[source_id] = SourceManager(
                source_id=source_id,
                initial_resources=self.calculate_initial_resources(source_id)
            )

        manager = self.source_managers[source_id]

        # Dynamic resource allocation based on performance
        await self.adaptive_controller.adjust_resources(manager)

        return await manager.process()

class SourceManager:
    def __init__(self, source_id: int, initial_resources: dict):
        self.source_id = source_id
        self.state = SourceState.ACTIVE
        self.resources = initial_resources
        self.performance_metrics = PerformanceTracker()

    async def process(self):
        if self.state == SourceState.SUSPENDED:
            return await self.attempt_recovery()

        # Process with dedicated resources
        return await self.execute_processing_pipeline()

This approach allows us to:

Isolate problematic sources without affecting others
Dynamically allocate resources based on source reliability
Maintain source-specific optimizations and learning
Scale processing power where it's needed most

Tier 2: Professional Management Interface

Our admin panel is built with Next.js 15 and React 19, but the architecture focuses on real-time data synchronization and performance optimization.

Real-Time WebSocket Architecture

// WebSocket coordinator for real-time updates
class WebSocketCoordinator {
  private connections: Map<string, WebSocket> = new Map();
  private subscriptions: Map<string, Set<string>> = new Map();

  async broadcastUpdate(channel: string, data: any) {
    const subscribers = this.subscriptions.get(channel) || new Set();

    const updatePayload = {
      timestamp: Date.now(),
      channel,
      data: await this.optimizePayload(data)
    };

    subscribers.forEach(connectionId => {
      const ws = this.connections.get(connectionId);
      if (ws?.readyState === WebSocket.OPEN) {
        ws.send(JSON.stringify(updatePayload));
      }
    });
  }

  private async optimizePayload(data: any): Promise<any> {
    // Compress and optimize data for real-time transmission
    return this.compressionService.compress(data);
  }
}

Component-Based Performance Optimization

We implemented a priority-based rendering system where critical components get updated first:

// Priority-based component updates
const SystemHealthCard = memo(({ priority = 'high' }: ComponentProps) => {
  const { data, loading } = useWebSocketData('system-health', {
    updatePriority: priority,
    bufferUpdates: priority === 'low'
  });

  return (
    <Card className="real-time-updates">
      <CPUMetrics data={data.cpu} />
      <MemoryMetrics data={data.memory} />
      <QueueMetrics data={data.queues} />
    </Card>
  );
});

Tier 3: Research Platform (Public Interface)

Our public platform focuses on performance-first architecture with advanced caching strategies.

Multi-Layer Caching Architecture

class CacheOrchestrator {
  private layers = [
    new BrowserCache(60000), // 1 minute
    new CDNCache(300000),    // 5 minutes  
    new RedisCache(900000),  // 15 minutes
    new DatabaseCache()      // Persistent
  ];

  async get(key: string): Promise<any> {
    for (const cache of this.layers) {
      try {
        const result = await cache.get(key);
        if (result) {
          // Populate higher-priority caches
          await this.backfillCaches(key, result, cache);
          return result;
        }
      } catch (error) {
        // Continue to next cache layer
        continue;
      }
    }

    // Cache miss - fetch from source
    return null;
  }

  private async backfillCaches(key: string, data: any, sourceCache: Cache) {
    const sourceIndex = this.layers.indexOf(sourceCache);

    // Populate all higher-priority caches
    for (let i = 0; i < sourceIndex; i++) {
      await this.layers[i].set(key, data);
    }
  }
}

Database Architecture: Optimized for Scale

We use PostgreSQL with advanced optimization strategies:

Connection Pooling and Query Optimization

# Optimized database session management
class DatabaseManager:
    def __init__(self):
        self.pool = create_async_engine(
            DATABASE_URL,
            poolclass=StaticPool,
            pool_pre_ping=True,
            pool_recycle=3600,
            echo=False
        )
        self.priority_session_factory = self.create_priority_factory()

    def create_priority_factory(self):
        return sessionmaker(
            bind=self.pool,
            class_=AsyncSession,
            expire_on_commit=False
        )

    async def get_priority_session(self):
        # Admin requests get priority database connections
        return self.priority_session_factory()

Intelligent Indexing Strategy

-- Performance-optimized indexes for high-volume queries
CREATE INDEX CONCURRENTLY idx_articles_created_at_btree 
ON articles USING btree(created_at DESC);

CREATE INDEX CONCURRENTLY idx_articles_source_id_status 
ON articles(source_id, processing_status) 
WHERE processing_status IN ('pending', 'processing');

-- Partial indexes for common query patterns
CREATE INDEX CONCURRENTLY idx_articles_importance_high 
ON articles(importance_score DESC) 
WHERE importance_score > 0.7;

Deployment and Scalability Architecture

Containerized Microservices

Our deployment uses Docker with service-specific optimizations:

# Multi-stage build for optimal container size
FROM python:3.11-slim as base

# Install system dependencies
RUN apt-get update && apt-get install -y \
    gcc \
    g++ \
    && rm -rf /var/lib/apt/lists/*

FROM base as dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

FROM dependencies as production
COPY . .
EXPOSE 8001

# Optimized for production workloads
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8001", "--workers", "4"]

Auto-Scaling Architecture

# Kubernetes HPA configuration
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ultranews-backend-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ultranews-backend
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

Performance Results and Metrics

Our architectural decisions deliver measurable results:

Processing Speed: 15,000+ articles processed daily
AI Accuracy: 98.7% across all processing tasks
Response Time: Sub-second API response times
Uptime: 99.9% availability with intelligent failover
Scalability: Linear scaling to 100,000+ daily articles tested

Key Architectural Lessons

Embrace Failure: Design for component failures rather than trying to prevent them
Intelligent Escalation: Automated fallback strategies are more reliable than perfect systems
Resource Isolation: Per-source processing prevents cascade failures
Multi-Provider Strategy: Never depend on a single external service
Performance Monitoring: Real-time metrics enable proactive optimization

The Future: Continuous Architectural Evolution

UltraNews continues evolving with new architectural patterns:

Edge Computing: Processing content closer to sources for reduced latency
Federated Learning: Improving AI models without centralizing sensitive data
Event-Driven Architecture: Moving toward fully reactive systems
Quantum-Ready Encryption: Future-proofing security architecture

Conclusion

Building UltraNews taught us that modern applications need to be intelligent by design. Traditional architectural patterns work for traditional problems, but when you're processing global information at scale with multiple AI providers, you need architectures that can think, adapt, and optimize themselves.

The result is a platform that doesn't just aggregate news—it transforms information into intelligence through thoughtful architectural design and continuous optimization.

Want to learn more about building intelligent systems? Follow RapierCraft for more insights on AI-first architecture and scalable system design.

Tags: #architecture #ai #scalability #fastapi #nextjs #python #typescript #devops #microservices #realtimedata