Forem: Devansh

Future of Internet Crawling: WebMCP

Devansh — Wed, 18 Feb 2026 12:43:22 +0000

For decades, the internet has been crawled the same way humans browse it. Bots download pages, parse HTML, click links, and try to guess what each element does. Search engines do it. Automation tools do it. And now AI agents do it too.

But there is a problem.

Modern websites are not simple documents anymore. They are dynamic apps, full of JavaScript, state, authentication, and complex user flows. For AI agents, interacting with them today is like navigating a city by looking at satellite images and guessing where the doors are.

WebMCP changes that.

And it may redefine how the internet is crawled and interacted with in the AI era.

The problem with traditional crawling

Let’s understand how AI agents currently work on websites.

Most agents follow a loop:

Load the page
Take a screenshot or parse the DOM
Send it to a model
Decide what to click
Repeat

This approach is:

Slow, Expensive in tokens, Fragile when UI changes, Often inaccurate
and in many cases, the agent is guessing.

One report describes this well: today’s agents often rely on screenshots and raw HTML, which forces them to infer where buttons and forms are, consuming large amounts of context just to understand the page.

This is not scalable for an internet where AI agents may become primary users.

Visual understanding: Old Web vs WebMCP

Traditional crawling

User request: “Book a flight from Mumbai to Delhi”

Agent workflow:

Page → Screenshot → Vision model → Find form → Type → Submit → Wait → Parse results

Each step adds latency and token cost.

WebMCP workflow

Website exposes:

tool: searchFlights
inputs: origin, destination, date

Agent:

searchFlights({
  origin: "BOM",
  destination: "DEL",
  date: "2026-03-01"
})

No UI interaction. No guessing. No screenshots.

This structured approach improves efficiency dramatically. Some implementations report up to 89% token savings compared to screenshot-based methods.

WebMCP shifts the web from:

Document web → Action web

Before:

Crawlers indexed content
Automation simulated humans

Now:

Websites expose capabilities
Agents perform tasks directly
Think of it like the difference between:
Reading a restaurant menu (HTML)
Calling orderFood() (WebMCP)

Real-world impact scenarios

E-commerce Agent Prompt: “Buy the cheapest noise-cancelling headphones”

Instead of:

Navigating pages
Sorting filters
Clicking add to cart

The site exposes:

searchProducts()
addToCart()
checkout()

The purchase becomes an API call.

What this means for the future of crawling

Instead of crawling full pages, sites provide structured context and resources. This reduces processing by more than 60 percent in some evaluations while maintaining high task success rates.

Less scraping, more structured access
Ranking may change
Invisible websites risk

If an AI agent can complete a task directly on a competitor’s site via WebMCP, your UI might never be visited.

The internet was built for humans. Then it was optimized for search engines. Now it is being redesigned for AI agents.

WebMCP represents a fundamental shift from crawling pages to executing intentions.

If this standard succeeds, the future crawler will not scrape your HTML. It will ask your website what it can do.

And the websites that answer clearly will become the new gateways of the AI web.

Starvation in Javascript: When your program remains Hungry

Devansh — Mon, 05 May 2025 18:55:31 +0000

If you've spent time working with JavaScript's asynchronous model, you know that its single-threaded nature is both a blessing and a curse. On one hand, you don't have to worry about complex locking mechanisms. On the other, you might stumble into weird issues where your code just... never runs. One such issue is starvation.

What's Starvation?

In the context of JavaScript, starvation refers to a situation where a task is perpetually delayed because the event loop is constantly busy with other tasks—usually of higher priority or those scheduled more frequently.

Since JavaScript executes code in a single-threaded environment using an event loop, starvation can occur when:

Long-running synchronous code blocks the event loop.
Microtasks (Promises, MutationObservers, etc.) continuously prevent macrotasks (like setTimeout, setInterval) from executing.

Eg : Macrotask Starvation

function scheduleMacrotask() {
  setTimeout(() => {
    console.log('Macrotask executed');
  }, 0);
}

function floodMicrotasks() {
  for (let i = 0; i < 1e5; i++) {
    Promise.resolve().then(() => {
      // Simulate quick microtasks
    });
  }
}

scheduleMacrotask();
floodMicrotasks();

You might expect to see "Macrotask executed" logged right away. But the setTimeout callback (a macrotask) gets pushed back behind a flood of microtasks.

Microtasks in JavaScript are executed immediately after the currently executing script and before any macrotask. So, if you continuously add microtasks, the macrotask keeps getting pushed back. That's starvation.

Why Does Starvation Happen?

Starvation usually happens due to:

Excessive microtask generation
- Frequent Promise.resolve().then() in tight loops.
- Recursive promise chains.
Long synchronous blocks
- Functions doing CPU-heavy work without yielding back to the event loop.
Unbalanced priority
- When you schedule many tasks at one level (e.g., microtasks) but never give time for other levels (e.g., macrotasks).

Solution and Workarounds

Yield with setTimeoutBreak long synchronous code into smaller chunks using setTimeout to allow the event loop to breathe:

function longTask() {
  for (let i = 0; i < 1e6; i++) {
    if (i % 10000 === 0) {
      // Yield to the event loop
      setTimeout(() => longTask(i + 1), 0);
      return;
    }
    // Do work
  }
}

Use await with care
If using async/await, be cautious of chaining too many await Promise.resolve() calls without breaks.
Consider requestIdleCallback (for non-critical work)

requestIdleCallback(() => {
  // Low-priority background task
});

Queue microtasks responsibly Avoid flooding the microtask queue. Instead of a huge batch, spread tasks out:

function chunkMicrotasks(data) {
  function processChunk(index) {
    if (index >= data.length) return;

    // Process a small piece
    for (let i = index; i < index + 100 && i < data.length; i++) {
      // Do lightweight work
    }

    // Schedule next chunk
    setTimeout(() => processChunk(index + 100), 0);
  }

  processChunk(0);
}

Starvation in JavaScript isn't just a theoretical concern—it can cause real performance and responsiveness issues in your applications.

Understanding how the event loop, microtasks, and macrotasks work together gives you the power to avoid these pitfalls.

Next time your setTimeout doesn’t seem to fire or your app feels sluggish, take a closer look at your microtask usage and the structure of your event loop.

Happy 😋 coding!

Other resources

https://web.dev/articles/optimize-long-tasks

We had a socket problem! Working with sockets and APIs

Devansh — Fri, 27 Dec 2024 14:40:29 +0000

Hi, I'm Devansh. A couple of months ago, after integrating sockets on our page that displays order details, our production monitors started going down randomly. They would go down at various times throughout the day without any API failures. We tried reproducing the issue but were unsuccessful. Then, we tracked the production monitor's activities and replicated them exactly in the development environment, where we finally observed the failure.

On the load of the page, an API call and a socket connection were established on that page to fetch the complete order details. Whenever an event update occurred for that order, only the fields with updated values were sent via the socket response.

The Issue

Since the database in production is much larger than the one in the testing environment, API calls sometimes take longer. In such cases, if an update for the order occurs and a socket event is received, the details object would be updated with incomplete but new data. This incomplete data might be missing some necessary keys required by the frontend to load the page, resulting in page errors due to page getting rendered with incomplete data.

The Solution

We implemented a solution by maintaining a state to temporarily store the socket data. This way, we only update the order details object once the complete response from the API is received. This ensures that the data remains consistent and complete, preventing the frontend from encountering errors due to missing keys