Forem: lazyasscoder

You've Never Seen 90% of the Internet. Neither Has Google.

lazyasscoder — Sat, 04 Apr 2026 21:56:00 +0000

Google indexes billions of web pages. That sounds like a lot until you realize it might be less than 10% of the total web. The rest, the overwhelming majority of online content, is invisible to every search engine that exists.

Not hidden on purpose. Not encrypted on the dark web. Just... inaccessible to anything that crawls the web the way search engines do.

I'd heard the "deep web" statistic before. Most people have. But I always assumed it was mostly junk — expired pages, duplicate databases, internal server logs. It wasn't until I started doing real market research across dozens of industry sites that I realized: the data I needed most was almost always in that invisible 90%. And once I saw it, I couldn't unsee it.

First, Let's Clear Up the Confusion

Whenever someone says "90% of the internet is hidden," half the room immediately thinks about the dark web — Tor, anonymous marketplaces, stolen credentials. That's not what we're talking about.

The web has three layers, and people mix them up constantly.

Surface web is everything Google can index. Public pages, blog posts, Wikipedia articles, news sites. This is where you spend most of your browsing time. Estimates vary — typically cited as 4-10% of all web content depending on methodology and what you count — but the point is consistent: it's a small fraction.

Deep web is everything behind a barrier that prevents search engine crawlers from accessing it. The flight prices that only appear after you enter dates and destinations. The supplier portal that requires a login. None of this is sinister — it's just not publicly crawlable. This makes up the vast majority of the web, around 90-96%.

Dark web is a tiny subset of the deep web that requires specialized software like Tor to access. According to Britannica, it's roughly 0.01% of the deep web — far smaller than most people imagine. It gets all the headlines, but in terms of size it's a rounding error.

The deep web is boring. That's the point. It's boring, and enormous, and full of exactly the kind of data that businesses desperately need.

What Actually Lives Behind the Wall?

Here's what makes the invisible web interesting: it's not a wasteland of forgotten pages. It's where the most valuable, most current, most actionable data on the internet lives.

Dynamic pricing and inventory. Every airline, hotel booking system, and e-commerce platform generates pricing based on inputs — dates, locations, user profiles. The price you see is computed on the fly. It doesn't exist as a static page for Google to crawl. If you want competitor pricing data, you have to interact with the site to generate it.

Authenticated portals. Government databases, insurance claim portals, enterprise SaaS dashboards, supplier catalogs — all behind login walls. The data is there, it's just not public. A procurement team that needs to compare pricing across 200 supplier portals can't Google their way to an answer. Each portal requires authentication, has a different interface, a different workflow.

Interactive search results. LinkedIn People Search. Zillow's filtered listings. Patent databases. Academic paper repositories. The results only exist after you type a query and apply filters. Before that interaction, the data is invisible to crawlers.

Form-gated content. Reports behind download forms. Tools that generate output based on user input. Calculators, configurators, quote generators. All invisible until a human (or something that acts like one) fills in the fields.

Single-page applications. Modern web apps built with React, Vue, or Angular often load a shell page and then fetch content dynamically. A search engine crawler that doesn't execute JavaScript sees an empty skeleton. The actual content only renders in a real browser environment.

This isn't obscure stuff. This is where most business-critical data lives today.

Why Search Engines Can't Fix This

Why can't Google just get better at this?

The answer is architectural. Search engines are built on a specific model: send a crawler to a URL, download what's there, index it, rank it. That model assumes content is static, public, and available at a fixed address. It's an incredibly powerful model for the surface web. But it fundamentally cannot handle content that requires interaction to exist.

Google's crawler can't log into your competitor's supplier portal. It can't fill out a form with your specific parameters to generate a custom quote. It can't scroll through an infinite-loading feed, click "next page" 47 times, and filter results by date range. It can't provide credentials, handle two-factor authentication, or navigate a multi-step checkout flow.

This isn't a limitation that gets fixed by better crawling technology. It's a limitation of the crawling paradigm itself. Crawling is about reading pages. The invisible web requires doing things on pages.

Google knows this, by the way. There's a reason Google Hotels uses third-party web agents to aggregate hotel inventory from thousands of Japanese booking sites that their own crawlers can't reach. When the company that built web search can't access web data with search technology, that tells you something about the structural boundary.

So What Can Reach It?

This is where the landscape gets interesting, and where I've been spending most of my time learning over the past few days.

"Agentic search" tools like Perplexity and Google's AI Overviews try to bridge the gap by synthesizing information from multiple sources. They're better than raw search for getting summarized answers, but they're still ultimately constrained by what's been indexed. They're smarter librarians, but the library hasn't gotten bigger.

Content extraction tools like Firecrawl go a step further — they can visit a URL, render JavaScript, and return clean content. This handles the single-page app problem. But they still can't interact with pages. If the data requires filling a form or clicking through filters, you're stuck.

Browser agents like Browser Use and OpenAI Operator are where things start to change. These are AI systems that actually navigate pages — clicking, typing, scrolling, filling forms, handling pop-ups. They can reach content that requires interaction. The limitation I've found is orchestration: when you need to run the same task across dozens or hundreds of sites in parallel, managing that yourself becomes its own infrastructure project.

Remote web agent platforms like TinyFish and Browserbase handle the orchestration part. Cloud-hosted browsers, parallel execution, structured output. I wrote about my experience testing several of these — the shift from "automate clicks" to "describe what you want" is real.

The framing I keep coming back to is the distinction between searching the web and operating on it. Search is about finding pages. Operating is about interacting with them — logging in, navigating workflows, extracting data from dynamic interfaces. These are fundamentally different activities, and they require fundamentally different tools. TinyFish's blog has some interesting writing on this if you want to go deeper.

The Economic Problem Nobody Talks About

Here's the angle that doesn't get enough attention: the invisible web isn't just a technical problem. It's an economic one.

Consider a procurement team that needs competitive pricing across 200 supplier portals. Each portal requires a login, has a unique interface, a different navigation flow. You could hire someone to manually check all 200. But the labor cost makes it prohibitive. So you check 5. Maybe 10. You make decisions based on incomplete data because complete data is economically inaccessible.

Or think about a pharmaceutical company matching patients to clinical trials. Eligibility criteria are scattered across thousands of fragmented research sites, each with its own search interface and data structure. No search engine indexes this. No API aggregates it.

Or an insurance company that needs to monitor prior authorization statuses across 50+ health plan portals — each one a different website, different login, different workflow to check status.

In all these cases, the data exists. It's not secret. But the cost of accessing it at scale, manually, is so high that most organizations simply don't. They make decisions with partial information and accept the inefficiency.

This is what makes the web agent category genuinely interesting to me — not as a cool technology demo, but as an economic unlock. When you can automate the interactive parts of the web at scale, you make data accessible that was previously too expensive to touch. That's not an incremental improvement. That's a category of decisions that was previously impossible to make.

One More Thing: WebMCP and the Future

There's a new W3C standard in early development called WebMCP that could eventually shrink the invisible web. The idea is that websites would publish structured tools that AI agents can call directly, instead of agents having to navigate the visual interface. I wrote a deeper explainer on how it works.

But here's the realistic take: WebMCP depends on website owners voluntarily adopting a new standard. The sites with the most valuable hidden data — legacy portals, government systems, enterprise SaaS — are the slowest to adopt anything. The invisible web will stay invisible for a long time. The question is who builds the bridge.

What This Means for Us

If you're building anything that depends on web data — competitive intelligence, market research, lead enrichment, pricing optimization — it's worth asking: how much of the data I need actually shows up in search results?

My guess is less than you think.

The surface web is the tip of the iceberg we've all been staring at. The real depth is underneath — behind logins, inside interactive interfaces, generated dynamically by forms and filters. It's not hidden in any dramatic sense. It's just waiting for something that can interact with it.

That's the gap web agents are filling. Not by indexing more pages, but by operating on the ones that already exist.

Further reading:

W3C WebMCP Specification — the emerging standard for making the web agent-readable
Firecrawl's Guide to Browser Agents — landscape overview from a content extraction perspective
Browser Use Documentation — open-source browser agent framework
Why We Launched With a Story About a Tiny Japanese Hotel — TinyFish on how invisible hotel data became accessible through web agents

WebMCP Explained: The New Standard That Turns Websites Into APIs for AI Agents

lazyasscoder — Fri, 03 Apr 2026 19:17:00 +0000

Here's a question that's been bugging me since I started digging into web agents: why do AI agents have to pretend to be humans?

Think about it. When an AI agent needs to book a flight on a travel site, it takes a screenshot of the page, sends that image to a vision model, waits for the model to figure out which pixel to click, then simulates a mouse click. Then it takes another screenshot. Then another model call. Repeat for every single interaction.

It's like asking someone to order food at a restaurant by reading the menu through binoculars from across the street, then shouting their order through the window. It works. Technically. But nobody would design it that way on purpose.

That's exactly what WebMCP is trying to fix.

What Is WebMCP, in Plain English?

WebMCP (Web Model Context Protocol) is a new browser API that lets websites say to AI agents: "Here's what I can do. Here are the buttons you'd normally click, but as structured functions you can call directly."

Instead of an agent squinting at a rendered page and guessing where the search box is, a WebMCP-enabled site publishes something like:

navigator.modelContext.registerTool({
  name: "searchProducts",
  description: "Search the product catalog by keyword",
  inputSchema: {
    type: "object",
    properties: {
      query: { type: "string" },
      category: { type: "string" }
    },
    required: ["query"]
  },
  execute: async (input) => await searchCatalog(input.query, input.category)
});

The agent sees: "Oh, this site has a searchProducts tool. It takes a query and an optional category. I'll just call it." No screenshots. No DOM parsing. No guessing.

It's the difference between handing someone a structured menu with item numbers versus making them read a chalkboard from across a noisy room.

Wait, Isn't This Just Anthropic's MCP?

No. And I'll admit this confused me for about two hours before I sorted it out.

Anthropic's MCP (Model Context Protocol) is a protocol for connecting AI agents to backend services — databases, APIs, file systems. It runs server-side. If you've used Claude's MCP integrations to connect to GitHub or Google Drive, that's Anthropic MCP.

WebMCP is a browser-native standard for connecting AI agents to web page interfaces. It runs client-side, inside the browser. It was announced by Google's Chrome team and is being co-developed with Microsoft through the W3C Web Machine Learning Community Group.

Anthropic MCP lets an agent talk to your server. WebMCP lets an agent talk to your website. Different layers, complementary purposes.

The naming overlap is genuinely unfortunate. But the relationship is intentional — WebMCP aligns its primitives (tools, schemas, execution) with MCP so that an agent speaking MCP can interact with WebMCP tools through the browser with minimal translation.

Two Ways to Add WebMCP to a Site

The spec offers two paths, and which one you'd use depends on how your site works.

The Declarative Path — For Existing Forms

If your site already has well-structured HTML forms, this is the low-effort path. You add a few attributes to your existing markup:

<form toolname="search-flights"
      tooldescription="Search for available flights by route and date">
  <input name="origin" type="text" placeholder="From (e.g., SFO)">
  <input name="destination" type="text" placeholder="To (e.g., NRT)">
  <input name="date" type="date">
  <button type="submit">Search</button>
</form>

The browser reads those attributes and automatically creates a structured tool schema from the form. An AI agent can then "fill out" the form by calling the tool with typed parameters — no DOM manipulation needed.

If your forms are already clean and semantic, you're most of the way there. A few HTML attributes and your site speaks agent.

The Imperative Path — For Complex Interactions

For anything beyond simple form submission — multi-step workflows, conditional logic, dynamic filtering, cart operations — you use JavaScript to register tools explicitly through navigator.modelContext.registerTool().

This is where things get powerful. You're essentially wrapping your existing frontend functions as AI-callable tools. The shopping cart logic you already wrote? Wrap it as a tool. Your search-and-filter pipeline? Same thing. You're reusing code, not writing new backend infrastructure.

The neat thing about this approach is that it keeps the human UI and the agent interface in sync. Both the user clicking "Add to Cart" and the agent calling add_to_cart() go through the same underlying code path. One interface, two consumers.

Why Should You Care?

If you're a web developer, you might be thinking: "Okay, cool spec. But why does this matter to me right now?"

Three reasons.

Cost. Every time an AI agent takes a screenshot of your page to figure out what's on it, that costs tokens. A single screenshot can run a couple thousand tokens through a vision model. When you're running agents at any kind of scale, the difference between "interpret a rendered screenshot" and "call a typed function" adds up fast. Early estimates suggest the token savings are dramatic — potentially an order of magnitude — though hard numbers will depend on the specific implementation and use case.

Reliability. Screenshot-based agents break when your site redesigns. Selector-based agents break when your CSS classes change. An agent calling searchProducts({ query: "laptop" }) doesn't care what your page looks like. The function contract stays stable even when the visual layer changes. It's the same reason APIs are more reliable than scraping — you're exposing intent, not implementation details.

Control. Right now, AI agents interact with your site by reverse-engineering it. You have no say in how they navigate, what they extract, or what actions they take. WebMCP flips that: you define which tools are available, what parameters they accept, and what they return. You're inviting agents in through the front door instead of watching them climb through the window.

What About Web Agent Platforms?

Here's where it gets interesting for the web agent space.

I wrote previously about how platforms like TinyFish, Browser Use, and Browserbase handle web automation today — by navigating sites visually, interpreting page content with LLMs, and executing browser actions. It works, and for the vast majority of the web, it's the only option.

WebMCP doesn't replace these platforms. It makes them faster on the sites that support it.

When a web agent arrives at a site that supports WebMCP, it can skip the entire perception layer — no screenshots, no DOM parsing — and call functions directly. When it arrives at a site that doesn't (which will be most sites for years to come), it falls back to the full visual navigation approach.

WebMCP is an "acceleration layer," not a replacement for agent architecture. The sites that most need automation — legacy portals, government systems, enterprise SaaS with no API — are precisely the ones least likely to adopt WebMCP quickly. A web agent platform still needs the full stack for those.

If anything, WebMCP makes the case for web agent platforms stronger, not weaker. Someone needs to handle both paradigms seamlessly — structured tool calls when available, full browser automation when not.

So Do Agents Still Have to Pretend to Be Human?

This brings us back to the question I opened with.

WebMCP gives us a glimpse of a web where agents don't have to pretend. On a site that publishes structured tools, an agent can just be an agent — calling functions, passing typed parameters, getting structured results. No screenshots. No guessing. No pretending.

But that's the future for a small slice of the web. Right now, and for years to come, the vast majority of sites won't support WebMCP. Government portals, legacy enterprise systems, the long tail of e-commerce — these are the sites where the most valuable data lives, and they'll be the last to adopt any new standard.

On those sites, agents still have to navigate like humans. They still need to interpret page layouts, click buttons, fill forms, handle CAPTCHAs. The "pretending" isn't going away. It's just that the best platforms are getting remarkably good at it.

This is where tools like TinyFish, Browser Use, and Browserbase become more relevant, not less. The real value of a web agent platform in a WebMCP world is being able to do both: call structured tools where they exist, and navigate the messy human-designed web everywhere else. The transition will take years. Someone has to bridge it.

The Reality Check

Let's be honest about where this stands.

Browser support: Chrome 146 Canary only, behind a feature flag. Microsoft is co-authoring the spec — Patrick Brosset from the Edge team has been actively involved — so Edge support is likely to follow. Firefox and Safari haven't committed to timelines.

Spec maturity: This is a W3C Draft Community Group Report, not a finalized standard. The API surface is still evolving — provideContext() has faced security concerns and is being phased out in favor of the more granular registerTool() approach. Expect breaking changes if you build on it today.

Adoption: Zero mainstream sites support it in production right now. Even optimistically, meaningful adoption is 12-18 months away for early movers, and years away for the long tail of the web.

Scope: The spec explicitly states that headless browsing and fully autonomous operation without human oversight are non-goals. WebMCP is designed for human-in-the-loop scenarios where users and agents collaborate in the same browser context.

If you're building a production system today, you don't build on WebMCP alone. But if you're architecting something with a 12-18 month horizon, you should be thinking about it.

What This Means for the Future of the Web

Here's the bigger picture that I keep thinking about.

For 30 years, the web has had one interface: the visual layer designed for human eyes and hands. Every attempt to interact with it programmatically — scraping, browser automation, headless Chrome — has been a workaround built on top of that human interface. Agents have had to pretend to be human because the web gave them no other option.

WebMCP is the first serious attempt to change that. Not by replacing the human web, but by adding a second layer alongside it — a structured, schema-driven interface that agents can interact with natively. The visual experience stays exactly the same for humans. But a parallel channel opens up for machines.

This is similar to what happened with accessibility APIs. The web originally had no structured way for screen readers to understand page content. Then ARIA came along and let developers annotate their pages with semantic meaning. The visual experience didn't change. But a parallel channel opened up for assistive technology.

WebMCP is doing the same thing, but for AI agents. And just as ARIA eventually became a standard expectation for web development, there's a version of the future where "agent-ready" becomes as normal as "mobile-responsive."

Some people are already calling the next wave of optimization "AEO" — Agent Experience Optimization — as a parallel to SEO. Whether that term sticks or not, the idea is real: if AI agents are increasingly how people interact with the web, making your site agent-friendly becomes a business advantage, not just a technical nice-to-have.

What I'd Do Right Now

If you're a web developer curious about WebMCP, here's what I'd suggest:

Start by making your existing forms clean and semantic. Well-structured HTML with clear labels and logical form fields is already 80% of the work for the declarative WebMCP path. This is good practice regardless of whether you ever add WebMCP attributes.

If you want to experiment, enable the flag in Chrome 146 Canary (chrome://flags/#enable-webmcp-testing) and try registering a simple tool. The W3C proposal has walkthrough examples, and there's a solid community example repo with implementations for vanilla TypeScript, React, Rails, Angular, and even Phoenix LiveView.

And keep an eye on the spec. It's moving fast. Google and Microsoft are actively iterating, and the API surface is likely to change before it stabilizes. Patrick Brosset from the Edge team has a good update post that covers the latest changes.

This is one of those moments where a web standard is being shaped in the open. If you have opinions about how AI agents should interact with websites, now is literally the time to get involved.

Further reading:

W3C WebMCP Specification Draft — the actual spec
WebMCP Proposal & Explainer — Google's detailed proposal document
Patrick Brosset: WebMCP Updates & Clarifications — latest status from the Edge team
WebMCP Example Repos — React, vanilla JS, Rails implementations
Christian Heilmann: WebMCP — Agents Play With Rather Than Against the Web — a thoughtful take from a web standards veteran
Why 90% of the Internet Is Invisible — context on why agent-web interaction matters

Web Scraping Is Dead. Web Agents Just Replaced It.

lazyasscoder — Wed, 01 Apr 2026 16:22:00 +0000

Last month, I needed to pull together a competitive landscape report. Nothing exotic — pricing trends, feature comparisons, market positioning across about 30 to 40 industry sites and competitor pages.

Sounds manageable, right? Here's what actually happened.

Half the sites loaded data dynamically — you scroll down, content appears, but view-source shows empty divs. A few required creating accounts just to see pricing. Others buried the good stuff behind three layers of filters and dropdown menus. And two sites hit me with CAPTCHAs the moment I tried to load them in an automated way.

I did what I always do: opened 40 Chrome tabs, spent two days copying and pasting into a spreadsheet, and ended up with a document that was already going stale by the time I finished formatting it.

That experience sent me down a rabbit hole. What I found changed how I think about web automation entirely.

Why Traditional Scraping Doesn't Cut It Anymore

Look, I'm not here to trash Beautiful Soup or Scrapy. If your target is a handful of static HTML pages that rarely change, those tools still work beautifully. No shame in using them. I still do for simple stuff.

But the web I was dealing with, the 2026 web, is a fundamentally different animal.

Most modern sites render content with JavaScript. Your scraper sends a request, gets back a skeleton of empty <div> tags, and has no idea that the actual data loads three seconds later through an API call triggered by a React component. Single-page apps mean the URL stays the same while the content changes entirely. Data hides behind interactive elements: filters, dropdowns, "load more" buttons, infinite scroll. And anti-bot systems have gotten genuinely sophisticated — they can tell the difference between a real browser with a real human behind it and a Python script pretending to be one.

Traditional scraping is like using a paper map in a city that redraws its streets every week. The map was accurate when you printed it. The city just didn't care.

The problem isn't that your code is bad. The problem is that the web wasn't designed to be scraped. It was designed to be used — by humans, in browsers, with clicks and scrolls and waiting for things to load. And every year it gets better at enforcing that assumption.

So I started looking at what else is out there. The landscape has changed a lot more than I expected.

I Tried a Bunch of Tools. Here's What Actually Happened.

Instead of just reading comparison articles, I spent a weekend actually testing different approaches against my real market research task. Here's the honest breakdown.

Search APIs (Exa, Tavily)

I tried these first because they're the fastest path to structured data. Give them a query, get back JSON with titles, snippets, and URLs. Some of them, like Exa, maintain their own semantic index that's optimized for LLM consumption. Tavily focuses on research-grade search results. Both are impressive for what they do.

But I hit the wall pretty quickly. They surface what's already been indexed and the data I needed wasn't sitting on pages that Google had crawled. It was behind login walls, inside interactive dashboards, and loaded dynamically after user interaction.

My takeaway: search APIs are perfect for discovery — figuring out WHICH pages have the data you want. But they can't go get it for you. It's like asking a librarian which shelf a book is on. Helpful. But they can't read it for you.

Content Extraction (Firecrawl)

Firecrawl was the logical next step. You give it a URL, it renders the JavaScript, and returns clean, structured content. It bridges the gap between "finding a page" and "reading a page."

I liked it for known URLs where I just needed to pull the rendered content. But when I needed to interact with the page — click through filters, paginate through results, fill out a form to reveal pricing — it wasn't built for that. It's still reading, not doing.

Browser Agents (Browser Use, OpenAI Operator)

This is where things got genuinely interesting. These are AI systems that can actually navigate pages — clicking buttons, filling forms, interpreting what's on screen and deciding what to do next.

Browser Use is open source and flexible. You can choose your own LLM, and their cloud offering means you're not tying up your own machine. I was impressed by how well the AI reasoned about page layouts.

OpenAI Operator has a polished interface and feels smooth for single-task workflows. Good for "go to this site and do this thing."

But here's where I ran into friction: I needed to run the same basic task across 30+ sites. Not one site at a time. All of them, ideally in parallel. Orchestrating that myself — managing sessions, handling failures, collecting structured output — started feeling like I was building infrastructure rather than doing research.

Remote Web Agent Platforms (TinyFish, Browserbase)

This is the category I didn't know existed until I went looking.

Browserbase is impressive cloud browser infrastructure. They spin up headless browsers at scale and handle the hard parts — proxies, anti-detection, session management. But it's fundamentally an infrastructure layer. You connect your own automation code to their browsers. Powerful for teams that want full control, but you're still writing the agent logic yourself.

Then I tried TinyFish. You give it a URL and a goal in plain English. It gives you structured JSON back. No selectors. No Playwright scripts. No browser management. I didn't have to think about HOW to navigate each site — just WHAT I wanted from it.

That felt fundamentally different. Not a better scraper. Not a fancier browser automation. A different way of thinking about the problem.

Now, I want to be clear about something: every tool I tried has its sweet spot. Exa is fantastic for semantic search. Firecrawl is great for clean content extraction from known URLs. Browser Use gives you maximum flexibility with open-source transparency. Browserbase is the right call if your team wants to build custom agent infrastructure.

I'm not saying TinyFish is the only answer. I'm saying that the experience of "describe what you want, get it back" was the thing that made me rethink the whole approach. It was the moment I realized web automation might be going through the same kind of shift that happened when we went from writing assembly to writing in high-level languages. The abstraction layer matters.

The Real Shift: From "Automate Clicks" to "Describe Goals"

Here's the idea that stuck with me after trying all of this.

The old way to automate the web was procedural. You'd write something like: "Navigate to this URL. Find the element with class price-card. Click the dropdown labeled 'Enterprise'. Wait 2 seconds. Extract the text from the element with ID annual-price."

Every single step is hard-coded. Change the class name? It breaks. Add a cookie banner that overlays the dropdown? It breaks. Redesign the page? Everything breaks.

The new approach is goal-oriented. You say: "Go to this company's pricing page. Find the enterprise annual price. Return it as JSON."

The agent figures out the how. You only specify the what.

This might sound like a small difference. It isn't. It's the difference between giving a taxi driver turn-by-turn directions versus giving them an address. If the road is closed, turn-by-turn fails — the driver is stuck at the non-existent left turn. But if you gave them the destination, they just find another route. The goal hasn't changed. Only the path.

And it changes who can automate web tasks. You don't need to understand the DOM, write XPath expressions, or manage headless browser sessions. You need to be able to describe what you want. For someone like me — technical enough to understand what's happening under the hood, but not wanting to maintain Playwright infrastructure for a market research project — this shift feels overdue.

Why I Kept Digging Into This

After my initial test, I got curious about TinyFish's thinking. Not as a customer evaluating a purchase, more like a nerd who found something interesting and wanted to understand why it worked differently.

Their blog has this argument that I keep coming back to: search engines only index about 5% of online content. The other 90% or more sits behind logins, forms, dynamic interfaces, and interactive workflows. It's not hidden in the sense that it's secret. It's hidden because you have to interact with a website to reveal it.

That matched my experience exactly. The market data I needed wasn't hard to find. I could see it in my browser just fine. It was hard to extract programmatically without writing a fragile script for every single site.

I've only scratched the surface of what their platform can do, but the way they frame the problem — that the web needs to be operated on, not just searched or scraped, resonates with what I experienced firsthand.

Where This Is All Going: WebMCP

One more thing worth mentioning, because it signals where this whole space is heading.

Google and Microsoft are jointly developing a new W3C standard called WebMCP (Web Model Context Protocol) that would let websites publish structured tools directly for AI agents. Instead of agents having to visually interpret a page — taking screenshots, parsing the DOM, guessing where to click — a site could expose a function like get_pricing({ plan: "enterprise" }) that agents call directly through a new browser API called navigator.modelContext.

(Quick note for the confused: WebMCP is not the same thing as Anthropic's MCP. Anthropic's MCP connects AI agents to backend services. WebMCP is a browser-native standard that connects agents to web page interfaces. Different protocols, complementary purposes. The naming is confusing — I know.)

Think of it this way: instead of making a delivery robot squint at a restaurant's chalkboard menu from across the room, you hand it a structured list of dishes and prices. Same information. Completely different efficiency.

But this is still extremely early. Most sites won't support it for years. And the sites that most need automation — legacy portals, government systems, enterprise SaaS with no API — are precisely the ones least likely to adopt it quickly. But it signals a future where the web isn't just built for human eyes. It's becoming readable by agents too.

What I'm Doing Next

The web automation landscape has genuinely shifted. "Scraping" as I knew it — writing brittle selector-based scripts — is being replaced by something more like "describing what you want and getting it back."

You don't have to throw out your working scripts overnight. Start with the workflows that break most often. The ones that cost you a day of copy-pasting every month. Those are your candidates for a different approach.

I'm planning to take this further: pick a real market research workflow I currently handle manually, run it through a proper hands-on test, and write up exactly what happens. The good, the bad, and the weird. Stay tuned for that.

If you've been fighting with web scrapers, I'd love to hear what's worked for you — and what hasn't. Drop a comment below.

Further reading if you want to go deeper:

Why 90% of the Internet Is Invisible (And Why AI Hasn't Fixed It) — TinyFish's blog on the invisible web problem
Firecrawl's Guide to Browser Agents — a good overview of the browser agent landscape from another player in the space
Browser Use Documentation — if you want to explore the open-source agent approach

How to Optimize Front-end Performance for Faster Load Times

lazyasscoder — Mon, 02 Dec 2024 16:02:30 +0000

How to Optimize Front-end Performance for Faster Load Times

As web developers, one of our primary goals is to ensure that our websites load swiftly and efficiently. Front-end performance directly impacts user experience, search engine rankings, and overall site usability. In an era where users expect instant access to information, optimizing front-end performance is not just a best practice—it's a necessity. This article delves into why front-end load speed is crucial and explores practical techniques for optimizing load times, including leveraging tools like EchoAPI for performance comparison.

The Importance of Front-end Load Speed

Front-end load speed refers to the time it takes for a web page to become fully interactive for the user. This aspect is critical for several reasons:

User Experience: Fast-loading pages offer a smoother and more satisfying user experience, reducing bounce rates and increasing engagement.
SEO Benefits: Search engines, especially Google, consider page load speed as a ranking factor, which can significantly impact your site's visibility in search results.
Conversion Rates: Faster websites typically yield higher conversion rates, as users are more likely to stay and complete actions on a site that loads quickly.
Mobile Accessibility: With an increasing number of users accessing the web via mobile devices, optimizing for load speed ensures better accessibility and usability across all devices.

Practical Examples of Front-end Optimization

Here are some effective techniques to optimize front-end performance:

1. Minimize HTTP Requests

Reduce the number of elements on your page (scripts, images, CSS) since each one requires an HTTP request. Use CSS Sprites to combine multiple images into one.

<!-- Combine multiple CSS files into one -->
<link rel="stylesheet" href="styles.min.css">

2. Use Content Delivery Networks (CDNs)

CDNs distribute your content closer to the user's geographical location, reducing load times.

<link rel="stylesheet" href="https://cdn.example.com/styles.min.css">
<script src="https://cdn.example.com/scripts.min.js"></script>

3. Optimize Images

Compress images using tools like TinyPNG or ImageOptim without sacrificing quality. Prefer modern formats like WebP over traditional ones.

<img src="image.webp" alt="Optimized Image" loading="lazy">

4. Leverage Browser Caching

Set appropriate cache headers to store static resources in the user's browser, reducing the number of requests for subsequent visits.

<IfModule mod_expires.c>
  ExpiresActive On
  ExpiresByType image/jpg "access 1 year"
  ExpiresByType image/jpeg "access 1 year"
  ExpiresByType image/gif "access 1 year"
  ExpiresByType image/png "access 1 year"
  ExpiresByType text/css "access 1 month"
  ExpiresByType application/pdf "access 1 month"
  ExpiresByType text/x-javascript "access 1 month"
  ExpiresByType application/x-shockwave-flash "access 1 month"
  ExpiresByType image/x-icon "access 1 year"
</IfModule>

5. Minify and Compress Files

Minify CSS, JavaScript, and HTML files. Use Gzip or Brotli compression to reduce the size of these files.

# Example using Gulp to minify JavaScript
const gulp = require('gulp');
const uglify = require('gulp-uglify');

gulp.task('minify-js', function() {
  return gulp.src('src/**/*.js')
    .pipe(uglify())
    .pipe(gulp.dest('dist'));
});

6. Implement Lazy Loading

Delay loading non-critical resources such as images and videos until they're needed.

<img src="image.jpg" alt="Lazy Loaded Image" loading="lazy">
<video width="600" controls loading="lazy">
  <source src="movie.mp4" type="video/mp4">
</video>

Conclusion

Optimizing front-end performance is an ongoing process that requires a comprehensive approach, combining best practices and modern tools. By minimizing HTTP requests, leveraging CDNs, optimizing images, implementing caching, minifying files, and using lazy loading, you can significantly improve load times. Tools like EchoAPI further aid in monitoring and comparing performance, ensuring that optimizations lead to measurable benefits. Faster load times enhance user experience, boost SEO rankings, and drive higher conversion rates—key factors for any successful web project. Happy optimizing!