Forem: PROXYCLAW

LangChain + ProxyClaw: Build an Agent That Can Actually Browse the Web

PROXYCLAW — Mon, 13 Apr 2026 14:21:45 +0000

You've built a LangChain agent. It can reason. It can use tools. It can call APIs.

Then you point it at the web — and it immediately breaks.

CAPTCHAs. IP bans. Cloudflare blocks. JavaScript-heavy pages that return blank HTML. The web was not built for agents, and it actively fights back.

This post shows you how to fix that — using ProxyClaw as a drop-in web access layer for your LangChain agent.

The Problem in One Line

When your LangChain agent tries to fetch a URL, it's making a raw HTTP request from a cloud IP. Every anti-bot system on earth knows that pattern and blocks it instantly.

ProxyClaw routes those requests through 2M+ real residential IPs, solves CAPTCHAs automatically, and returns clean Markdown instead of raw HTML — exactly what an LLM needs.

Install

pip install langchain-proxyclaw

Or if you're using the core SDK:

pip install iploop-sdk

3-Line Integration

Here's the minimal setup:

from langchain_proxyclaw import ProxyClawLoader

loader = ProxyClawLoader(url="https://amazon.com/dp/B09X7CRKRZ")
docs = loader.load()
print(docs[0].page_content)

That's it. ProxyClaw handles the proxy rotation, CAPTCHA solving, and Markdown conversion. You get clean text back, ready to feed into your chain.

Building a Real Agent

Let's build a LangChain agent that can browse any URL and answer questions about it.

from langchain.agents import initialize_agent, AgentType
from langchain.tools import Tool
from langchain_openai import ChatOpenAI
from iploop import ProxyClawClient

client = ProxyClawClient(api_key="your_api_key")  # Free tier: proxyclaw.ai

def browse_url(url: str) -> str:
    """Fetch a URL and return clean Markdown content."""
    result = client.fetch(url, format="markdown")
    return result.content[:4000]  # trim for context window

browse_tool = Tool(
    name="BrowseURL",
    func=browse_url,
    description="Fetches a URL and returns the page content as clean Markdown. Use this to read web pages, product listings, documentation, or any public URL."
)

llm = ChatOpenAI(model="gpt-4o", temperature=0)

agent = initialize_agent(
    tools=[browse_tool],
    llm=llm,
    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    verbose=True
)

result = agent.run("What are the top reviews for https://amazon.com/dp/B09X7CRKRZ?")
print(result)

The agent now has real web access. It can read product pages, scrape pricing, pull documentation, or gather competitive intel — without getting blocked.

Why Markdown Instead of HTML?

Raw HTML is toxic for LLMs. A typical product page is 90% boilerplate — nav bars, scripts, ads, footer links. An LLM wastes context tokens on noise and misses the signal.

ProxyClaw strips all of that and returns just the content: headings, paragraphs, lists, prices, reviews. The LLM gets clean signal, not garbage.

# Raw HTML approach — bad
import requests
html = requests.get("https://example.com").text
# Result: 50KB of scripts, nav, ads, and maybe 500 words of actual content

# ProxyClaw approach — good
result = client.fetch("https://example.com", format="markdown")
# Result: 800 words of clean, structured Markdown

Handling Dynamic Sites

Some sites only render content via JavaScript — React, Vue, Next.js apps that return an empty shell on first load. ProxyClaw handles this automatically with headless rendering.

result = client.fetch(
    url="https://some-spa-app.com/products",
    format="markdown",
    render_js=True  # enables headless browser rendering
)
print(result.content)

No Playwright setup. No Selenium. No managing browser instances in your agent infrastructure. ProxyClaw does it remotely.

Free Tier

ProxyClaw has a free tier: 0.5GB per month, no credit card required.

That's enough for thousands of page fetches in development. Paid plans start at $1.50/GB — compare that to BrightData at $8-15/GB.

What's Next

In the next article: scraping Zillow, Amazon, and Reddit with 3 lines of Python — no browser, no proxy management, no blocked requests.

GitHub: github.com/Iploop/proxyclaw
Docs: iploop.io/docs

Scraping Zillow, Amazon, and Reddit with 3 Lines of Python (No Browser, No Blocks)

PROXYCLAW — Mon, 13 Apr 2026 14:21:09 +0000

Zillow blocks scrapers. Amazon detects bots in milliseconds. Reddit rate-limits anything that isn't a human clicking slowly.

These are the three sites developers most want to scrape — and the three sites that are hardest to scrape reliably.

This post shows you how to scrape all three with 3 lines of Python each. No Playwright. No Selenium. No proxy rotation code. No CAPTCHA solving logic. Just clean data.

Why Scrapers Fail on These Sites

Zillow: JavaScript-rendered listings, Cloudflare protection, and aggressive bot fingerprinting. A raw requests.get() returns a 403 before it even loads the page.

Amazon: Product pages require residential IPs. Cloud IPs (AWS, GCP, Azure) are blacklisted. They also serve fake "bot detected" pages that look like real HTML but contain no product data.

Reddit: Rate-limits at the HTTP level, serves CAPTCHAs to anything that looks automated, and their official API now costs money for anything above toy usage.

The common thread: all three detect non-human traffic patterns and block them at the network level.

The Fix: Residential Proxies + Clean Output

ProxyClaw routes requests through real residential IPs (2M+ across 195 countries), handles CAPTCHA solving automatically, and returns clean Markdown instead of raw HTML.

pip install iploop-sdk

Get a free API key at proxyclaw.ai — 0.5GB free, no credit card.

Scraping Zillow

3 lines:

from iploop import ProxyClawClient

client = ProxyClawClient(api_key="your_api_key")
result = client.fetch("https://www.zillow.com/homes/for_sale/New-York_rb/", format="markdown")
print(result.content)

Output: Clean Markdown with listing addresses, prices, bed/bath counts, and square footage. No HTML parsing. No BeautifulSoup selectors that break every time Zillow redesigns their site.

Getting Structured Data

result = client.fetch(
    "https://www.zillow.com/homes/for_sale/New-York_rb/",
    format="json",
    extract="listings"
)

for listing in result.data["listings"]:
    print(f"{listing['address']} — ${listing['price']:,} — {listing['beds']}bd/{listing['baths']}ba")

Scraping Amazon

Product pages, reviews, pricing — 3 lines:

from iploop import ProxyClawClient

client = ProxyClawClient(api_key="your_api_key")
result = client.fetch("https://www.amazon.com/dp/B09X7CRKRZ", format="markdown")
print(result.content)

You get the product title, description, features, pricing, and top reviews in clean Markdown. No fake "Robot Check" pages. No blank HTML shells.

Price Monitoring Example

import json
from iploop import ProxyClawClient

client = ProxyClawClient(api_key="your_api_key")

def get_amazon_price(asin: str) -> dict:
    result = client.fetch(
        f"https://www.amazon.com/dp/{asin}",
        format="json",
        extract="product"
    )
    return {
        "asin": asin,
        "title": result.data["title"],
        "price": result.data["price"],
        "rating": result.data["rating"],
        "review_count": result.data["review_count"]
    }

asins = ["B09X7CRKRZ", "B08N5WRWNW", "B07ZPKN6YR"]
prices = [get_amazon_price(asin) for asin in asins]
print(json.dumps(prices, indent=2))

Run this as a cron job, store to a database, and you have a price tracker with zero browser overhead.

Scraping Reddit

Threads, comments, upvotes — without the API costs:

from iploop import ProxyClawClient

client = ProxyClawClient(api_key="your_api_key")
result = client.fetch("https://www.reddit.com/r/MachineLearning/top/?t=week", format="markdown")
print(result.content)

Pulling Thread Comments

result = client.fetch(
    "https://www.reddit.com/r/python/comments/1abc123/some_thread/",
    format="json",
    extract="comments"
)

for comment in result.data["comments"][:10]:
    print(f"[{comment['score']} pts] {comment['author']}: {comment['body'][:200]}")

Useful for sentiment analysis, trend detection, or feeding community discussion into an LLM pipeline.

Combining All Three: A Market Research Agent

from iploop import ProxyClawClient
from langchain_openai import ChatOpenAI
from langchain.schema import HumanMessage

client = ProxyClawClient(api_key="your_api_key")
llm = ChatOpenAI(model="gpt-4o")

def research_market(product_query: str, amazon_asin: str):
    amazon = client.fetch(f"https://www.amazon.com/dp/{amazon_asin}", format="markdown")
    reddit = client.fetch(
        f"https://www.reddit.com/search/?q={product_query.replace(' ', '+')}&sort=top",
        format="markdown"
    )
    context = f"## Amazon\n{amazon.content[:2000]}\n\n## Reddit\n{reddit.content[:2000]}"
    response = llm.invoke([HumanMessage(content=
        f"Summarize the market for '{product_query}': {context}"
    )])
    return response.content

print(research_market("wireless earbuds", "B09X7CRKRZ"))

Three data sources. One LLM call. Zero blocked requests.

Performance Notes

ProxyClaw adds ~1-3 seconds of latency per request. For price monitoring, market research, and data pipelines, this is completely acceptable.

For high-frequency scraping, use the batch API:

urls = [
    "https://amazon.com/dp/B09X7CRKRZ",
    "https://amazon.com/dp/B08N5WRWNW",
    "https://zillow.com/homedetails/123-main-st/12345_zpid/"
]

results = client.fetch_batch(urls, format="markdown")
for r in results:
    print(r.url, "—", len(r.content), "chars")

Free Tier

0.5GB free per month. No credit card. Roughly 5,000–10,000 page fetches — enough to build and test a full scraping pipeline.

Paid plans: $1.50/GB vs BrightData at $8–15/GB.

What's Next

Wire it into a full agent pipeline — ProxyClaw + LangChain memory + structured output parsers = agents that can autonomously research, monitor, and report on anything on the web.

GitHub: github.com/Iploop/proxyclaw
Docs: iploop.io/docs

Why Your AI Agent Can't Browse the Web (And How to Fix It in 3 Lines of Python)

PROXYCLAW — Sun, 12 Apr 2026 10:23:37 +0000

You've built a LangChain agent. It can reason, plan, and use tools. Then you try to give it access to the web — and it immediately hits a wall.

CAPTCHAs. IP bans. Cloudflare challenges. JavaScript-heavy pages that return blank HTML. Sites that detect non-browser traffic and block it instantly.

The web was built for humans. And it actively fights back against anything that looks like a bot.

This post explains exactly why this happens — and shows you how to fix it in 3 lines of Python.

The Problem: The Web Blocks Agents

When your AI agent tries to fetch a webpage, here's what actually happens:

1. No browser fingerprint

Real browsers send hundreds of signals: user-agent strings, TLS fingerprints, canvas/WebGL data, screen resolution, installed fonts, timing patterns. Sites like Cloudflare, Akamai, and Datadome analyze all of this in milliseconds. A raw requests.get() call fails every single check.

2. Datacenter IP = instant block

When you run an agent on AWS, GCP, or any cloud VM, your requests come from a datacenter IP range. Most anti-bot systems maintain blocklists of these ranges. Your request never even reaches the server logic — it's dropped at the edge.

3. JavaScript rendering

Roughly 60% of modern websites render their content via JavaScript. A standard HTTP request gets you the skeleton HTML — the actual data is loaded dynamically after JS runs. Without a full browser engine, you get empty pages.

4. CAPTCHAs

Even if you get past the above, CAPTCHAs are the last line of defense. Your agent can't solve them. The request dies there.

Here's what this looks like in practice:

import requests

# This fails on ~60% of real websites
r = requests.get("https://www.zillow.com/homes/NYC_rb/")
print(r.status_code)  # 403 or 200 with empty/blocked content

Why Existing Proxy Solutions Don't Work for Agents

You might think: "I'll just use a proxy service." The problem is that existing proxy providers — BrightData, Oxylabs, Smartproxy — were built for human-operated scraping workflows, not autonomous AI agents.

The issues:

They return raw HTML. Your LLM has to parse HTML, extract content, handle encoding, strip scripts and styles. That's hundreds of extra tokens just for noise.
Clunky SDKs designed for scraping pipelines, not agent tool calls.
Per-GB pricing that doesn't match how agents consume data.
No native integration with LangChain, CrewAI, LangGraph, or any agent framework.

The Fix: A Web Access Layer Built for Agents

We built ProxyClaw specifically to solve this. It's an open source web access layer that routes your agent's requests through 2M+ residential IPs, handles anti-bot bypass automatically, and returns clean Markdown or JSON — not raw HTML.

Here's the 3-line fix:

from iploop import IPLoop

client = IPLoop(api_key="YOUR_KEY", country="US")
r = client.fetch("https://www.zillow.com/homes/NYC_rb/")

print(r)  # Clean Markdown, ready for your LLM

That's it. The SDK handles everything underneath:

Routes through a residential IP (not a datacenter)
Applies browser fingerprinting and stealth mode
Solves CAPTCHAs automatically
Renders JavaScript if needed
Returns Markdown (not raw HTML)

Real-World Results

We tested 66 of the most bot-protected sites — Amazon, Reddit, Zillow, LinkedIn, Cloudflare-protected pages, Akamai-protected pages — and ran each one 5 times back-to-back.

66/66 passed. 100% success rate.

Here's a sample from the test suite:

client = IPLoop(api_key="YOUR_KEY")

# E-commerce (Cloudflare protected)
r = client.fetch("https://www.amazon.com/s?k=laptops")       # ✅ 2.1MB
r = client.fetch("https://www.walmart.com/browse/electronics") # ✅ 2.5MB

# Real estate (heavy JS + bot protection)  
r = client.fetch("https://www.zillow.com/homes/NYC_rb/")      # ✅ 1.3MB

# Social (aggressive bot detection)
r = client.fetch("https://www.reddit.com/r/python/")          # ✅ 890KB

LangChain Integration

Plugging ProxyClaw into a LangChain agent takes about 10 lines:

from langchain.agents import AgentExecutor, create_react_agent
from langchain.tools import tool
from iploop import IPLoop

client = IPLoop(api_key="YOUR_KEY")

@tooldef browse_web(url: str) -> str:
    """Browse a URL and return its content as Markdown."""
    return client.fetch(url)

# Add browse_web to your agent's tool list
tools = [browse_web]

Your agent can now browse any website, read product pages, pull news, scrape data — all without hitting bot blocks.

Node.js? Also covered.

const { IPLoop } = require('iploop');

const client = new IPLoop('YOUR_API_KEY');
const result = await client.fetch('https://example.com', { country: 'US' });

Free Tier + Earn Credits

ProxyClaw has a free tier — 0.5 GB with no credit card required. Get your API key at platform.iploop.io.

There's also an unusual earn model: run a Docker node and share bandwidth to earn proxy credits.

docker run -d --name iploop-node --restart=always ultronloop2026/iploop-node:latest

Running it 24/7 earns ~70 GB/month free. It's like mining, but for proxy credits.

Pricing beyond the free tier starts at $1.50/GB — vs $8-15/GB on BrightData.

Summary

If your AI agent needs to browse the web, here's what you're up against:

Problem	Why it happens	ProxyClaw fix
403 / blocked	Datacenter IP detected	Routes through residential IPs
CAPTCHA	Bot fingerprint detected	Stealth mode + auto CAPTCHA solve
Empty HTML	JavaScript rendering needed	Full JS render on request
Raw HTML to parse	Legacy scraping tools	Returns clean Markdown/JSON

Fix it with 3 lines. Free to start.

pip install iploop-sdk

GitHub (MIT): github.com/Iploop/proxyclaw

Use code OPENCLAW for 20% off any paid plan.