How to Scrape Any Website Using Python, Bright Data, and MCP Servers

brd-customer-hl_b2705b7f-zone-mcp_scraping_browser:50wpsh0oa734

uv pip install -r requirements.txt

MISTRALAI_API_KEY=your_mistralai_key
API_TOKEN=your_brightdata_token
BROWSER_AUTH=your_browser_api_key
WEB_UNLOCKER_ZONE=your_zone_name

# Imports
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from langchain_mcp_adapters.tools import load_mcp_tools
from langgraph.prebuilt import create_react_agent
from langchain_mistralai import ChatMistralAI
from dotenv import load_dotenv
import asyncio
import os

# Load environment variables
load_dotenv()

# Initialize the model
model = ChatMistralAI(model="mistral-large-latest", api_key=os.getenv("MISTRALAI_API_KEY"))

# Initialize the server parameters
server_params = StdioServerParameters(
    command="C:\\Program Files\\nodejs\\npx.cmd",   # In Windows, other OS you can use "npx"
    env={
        "API_TOKEN": os.getenv("API_TOKEN"),
        "BROWSER_AUTH": os.getenv("BROWSER_AUTH"),
        "WEB_UNLOCKER_ZONE": os.getenv("WEB_UNLOCKER_ZONE"),
    },
    args=["@brightdata/mcp"],
)

# Define the chat function
async def chat_with_agent():
    # Initialize the client
    async with stdio_client(server_params) as (read, write):
        # Initialize the session
        async with ClientSession(read, write) as session:
            # Initialize the session
            await session.initialize()
            # Load the tools
            tools = await load_mcp_tools(session)
            # Create the agent
            agent = create_react_agent(model, tools)

            # Start conversation history, this is the initial message from the system prompt
            messages = [
                {
                    "role": "system",
                    "content": "You can use multiple tools in sequence to answer complex questions. Think step by step.",
                }
            ]

            # Start the conversation
            print("Type 'exit' or 'quit' to end the chat.")
            while True:
                # Get the user's message
                user_input = input("\nYou: ")

                # Check if the user wants to end the conversation
                if user_input.strip().lower() in {"exit", "quit"}:
                    print("Goodbye!")
                    break

                # Add the user's message to the conversation history
                messages.append({"role": "user", "content": user_input})

                # Invoke the agent with the full message history
                agent_response = await agent.ainvoke({"messages": messages})

                # Get the agent's reply
                ai_message = agent_response["messages"][-1].content

                # Add the agent's reply to the conversation history
                messages.append({"role": "system", "content": ai_message})

                # Print the agent's reply
                print(f"Agent: {ai_message}")

# Run the chat function
if __name__ == "__main__":
    # Run the chat function asynchronously
    asyncio.run(chat_with_agent())

uv run main.py

(.venv) D:\GitHub\mcp-scrape-web-article>uv run main.py
Type 'exit' or 'quit' to end the chat.

You: search linkedin and indeed for remote ai developer jobs. for the top 2 jobs each job extract: job title, com
pany, location, salary, date and url
Agent: The top 2 remote AI developer jobs from LinkedIn are:

1. **Artificial Intelligence (AI) Engineer/Developer (Remote)**
   - **Company**: Statherósè
   - **Location**: Remote
   - **Date**: Not specified
   - **Job Title**: Artificial Intelligence (AI) Engineer/Developer
   - **Salary**: Not specified
   - **Job Type**: Full-time
   - **Experience Level**: Mid-Senior level
   - **Industries**: Computer Software
   - **Job Function**: Information Technology
   - **URL**: https://www.linkedin.com/jobs/view/artificial-intelligence-ai-engineer-developer-remote-at-stather%C3%B3s%C2%AE-4089407234

2. **AI/ML Developer - REMOTE WORK**
   - **Company**: Primus Global Technologies Pvt Ltd
   - **Location**: Remote
   - **Date**: Not specified
   - **Job Title**: AI/ML Developer
   - **Salary**: Not specified
   - **Job Type**: Contract
   - **Experience Level**: Not specified
   - **Industry**: Information Technology & Services
   - **Job Function**: Engineering
   - **URL**: https://www.linkedin.com/jobs/view/ai-ml-developer-remote-work-at-primus-global-technologies-pvt-ltd-4182993980

The top 2 remote AI developer jobs from Indeed are:

1. **Hackathon AI Curriculum Developer**
   - **Company**: Devpost
   - **Location**: Remote in New York, NY
   - **Date**: Not specified
   - **Salary**: $25 - $100 per hour
   - **Experience Level**: 3+ years of experience teaching or creating educational content for developers, including AI content that reaches a wide audience.
   - **Job Type**: Part-time
   - **Industry**: Not specified
   - **Job Function**: Not specified
   - **URL**: https://www.indeed.com/company/Devpost/jobs?jk=13e7814d20d9b310&from=serp&spa=1&utm_campaign=serp-more

2. **Head of Artificial Intelligence (AI)**
   - **Company**: Divergent Talent
   - **Location**: Remote in Palo Alto, CA
   - **Date**: Not specified
   - **Salary**: $200,000 - $275,000 per year
   - **Experience Level**: Not specified
   - **Job Type**: Full-time
   - **Industry**: Not specified
   - **Job Function**: Not specified
   - **URL**: https://www.indeed.com/company/divergent-talent/jobs?jk=a9d9ec0c913a3f1f&from=serp&spa=2&utm_campaign=serp-more

Type 'exit' or 'quit' to end the chat.

You: search reddit and return the top 5 complaints about SaaS products
Agent: [{'type': 'text', 'text': 'Here are the top 5 complaints:\n\n1. **Subscription Traps**: Users often complain about being trapped into subscriptions with no clear way to cancel without contacting the company directly. This is seen as a deceptive practice that locks users into unwanted payments'}, {'type': 'reference', 'reference_ids': [1]}, {'type': 'text', 'text': '.\n\n2. **Scalability Issues**: As the user base expands, many SaaS products face significant challenges in scaling their infrastructure. This can lead to performance degradation and a poor user experience, which is a common frustration among users'}, {'type': 'reference', 'reference_ids': [2]}, {'type': 'text', 'text': '.\n\n3. **Lack of Product-Market Fit**: A frequent complaint is that many SaaS products fail because they do not meet the needs of their target market. This misalignment leads to dissatisfaction and low adoption rates'}, {'type': 'reference', 'reference_ids': [3]}, {'type': 'text', 'text': '.\n\n4. **Customer Support Problems**: Poor customer support is a recurring issue, with users often frustrated by the lack of timely and effective assistance when they encounter problems with the SaaS product'}, {'type': 'reference', 'reference_ids': [4]}, {'type': 'text', 'text': '.\n\n5. **Complex Onboarding Process**: Users often struggle with complicated onboarding processes that hinder their ability to quickly realize the value of the SaaS solution. A non-intuitive setup can lead to frustration and a reluctance to fully adopt the product'}, {'type': 'reference', 'reference_ids': [5]}, {'type': 'text', 'text': '.'}]