Forem: Zied

Py-Parkour

Zied — Thu, 11 Dec 2025 08:32:20 +0000

  _____       _--_          _                   
 |  __ \     |  _ \        | |                  
 | |__) |   _| |_) |__ _ __| | ___ ___  _   _ _ __ 
 |  ___/ | | |  _ // _` | '__| |/ / _ \| | | | '__|
 | |   | |_| | | \ \ (_| | |  |   < (_) | |_| | |  
 |_|    \__, |_|  \_\__,_|_|  |_|\_\___/ \__,_|_|  
         __/ |                                     
        |___/

Version: 2.0.0

Author: zinzied (zinzied@gmail.com)

🏃 Py-Parkour: The Hybrid Scraper Framework

Py-Parkour is a lightweight automation utility designed to solve the biggest annoyances in modern web scraping:

🍪 Cookie Consents: Detecting and destroying GDPR/modal popups.
🧭 Pagination: Auto-detecting "Next" buttons or infinite scroll.
🎭 Verification Gates: Generating temporary identities (Email/SMS) for signups.
👻 Hybrid Scraping: Break in with the browser, then steal the session for fast API calls.
📡 API Discovery: Automatically detect hidden JSON APIs.

It turns your scraper into a workflow automation platform.

📦 Installation

pip install py-parkour[full]

Or for development:

pip install -r requirements.txt
playwright install

🚀 How to Use It

1. The "Unified" Bot

The ParkourBot is your main entry point. It wraps a Playwright browser and gives you access to all gadgets.

import asyncio
from py_parkour import ParkourBot

async def main():
    bot = ParkourBot(headless=False)
    await bot.start()
    await bot.goto("https://target-website.com")
    # ... use gadgets here ...
    await bot.close()

asyncio.run(main())

2. 🍪 Gadget: Crusher (Cookie Bypasser)

Don't write brittle selectors for every "Accept Cookies" button.

await bot.crush_cookies()

3. 🧭 Gadget: Compass (Auto-Pagination)

Stop guessing if the site uses ?page=2 or a "Next >" button.

async for page_number in bot.crawl(max_pages=10):
    print(f"Scraping Page {page_number}: {bot.current_url}")

4. 🎭 Gadget: Disguises (Temp Identity)

Need to sign up to view data? Generate a burner identity.

identity = await bot.identity.generate_identity(country="US")
print(f"Using email: {identity.email}")

code = await bot.identity.wait_for_code()
await bot.driver.page.fill("#otp-input", code)

5. ✨ The "Magic" Auto-Setup

Try to automate the entire signup flow (Experimental).

await bot.auto_setup_identity("https://example.com/signup")

6. 👻 Gadget: Shadow (Session Bridge) ⭐ NEW

Stop choosing between "fast" (requests) and "capable" (browser). Use both.
Break in with the browser, then steal the session for high-speed API calls.

# 1. Login with the browser
await bot.goto("https://target.com/login")
# ... do login stuff ...

# 2. Transfer the session to a fast aiohttp client
async with await bot.shadow.create_session() as session:
    async with session.get("https://target.com/api/secret-data") as resp:
        print(await resp.json())

7. 📡 Gadget: Radar (API Detector) ⭐ NEW

Why scrape HTML if there's a hidden API? Radar listens to background traffic.

await bot.goto("https://complex-spa-site.com")

# Check what we found
print(f"Latest JSON found: {bot.radar.latest_json}")

# Replay captured requests
for req in bot.radar.requests:
    if "api/v1/users" in req['url']:
        print(f"Found User API: {req['url']}")

🎯 Where to use it?

Py-Parkour is best for:

Complex Scraping: Sites that require login or have heavy popups.
QA Automation: Testing "User Registration" flows without using real email addresses.
Bot Development: Quickly spinning up bots that need to pass "verify your email" checks.
API Hunting: Discovering undocumented APIs behind SPAs.

🏗 Architecture

Core: Async Playwright wrapper.
Gadgets: Modular tools attached to the bot (.crusher, .compass, .identity, .shadow, .radar).

Built with ❤️ for Scrapers who hate boilerplate.

TLS-Chameleon

Zied — Thu, 11 Dec 2025 08:28:34 +0000

TLS-Chameleon

Anti-Fingerprinting HTTP client that spoofs real browser TLS fingerprints with a simple, requests-like API.

🚀 Features

TLS Fingerprint Spoofing: Built-in profiles for Chrome, Firefox, Safari (uses curl_cffi for realistic signatures).
Persistent Sessions: Proper cookie handling and connection pooling (just like requests.Session).
Magnet Module 🧲: One-line data extraction (Emails, Tables, Forms, JSON-LD, Links).
Smart Static ⚡: Automatically fetch page assets (CSS/JS/Images) to mimic real browser traffic.
Auto-Form 📝: Find and submit forms automatically, handling hidden inputs and CSRF tokens.
Humanize 🧠: Built-in delays to mimic human reading/typing speed.
Resilience: Auto-rotation of proxies/profiles upon blocking (403/429/Cloudflare).

🆚 Why use this vs curl_cffi?

Feature	Raw curl_cffi	TLS-Chameleon
TLS Spoofing	You must manually set `impersonate="chrome110"`	Auto-Rotation: It likely has logic to rotate these so you don't get stuck on one.
Asset Loading	You just get the HTML.	`mimic_assets=True`: It parses the HTML and fetches CSS/JS/Images to look like a "real" browser visit (very important for some anti-bots).
Forms	You must manually parse CSRF tokens and hidden fields.	`client.submit_form()`: It finds the form, keeps the hidden tokens, and submits for you.
Data Extraction	You need to use BeautifulSoup manually.	Magnet Module: It has built-in extractors for emails, tables, and json_ld.

📦 Install

You can install TLS-Chameleon directly from PyPI:

pip install tls-chameleon[curl]

Note: The [curl] extra is required for TLS fingerprint spoofing.

⚡ Quick Start

1. Simple Requests (Drop-in)

from tls_chameleon import get

# One-line spoofing
r = get("https://httpbin.org/get", fingerprint="chrome_124")
print(r.json())

2. Persistent Session (Recommended)

Use Session (alias for TLSChameleon) to maintain cookies across requests:

from tls_chameleon import Session

with Session(fingerprint="chrome_120") as client:
    # First request sets cookies
    client.get("https://github.com/login")

    # Second request sends them back!
    r = client.get("https://github.com/settings")

3. Magnet Extraction 🧲

AI Extraction (New! ✨)

Use Gemini, Claude, or OpenAI (ChatGPT/Grok) to extract data intelligently without regex.

# Install AI support
pip install tls-chameleon[ai]

r = client.get("https://news.ycombinator.com")

# 1. Google Gemini (Default)
print(r.magnet.ask("Summary", provider="gemini"))

# 2. Anthropic Claude
print(r.magnet.ask("Summary", provider="anthropic", model="claude-3-opus-20240229"))

# 3. OpenAI / Grok
# (Set OPENAI_API_KEY or pass api_key=...)
print(r.magnet.ask("Summary", provider="openai", model="gpt-4o"))

Standard Extractors

Don't write regex. Let Magnet do it.

r = client.get("https://example.com/contact")

emails = r.magnet.emails()        # ['support@example.com']
tables = r.magnet.tables()        # [['Row1', 'Val1'], ...]
links  = r.magnet.links()
forms  = r.magnet.get_forms()     # List of parsed forms
json_data = r.magnet.json_ld()    # Schema.org data

4. Cookie Persistence 🍪

Save your session to a Netscape-formatted file (compatible with wget/curl) to use later.

# Save
client.save_cookies("cookies.txt")

# Load later
client.load_cookies("cookies.txt")

4. Smart Features

Mimic Real Browser Traffic (Fetches static assets in background):

client.get("https://example.com", mimic_assets=True)

Auto-Submit Forms (Handles hidden fields automatically):

# Finds <form>, fills 'user'/'pass', keeps hidden tokens, POSTs to action.
client.submit_form("https://site.com/login", {
    "username": "myuser",
    "password": "mypassword"
})

Humanize Delays:

client.human_delay(reading_speed="fast") # Sleeps randomly based on speed

🛠 API Reference

`Session(fingerprint=..., site=..., ...)`

fingerprint: "chrome_120", "firefox_120", "mobile_safari_17".
site: "cloudflare" or "akamai" (presets for retries/headers).
randomize_ciphers: True/False (shuffles cipher suite order).
proxies: http://user:pass@host:port

Response Object

The response object wraps curl_cffi.Response or httpx.Response but adds:

.magnet: Access extraction tools.
.json_fuzzy(): Parse broken/JSONP responses.

🤝 Contributing

Issues and Pull Requests welcome!

📜 License

MIT

🚨 Is this library failing on a specific site?

Please open an issue with the URL! I need test cases to improve the fingerprinting logic.

AI-cloudscraper

Zied — Thu, 11 Dec 2025 07:12:48 +0000

CloudScraper v3.5.0 🚀 - AI & Hybrid Engine Update

A powerful, feature-rich Python library to bypass Cloudflare's anti-bot protection with 10 production-ready bypass strategies, cutting-edge advanced stealth capabilities, async support, and comprehensive monitoring. This Hybrid Edition includes the revolutionary Hybrid Engine, integrating TLS-Chameleon and Py-Parkour for the ultimate bypass capability now powered by Google Gemini AI.

🔥 NEW: AI Captcha Bypass (v3.4.0) - Vision-Powered Solving

The scraper now deeply integrates Google Gemini 1.5 Flash to solve complex visual challenges like reCAPTCHA v2:

Visual Understanding: Analyzes instruction images (e.g., "Select all traffic lights") and identifies target objects.
Intelligent Solving: Visually inspects every tile, matches objects, and solves the puzzle just like a human.
Fast & Cheap: Uses Gemini 1.5 Flash for millisecond latency.

✅ Verified Features

Feature	Status
reCAPTCHA v2 Solving	✅ Tested
Text Captcha (Generic)	✅ Tested
Hybrid Engine	✅ Tested
Cloudflare Bypass	✅ Tested

# Pass your Google API Key to enable AI Solving
scraper = cloudscraper.create_scraper(
    interpreter='hybrid',
    google_api_key='YOUR_GEMINI_API_KEY',
    # Proxies are automatically used for AI requests too!
    rotating_proxies=['http://user:pass@proxy:port']
)

# For Complicated Text Captchas (Non-Standard)
scraper = cloudscraper.create_scraper(
    interpreter='hybrid',
    google_api_key='YOUR_GEMINI_API_KEY',
    captcha={
        'text_captcha': {
            'selector': '#captcha-image',   # CSS selector for the image
            'input_selector': '#captcha-input', # CSS selector for the input
            'submit_selector': '#submit-btn'    # Optional: submit button
        }
    }
)

🔥 Hybrid Engine - The Ultimate Solution

The Hybrid Engine is a game-changer that combines two powerful technologies:

TLS-Chameleon (curl_cffi): Provides perfect TLS fingerprinting (JA3/JA4) to mimic real browsers at the network layer.
Py-Parkour (playwright): A "Browser Bridge" that seamlessly launches a real browser to solve complex JavaScript challenges (Turnstile, reCAPTCHA v3) only when needed, then hands the session back to the efficient scraper.

Why use Hybrid?

Speed: Uses lightweight HTTP requests for 99% of work.
Power: Falls back to a real browser only for seconds to solve a challenge.
Stealth: Perfect TLS fingerprints + Real Browser interactions.
Simplicity: No complex setup—just interpreter='hybrid'.

✨ Key Features

🛡️ Hybrid Engine: Automatically switches between lightweight requests and real browser solving
🤖 AI Captcha Solver: Solves reCAPTCHA v2 using Google Gemini Vision
🔐 TLS Fingerprinting: JA3 fingerprint rotation with real browser signatures (Chrome, Firefox, Safari) via tls-chameleon
🕵️ Traffic Pattern Obfuscation: Intelligent request spacing and behavioral consistency
🧠 Intelligent Challenge Detection: AI-powered challenge recognition
⚡ Async Support: Check async_cloudscraper for non-blocking operations

🚀 NEW: Phase 1 & 2 - Industrial Strength Bypass (v3.1.2+)

This version includes 10 production-ready bypass strategies:

Phase 1: Foundation Features

1. 🧬 The Hybrid Engine (Introduced in v3.3.0)

The most powerful mode available. Requires cloudscraper[hybrid].

# Install with: pip install cloudscraper[hybrid]

scraper = cloudscraper.create_scraper(
    interpreter='hybrid',
    impersonate='chrome120', # Optional: Force specific fingerprint
    google_api_key='YOUR_API_KEY' # Optional: For AI Captcha solving
)
scraper.get("https://hight-security-site.com")

2. 🍪 Cookie Harvesting & Persistence

Auto-saves cf_clearance cookies after successful bypasses
Reuses cookies for 30-60 minutes (configurable TTL)
70-90% reduction in repeat challenge encounters
Storage: ~/.cloudscraper/cookies/

# Enabled by default!
scraper = cloudscraper.create_scraper(
    enable_cookie_persistence=True,
    cookie_ttl=1800  # 30 minutes
)

2. 🎯 Hybrid Captcha Solver

Tries AI OCR → AI Object Detection → 2Captcha in sequence
Automatic fallback on failure
3-5x higher solve rate vs single solver

scraper = cloudscraper.create_scraper(
    captcha={
        'provider': 'hybrid',
        'primary': 'ai_ocr',
        'fallbacks': ['ai_obj_det', '2captcha'],
        '2captcha': {'api_key': 'YOUR_KEY'}
    }
)

3. 🌐 Browser Automation Helper

Uses Playwright to launch real browser when all else fails
Ultimate fallback with 99% success rate

from cloudscraper.browser_helper import create_browser_helper

browser = create_browser_helper(headless=False)
cookies = browser.solve_challenge_and_get_cookies(url)
scraper.cookies.update(cookies)

4. ⏱️ Enhanced Human Behavior Simulation

Content-aware delays (text vs images vs API)
Mouse movement simulation
Fingerprint resistance

Phase 2: Advanced Strategies

5. 🔌 Circuit Breaker Pattern

Prevents infinite retry loops
Opens after 3 consecutive failures (configurable)
Auto-retry after timeout

# Enabled by default!
scraper = cloudscraper.create_scraper(
    enable_circuit_breaker=True,
    circuit_failure_threshold=3,
    circuit_timeout=60
)

6. 🔄 Session Pool (Multi-Fingerprint Distribution)

Maintains pool of 3-10 scraper instances
Each with unique browser fingerprint
Round-robin / random / least-used rotation

from cloudscraper.session_pool import SessionPool

pool = SessionPool(pool_size=5, rotation_strategy='round_robin')
resp = pool.get('https://protected-site.com')

7. ⚡ Smart Rate Limiter

Adaptive per-domain delays
Learns from 429/503 responses
Burst prevention

from cloudscraper.rate_limiter import SmartRateLimiter

limiter = SmartRateLimiter(default_delay=1.0, burst_limit=10)
limiter.wait_if_needed(domain)

8. 🔐 TLS Fingerprint Rotator

6+ real browser JA3 signatures (Chrome, Firefox, Safari, Edge)
Auto-rotation every N requests

from cloudscraper.tls_rotator import TLSFingerprintRotator

rotator = TLSFingerprintRotator(rotation_interval=10)
fp = rotator.get_fingerprint()  # chrome_120, firefox_122, etc.

9. 🧠 Challenge Prediction System (ML-based)

Learns which domains use which challenges
Auto-configuration based on history
SQLite storage: ~/.cloudscraper/challenges.db

from cloudscraper.challenge_predictor import ChallengePredictor

predictor = ChallengePredictor()
predicted = predictor.predict_challenge('example.com')
config = predictor.get_recommended_config('example.com')
scraper = cloudscraper.create_scraper(**config)

10. 🎭 Enhanced Timing (from Phase 1)

Content-type aware delays
Adaptive reading time calculation

📊 Success Rate Comparison

Configuration	Success Rate	Speed	Use Case
Default (V1 + Cookies + Circuit Breaker)	70-80%	Fast	Most sites
+ Hybrid Solver	85-95%	Medium	Sites with captchas
+ Session Pool	90-95%	Medium	Pattern detection
+ Browser Fallback	99%+	Slow	Hardest sites

📚 Documentation

See ENHANCED_FEATURES.md for detailed documentation on all bypass strategies.

☕ Support This Project

If you find this library useful, consider supporting its development:

Installation

[!NOTE]
This is a maintained fork of the original cloudscraper library.
You can use this version (ai-cloudscraper) as a drop-in replacement while waiting for updates to the original library, or continue using it as your primary driver as we will consistently update it with the latest anti-detection technologies.

# Install maintained version (Recommended)
pip install ai-cloudscraper

# Install with AI solvers (Phase 1)
pip install ai-cloudscraper[ai]

# Install with browser automation (Phase 1)
pip install ai-cloudscraper[browser]

# Or install from source (Development)
pip install -e .

🚀 Quick Start

Basic Usage

import cloudscraper

# Create a CloudScraper instance (cookie persistence + circuit breaker enabled by default)
scraper = cloudscraper.create_scraper()

# Use it like a regular requests session
response = scraper.get("https://protected-site.com")
print(response.text)

Using Phase 1 & 2 Features

import cloudscraper
from cloudscraper.session_pool import SessionPool
from cloudscraper.challenge_predictor import ChallengePredictor

# Option 1: Default (Recommended for most sites)
scraper = cloudscraper.create_scraper()
resp = scraper.get('https://protected-site.com')

# Option 2: With hybrid solver
scraper = cloudscraper.create_scraper(
    captcha={
        'provider': 'hybrid',
        'fallbacks': ['ai_ocr', '2captcha'],
        '2captcha': {'api_key': 'YOUR_KEY'}
    }
)

# Option 3: Session pool for maximum stealth
pool = SessionPool(pool_size=5, rotation_strategy='round_robin')
resp = pool.get('https://protected-site.com')

# Option 4: Challenge predictor for smart configuration
predictor = ChallengePredictor()
config = predictor.get_recommended_config('target-domain.com')
scraper = cloudscraper.create_scraper(**config)

How It Works

Cloudflare's anti-bot protection works by presenting JavaScript challenges that must be solved before accessing the protected content. cloudscraper:

Detects Cloudflare challenges automatically
Solves JavaScript challenges using embedded interpreters
Maintains session state and cookies
Returns the protected content seamlessly

Dependencies

Python 3.8+
requests >= 2.32.0
requests_toolbelt >= 1.0.0
js2py >= 0.74 (default JavaScript interpreter)
Additional dependencies listed in requirements.txt

Optional Dependencies

Phase 1 AI Solvers:

pip install ddddocr ultralytics pillow

Phase 1 Browser Automation:

pip install playwright
playwright install chromium

Phase 2 features require NO additional dependencies - everything is included!

JavaScript Interpreters

cloudscraper supports multiple JavaScript interpreters:

js2py (default) - Pure Python implementation
nodejs - Requires Node.js installation
native - Built-in Python solver

Basic Configuration

Browser Selection

# Use Chrome fingerprint
scraper = cloudscraper.create_scraper(browser='chrome')

# Use Firefox fingerprint  
scraper = cloudscraper.create_scraper(browser='firefox')

Proxy Support

# Single proxy
scraper = cloudscraper.create_scraper()
scraper.proxies = {
    'http': 'http://proxy:8080',
    'https': 'http://proxy:8080'
}

CAPTCHA Solver Integration

scraper = cloudscraper.create_scraper(
    captcha={
        'provider': '2captcha',
        'api_key': 'your_api_key'
    }
)

Supported CAPTCHA providers:

2captcha
anticaptcha
CapSolver
CapMonster Cloud # Try maximum stealth configuration scraper = cloudscraper.create_scraper( enable_tls_fingerprinting=True, enable_anti_detection=True, enable_enhanced_spoofing=True, spoofing_consistency_level='high', enable_adaptive_timing=True, behavior_profile='research', # Slowest, most careful stealth_options={ 'min_delay': 3.0, 'max_delay': 10.0, 'human_like_delays': True } )

Enable maximum stealth mode

scraper.enable_maximum_stealth()


**Challenge detection not working?**

python

Add custom challenge patterns

scraper.intelligent_challenge_system.add_custom_pattern(
domain='problem-site.com',
pattern_name='Custom Challenge',
patterns=[r'custom.+challenge.+text'],
challenge_type='custom',
response_strategy='delay_retry'
)


**Want to optimize for specific domains?**

python

Make several learning requests first

for i in range(5):
try:
response = scraper.get('https://target-site.com/test')
except Exception:
pass

Then optimize for the domain

scraper.optimize_for_domain('target-site.com')


**Check enhanced system status:**

python
stats = scraper.get_enhanced_statistics()
for system, status in stats.items():
print(f"{system}: {status}")

Get ML optimization report

if hasattr(scraper, 'ml_optimizer'):
report = scraper.ml_optimizer.get_optimization_report()
print(f"Success rate: {report.get('global_success_rate', 0):.2%}")


### Common Issues

**Challenge solving fails:**

python

Try different interpreter

scraper = cloudscraper.create_scraper(interpreter='nodejs')

Increase delay

scraper = cloudscraper.create_scraper(delay=10)

Enable debug mode

scraper = cloudscraper.create_scraper(debug=True)


**403 Forbidden errors:**

python

Enable stealth mode

scraper = cloudscraper.create_scraper(
enable_stealth=True,
auto_refresh_on_403=True
)


**Slow performance:**

python

Use faster interpreter

scraper = cloudscraper.create_scraper(interpreter='native')


### Debug Mode

Enable debug mode to see what's happening:

python
scraper = cloudscraper.create_scraper(debug=True)
response = scraper.get("https://example.com")

Debug output shows:

- Challenge type detected

- JavaScript interpreter used

- Challenge solving process

- Final response status


## 🔧 Enhanced Configuration Options

### 🔥 **Enhanced Bypass Parameters** (NEW)

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `enable_tls_fingerprinting` | boolean | True | Enable advanced TLS fingerprinting |
| `enable_tls_rotation` | boolean | True | Rotate TLS fingerprints automatically |
| `enable_anti_detection` | boolean | True | Enable traffic pattern obfuscation |
| `enable_enhanced_spoofing` | boolean | True | Enable Canvas/WebGL spoofing |
| `spoofing_consistency_level` | string | 'medium' | Spoofing consistency ('low', 'medium', 'high') |
| `enable_intelligent_challenges` | boolean | True | Enable AI challenge detection |
| `enable_adaptive_timing` | boolean | True | Enable human behavior simulation |
| `behavior_profile` | string | 'casual' | Timing profile ('casual', 'focused', 'research', 'mobile') |
| `enable_ml_optimization` | boolean | True | Enable ML-based bypass optimization |
| `enable_enhanced_error_handling` | boolean | True | Enable intelligent error recovery |

### 🎭 **Enhanced Stealth Options**

python
stealth_options = {
'min_delay': 1.0, # Minimum delay between requests
'max_delay': 4.0, # Maximum delay between requests

'human_like_delays': True, # Use human-like delay patterns
'randomize_headers': True, # Randomize request headers
'browser_quirks': True, # Enable browser-specific quirks
'simulate_viewport': True, # Simulate viewport changes
'behavioral_patterns': True # Use behavioral pattern simulation
}


### 🤖 **Complete Enhanced Configuration Example**

python
import cloudscraper

Ultimate bypass configuration

scraper = cloudscraper.create_scraper(
# Basic settings
debug=True,
browser='chrome',
interpreter='js2py',

# Enhanced bypass features
enable_tls_fingerprinting=True,
enable_tls_rotation=True,
enable_anti_detection=True,
enable_enhanced_spoofing=True,
spoofing_consistency_level='medium',
enable_intelligent_challenges=True,
enable_adaptive_timing=True,
behavior_profile='focused',
enable_ml_optimization=True,
enable_enhanced_error_handling=True,

# Stealth mode
enable_stealth=True,
stealth_options={
    'min_delay': 1.5,
    'max_delay': 4.0,
    'human_like_delays': True,
    'randomize_headers': True,
    'browser_quirks': True,
    'simulate_viewport': True,
    'behavioral_patterns': True
},

# Session management
session_refresh_interval=3600,
auto_refresh_on_403=True,
max_403_retries=3,

# Proxy rotation
rotating_proxies=[
    'http://proxy1:8080',
    'http://proxy2:8080',
    'http://proxy3:8080'
],
proxy_options={
    'rotation_strategy': 'smart',
    'ban_time': 600
},

# CAPTCHA solving
captcha={
    'provider': '2captcha',
    'api_key': 'your_api_key'
}

)

Monitor bypass performance

stats = scraper.get_enhanced_statistics()
print(f"Active bypass systems: {len(stats)}")


### 📈 **Behavior Profiles**

| Profile | Description | Use Case |
|---------|-------------|----------|
| `casual` | Relaxed browsing patterns | General web scraping |
| `focused` | Efficient but careful | Targeted data collection |
| `research` | Slow, methodical access | Academic or detailed research |
| `mobile` | Mobile device simulation | Mobile-optimized sites |

### 📉 **Spoofing Consistency Levels**

| Level | Fingerprint Stability | Detection Resistance | Performance |
|-------|----------------------|---------------------|-------------|
| `low` | Minimal changes | Good | Fastest |
| `medium` | Moderate variations | Excellent | Balanced |
| `high` | Significant obfuscation | Maximum | Slower |

## Configuration Options

### Common Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `debug` | boolean | False | Enable debug output |
| `delay` | float | auto | Override challenge delay |
| `interpreter` | string | 'js2py' | JavaScript interpreter |
| `browser` | string/dict | None | Browser fingerprint |
| `enable_stealth` | boolean | True | Enable stealth mode |
| `allow_brotli` | boolean | True | Enable Brotli compression |

### Challenge Control

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `disableCloudflareV1` | boolean | False | Disable v1 challenges |
| `disableCloudflareV2` | boolean | False | Disable v2 challenges |
| `disableCloudflareV3` | boolean | False | Disable v3 challenges |
| `disableTurnstile` | boolean | False | Disable Turnstile |

### Session Management

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `session_refresh_interval` | int | 3600 | Session refresh time (seconds) |
| `auto_refresh_on_403` | boolean | True | Auto-refresh on 403 errors |
| `max_403_retries` | int | 3 | Max 403 retry attempts |

### Example Configuration

python
scraper = cloudscraper.create_scraper(
debug=True,
delay=5,
interpreter='js2py',
browser='chrome',
enable_stealth=True,
stealth_options={
'min_delay': 2.0,
'max_delay': 5.0,
'human_like_delays': True,
'randomize_headers': True,
'browser_quirks': True
}
)


## Utility Functions

### Get Tokens

Extract Cloudflare cookies for use in other applications:

python
import cloudscraper

Get cookies as dictionary

tokens, user_agent = cloudscraper.get_tokens("https://example.com")
print(tokens)

{'cf_clearance': '...', '__cfduid': '...'}

Get cookies as string

cookie_string, user_agent = cloudscraper.get_cookie_string("https://example.com")
print(cookie_string)

"cf_clearance=...; __cfduid=..."


### Integration with Other Tools

Use cloudscraper tokens with curl or other HTTP clients:

python
import subprocess
import cloudscraper

cookie_string, user_agent = cloudscraper.get_cookie_string('https://example.com')

result = subprocess.check_output([
'curl',
'--cookie', cookie_string,
'-A', user_agent,
'https://example.com'
])




## License

MIT License. See LICENSE file for details.

## 📁 **Enhanced Features Documentation**

For detailed documentation about the enhanced bypass capabilities, see:
- **[ENHANCED_FEATURES.md](ENHANCED_FEATURES.md)** - Complete technical documentation
- **[examples/enhanced_bypass_demo.py](examples/enhanced_bypass_demo.py)** - Comprehensive usage examples
- **[tests/test_enhanced_features.py](tests/test_enhanced_features.py)** - Feature validation tests

### 🔍 **Quick Feature Reference**

| Feature | Module | Description |
|---------|--------|--------------|
| TLS Fingerprinting | `tls_fingerprinting.py` | JA3 fingerprint rotation |
| Anti-Detection | `anti_detection.py` | Traffic pattern obfuscation |
| Enhanced Spoofing | `enhanced_spoofing.py` | Canvas/WebGL fingerprint spoofing |
| Challenge Detection | `intelligent_challenge_system.py` | AI-powered challenge recognition |
| Adaptive Timing | `adaptive_timing.py` | Human behavior simulation |
| ML Optimization | `ml_optimization.py` | Machine learning bypass optimization |
| Error Handling | `enhanced_error_handling.py` | Intelligent error recovery |

---

🎉 **Enhanced CloudScraper** - Bypass the majority of Cloudflare protections with cutting-edge anti-detection technology!

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## Disclaimer

This tool is for educational and testing purposes only. Always respect website terms of service and use responsibly.

HumanFuzz

Zied — Tue, 13 May 2025 15:40:56 +0000

Description:

A human-like web application fuzzing library that simulates real user interactions to discover security vulnerabilities. HumanFuzz provides both a Python API and a comprehensive command-line interface for scanning web applications and APIs.

Features

Human-like Interaction: Uses a headless browser to interact with web applications like a real user
Automatic Form Discovery: Identifies and interacts with forms, buttons, and other interactive elements
Smart Payload Generation: Generates and mutates payloads based on context and target responses
Multiple Vulnerability Classes: Supports testing for XSS, SQL injection, SSRF, and more
Session Awareness: Maintains authentication and session context throughout testing
Extensible Architecture: Easily add custom payload modules and detection rules
Comprehensive Reporting: Generates detailed reports of findings with proof-of-concept examples
Animated Interface: Real-time visual feedback during scanning operations
Enhanced Protection Bypass: Advanced techniques for bypassing web application firewalls and protections
Improved HTTP Handling: Enhanced HTTP client for better request handling and performance

Installation

Basic Installation

pip install humanfuzz

Enhanced Installation (Recommended)

# Install the required dependencies
pip install -r requirements.txt

# Install additional libraries for enhanced scanning
pip install humanfuzz[enhanced]

Using the Python API

Quick Start

from humanfuzz import HumanFuzzer

# Create a fuzzer instance
fuzzer = HumanFuzzer()

# Start a fuzzing session
fuzzer.start_session("https://example.com")

# Authenticate (if needed)
fuzzer.authenticate(
    login_url="https://example.com/login",
    username_field="username",
    password_field="password",
    username="test_user",
    password="test_password"
)

# Discover and fuzz forms automatically
results = fuzzer.fuzz_site()

# Generate a report
fuzzer.generate_report("vulnerability_report.html")

Enhanced API Usage

from humanfuzz import HumanFuzzer

# Create an enhanced fuzzer instance
fuzzer = HumanFuzzer(
    headless=True,
    browser_type="chromium",
    bypass_cloudflare=True,  # Enable Cloudflare bypass
    enhanced_http=True       # Use enhanced HTTP client
)

# Start a fuzzing session
fuzzer.start_session("https://example.com")

# Perform API scanning
fuzzer.api_scan(
    base_url="https://api.example.com",
    endpoints=["/users", "/products", "/orders"],
    auth_headers={"Authorization": "Bearer YOUR_TOKEN"}
)

# Generate a comprehensive report
fuzzer.generate_report("vulnerability_report.html")

Using the Command-Line Interface

HumanFuzz provides a comprehensive command-line interface for easy scanning operations.

Basic Usage

humanfuzz_cli.py [-h] {scan,api,report,version} ...

Commands

scan: Scan a website for vulnerabilities
api: Scan API endpoints
report: Generate a report from saved results
version: Show version information

Scan Command

The scan command is used to scan a website for vulnerabilities:

humanfuzz_cli.py scan https://example.com [options]

Options

--depth, -d DEPTH: Maximum crawl depth (default: 2)
--pages, -p PAGES: Maximum pages to crawl (default: 10)
--visible, -v: Show browser window
--simulate, -s: Run in simulation mode (no actual browser)
--output, -o OUTPUT: Output file for the report (default: auto-generated)
--format, -f {html,json,md}: Report format (default: html)
--auth, -a: Enable authentication
--username, -u USERNAME: Username for authentication
--password, -pw PASSWORD: Password for authentication
--login-url, -l LOGIN_URL: Login URL (if different from main URL)
--username-field USERNAME_FIELD: Username field name/ID (default: username)
--password-field PASSWORD_FIELD: Password field name/ID (default: password)
--submit-button SUBMIT_BUTTON: Submit button selector (optional)
--cookies, -c COOKIES: Cookies to use (format: name1=value1;name2=value2)
--headers HEADERS: Custom headers (format: name1=value1;name2=value2)
--user-agent USER_AGENT: Custom User-Agent
--timeout TIMEOUT: Request timeout in seconds (default: 30)
--delay DELAY: Delay between requests in seconds (default: 0)
--bypass-cloudflare: Enable Cloudflare bypass using cloudscraper25
--enhanced-http: Use urllib4-enhanced for HTTP requests
--verbose: Enable verbose output
--save-results SAVE_RESULTS: Save raw results to JSON file

API Command

The api command is used to scan API endpoints:

humanfuzz_cli.py api https://api.example.com [options]

Options

--endpoints, -e ENDPOINTS: Comma-separated list of endpoints to scan
--endpoints-file ENDPOINTS_FILE: File containing endpoints (one per line)
--methods, -m METHODS: HTTP methods to use (comma-separated, default: GET,POST,PUT,DELETE)
--auth-type {none,basic,bearer,api-key}: Authentication type
--auth-user AUTH_USER: Username for Basic Auth
--auth-pass AUTH_PASS: Password for Basic Auth
--auth-token AUTH_TOKEN: Token for Bearer Auth
--api-key-name API_KEY_NAME: API key header name (default: X-API-Key)
--api-key-value API_KEY_VALUE: API key value
--headers HEADERS: Custom headers (format: name1=value1;name2=value2)
--output, -o OUTPUT: Output file for the report (default: auto-generated)
--format, -f {html,json,md}: Report format (default: html)
--simulate, -s: Run in simulation mode
--verbose: Enable verbose output
--save-results SAVE_RESULTS: Save raw results to JSON file

Report Command

The report command is used to generate a report from saved results:

humanfuzz_cli.py report --input results.json --output report.html [options]

Options

--input, -i INPUT: Input JSON file with scan results (required)
--output, -o OUTPUT: Output report file (required)
--format, -f {html,json,md}: Report format (default: html)
--template, -t TEMPLATE: Custom template file for the report

Version Command

The version command shows version information:

humanfuzz_cli.py version

CLI Examples

Basic Website Scan

humanfuzz_cli.py scan https://example.com

Scan with Authentication

humanfuzz_cli.py scan https://example.com --auth --username admin --password secret

Scan with Cloudflare Bypass and Enhanced HTTP

humanfuzz_cli.py scan https://example.com --bypass-cloudflare --enhanced-http

API Scan with Bearer Authentication

humanfuzz_cli.py api https://api.example.com --endpoints /users,/products --auth-type bearer --auth-token "YOUR_TOKEN"

Generate Report from Saved Results

humanfuzz_cli.py report --input scan_results.json --output vulnerability_report.html

Enhanced Features

Cloudflare Bypass

The --bypass-cloudflare option enables Cloudflare protection bypass. This allows scanning websites protected by Cloudflare, which would normally block automated scanning tools.

Benefits:

Bypass Cloudflare's anti-bot protection
Access protected content for scanning
Maintain session context through Cloudflare challenges

Enhanced HTTP Requests

The --enhanced-http option enables the use of enhanced HTTP libraries for requests, providing improved capabilities for more effective scanning.

Benefits:

Automatic retry mechanisms for failed requests
Better error handling and reporting
Improved performance for large-scale scanning
Enhanced cookie and session management
Support for complex authentication schemes

Advanced Usage

See the documentation for advanced usage examples, including:

Custom payload modules
Targeted fuzzing of specific forms or endpoints
Recording and replaying user flows
Integration with CI/CD pipelines
Creating custom scan profiles

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Copyright

ai-urllib4

Zied — Mon, 12 May 2025 07:20:22 +0000

# urllib4: An Enhanced HTTP Client for Python

ai-urllib4 is a modern HTTP client for Python that builds upon the foundation of urllib3 while adding enhancements for modern web applications. It provides a powerful yet user-friendly interface for making HTTP requests with advanced features.

Features

urllib4 provides a comprehensive set of features for modern web applications:

✅ Core Features:

Connection pooling and thread safety
Basic URL parsing and manipulation
HTTP header handling and collections
Multipart form data encoding
SSL/TLS utility functions
File upload functionality
HTTP/HTTPS request handling
Proxy support
Retry mechanisms
Redirect handling
Compression support

✅ Advanced Features:

Enhanced HTTP/2 Support
WebSocket capabilities with extensions and subprotocols
Improved security features
HTTP/3 (QUIC) support with Multipath QUIC

Usage Example

You can use urllib4 for your HTTP requests with a simple, intuitive API:

# Simple GET request
>>> import urllib4
>>> resp = urllib4.request("GET", "http://httpbin.org/robots.txt")
>>> resp.status
200
>>> resp.data
b"User-agent: *\nDisallow: /deny\n"

# POST request with JSON data
>>> import urllib4
>>> import json
>>> data = {"name": "John", "age": 30}
>>> resp = urllib4.request(
...     "POST",
...     "http://httpbin.org/post",
...     headers={"Content-Type": "application/json"},
...     body=json.dumps(data).encode()
... )
>>> resp.status
200

Installation

You can install urllib4 with pip:

$ pip install urllib4

For additional features, you can install optional dependencies:

# For HTTP/3 support
$ pip install ai-urllib4[http3]

# For WebSocket subprotocols
$ pip install ai-urllib4[websocket]

# For all optional features
$ pip install ai-urllib4[all]

Alternatively, you can install from source:

$ git clone https://github.com/zinzied/urllib4.git
$ cd urllib4
$ pip install -e .

Development and Testing

To set up a development environment:

$ git clone https://github.com/zinzied/urllib4.git
$ cd urllib4
$ pip install -e ".[dev]"

To run tests:

$ python -m pytest

Note that many tests are currently failing as the library is under active development.

Documentation

Documentation is currently limited to code comments and this README. As the project matures, more comprehensive documentation will be developed.

Roadmap

The following features are planned for future development:

HTTP/3 Enhancements

Connection migration for improved reliability
HTTP/3 priority management
WebTransport support

WebSocket Enhancements

Additional WebSocket extensions
WebSocket over HTTP/2

HTTP/2 Support

import urllib4
from urllib4.http2 import inject_into_urllib4, ConnectionProfile

# Enable HTTP/2 support
inject_into_urllib4()

# Create a pool manager with a specific connection profile
http = urllib4.PoolManager(http2_profile=ConnectionProfile.HIGH_PERFORMANCE)

# Make a request (automatically uses HTTP/2 if the server supports it)
response = http.request("GET", "https://nghttp2.org")
print(f"HTTP version: {response.version_string}")

WebSocket Support

from urllib4.websocket import WebSocketConnection

# Connect to a WebSocket server with compression and health monitoring
ws = WebSocketConnection(
    "wss://echo.websocket.org",
    enable_compression=True,
    compression_level=9,
    enable_health_monitoring=True,
    ping_interval=30.0,
    ping_timeout=5.0,
)

# Connect to the server
ws.connect()

# Send a text message
ws.send("Hello, WebSocket!")

# Send a binary message
ws.send(b"\x01\x02\x03\x04")

# Send a structured object using a subprotocol
ws = WebSocketConnection(
    "wss://echo.websocket.org",
    protocols=["json"],
)
ws.connect()
ws.send({"name": "John", "age": 30})  # Automatically encoded as JSON

# Receive a message with timeout
try:
    message = ws.receive(timeout=5.0)
    if isinstance(message, dict):
        print(f"Received JSON: {message}")
    else:
        print(f"Received: {message.text}")
except WebSocketTimeoutError:
    print("Timed out waiting for message")

# Close the connection
ws.close()

WebSocket Extensions and Subprotocols

urllib4 supports various WebSocket extensions and subprotocols:

Extensions:
- permessage-deflate: Compresses WebSocket messages for better performance
Subprotocols:
- json: Automatically encodes/decodes JSON data
- msgpack: Efficiently encodes/decodes MessagePack data (requires msgpack package)
- cbor: Efficiently encodes/decodes CBOR data (requires cbor2 package)

HTTP/3 Support

import urllib4
from urllib4.http3 import HTTP3Connection, HTTP3Settings, QUICSettings, inject_into_urllib4

# Direct HTTP/3 usage
quic_settings = QUICSettings(
    enable_multipath=True,  # Enable Multipath QUIC
    max_paths=4,
    enable_0rtt=True,       # Enable 0-RTT for faster connections
)

http3_settings = HTTP3Settings(
    quic=quic_settings,
)

# Create an HTTP/3 connection
conn = HTTP3Connection(
    "cloudflare-quic.com",  # A server that supports HTTP/3
    settings=http3_settings,
)

# Connect and make a request
conn.connect()
response = conn.request("GET", "/")
print(f"Status: {response.status}")
print(f"Body: {response.data.decode()[:100]}...")

# Close the connection
conn.close()

# Alternatively, inject HTTP/3 support into urllib4
inject_into_urllib4()

# Now all HTTPS requests will automatically use HTTP/3 when available
http = urllib5.PoolManager()
response = http.request("GET", "https://cloudflare-quic.com/")

🤖 AI-Powered Smart Config

ai-urllib4 includes an experimental AI module that suggests optimal connection parameters based on URL heuristics.

from ai_urllib4.ai import AISmartConfig, optimize_params_for

# Get a suggested timeout for an API endpoint
timeout = AISmartConfig.suggest_timeout("https://api.example.com/v1/users")
print(f"Suggested Timeout: {timeout}s")

# Get a full set of optimized parameters
params = optimize_params_for("https://cdn.example.com/big-file.zip")
print(f"Optimized Params: {params}")
# Output: {'timeout': 60.0, 'retries': 3}

🚀 AsyncIO Support

First-class support for asyncio and anyio is now available.

import asyncio
from ai_urllib4.async_connectionpool import connection_from_url

async def main():
    pool = connection_from_url("http://httpbin.org")
    response = await pool.urlopen("GET", "/get")
    print(response)
    await pool.close()

asyncio.run(main())

Enhanced Security Features (Planned)

# This is a planned API - not yet implemented
import urllib4
from urllib4.util.cert_verification import SPKIPinningVerifier, CertificateTransparencyPolicy
from urllib4.util.hsts import HSTSCache, HSTSHandler

# Create a pool manager with SPKI pinning
pins = {
    "example.com": {"pin-sha256:YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="}
}
http = urllib4.PoolManager(
    spki_pins=pins,
    cert_transparency_policy=CertificateTransparencyPolicy.BEST_EFFORT
)

# Create an HSTS handler
hsts_cache = HSTSCache()
hsts_handler = HSTSHandler(hsts_cache)

# Secure a URL if needed
url = "http://example.com/api"
secured_url = hsts_handler.secure_url(url)  # Returns https://example.com/api if in HSTS cache

Contributing

This project is in its early stages and contributions are welcome! Here's how you can help:

Bug Reports: If you find a bug, please open an issue with detailed information.
Feature Requests: Have ideas for new features? Open an issue to discuss.
Code Contributions: Pull requests are welcome for bug fixes or new features.
Documentation: Help improve or expand the documentation.
Testing: Help write or improve tests for the codebase.

Development Guidelines

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add some amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

Why ai-urllib4? (vs urllib3)

While ai-urllib4 is built on the solid foundation of urllib3, it introduces key differences designed for modern, high-performance applications:

Feature	urllib3	ai-urllib4
Async Support	❌ Synchronous only (blocking)	✅ Native AsyncIO (`async`/`await`)
Configuration	Manual Tuning	🤖 AI Smart Config (Heuristic optimization)
HTTP/2 & 3	Experimental / Partial	✅ First-Class Support (Multipath QUIC, etc.)
WebSocket	Limited / Extension needed	✅ Built-in Advanced Support (Subprotocols, etc.)

Key Differentiators:

🚀 AsyncIO Support: ai-urllib4 allows non-blocking requests perfect for apps using FastAPI, Quart, or standard asyncio.
🤖 AI-Powered Tuning: The AISmartConfig module automatically suggests timeout and retry settings based on the target URL type (API vs Download).
🔒 Modern Security: Stronger focus on Certificate Transparency and SPKI Pinning out of the box.

Security Considerations

urllib4 is designed with security in mind, providing robust SSL/TLS verification, certificate transparency checking, and HSTS support. It's suitable for use in production environments where security is a priority.

Acknowledgements

This project builds on concepts from the urllib3 project and other Python HTTP libraries. We extend our gratitude to the authors and maintainers of these projects for their foundational work.

License

This project is licensed under the MIT License - see the LICENSE file for details.