Proxy Rotation in Crawlee for Python: A Developer's Guide to Residential Proxies

Proxy rotation in Crawlee for Python is the difference between a scraper that runs for weeks and one that dies after 200 requests. Crawlee's architecture — a unified request queue, an autoscaled pool, and a SessionPool that binds cookies and fingerprints to IP addresses — gives you the building blocks. But you still need to wire those blocks to a reliable residential proxy gateway and handle blocks gracefully. This guide walks through the idiomatic way to do that with ProxyHat.

Legal caveat: This guide covers scraping publicly accessible data. Always review a site's Terms of Service, honor robots.txt, respect rate limits, and consider whether an official API exists. Unauthorized scraping may violate the Computer Fraud and Abuse Act (CFAA) in the US and GDPR in the EU if personal data is involved. Consult counsel for your specific use case.

Why Proxy Rotation in Crawlee for Python Matters

Modern anti-bot systems like Cloudflare, DataDome, and Akamai Bot Manager don't just check IP reputation — they correlate IP, TLS fingerprint, browser fingerprint, cookie state, and behavioral signals across requests. A single datacenter IP making 50 requests per minute to an e-commerce site is a red flag regardless of headers. Rotating IPs per session disrupts that correlation, and residential IPs — addresses assigned to real ISPs — are far less likely to appear on blocklists than datacenter ranges owned by AWS, DigitalOcean, or Hetzner.

Crawlee for Python (the Apify Crawlee framework) gives you the plumbing: a ProxyConfiguration class, a SessionPool that manages cookie jars and fingerprints, and an autoscaled HTTP client. The gap most developers hit is connecting those primitives to a proxy provider in a way that survives blocks at scale.

Crawlee's Architecture: Request Queue, Autoscaling, and SessionPool

Before diving into proxy code, it helps to understand the three layers Crawlee uses to manage crawling:

• Unified Request Queue: Every crawler (BeautifulSoupCrawler, PlaywrightCrawler, etc.) pulls from the same RequestQueue. This means you can enqueue URLs once and let the framework handle retries, deduplication, and ordering. The queue persists state locally or to storage, so a crashed run can resume.
• Autoscaled Pool: Crawlee dynamically adjusts concurrency based on system resources and response times. You set min_concurrency and max_concurrency, and the framework scales within that range. This is where proxy concurrency matters — if your proxy provider caps concurrent sessions, you need max_concurrency to stay under that limit.
• SessionPool: Each Session object holds a cookie jar, a browser fingerprint (for Playwright), and — critically — a proxy URL. When you bind a proxy to a session, Crawlee reuses that IP for all requests in that session's lifecycle. This is the key to maintaining login state and avoiding CAPTCHA re-challenges.

The SessionPool rotates sessions automatically. When a session gets blocked (detected via status codes, CAPTCHA pages, or custom error handlers), you call session.retire() to discard it. The pool then creates a new session with a fresh proxy, cookies, and fingerprint.

BeautifulSoupCrawler vs PlaywrightCrawler

Crawlee offers two primary HTTP-based crawlers and one browser-based crawler:

Start with BeautifulSoupCrawler. Only escalate to PlaywrightCrawler when the target genuinely requires JavaScript execution or challenge solving. Browser crawling is 5–20x slower and far more resource-intensive.

The Idiomatic ProxyConfiguration Class

Crawlee's ProxyConfiguration is the framework-native way to manage proxies. Instead of manually injecting proxy URLs into each request, you create a ProxyConfiguration and pass it to the crawler. The configuration handles rotation, session pinning, and error fallback.

Round-Robin vs Session-Bound Rotation

There are two rotation strategies:

• Round-robin: Each request gets a new proxy IP. Good for simple, stateless scraping where you don't need cookies. Use proxy_configuration.new_url() without a session ID.
• Session-bound (sticky): All requests within a session share the same IP. Use proxy_configuration.new_url(session_id="abc123") to pin an IP. This is essential for sites that issue session cookies, CSRF tokens, or multi-step flows.

For residential proxies behind anti-bot systems, session-bound rotation is almost always the right choice. Round-robin with residential IPs can trigger anomaly detection because no real user changes IP every 3 seconds.

Residential Proxies vs Datacenter: Why It Matters for Crawlee

Datacenter proxies are fast and cheap — often $0.5–$2 per GB — but they come from ASN ranges that anti-bot vendors flag. Cloudflare's Bot Management score, for instance, weights IP reputation heavily, and datacenter IPs from major cloud providers are low-trust by default. Residential proxies cost more ($3–$15 per GB) but come from real ISP assignments (Comcast, Vodafone, Deutsche Telekom), making them appear as organic user traffic.

For Crawlee specifically, the proxy type affects your SessionPool strategy:

• Datacenter: High rotation frequency needed. Sessions get blocked fast. Expect 20–40% block rates on protected targets. Use max_session_rotations aggressively.
• Residential: Sessions last longer (hours to days). Lower block rates (2–10% on most targets). Pin sessions and rotate only on blocks.
• Mobile: Highest trust score. Best for social media and ticketing. Most expensive. Use sparingly for high-value targets.

A tiered approach works well in production: start with residential proxies for the initial request, fall back to mobile proxies on persistent blocks, and retire the session entirely after N failures.

Runnable Example: BeautifulSoupCrawler with ProxyHat Residential Proxies

Here's a complete, runnable example that uses Crawlee's ProxyConfiguration with ProxyHat residential proxies. The key is encoding the session ID and country in the proxy username so each Crawlee session pins to a specific residential IP.

import asyncio
from crawlee.beautifulsoup_crawler import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
from crawlee.proxy_configuration import ProxyConfiguration
from crawlee.sessions import SessionPool
import uuid

# ProxyHat credentials
PROXYHAT_USER = "your_username"
PROXYHAT_PASS = "your_password"

# Generate a per-session proxy URL with country targeting and session pinning
def make_proxy_url(session_id: str, country: str = "US") -> str:
    return (
        f"http://{PROXYHAT_USER}-country-{country}-session-{session_id}"
        f":{PROXYHAT_PASS}@gate.proxyhat.com:8080"
    )

async def main():
    # Create a ProxyConfiguration that generates session-bound URLs
    proxy_config = ProxyConfiguration(
        proxy_urls=[make_proxy_url(str(uuid.uuid4()))],
    )

    crawler = BeautifulSoupCrawler(
        proxy_configuration=proxy_config,
        max_requests_per_crawl=500,
        max_request_retries=3,
        max_session_rotations=5,
        request_handler_timeout=30,
        min_concurrency=2,
        max_concurrency=10,
    )

    @crawler.router.default_handler
    async def handler(ctx: BeautifulSoupCrawlingContext) -> None:
        ctx.log.info(f"Scraping {ctx.request.url}")
        title = ctx.soup.find("title")
        if title:
            ctx.log.info(f"Title: {title.text.strip()}")

        # Extract product data, SERP results, etc.
        links = ctx.soup.select("a[href]")
        for link in links[:10]:
            href = link.get("href")
            if href and href.startswith("http"):
                await ctx.enqueue_links([href], label="LINK")

    await crawler.run(["https://example.com"])

if __name__ == "__main__":
    asyncio.run(main())

In this example, each session gets a unique UUID appended to the ProxyHat username via the -session- flag. This tells ProxyHat's gateway to assign a sticky residential IP for that session's lifetime. When Crawlee retires a session (due to a block or error), the next session gets a new UUID and therefore a new residential IP.

Advanced: Custom Proxy URL Generator with Session Pool Integration

For more control, you can subclass or use a custom proxy URL generator that ties into Crawlee's SessionPool directly:

from crawlee.proxy_configuration import ProxyConfiguration
import hashlib

PROXYHAT_USER = "your_username"
PROXYHAT_PASS = "your_password"

class ProxyHatProxyConfiguration(ProxyConfiguration):
    async def new_url(self, session_id: str | None = None) -> str:
        """Generate a ProxyHat URL with session-bound residential IP."""
        sid = session_id or "default"
        # Hash session ID for consistent, short proxy identifiers
        short_id = hashlib.md5(sid.encode()).hexdigest()[:8]
        return (
            f"http://{PROXYHAT_USER}-country-US-session-{short_id}"
            f":{PROXYHAT_PASS}@gate.proxyhat.com:8080"
        )

    async def new_urls(self, count: int, session_id: str | None = None) -> list[str]:
        """Generate multiple URLs for fallback rotation within a session."""
        sid = session_id or "default"
        return [await self.new_url(f"{sid}-{i}") for i in range(count)]

This pattern lets you implement tiered fallback: generate 2–3 proxy URLs per session (residential US, residential DE, mobile US) and rotate through them before retiring the session entirely.

Production Patterns: Retiring Sessions, Retries, and Concurrency

Handling Blocks with session.retire()

When a request fails with a 403, 429, or returns a CAPTCHA page, you should retire the session so the pool creates a fresh one with a new IP. Crawlee lets you detect blocks in the request handler:

@crawler.router.default_handler
async def handler(ctx: BeautifulSoupCrawlingContext) -> None:
    status = ctx.http_response.status_code
    
    # Detect common block signals
    if status in (403, 429):
        ctx.log.warning(f"Blocked on {ctx.request.url} (HTTP {status})")
        await ctx.session.retire()
        raise RuntimeError(f"Blocked: HTTP {status}")
    
    # Check for CAPTCHA / challenge pages
    captcha_selectors = ["#captcha", ".cf-challenge", ".g-recaptcha", "#challenge-form"]
    for selector in captcha_selectors:
        if ctx.soup.select_one(selector):
            ctx.log.warning(f"CAPTCHA detected on {ctx.request.url}")
            await ctx.session.retire()
            raise RuntimeError("CAPTCHA challenge encountered")
    
    # Normal processing
    title = ctx.soup.find("title")
    if title:
        await ctx.push_data({"url": ctx.request.url, "title": title.text.strip()})

When session.retire() is called, Crawlee discards the current session (cookies, fingerprint, proxy IP) and creates a new one. The failed request is retried with the fresh session, up to max_request_retries times. If retries are exhausted, the request is marked as failed and logged.

Configuring Retries and Concurrency

Key crawler parameters for proxy-driven scraping:

• max_request_retries=3: Retry each request up to 3 times before giving up. Lower this if your proxy costs are high.
• max_session_rotations=5: Maximum number of session rotations per request. After this, the request fails permanently.
• max_concurrency=10: Cap concurrency to stay within your proxy provider's concurrent session limit. ProxyHat supports high concurrency, but 10–20 is a safe starting point for most targets.
• request_handler_timeout=30: Timeout per request handler. Increase to 60s for slow targets or browser crawling.

Error Handling and Dead Letter Queue

@crawler.router.default_handler
async def handler(ctx: BeautifulSoupCrawlingContext) -> None:
    try:
        response = ctx.http_response
        if response.status_code >= 500:
            ctx.log.error(f"Server error {response.status_code} on {ctx.request.url}")
            # Don't retire session on 5xx — server issue, not block
            raise RuntimeError(f"Server error: {response.status_code}")
        
        # Process normally
        data = extract_data(ctx.soup)
        await ctx.push_data(data)
        
    except Exception as e:
        ctx.log.error(f"Handler error on {ctx.request.url}: {e}")
        # Log failed URL for manual review
        await ctx.push_data({
            "failed_url": ctx.request.url,
            "error": str(e),
            "session_id": ctx.session.id if ctx.session else None,
        })
        raise  # Let Crawlee handle retry logic

When NOT to Use Browser Crawling

PlaywrightCrawler is powerful but expensive. A single browser instance consumes 150–300 MB of RAM and takes 2–5 seconds per page. Running 10 concurrent browsers needs a machine with at least 4 GB of free memory. For most scraping tasks — SERP tracking, price monitoring, e-commerce catalog extraction — BeautifulSoupCrawler with residential proxies is sufficient and 10–20x faster.

Reach for PlaywrightCrawler only when:

• The target renders content via JavaScript (React/Vue/Angular SPAs).
• The site uses Cloudflare's JS challenge (not just the managed challenge).
• You need to interact with the page (clicking, scrolling, form submission).
• You need to capture network requests or API calls made by the page.

Even then, consider whether the site has a hidden JSON API you can call directly. Many SPAs load data from /api/... endpoints that return clean JSON — no browser needed. Inspect network traffic in DevTools before committing to browser crawling.

Scaling Patterns: Containerization and Headless Fleets

For production scraping at scale, containerize your Crawlee crawler and run multiple instances behind a job queue:

• Docker: Package your crawler with httpx, crawlee, and dependencies. Set max_concurrency based on container CPU/memory limits.
• Job queue: Use Redis or RabbitMQ to distribute URL batches across containers. Each container runs its own SessionPool with its own ProxyHat sessions.
• Headless fleet: For PlaywrightCrawler, run headless Chromium in containers with --disable-gpu --no-sandbox. Limit to 2–3 browsers per container to avoid OOM.
• Storage: Use Crawlee's storage abstraction to write results to S3, GCS, or a database. Avoid writing large datasets to local disk in containers.

ProxyHat's proxy locations span 90+ countries, so you can geo-distribute your scraping fleet. Run US-targeted containers with -country-US, EU containers with -country-DE, etc. This reduces per-IP request volume and improves success rates.

Ethics and Legal Considerations

Scraping is a legal gray area. Key principles:

• Public data only: Don't scrape behind logins unless you have explicit permission. Accessing non-public data may violate the CFAA.
• Honor robots.txt: Crawlee doesn't automatically check robots.txt. Review it manually and exclude disallowed paths from your request queue.
• Rate limits: Even with residential proxies, don't hammer a site. 1–5 requests per second per domain is a reasonable ceiling for most targets.
• GDPR: If you collect personal data (names, emails, profile info) from EU residents, you're subject to GDPR regardless of where you're based. Avoid storing personal data unless you have a lawful basis.
• Prefer APIs: Many platforms offer official APIs (Google, Amazon, Twitter/X). Check ProxyHat's documentation for integration guides, and always prefer official endpoints when available.

ProxyHat Setup and Pricing

ProxyHat provides residential, mobile, and datacenter proxies through a single gateway at gate.proxyhat.com. The username-based targeting system lets you control country, city, and session pinning without a separate API call:

• user-country-US:pass@gate.proxyhat.com:8080 — US residential, rotating IP per request.
• user-country-US-session-abc123:pass@gate.proxyhat.com:8080 — US residential, sticky session.
• user-country-DE-city-berlin-session-xyz789:pass@gate.proxyhat.com:8080 — Berlin residential, sticky session.
• socks5://user-country-US:pass@gate.proxyhat.com:1080 — US residential via SOCKS5.

Check ProxyHat pricing for current rates. Residential proxies are billed by traffic (GB), so estimate your crawl volume before committing. A 500-URL crawl of average pages (~200 KB each) uses roughly 100 MB of proxy traffic.

For use-case-specific guidance, see web scraping and SERP tracking.

Key Takeaways

• Use session-bound rotation (proxy_configuration.new_url(session_id=...)) for residential proxies. Round-robin triggers anti-bot anomaly detection.
• Start with BeautifulSoupCrawler. Only escalate to PlaywrightCrawler when JavaScript rendering is genuinely required.
• Retire sessions on blocks using session.retire(). Let Crawlee's retry logic handle the rest with max_request_retries=3 and max_session_rotations=5.
• Cap concurrency to stay within your proxy plan's session limits. max_concurrency=10 is a safe default for most residential proxy setups.
• Encode session IDs in the ProxyHat username to pin residential IPs per Crawlee session. Use UUIDs or hashes for uniqueness.
• Residential proxies beat datacenter on Cloudflare/DataDome-protected targets. Expect 2–10% block rates vs 20–40% with datacenter IPs.
• Honor robots.txt and rate limits. Legal compliance isn't optional — the CFAA and GDPR have real teeth.

FAQ

What is Proxy Rotation in Crawlee for Python?

Proxy rotation in Crawlee for Python is the practice of assigning different proxy IP addresses to crawler sessions using Crawlee's ProxyConfiguration class. You can rotate per-request (round-robin) or per-session (sticky). The idiomatic approach uses proxy_configuration.new_url(session_id=...) to pin a residential IP to each session, ensuring cookies and fingerprints stay consistent with the IP. When a session is blocked, Crawlee retires it and creates a new one with a fresh proxy.

Why does Proxy Rotation in Crawlee for Python matter for proxy users?

Without proxy rotation, a single IP making hundreds of requests will get blocked by anti-bot systems like Cloudflare and DataDome. Crawlee's SessionPool ties cookies, TLS fingerprints, and browser fingerprints to specific proxy IPs. If you rotate IPs without rotating sessions, you break cookie continuity and trigger anomaly detection. Proper proxy rotation within Crawlee's session model keeps each session's identity consistent while letting you cycle IPs when blocks occur, dramatically improving crawl success rates.

Which proxy type works best for Proxy Rotation in Crawlee for Python?

Residential proxies are the best default for Crawlee proxy rotation. They come from real ISP assignments, so anti-bot systems treat them as organic traffic with high trust scores. Datacenter proxies are faster and cheaper but have block rates of 20–40% on protected targets versus 2–10% for residential. Mobile proxies offer the highest trust but are the most expensive. A tiered strategy — residential first, mobile fallback — works well for high-value targets. Use datacenter only for unprotected or low-security sites.

How do you avoid blocks when implementing Proxy Rotation in Crawlee for Python?

Use session-bound rotation (not round-robin), call session.retire() when you detect 403/429 responses or CAPTCHA pages, set max_session_rotations=5 to limit retries, cap concurrency at 10–20 to avoid triggering rate limits, and add realistic headers and delays between requests. Always use residential proxies for protected targets, and implement custom block detection in your request handler by checking for known CAPTCHA selectors and challenge page patterns.