DrissionPage Proxy Guide: One Python Tool for HTTP and Chromium Scraping

If you've spent time building Python scrapers, you've probably juggled two tools: requests or httpx for simple HTML pages, and Selenium or Playwright for JavaScript-heavy targets. DrissionPage is a Python framework that unifies both paradigms — HTTP-style requests and Chromium browser automation — into a single API, with a WebPage class that can switch modes mid-session while sharing cookies and state. For scrapers who need residential proxies to reach hard targets, this hybrid model means you can run cheap HTTP requests 90% of the time and only escalate to a full browser when the page actually requires JavaScript rendering.

Legal note: This guide covers scraping publicly available data. Unauthorized access to non-public systems may violate the U.S. Computer Fraud and Abuse Act (CFAA) or the EU's GDPR. Always respect robots.txt, site Terms of Service, and rate limits. When an official API exists, prefer it over scraping.

Why DrissionPage Matters for Proxy Users

DrissionPage matters for proxy users because it lets you match the cost of your request to the difficulty of the target. A residential proxy request through an HTTP client costs a fraction of what a headless-browser session costs — both in proxy bandwidth and in compute. According to the official DrissionPage documentation, the framework was designed specifically to avoid the overhead of launching a browser for every page when a simple HTTP request would suffice.

The framework's name comes from "Driver + Session." It gives you three primary page types:

• SessionPage — an HTTP client (built on requests) for static or server-rendered pages. Fast, lightweight, no browser process.
• ChromiumPage — a CDP-driven browser controller that communicates with Chromium via the Chrome DevTools Protocol, not WebDriver. This means no chromedriver binary to manage and lower detection surface.
• WebPage — a hybrid that can switch between SessionPage and ChromiumPage modes on the same object, preserving cookies, headers, and session state across the switch.

For proxy users, this architecture translates directly to savings. If you're scraping 10,000 product pages and only 500 require JavaScript rendering, you can run 9,500 requests through SessionPage at HTTP speed and only spin up Chromium for the 500 that need it. With residential proxies billed per GB or per request, that difference compounds quickly.

The Idiomatic DrissionPage API

DrissionPage's locator system is one of its most distinctive features. Instead of relying solely on CSS selectors or XPath, it uses a compact syntax that blends convenience with precision.

Locators: ele(), eles(), and the @ Syntax

The ele() method returns a single element; eles() returns a list. Locators support several syntaxes:

from DrissionPage import WebPage

page = WebPage()
page.get('https://example.com')

# Tag-based locator
title = page.ele('tag:h1').text

# Attribute-based locator with @ syntax
links = page.eles('@class=product-link')

# XPath
price = page.ele('xpath://span[@id="price"]').text

# Chained locators
container = page.ele('@id=results')
items = container.eles('tag:div@class=item')

The @ prefix is DrissionPage's shorthand for attribute matching. '@class=product-link' finds any element whose class attribute equals product-link. You can also use @@ for multi-attribute matching (AND logic) and @@ with | for OR logic. This is more concise than writing XPath for common cases while still falling back to XPath when you need it.

ChromiumOptions for Browser Configuration

When you need ChromiumPage, you configure it through ChromiumOptions — the idiomatic way to set browser arguments, proxy settings, user-agent, and headless mode:

from DrissionPage import ChromiumOptions, ChromiumPage

co = ChromiumOptions()
co.headless(True)
co.set_argument('--no-sandbox')
co.set_argument('--disable-gpu')
co.set_user_agent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) ...')
co.set_proxy('http://gate.proxyhat.com:8080')

page = ChromiumPage(co)
page.get('https://example.com')

Note that set_proxy() on ChromiumOptions sets the proxy at the browser level — every request the browser makes, including XHR/fetch calls from JavaScript, goes through that proxy. This is critical for targets that load data via background API calls after the initial page load.

Packet Capture with listen.start()

One of DrissionPage's most powerful features for scrapers is its built-in packet listener. Instead of reverse-engineering API endpoints by reading minified JavaScript, you can instruct the browser to capture network traffic and extract JSON responses directly:

from DrissionPage import ChromiumPage, ChromiumOptions

co = ChromiumOptions().headless(True)
co.set_proxy('http://user-country-US:pass@gate.proxyhat.com:8080')
page = ChromiumPage(co)

# Start listening for XHR responses containing JSON
page.listen.start('api/products')
page.get('https://example.com/shop')

# Wait for the packet to arrive
packet = page.listen.wait()
print(packet.response.body)  # Raw JSON body

page.listen.stop()

This pattern is invaluable for discovering hidden APIs. Many modern sites render a shell HTML page and then fetch product data, pricing, or reviews via authenticated XHR endpoints. By capturing these packets, you can often identify the underlying API and switch to SessionPage for subsequent requests — cutting your browser usage dramatically.

Configuring Proxies in DrissionPage

Proxy configuration differs between SessionPage and ChromiumPage, and understanding the difference is essential for production scraping.

SessionPage: set_proxies()

SessionPage wraps the requests library, so proxy configuration follows the standard requests convention:

from DrissionPage import SessionPage

page = SessionPage()
page.set.proxies({
    'http': 'http://user-country-US-session-abc123:pass@gate.proxyhat.com:8080',
    'https': 'http://user-country-US-session-abc123:pass@gate.proxyhat.com:8080'
})
page.get('https://example.com')

The set.proxies() method (accessible via the set property) accepts a dictionary mapping protocols to proxy URLs. For ProxyHat residential proxies, the username encodes geo-targeting and session persistence: user-country-US-session-abc123 pins the exit IP to a specific US residential address for the duration of the session.

ChromiumPage: ChromiumOptions.set_proxy()

For the browser path, you set the proxy before launching Chromium via ChromiumOptions.set_proxy(). The proxy URL format is the same, but it applies to all browser traffic:

from DrissionPage import ChromiumOptions

co = ChromiumOptions()
co.set_proxy('http://user-country-DE-city-berlin-session-xyz789:pass@gate.proxyhat.com:8080')

One important caveat: Chromium applies the proxy at startup. If you need to change the proxy mid-session, you must restart the browser. This is why per-session proxy pinning (using the session- flag) is the recommended pattern — you assign each browser instance a sticky session ID and keep that IP for the instance's lifetime.

Why Residential Proxies for Hard Targets

Datacenter IPs are fast and cheap, but they're trivially detectable. Most anti-bot systems (Cloudflare, Akamai Bot Manager, DataDome) maintain IP reputation databases that flag datacenter ranges. Residential proxies use IPs assigned by ISPs to real households, making them indistinguishable from organic traffic at the IP layer.

For DrissionPage users, this matters because the framework's hybrid model is designed to minimize browser usage. If your HTTP requests from SessionPage are blocked by IP reputation checks, you're forced to escalate to ChromiumPage more often — defeating the cost advantage. Residential proxies keep your SessionPage requests succeeding, so you only use the browser when the page genuinely requires JavaScript.

Runnable Example: WebPage with Mode Switching

Here's a complete, runnable example that demonstrates the core value proposition: start in HTTP mode with a residential proxy, then escalate to Chromium when needed — all on the same WebPage object with shared cookies.

from DrissionPage import WebPage
import hashlib
import time

def build_proxy_url(country='US', session_id=None, city=None):
    """Build a ProxyHat residential proxy URL with geo + session flags."""
    username = f'user-country-{country}'
    if city:
        username += f'-city-{city}'
    if session_id:
        username += f'-session-{session_id}'
    return f'http://{username}:pass@gate.proxyhat.com:8080'

# Generate a stable session ID for IP pinning
session_id = hashlib.md5(str(time.time()).encode()).hexdigest()[:12]
proxy_url = build_proxy_url(country='US', session_id=session_id)

# Initialize WebPage — starts in SessionPage (HTTP) mode by default
page = WebPage()
page.set.proxies({
    'http': proxy_url,
    'https': proxy_url
})

# Step 1: Fetch the listing page via HTTP (fast, cheap)
page.get('https://example-shop.com/products')
product_links = page.eles('@class=product-card-link')
print(f'Found {len(product_links)} products via HTTP')

# Step 2: Switch to Chromium mode for a JS-rendered product page
# Cookies and session state carry over automatically
page.change_mode()  # Now in ChromiumPage mode

# Set the same proxy for the browser
from DrissionPage import ChromiumOptions
co = ChromiumOptions()
co.headless(True)
co.set_proxy(proxy_url)
page.set_options(co)

# Listen for background API calls
page.listen.start('api/reviews')
page.get('https://example-shop.com/product/12345')

# Capture the reviews XHR
review_packet = page.listen.wait(timeout=10)
if review_packet:
    print(f'Reviews JSON: {review_packet.response.body[:200]}')

page.listen.stop()
page.close()

This example illustrates the key workflow: HTTP for the easy parts, browser for the hard parts, same proxy identity throughout. The change_mode() call is what makes DrissionPage unique — it preserves cookies and headers so the target site sees a continuous session, not two separate visitors.

Production Patterns for DrissionPage Scraping

Per-Session Proxy Pinning

For sites that track IP consistency across a session, rotating IPs mid-session can trigger anti-bot challenges. The solution is to generate a unique session ID per scraping task and use it in the ProxyHat username:

import uuid

def get_pinned_proxy(country='US'):
    sid = uuid.uuid4().hex[:12]
    return f'http://user-country-{country}-session-{sid}:pass@gate.proxyhat.com:8080'

# Each worker gets its own sticky residential IP
worker_proxy = get_pinned_proxy('US')

With ProxyHat's session flag, the same session ID maps to the same exit IP for up to 30 minutes of inactivity. This gives you IP consistency without manual IP management.

Retries with Backoff

Even with residential proxies, requests can fail due to network blips or temporary blocks. Wrap your calls in a retry loop with exponential backoff:

import time
from DrissionPage import SessionPage

def fetch_with_retry(page, url, max_retries=3):
    for attempt in range(max_retries):
        try:
            resp = page.get(url, timeout=15)
            if resp.status_code == 200:
                return resp
            if resp.status_code == 403:
                # IP might be flagged — rotate by changing session
                time.sleep(2 ** attempt)
                continue
        except Exception as e:
            time.sleep(2 ** attempt)
    return None

For 403 responses specifically, consider rotating to a new session ID (new exit IP) rather than simply retrying with the same IP. A 403 often means the current IP has been flagged.

Discovering Hidden APIs with listen

Many e-commerce sites load product data via XHR after the initial HTML loads. Instead of scraping the rendered DOM (slow, brittle), use DrissionPage's packet listener to capture the API response and then replay that API via SessionPage:

1. Load the page once in ChromiumPage mode with listen.start() targeting likely API paths.
2. Capture the request URL, headers, and payload from the packet.
3. Replay the same request via SessionPage with your residential proxy — no browser needed.
4. Scale the SessionPage approach to thousands of product IDs.

This pattern can reduce your browser usage by 80–95% on sites that use client-side rendering with discoverable APIs.

Concurrency and Scaling

DrissionPage's SessionPage mode is thread-safe for simple use cases, but for high concurrency, use a process pool or async approach. Each SessionPage instance maintains its own requests.Session, so running 50–100 concurrent SessionPage workers is feasible on a single machine. For ChromiumPage, each instance launches a separate browser process — plan for roughly 150–250 MB of RAM per headless Chromium instance.

For containerized deployments, package DrissionPage with Chromium in a Docker image. Use a process supervisor (like supervisord or Kubernetes) to manage browser lifecycle, and set --no-sandbox and --disable-dev-shm-usage flags for container compatibility. The Chrome DevTools Protocol specification documents the full range of runtime parameters available for tuning browser behavior.

When NOT to Escalate to the Browser

The temptation with DrissionPage is to default to ChromiumPage because it "just works" on more sites. Resist this. Browser escalation should be a deliberate decision, not a fallback. Here's when you should stay in SessionPage:

• The HTML contains the data you need. Check the page source with page.html before assuming JavaScript is required. Many sites server-render product data and then hydrate it client-side — the data is already in the initial HTML.
• You've discovered the underlying API. If listen.start() revealed a JSON endpoint, replay it with SessionPage. It's 10–50x faster and uses a fraction of the bandwidth.
• The target is a simple form submission or redirect chain. SessionPage handles cookies, redirects, and form posts natively.
• You're doing high-volume serial scraping. If you need 50,000 pages and none require JS, running a browser is pure waste.

Escalate to ChromiumPage only when:

• The page requires JavaScript to render content (check by disabling JS in your browser and reloading).
• The site uses challenge-based bot detection (Cloudflare Turnstile, hCaptcha) that needs a real browser environment.
• You need to capture XHR traffic to discover hidden APIs.
• You need to interact with the page (clicking, scrolling, form filling) before data appears.

Ethics and Legal Considerations

Scraping public data is generally legal in many jurisdictions, but the legal landscape is nuanced. In the U.S., the Computer Fraud and Abuse Act (18 U.S.C. § 1030) criminalizes unauthorized access to protected computers. Courts have generally held that accessing public web pages does not constitute "unauthorized access," but circumventing authentication or access controls (including CAPTCHAs designed as access barriers) is riskier territory.

In the EU, GDPR applies to personal data — scraping names, emails, or other personally identifiable information from public pages may still constitute processing under GDPR, requiring a lawful basis. If your scraping targets involve any personal data, consult legal counsel.

Practical ethical guidelines:

• Respect robots.txt directives. DrissionPage doesn't automatically check robots.txt — you'll need to parse it yourself or use a library like urllib.robotparser.
• Honor rate limits. If a site responds with 429 Too Many Requests, back off. Residential proxies don't exempt you from being a good citizen.
• Prefer official APIs when available. They're faster, more reliable, and legally safer.
• Don't scrape behind authentication unless you have explicit permission.

ProxyHat Setup for DrissionPage

Configuring ProxyHat with DrissionPage is straightforward. Your ProxyHat credentials go directly into the proxy URL — no SDK installation required, though you can build a helper function for username construction as shown in the runnable example above.

Key configuration points:

• Gateway: gate.proxyhat.com on port 8080 (HTTP) or 1080 (SOCKS5)
• Username format: user-country-{CC}-city-{city}-session-{id}
• Geo-targeting: Use country codes like US, DE, GB, JP — see all available proxy locations
• Session persistence: Append -session-{id} to pin an exit IP for sticky sessions

For detailed authentication and configuration reference, see the ProxyHat documentation. To estimate costs for your workload, check the ProxyHat pricing page — residential proxy pricing scales with traffic, so the DrissionPage hybrid model (HTTP for most requests, browser only when needed) directly reduces your proxy bill.

For broader context on how residential proxies fit into scraping architectures, see our web scraping use case guide and SERP tracking overview.

Key Takeaways

• DrissionPage's hybrid model is its core value: SessionPage for HTTP, ChromiumPage for browser, WebPage to switch mid-session with shared state. This cuts both compute cost and proxy bandwidth.
• Use set.proxies() for SessionPage and ChromiumOptions.set_proxy() for ChromiumPage. Both accept the same ProxyHat URL format with geo and session flags in the username.
• Residential proxies keep SessionPage viable. If datacenter IPs get blocked, you're forced into browser mode more often, erasing the cost advantage of the hybrid approach.
• listen.start() is your API discovery tool. Capture XHR traffic in browser mode, then replay endpoints in HTTP mode at scale.
• Pin sessions per worker. Use the -session-{id} flag to maintain IP consistency for login flows and anti-bot evasion.
• Escalate to the browser deliberately, not reflexively. Check if data is in the initial HTML before assuming JavaScript is required.

FAQ

What is DrissionPage?

DrissionPage is a Python web automation framework that combines HTTP requests (SessionPage) and Chromium browser control (ChromiumPage) in a single API. Its WebPage class can switch between HTTP and browser modes while sharing cookies and session state, allowing scrapers to use the cheapest effective method for each page.

Why does DrissionPage matter for proxy users?

DrissionPage matters for proxy users because it lets you match request cost to target difficulty. HTTP requests through SessionPage are fast and use minimal proxy bandwidth, while browser requests through ChromiumPage are heavier. By running most requests in HTTP mode and only escalating to the browser when JavaScript is required, you reduce both proxy costs and compute overhead. Residential proxies are essential to keep SessionPage requests from being blocked by IP reputation checks.

Which proxy type works best for DrissionPage?

Residential proxies work best for DrissionPage when targeting sites with anti-bot protection, because they use real ISP-assigned IPs that are harder to detect than datacenter ranges. For SessionPage (HTTP mode), configure proxies via set.proxies() with the ProxyHat gateway at gate.proxyhat.com:8080. For ChromiumPage, use ChromiumOptions.set_proxy() with the same gateway. Use the session flag (-session-{id}) for sticky IPs when consistency matters.

How do you avoid blocks when implementing DrissionPage?

To avoid blocks, use residential proxies with geo-targeting, pin sessions per worker with the -session-{id} flag, implement retry logic with exponential backoff for 403/429 responses, and rotate session IDs when an IP appears flagged. Use DrissionPage's listen.start() to discover underlying APIs and replay them via SessionPage to reduce browser fingerprinting exposure. Always respect robots.txt and rate limits, and set realistic user-agent strings via ChromiumOptions.set_user_agent().

Can DrissionPage capture background XHR and JSON responses?

Yes. DrissionPage's listen.start() method activates a packet listener on ChromiumPage that captures network traffic matching a URL pattern. After calling page.get(), use page.listen.wait() to retrieve the matching response packet, including its body, headers, and request details. This is useful for discovering hidden API endpoints that sites use to load data via JavaScript after the initial page render.