Scraping With nodriver: A Practical Guide to Undetected Async Browser Automation

Scraping With nodriver has become the go-to approach for Python engineers who need to extract data from pages protected by Cloudflare, Imperva, or DataDome. If you are migrating away from undetected-chromedriver — or building a new automation pipeline from scratch — this guide walks through nodriver's architecture, its idiomatic async API, and how to pair it with ProxyHat residential proxies for reliable, large-scale collection.

Before you begin: Only scrape data you are authorized to access. Respect robots.txt, site Terms of Service, and applicable laws such as the Computer Fraud and Abuse Act (CFAA) and the GDPR Article 5. This article covers techniques for public-data collection and authorized testing only.

Why Scraping With nodriver Replaces undetected-chromedriver

undetected-chromedriver (uc) was a patch-layer on top of Selenium WebDriver. It worked by modifying the chromedriver binary to remove the $cdc_ variables that anti-bot scripts fingerprinted. But the landscape shifted: Chrome 111+ introduced CDP improvements, and anti-bot vendors began detecting WebDriver itself — not just chromedriver artifacts. The navigator.webdriver flag, WebDriver command traces, and CDP leak vectors made Selenium-based approaches increasingly fragile.

nodriver is the successor written by the same author (ultrafunkamsterdam). It drops Selenium entirely and communicates with Chrome through the DevTools Protocol over a WebSocket connection. No chromedriver binary, no WebDriver session, no navigator.webdriver = true. This architectural change eliminates entire categories of detection signals.

Core architectural differences

nodriver Architecture: Zero WebDriver, Direct CDP

nodriver launches Chrome with --remote-debugging-port and connects to the DevTools WebSocket endpoint. Every action — navigation, DOM query, click, screenshot — is a CDP command sent over the socket. Because there is no WebDriver layer, the browser process looks like a normal user-launched Chrome instance to most fingerprinting scripts.

The key anti-detection advantages:

• No navigator.webdriver flag. Selenium sets this to true; uc patches it to false, but the patch itself can be detected via property descriptor inspection. nodriver never injects a WebDriver context, so the property is absent.
• No chromedriver artifacts. Anti-bot scripts scan for $cdc_ variables and cdc_ strings in the driver process. nodriver has no driver process to scan.
• Fewer CDP leak vectors. While raw CDP can still be detected through Runtime.evaluate timing or Page.addScriptToEvaluateOnNewDocument traces, nodriver minimizes the surface by not layering additional automation frameworks on top.

For Cloudflare Turnstile and Imperva Incapsula, which combine browser fingerprinting with IP reputation checks, removing WebDriver signals is necessary but not sufficient. You also need clean IP addresses — which is where residential proxies enter the picture.

The Idiomatic nodriver API

nodriver's API is async-first. You do not write driver.find_element_by_xpath() or WebDriverWait(driver, 10).until(...). Instead, you await CDP operations and register event listeners.

Starting a browser

import nodriver as uc
import asyncio

async def main():
    browser = await uc.start(
        headless=False,
        browser_args=["--disable-blink-features=AutomationControlled"]
    )
    tab = await browser.get("https://example.com")
    print(await tab.get_content())
    browser.stop()

if __name__ == "__main__":
    uc.loop().run_until_complete(main())

uc.start() returns a Browser object. browser.get(url) opens a new Tab and navigates to the URL. Each tab is an independent CDP target — you can run dozens of tabs concurrently within a single browser process.

Finding elements and interacting

# Select an element by CSS selector
button = await tab.select("button#submit", timeout=10)
await button.click()

# Find multiple elements
items = await tab.find("div.product-card", best_match=False)
for item in items:
    text = await item.get_attribute("data-product-id")
    print(text)

tab.select() returns the best-matching element (or raises TimeoutError). tab.find() returns a list. Neither uses WebDriver's implicit/explicit wait model — they poll internally via CDP and resolve as soon as the DOM matches.

Event hooks instead of WebDriver waits

nodriver exposes CDP events through tab.add_handler(). This is the idiomatic replacement for WebDriverWait:

async def on_request(event):
    if event.request.url.endswith("/api/data"):
        print(f"Intercepted: {event.request.url}")

tab.add_handler(uc.cdp.network.RequestWillBeSent, on_request)
await tab.get("https://protected-site.com")

You can listen for Network.responseReceived, Page.loadEventFired, Runtime.consoleAPICalled, and dozens of other CDP events. This is far more efficient than polling the DOM in a loop.

Configuring a nodriver Proxy with ProxyHat

nodriver does not include built-in proxy rotation or a proxy manager. You configure proxies the same way you would for a manual Chrome launch: via the --proxy-server command-line flag. The challenge with authenticated proxies is that Chrome's --proxy-server flag does not accept credentials inline — you need an authentication extension or a proxy that supports IP whitelisting.

ProxyHat supports both IP whitelisting (recommended for browser automation) and username:password authentication. For nodriver, IP whitelisting is the cleaner path because it avoids injecting a Manifest V2/V3 extension that may itself be fingerprinted.

Building the proxy URL

The ProxyHat gateway endpoint is gate.proxyhat.com:8080 for HTTP and gate.proxyhat.com:1080 for SOCKS5. Geo-targeting and session flags are embedded in the username:

PROXY_HOST = "gate.proxyhat.com"
PROXY_PORT = 8080

# US residential, sticky session
PROXY_USER = "user-country-US-session-abc123"
PROXY_PASS = "your_password"

proxy_url = f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"

# For Chrome --proxy-server, use the host:port form
chrome_proxy_arg = f"--proxy-server=http://{PROXY_HOST}:{PROXY_PORT}"

When using --proxy-server with authenticated proxies, you have two options:

1. IP whitelisting — add your server IP in the ProxyHat dashboard, then use --proxy-server=http://gate.proxyhat.com:8080 with no credentials. This is the most stealth-friendly approach.
2. Extension-based auth — generate a small Chrome extension that sets webRequest.onAuthRequired credentials. nodriver supports loading unpacked extensions via browser_args=["--load-extension=/path/to/auth-ext"].

Why residential IPs matter for headful stealth

Even with perfect browser fingerprinting, a datacenter IP on an ASN belonging to AWS, DigitalOcean, or OVH will trigger Cloudflare's IP reputation engine. Residential proxies route traffic through ISP-assigned IPs that look like real home users. For protected targets, residential proxies typically achieve 2x to 5x higher success rates than datacenter proxies on the same browser setup.

You can explore available geolocations on the ProxyHat locations page and compare plan options on the pricing page.

Runnable Example: nodriver + ProxyHat Residential Proxy

Here is a complete, runnable script that launches nodriver through a US residential sticky session, navigates to a protected page, waits for a JSON API response via CDP event interception, and extracts the payload.

import nodriver as uc
import asyncio
import json

PROXY_HOST = "gate.proxyhat.com"
PROXY_PORT = 8080
PROXY_USER = "user-country-US-session-abc123"
PROXY_PASS = "your_password"

class ProxyHatConfig:
    """Builds proxy configuration for nodriver browser_args."""
    def __init__(self, host, port, user, password):
        self.host = host
        self.port = port
        self.user = user
        self.password = password

    @property
    def chrome_arg(self):
        # Use IP whitelisting for browser automation
        return f"--proxy-server=http://{self.host}:{self.port}"

    @property
    def curl_proxy(self):
        return f"http://{self.user}:{self.password}@{self.host}:{self.port}"

async def main():
    config = ProxyHatConfig(PROXY_HOST, PROXY_PORT, PROXY_USER, PROXY_PASS)

    browser = await uc.start(
        headless=True,
        browser_args=[
            config.chrome_arg,
            "--disable-blink-features=AutomationControlled",
            "--no-sandbox",
            "--disable-dev-shm-usage",
        ],
    )

    tab = await browser.get("https://protected-target.com/dashboard")

    captured_json = []

    async def on_response(event):
        url = event.response.url
        if "/api/products" in url and event.response.mimeType == "application/json":
            body = await tab.send(uc.cdp.network.getResponseBody(
                requestId=event.requestId
            ))
            captured_json.append(json.loads(body[0]))

    tab.add_handler(uc.cdp.network.ResponseReceived, on_response)

    # Trigger the API call by interacting with the page
    load_button = await tab.select("button#load-products", timeout=15)
    await load_button.click()

    # Wait for the response
    await asyncio.sleep(3)
    browser.stop()

    if captured_json:
        print(f"Captured {len(captured_json[0])} products")
        print(json.dumps(captured_json[0][:2], indent=2))
    else:
        print("No API response intercepted")

if __name__ == "__main__":
    uc.loop().run_until_complete(main())

This example demonstrates the idiomatic nodriver pattern: launch with browser_args, register a CDP event handler for network.ResponseReceived, interact with the page to trigger XHR/fetch calls, and capture the JSON response directly from the network layer — no HTML parsing required.

Scaling: Concurrent Tabs, Docker Fleet, Per-Context Proxies

nodriver is designed for concurrency. A single browser process can manage 20–50 concurrent tabs on a machine with 8 GB RAM and 4 vCPUs. Beyond that, you should run multiple browser instances — each with its own proxy assignment — in containers.

Concurrent tabs with per-tab proxy rotation

Chrome's --proxy-server flag applies to the entire browser process. To assign different proxies per tab, you have two options:

1. Multiple browser instances — launch one uc.start() per proxy, each with its own --proxy-server arg. This is the simplest and most isolated approach.
2. CDP Network.setRequestInterception — intercept requests at the protocol level and route them through different upstream proxies. This is complex and not well-supported in nodriver's high-level API.

import nodriver as uc
import asyncio

async def scrape_with_proxy(country, session_id, url):
    proxy_arg = f"--proxy-server=http://gate.proxyhat.com:8080"
    browser = await uc.start(
        headless=True,
        browser_args=[proxy_arg, "--no-sandbox", "--disable-dev-shm-usage"],
    )
    try:
        tab = await browser.get(url)
        title = await tab.evaluate("document.title")
        return {"country": country, "session": session_id, "title": title}
    finally:
        browser.stop()

async def run_fleet():
    tasks = []
    targets = [
        ("US", "sess-001", "https://example.com"),
        ("DE", "sess-002", "https://example.com"),
        ("GB", "sess-003", "https://example.com"),
    ]
    for country, sid, url in targets:
        tasks.append(scrape_with_proxy(country, sid, url))
    results = await asyncio.gather(*tasks, return_exceptions=True)
    for r in results:
        if isinstance(r, Exception):
            print(f"Error: {r}")
        else:
            print(r)

if __name__ == "__main__":
    uc.loop().run_until_complete(run_fleet())

Docker headless fleet

For production scraping, containerize each browser instance. A typical Dockerfile for nodriver:

FROM python:3.11-slim

RUN apt-get update && apt-get install -y \
    chromium \
    chromium-driver \
    fonts-liberation \
    libappindicator3-1 \
    libasound2 \
    libatk-bridge2.0-0 \
    libatk1.0-0 \
    libcups2 \
    libdbus-1-3 \
    libgdk-pixbuf2.0-0 \
    libnspr4 \
    libnss3 \
    libx11-xcb1 \
    libxcomposite1 \
    libxdamage1 \
    libxrandr2 \
    xdg-utils \
    && rm -rf /var/lib/apt/lists/*

RUN pip install nodriver

COPY scraper.py /app/scraper.py
WORKDIR /app

CMD ["python", "scraper.py"]

Scale horizontally with Docker Compose or Kubernetes. Each pod runs one browser instance with one proxy. Use a graceful shutdown handler to call browser.stop() on SIGTERM — this closes the CDP connection cleanly and avoids zombie Chrome processes.

import signal
import sys

browser = None

def graceful_shutdown(signum, frame):
    if browser:
        browser.stop()
    sys.exit(0)

signal.signal(signal.SIGTERM, graceful_shutdown)
signal.signal(signal.SIGINT, graceful_shutdown)

When NOT to Use a Full Browser

nodriver is powerful but expensive. A headless Chrome instance consumes 200–400 MB of RAM per browser process and adds 500ms to 2s of overhead per page load compared to raw HTTP requests. If your target does not require JavaScript rendering or challenge solving, you are wasting resources.

For static HTML or JSON APIs, use curl_cffi with ProxyHat HTTP proxies:

from curl_cffi import requests

response = requests.get(
    "https://api.example.com/data",
    proxies={"https": "http://user-country-US:pass@gate.proxyhat.com:8080"},
    impersonate="chrome120",
)
print(response.json())

curl_cffi impersonates Chrome's TLS fingerprint (JA3/JA4) at the HTTP layer — no browser process needed. It handles many Cloudflare challenges that check only TLS and HTTP/2 fingerprints, not full JavaScript execution. For SERP tracking and simple API scraping, this approach is 10x cheaper than running a browser fleet. See our SERP tracking use case for details.

Use nodriver when:

• The target requires JavaScript execution (SPA, client-side rendering).
• Cloudflare Turnstile or Imperva presents a interactive challenge.
• You need to capture XHR/fetch responses from the network layer.
• The site uses canvas/WebGL fingerprinting that only a real browser can satisfy.

Use HTTP + curl_cffi when:

• The page is server-rendered HTML.
• The data is available through a JSON API endpoint you can call directly.
• The anti-bot check is TLS-fingerprint-only (no JS challenge).

For broader guidance on choosing the right approach, see our web scraping use case overview.

Common Mistakes and Edge Cases

• Using headless=True without stealth flags. Headless Chrome 112+ is less detectable than older versions, but some sites still fingerprint navigator.webdriver or User-Agent headless strings. Always add --disable-blink-features=AutomationControlled.
• Reusing a single sticky session indefinitely. A sticky session keeps the same exit IP, which is great for login flows but will eventually get rate-limited. Rotate sessions every 50–200 requests depending on the target's sensitivity.
• Not handling CDP disconnections. If Chrome crashes or the WebSocket drops, nodriver raises ConnectionError. Wrap your scraping logic in try/except and implement retry with a fresh browser instance.
• Forgetting to call browser.stop(). Without explicit cleanup, Chrome processes accumulate and exhaust system resources. Use finally: blocks or context managers.
• Running too many tabs per browser. Beyond 30 concurrent tabs, Chrome's memory usage spikes and tab crashes become frequent. Prefer multiple browser instances with fewer tabs each.

Key Takeaways

• nodriver eliminates WebDriver detection by communicating directly via CDP — no chromedriver, no navigator.webdriver, no $cdc_ artifacts.
• Proxy configuration is via --proxy-server browser args — nodriver has no built-in proxy manager. Use IP whitelisting with ProxyHat for the cleanest setup.
• Residential proxies are essential for protected targets — datacenter IPs fail IP reputation checks even with a perfect browser fingerprint.
• Scale with multiple browser instances, not dozens of tabs in one process. Containerize with Docker and assign one proxy per instance.
• Use HTTP + curl_cffi for non-JS targets — it is 10x cheaper and handles TLS-fingerprint-only challenges.
• Always respect legal boundaries — CFAA, GDPR, robots.txt, and site ToS apply to all automated collection.

FAQ

What is Scraping With nodriver?

Scraping With nodriver refers to using the nodriver Python library — the async successor to undetected-chromedriver — to automate Chrome via the Chrome DevTools Protocol without Selenium or WebDriver. It removes common detection signals like navigator.webdriver and chromedriver artifacts, making it harder for Cloudflare, Imperva, and DataDome to identify automated traffic.

Why does Scraping With nodriver matter for proxy users?

nodriver handles browser-level stealth, but anti-bot systems also check IP reputation. A datacenter IP on an AWS or OVH ASN will trigger blocks even with a perfect browser fingerprint. Pairing nodriver with residential proxies ensures that both the browser fingerprint and the IP address look like a real user, dramatically improving success rates on protected targets.

Which proxy type works best for Scraping With nodriver?

Residential proxies are the best choice for nodriver scraping of protected sites. They route traffic through ISP-assigned IPs that pass IP reputation checks. Mobile proxies offer even higher trust scores but at higher cost. Datacenter proxies are suitable only for non-protected targets or internal testing. For nodriver, configure proxies via the --proxy-server Chrome flag with IP whitelisting in the ProxyHat dashboard.

How do you avoid blocks when implementing Scraping With nodriver?

Combine four strategies: (1) use nodriver with --disable-blink-features=AutomationControlled to minimize browser fingerprint leaks; (2) route traffic through residential proxies with geo-targeting matching your target's expected audience; (3) rotate sticky sessions every 50–200 requests to avoid rate limits; (4) add realistic delays and human-like interaction patterns. Never reuse a single IP indefinitely on sensitive targets.

Can nodriver handle SOCKS5 proxies?

Yes. Chrome's --proxy-server flag supports SOCKS5. Use --proxy-server=socks5://gate.proxyhat.com:1080 with IP whitelisting configured in the ProxyHat dashboard. SOCKS5 can be preferable when the target site blocks HTTP proxy headers or when you need UDP support, though for most web scraping scenarios HTTP proxies on port 8080 work equally well.