Proxy Setup for AI Agents: browser-use, Crawl4AI and an MCP Server

An AI agent on a cloud VM leaves from a datacenter IP. Sites see that IP before they see anything else, so the agent gets results for the wrong country, hits rate limits it never caused, or lands in a CAPTCHA loop it can't get out of. A good proxy setup for AI agents fixes the egress problem in the browser config, never in the prompt. Multi-step tasks keep one sticky IP from start to finish, and parallel fetches rotate. Below is working configuration for browser-use, Crawl4AI and an MCP server, plus the cost controls and the failure modes that come up most often.

Why a proxy setup for AI agents matters more than for scripts

Agents need a proxy for the same three reasons scrapers do: geography, rate limits and IP reputation. They need it more urgently because they run unattended and make their own retry decisions. A script that gets blocked stops. An agent that gets blocked tries again, rephrases, opens a new tab and tries again, and every attempt costs bandwidth and LLM tokens.

• Geography. Prices, search results, stock and legal notices change by country. An agent comparing prices for a German customer from a US-East VM is collecting the wrong data. Country targeting in the proxy username fixes that without touching the agent's logic. How geo-targeted proxies work explains the mechanics.
• Cloud egress reputation. AWS, GCP, Azure and Hetzner ranges are public, and IP-reputation services score them as hosting, not consumer access. Many sites challenge or throttle these ranges on sight, whatever the request looks like. Our IP reputation deep-dive covers how that scoring works.
• Shared rate limits. Several agents behind one NAT gateway share a single public IP. Each one is polite on its own, but together they look like one aggressive client.

What a proxy does not fix: a residential IP doesn't make an automated browser look human. Behavioural and fingerprint detection still works, and our companion piece on how websites detect AI browser agents shows how. Use proxies to get correct, geo-accurate data and to spread load fairly. Don't use them to get around a site that has told automation to stay out.

Sticky sessions vs rotation: decide per task, not per project

Rule of thumb: if the task carries state, pin the IP; if it doesn't, rotate. A login → navigate → add-to-cart → checkout flow has to keep one exit IP from start to finish, because many sites bind session cookies, CSRF tokens or risk scores to the IP that opened the session. If the IP changes mid-flow, you'll often get logged out or re-challenged. Independent page fetches are the reverse: spreading them across IPs keeps each one under per-IP limits.

With a backconnect gateway you switch modes by changing the username. USERNAME-sid-abc123-ttl-10m keeps the same exit IP for as long as you reuse that session ID. A plain USERNAME rotates. USERNAME-country-US pins the country. The sticky vs rotating sessions guide covers session lifetimes in more depth.

browser-use: proxy configuration in the current API

browser-use takes a proxy=ProxySettings(...) argument on Browser (an alias of BrowserSession), with server, username, password and bypass fields. The credentials go to the browser and never into the LLM context. Under the hood, browser-use launches Chromium with --proxy-server and answers the proxy's auth challenge over CDP (Fetch.authRequired), so you never need to put credentials in the URL. Note the import path: ProxySettings lives in browser_use.browser, not the package root (see the browser parameters reference and profile.py).

import asyncio
import os
import uuid

from browser_use import Agent, Browser, ChatOpenAI
from browser_use.browser import ProxySettings

PROXY_USER = os.environ["PROXYHAT_USER"]
PROXY_PASS = os.environ["PROXYHAT_PASS"]


async def run_checkout_test() -> None:
    # One sticky session ID for the whole flow, so login -> cart -> checkout keeps one exit IP.
    session_id = uuid.uuid4().hex[:12]

    browser = Browser(
        headless=True,
        proxy=ProxySettings(
            server="http://gate.proxyhat.com:8080",
            username=f"{PROXY_USER}-sid-{session_id}-ttl-10m",
            password=PROXY_PASS,
            bypass="localhost,127.0.0.1",
        ),
        # Keep the agent on the site under test; it also limits where sensitive_data can be typed.
        allowed_domains=["https://staging.example-shop.com"],
        # Chromium switch: don't download images at all.
        args=["--blink-settings=imagesEnabled=false"],
    )

    agent = Agent(
        task="Log in with x_user / x_pass, add the cheapest USB-C cable to the cart, "
             "stop at the payment page and report the order total.",
        llm=ChatOpenAI(model=os.environ["OPENAI_MODEL"]),
        browser=browser,
        # The model sees the placeholders x_user / x_pass, never the values.
        sensitive_data={
            "https://staging.example-shop.com": {
                "x_user": os.environ["SHOP_USER"],
                "x_pass": os.environ["SHOP_PASS"],
            }
        },
        use_vision=False,   # no screenshots in every step: fewer tokens
        max_failures=2,     # default is 5; each failure is another round of page loads
    )
    history = await agent.run(max_steps=25)   # default is 500
    print(history.final_result())


asyncio.run(run_checkout_test())

Notes on this pattern:

• Use the HTTP port with authenticated proxies. Chromium's network stack docs state that no authentication methods are supported for SOCKSv5, so socks5://…:1080 with credentials will fail in any Chromium-based agent. Use http://gate.proxyhat.com:8080.
• Proxy credentials are not sensitive_data. sensitive_data is for values the agent has to type into pages. Proxy credentials go in ProxySettings, which the model never sees. Don't paste a proxy URL into the task string "so the agent knows".
• allowed_domains matters more than it looks. browser-use logs a warning if you pass sensitive_data without locking domains down, because a prompt-injected page could otherwise talk the agent into typing secrets somewhere else.
• Cloud mode is a different egress. Browser(use_cloud=True) runs the browser on Browser Use Cloud with its own proxies. Your ProxySettings apply to the local or cdp_url browser you launch yourself.
• Default extensions save bandwidth. enable_default_extensions is on by default and loads uBlock Origin Lite plus a cookie-banner handler, so most ads and trackers never count against your traffic.

Crawl4AI: per-crawl proxies, rotation and sticky sessions

In Crawl4AI, set the proxy per crawl on CrawlerRunConfig, either as proxy_config or through a proxy_rotation_strategy. The proxy docs mark the old BrowserConfig proxy argument as deprecated and recommend configuring proxies per request. For sticky behaviour, proxy_session_id pins one proxy from the strategy to every crawl that shares that ID.

A backconnect gateway is a single host, so "a list of proxies" means a list of session usernames. Each session ID is a slot with its own exit IP, and round-robin spreads URLs across the slots:

import asyncio
import os
import uuid

from crawl4ai import (AsyncWebCrawler, BrowserConfig, CacheMode,
                      CrawlerRunConfig, ProxyConfig, RoundRobinProxyStrategy)

USER = os.environ["PROXYHAT_USER"]
PASS = os.environ["PROXYHAT_PASS"]


def slot() -> ProxyConfig:
    # One sticky session per slot, so each slot keeps one exit IP.
    sid = uuid.uuid4().hex[:10]
    return ProxyConfig(
        server="http://gate.proxyhat.com:8080",
        username=f"{USER}-sid-{sid}-ttl-10m",
        password=PASS,
    )


strategy = RoundRobinProxyStrategy([slot() for _ in range(8)])

browser_cfg = BrowserConfig(
    headless=True,
    text_mode=True,   # tries to disable images and other heavy content
    avoid_ads=True,   # blocks common ad/tracker domains
)

bulk_cfg = CrawlerRunConfig(
    proxy_rotation_strategy=strategy,   # next slot for each URL
    cache_mode=CacheMode.BYPASS,
    check_robots_txt=True,
    max_retries=1,
)


async def main(urls: list[str]) -> None:
    async with AsyncWebCrawler(config=browser_cfg) as crawler:
        results = await crawler.arun_many(urls, config=bulk_cfg)
        for r in results:
            print(r.url, r.success, len(r.markdown or ""))

        # Stateful sequence: same proxy slot AND same browser page for every step.
        step_cfg = bulk_cfg.clone(proxy_session_id="acct-42", session_id="acct-42")
        await crawler.arun("https://example.com/login", config=step_cfg)
        await crawler.arun("https://example.com/account/orders", config=step_cfg)


asyncio.run(main(["https://example.com/a", "https://example.com/b"]))

Two settings are easy to mix up. session_id keeps the same browser page across arun() calls, and proxy_session_id keeps the same proxy. A stateful flow needs both. For geo-pinned rotation instead of sticky slots, use a single ProxyConfig with the username USERNAME-country-US.

A retry-amplification trap: proxy_config also accepts a list of proxies to fall back through, and the retry loop in async_webcrawler.py walks the whole list on each attempt. A blocked URL can therefore be fetched up to (max_retries + 1) × len(list) times. With 10 fallbacks and max_retries=3, that's 40 full page loads for one URL that was never going to load. Keep the list short and set max_retries on purpose.

Giving an MCP assistant proxied web access

You have two options: run Microsoft's Playwright MCP server with a proxy in its config, or write a small MCP tool that fetches through the proxy and returns text. Use the first when the assistant has to click and type. Use the second when it only needs to read pages, because that's far cheaper per call.

Option 1: Playwright MCP with an authenticated proxy

@playwright/mcp has --proxy-server and --proxy-bypass flags (README). For a proxy that needs credentials, use the JSON config file instead. Its browser.launchOptions is passed straight to Playwright, whose proxy option takes separate username and password fields:

{
  "browser": {
    "browserName": "chromium",
    "isolated": true,
    "launchOptions": {
      "headless": true,
      "proxy": {
        "server": "http://gate.proxyhat.com:8080",
        "username": "USERNAME-sid-mcp01-ttl-10m",
        "password": "PASSWORD"
      }
    }
  },
  "capabilities": ["core"]
}

npx @playwright/mcp@latest --config ./playwright-mcp.json --image-responses omit

Keep this file out of the repository and readable only by the user running the server. The session ID here means the whole assistant session shares one IP, which fits "log in and do X". Restart with a new ID when you want a fresh IP. --image-responses omit stops screenshots from going back to the model. "isolated": true (the --isolated flag) keeps the profile in memory, so cookies don't leak between tasks. The README also points out that --allowed-origins and --blocked-origins are not a security boundary. Treat them as hygiene, not protection.

Option 2: a minimal proxied fetch tool with the official MCP Python SDK

The Python SDK's v2 renamed FastMCP to MCPServer (migration guide). Code written for v1 with from mcp.server.fastmcp import FastMCP needs the new import. Here is a read-only tool that fetches through the gateway, caps the response size and never puts the proxy URL in its output:

import os
import re

import httpx
from mcp.server import MCPServer

mcp = MCPServer("proxied-fetch")

_USER = os.environ["PROXYHAT_USER"]
_PASS = os.environ["PROXYHAT_PASS"]
_MAX_BYTES = 1_500_000
_COUNTRY = re.compile(r"^[A-Za-z]{2}$")
_SESSION = re.compile(r"^[A-Za-z0-9]{1,24}$")


def _proxy(country: str | None, session: str | None) -> str:
    user = _USER
    # Validate model-supplied values: they end up inside the proxy username.
    if session:
        if not _SESSION.match(session):
            raise ValueError("session must be 1-24 alphanumerics")
        user = f"{_USER}-sid-{session}-ttl-10m"
    elif country:
        if not _COUNTRY.match(country):
            raise ValueError("country must be a 2-letter code")
        user = f"{_USER}-country-{country.upper()}"
    return f"http://{user}:{_PASS}@gate.proxyhat.com:8080"


@mcp.tool()
async def fetch_page(url: str, country: str | None = None, session: str | None = None) -> str:
    """Fetch a public http(s) page through a residential proxy and return its text.

    country: optional 2-letter exit country. session: optional ID to reuse one exit IP.
    """
    if not url.startswith(("http://", "https://")):
        return "error: only http(s) URLs are allowed"
    try:
        async with httpx.AsyncClient(proxy=_proxy(country, session), timeout=30,
                                     follow_redirects=True) as client:
            async with client.stream("GET", url) as resp:
                body = b""
                async for chunk in resp.aiter_bytes():
                    body += chunk
                    if len(body) > _MAX_BYTES:
                        break
        text = body.decode(resp.encoding or "utf-8", errors="replace")
        return f"HTTP {resp.status_code} {resp.url}\n\n{text[:200_000]}"
    except ValueError as e:
        return f"error: {e}"
    except httpx.HTTPError as e:
        # Only the exception type: some messages can echo the proxy URL.
        return f"error: fetch failed ({type(e).__name__})"


if __name__ == "__main__":
    mcp.run(transport="stdio")

Register it with your MCP client as a stdio server. Pass PROXYHAT_USER and PROXYHAT_PASS through the client's env block, never through tool arguments. In Claude Code, for example, that is claude mcp add --env PROXYHAT_USER=… --env PROXYHAT_PASS=… --transport stdio proxied-fetch -- python server.py. Keep --transport between the last --env and the server name, or the CLI reads the name as another env pair. The regex checks earn their place: a prompt-injected page can make the model call the tool with odd arguments, and unchecked input would let it write arbitrary text into your proxy username. For pages that need JavaScript, use option 1, or our Playwright rotating proxies guide if you're building the browser layer yourself.

Cost control: agents burn bandwidth in ways scripts don't

Residential traffic is billed by the gigabyte, and an agent loads full pages, often several times per step. These controls make the biggest difference, roughly in order:

1. Don't load what the model won't read. Images, video, fonts and ad scripts are most of a modern page's weight. Use text_mode=True and avoid_ads=True in Crawl4AI, the imagesEnabled=false switch or uBlock in browser-use, and a fetch tool instead of a browser when the page renders server-side.
2. Cap retries in two places. Framework retries (max_failures, max_retries) and agent-level re-planning multiply each other. An agent that "tries a different approach" after a 403 often just reloads the same blocked page. Set max_steps low enough that a stuck run dies cheaply.
3. Reuse contexts in stateful flows. A new browser context means fresh cookies, a fresh challenge and a fresh download of every cached asset. Within one task, keep the context (and the sticky IP) alive.
4. Stop on block signals. Treat a CAPTCHA page, a 403 from a bot-management vendor, or a 429 as a terminal state that gets reported to a human. Don't let the agent retry its way past it. Our CAPTCHA handling guide has the reasoning for backing off.
5. Measure per task, not per month. Log bytes per agent run. A sudden jump usually means a loop, not a bigger job.

Failure modes worth designing for

Most agent proxy incidents come from one of four causes: a CAPTCHA loop, retry amplification, a session that silently changes IP, or credentials leaking into model context.

• CAPTCHA loops. Vision-enabled agents will try to "solve" a challenge, fail, reload and try again. Detect challenge pages (known iframe hosts, titles such as "Just a moment…") and end the run. Answering challenges at scale isn't something a legitimate pipeline should do, and repeated failures from a residential IP hurt that IP's reputation for everyone using it.
• Retry amplification. Covered above: framework retries × proxy fallback lists × agent re-planning. Watch total attempts per URL, not just failures.
• Silent session breaks. A sticky session ends when its TTL runs out, and can end earlier if the upstream peer goes offline; the next request then leaves from a new IP. If a mid-flow request suddenly redirects to a login page, assume the IP changed. Restart the flow with a new session ID instead of letting the agent improvise.
• Credentials in the LLM context. Anything that reaches the model can end up in logs, traces, eval datasets or a prompt-injected exfiltration request. Keep proxy credentials in environment variables read by the browser or HTTP client. Don't put them in task strings, system prompts or tool output, and strip exception messages before returning them. Playwright MCP's secrets option can redact known strings from tool responses, but its own README calls that a convenience, not a security feature.

Before blaming the agent, test the proxy on its own: run the username variant you configured through the proxy checker and confirm the exit country and IP type are what you expect. For a broader comparison of IP types for agent workloads, see Best Proxies for AI Agents and LLM Web Data Collection.

What this setup does not solve

A proxy changes where traffic comes from, not how it behaves. It won't make an agent pass behavioural bot detection, and it isn't permission to access a site that blocks automation in its robots.txt or terms. It won't make collecting personal data lawful either. If a site offers an API, a feed, or a signed-agent programme, those routes are usually cheaper and more reliable than any proxy configuration. Use residential IPs for what they're good at: correct geography, fair load distribution, and not inheriting a cloud range's reputation.