Using Proxies in R: A Code-First Guide with httr2 and rvest

If you've ever run a web scraping script in R only to get empty pages, HTTP 403s, or CAPTCHA walls after a few dozen requests, you're not alone. Using proxies in R is the standard fix: route your HTTP traffic through residential IP addresses that look like real users rather than a datacenter block that gets flagged instantly. This guide walks through the modern R stack — httr2 for requests with req_proxy() and rvest for parsing — with runnable code for geo-targeting, session rotation, pagination, and JavaScript-rendered pages.

Why Using Proxies in R Matters

R has become a serious tool for data collection, not just analysis. Packages like httr2 and rvest make it straightforward to fetch and parse web pages, but they inherit a fundamental limitation: every request originates from your own IP address. When you hit a target site repeatedly — whether for SERP tracking, price monitoring, or research — that IP becomes a fingerprint. Anti-bot systems from Cloudflare, Datadome, and PerimeterX correlate request patterns, IP reputation, and TLS fingerprints to block scrapers within minutes.

The problem is structural. According to the rvest documentation, the package is designed for static HTML parsing, not for evading detection. That's by design — rvest is a parser, not a stealth engine. The same goes for httr2, which provides a clean request-builder API but sends traffic directly from your machine. When a site rate-limits by IP, blocks datacenter ranges, or geo-restricts content, you need a proxy layer between your R process and the target.

Residential vs Datacenter vs Mobile Proxies

Not all proxies solve the same problem. The IP type you choose determines whether your requests look like a real user in a specific city or a bot in a server farm.

Proxy Type	IP Source	Detection Risk	Typical Latency	Best For
Residential	ISP-assigned home connections	Low	200–500ms	Geo-walled content, anti-bot sites, SERP scraping
Datacenter	Cloud/hosting provider ranges	High	50–100ms	APIs, non-restricted endpoints, high-volume fetching
Mobile	Cellular carrier IPs	Very low	300–800ms	Social media, ticketing, high-trust targets

Residential proxies are the workhorse for blocked or geo-walled sources. A residential IP in London looks like a real user browsing from their home ISP, not a scraper in a Virginia datacenter. Mobile proxies go further — they're assigned by cellular carriers, which many anti-bot systems whitelist entirely. Datacenter proxies are fast and cheap but get flagged quickly on any site with modern bot protection.

Basic Proxy Setup with httr2 req_proxy

The httr2 package provides req_proxy(), which configures an HTTP or SOCKS5 proxy on a request object. Here's the minimal pattern:

library(httr2)
library(rvest)

# Build a request through ProxyHat's HTTP gateway
req <- request("https://httpbin.org/ip") |>
  req_proxy(
    proxy = "http://gate.proxyhat.com:8080",
    username = "user-country-US",
    password = "your_password"
  ) |>
  req_user_agent(
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
  ) |>
  req_timeout(30)

# Perform the request
resp <- req |>
  req_perform()

# Check the response
status <- resp_status(resp)
body <- resp_body_string(resp)
cat("Status:", status, "\n")
cat("Body:", body, "\n")

The req_proxy() call sets the proxy URL, username, and password. The username field does double duty: it carries authentication and routing flags like country, city, and session ID. This is the same pattern used by the ProxyHat SDK in Python and Node.js, so mixed-language pipelines share identical configuration logic.

Parsing HTML with rvest

Once you have the response body, pass it to rvest::read_html() and use CSS selectors or XPath to extract data:

library(httr2)
library(rvest)

req <- request("https://books.toscrape.com/") |>
  req_proxy(
    proxy = "http://gate.proxyhat.com:8080",
    username = "user-country-GB",
    password = "your_password"
  ) |>
  req_user_agent(
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
  )

resp <- req |> req_perform()
html <- resp_body_string(resp) |> read_html()

# Extract book titles and prices
titles <- html |>
  html_elements("h3 a") |>
  html_text2()

prices <- html |>
  html_elements(".price_color") |>
  html_text2()

tibble::tibble(title = titles, price = prices)

For tabular data, html_table() converts <table> elements directly into data frames:

# Parse all tables on a page
tables <- html |>
  html_table()

# Usually the first or second table is the one you want
df <- tables[[1]]
head(df)

Geo-Targeting and Sticky Sessions

ProxyHat encodes routing instructions in the proxy username. You can target a country, a city, or pin a specific IP for a sticky session. This is powerful for scraping geo-restricted content or maintaining login state across requests.

# Country-level targeting: United Kingdom
req_uk <- request("https://example.co.uk/data") |>
  req_proxy(
    proxy = "http://gate.proxyhat.com:8080",
    username = "user-country-GB",
    password = "your_password"
  )

# City-level targeting: London, UK
req_london <- request("https://example.co.uk/data") |>
  req_proxy(
    proxy = "http://gate.proxyhat.com:8080",
    username = "user-country-GB-city-london",
    password = "your_password"
  )

# Sticky session: same IP across multiple requests
req_sticky <- request("https://example.com/dashboard") |>
  req_proxy(
    proxy = "http://gate.proxyhat.com:8080",
    username = "user-session-abc123",
    password = "your_password"
  )

# Combine geo + session
req_geo_session <- request("https://example.com/dashboard") |>
  req_proxy(
    proxy = "http://gate.proxyhat.com:8080",
    username = "user-country-DE-city-berlin-session-xyz789",
    password = "your_password"
  )

Sticky sessions are essential when a target site issues cookies tied to your IP. If your IP rotates mid-session, the server sees a new IP with an old cookie and may flag it as suspicious. Use a fixed session ID for the duration of a login flow, then switch to a new session ID for the next logical unit of work.

SOCKS5 on Port 1080

For targets that block HTTP proxy protocols or when you need TCP-level tunneling, switch to SOCKS5 on port 1080:

# SOCKS5 proxy on port 1080
req_socks <- request("https://example.com/api/data") |>
  req_proxy(
    proxy = "socks5://gate.proxyhat.com:1080",
    username = "user-country-FR",
    password = "your_password"
  ) |>
  req_perform()

SOCKS5 proxies tunnel at the TCP layer, which can bypass some HTTP-aware filtering. The trade-off is slightly higher overhead — expect 10–30ms additional latency compared to HTTP CONNECT proxies on the same gateway.

Worked Example: Paginated Table Collection

Real-world scraping usually involves pagination. This example pulls a multi-page table into a tidy data frame, rotating the proxy session per page to distribute requests across IPs. It uses purrr::map_dfr() to iterate, req_retry() for transient failures, and req_throttle() to stay within polite rate limits.

library(httr2)
library(rvest)
library(purrr)
library(dplyr)
library(tibble)

# Configuration
BASE_URL <- "https://books.toscrape.com/catalogue/page-%d.html"
TOTAL_PAGES <- 50
PROXY_HOST <- "http://gate.proxyhat.com:8080"
PROXY_USER <- "user"
PROXY_PASS <- "your_password"

# Function to scrape a single page with a unique session
scrape_page <- function(page_num) {
  url <- sprintf(BASE_URL, page_num)
  session_id <- paste0("page-", page_num, "-", as.integer(Sys.time()))

  req <- request(url) |>
    req_proxy(
      proxy = PROXY_HOST,
      username = paste0(PROXY_USER, "-country-GB-session-", session_id),
      password = PROXY_PASS
    ) |>
    req_user_agent(
      "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
    ) |>
    req_headers(
      `Accept` = "text/html,application/xhtml+xml",
      `Accept-Language` = "en-GB,en;q=0.9"
    ) |>
    req_timeout(30) |>
    req_retry(max_tries = 5, max_seconds = 60) |>
    req_throttle(rate = 2 / 1)  # 2 requests per second

  resp <- tryCatch(
    req |> req_perform(),
    error = function(e) {
      message("Failed on page ", page_num, ": ", conditionMessage(e))
      return(NULL)
    }
  )

  if (is.null(resp)) return(tibble())

  html <- resp_body_string(resp) |> read_html()

  tibble(
    title = html |> html_elements("h3 a") |> html_text2(),
    price = html |> html_elements(".price_color") |> html_text2(),
    rating = html |> html_elements(".star-rating") |>
      html_attr("class") |> stringr::str_replace("star-rating ", ""),
    page = page_num
  )
}

# Scrape all pages, combining into one data frame
all_books <- 1:TOTAL_PAGES |>
  map_dfr(scrape_page)

cat("Total rows:", nrow(all_books), "\n")
glimpse(all_books)

Key design decisions in this example:

• Session rotation per page: Each page gets a unique session ID, so ProxyHat assigns a different residential IP. This prevents the target from correlating 50 requests to a single IP.
• req_retry(max_tries = 5): Automatically retries on 429, 500, 502, and 503 responses with exponential backoff. This handles transient failures without manual loop logic.
• req_throttle(rate = 2 / 1): Limits to 2 requests per second, keeping you within polite scraping norms. Adjust based on the target's tolerance.
• tryCatch wrapper: If a page fails after all retries, the function returns an empty tibble instead of aborting the entire run. map_dfr handles NULL gracefully by skipping it.

JavaScript-Rendered Pages with chromote

Many modern sites render content via JavaScript, so the initial HTML response is an empty shell. rvest alone can't execute JS. The solution is chromote, which drives a real Chrome instance via the DevTools Protocol. You can launch Chrome with a --proxy-server flag to route all browser traffic through ProxyHat.

library(chromote)
library(rvest)
library(dplyr)

# Launch Chrome with ProxyHat as the proxy server
chrome <- Chrome$new(
  extra_args = c(
    "--proxy-server=http://gate.proxyhat.com:8080",
    "--disable-blink-features=AutomationControlled",
    "--no-sandbox"
  )
)
cm <- Chromote$new(browser = chrome)
session <- cm$new_session()

# Navigate to a JS-rendered page
session$Page$navigate("https://quotes.toscrape.com/js/")
session$Page$loadEventFired(wait_ = TRUE)
Sys.sleep(3)  # Allow JS to render

# Extract the rendered HTML
result <- session$Runtime$evaluate(
  expression = "document.documentElement.outerHTML"
)
html <- read_html(result$result$value)

# Parse with rvest as usual
quotes <- html |>
  html_elements(".quote .text") |>
  html_text2()

authors <- html |>
  html_elements(".quote .author") |>
  html_text2()

df <- tibble(quote = quotes, author = authors)
print(df)

# Clean up
session$close()
cm$close()

For proxy authentication with chromote, Chrome's --proxy-server flag doesn't handle credentials directly. You need to handle the proxy auth challenge via the DevTools Protocol's Fetch.authRequired event, or use an authenticated proxy URL format if your provider supports it. With ProxyHat, credentials go in the username field — for Chrome, you'll typically intercept the auth challenge:

# Enable Fetch domain to intercept auth challenges
session$Fetch$enable(handleAuthRequests = TRUE)

# Listen for auth-required events
session$Fetch$authRequired <- function(event) {
  session$Fetch$continueWithAuth(
    requestId = event$requestId,
    authChallengeResponse = list(
      response = "ProvideCredentials",
      username = "user-country-US",
      password = "your_password"
    )
  )
}

# Also continue regular requests
session$Fetch$requestPaused <- function(event) {
  session$Fetch$continueRequest(requestId = event$requestId)
}

This pattern gives you full control over the browser's proxy authentication. The ProxyHat SDK mirrors this approach for Python and Node.js pipelines, so teams running mixed-language stacks can share the same proxy configuration and session management logic.

Realistic Headers and User Agents

Proxy IPs alone won't save you if your headers scream "bot." Always set a realistic user agent and accept headers that match a real browser:

req <- request("https://example.com/data") |>
  req_proxy(
    proxy = "http://gate.proxyhat.com:8080",
    username = "user-country-US",
    password = "your_password"
  ) |>
  req_user_agent(
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
  ) |>
  req_headers(
    `Accept` = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    `Accept-Language` = "en-US,en;q=0.9",
    `Accept-Encoding` = "gzip, deflate, br",
    `Connection` = "keep-alive",
    `Upgrade-Insecure-Requests` = "1"
  )

Avoid using the default httr2 user agent (which identifies the package) for any scraping work. Rotate user agents across requests if you're doing high-volume collection — a pool of 10–15 real browser UAs is usually sufficient.

Ethics, Compliance, and Best Practices

Proxy access doesn't exempt you from legal and ethical obligations. Before scraping any site:

• Check robots.txt: The RFC 9309 specification defines the robots.txt protocol. Respect crawl-delay directives and disallowed paths.
• Review the Terms of Service: Many sites prohibit automated access in their ToS. Violating ToS can lead to account bans or legal action.
• Prefer official APIs: If a site offers an API — even a paid one — it's almost always more reliable, faster, and legally safer than scraping.
• GDPR for EU data subjects: If you're collecting personal data from EU residents, the GDPR applies regardless of where you're based. Scraping names, emails, or other PII without a lawful basis is a violation.
• Rate limiting: Use req_throttle() to stay within reasonable limits. 1–3 requests per second per target is a common baseline.

Collecting publicly available, non-personal data for research or competitive analysis is generally acceptable, but the specifics depend on jurisdiction and the data type. When in doubt, consult legal counsel.

ProxyHat Setup and Integration

Getting started with ProxyHat in R takes minutes. Sign up at dashboard.proxyhat.com, grab your credentials from the dashboard, and plug them into the req_proxy() calls shown above.

Key integration points:

• Gateway: gate.proxyhat.com on port 8080 (HTTP) or 1080 (SOCKS5)
• Username flags: -country-XX, -city-yyy, -session-ZZZ
• Geo-targeting: Check available locations before targeting specific countries or cities
• Pricing: See ProxyHat pricing for residential, mobile, and datacenter plans
• Use cases: Explore web scraping and SERP tracking guides for R-specific patterns
• Full API docs: docs.proxyhat.com for SDK references and advanced configuration

Key Takeaways

Using proxies in R is straightforward with httr2's req_proxy() and rvest's parsing functions. The critical patterns to remember:

• Use residential proxies for anti-bot-protected or geo-restricted targets; datacenter IPs get flagged quickly.
• Encode geo-targeting and session IDs in the proxy username — user-country-GB-city-london-session-abc123.
• Rotate session IDs per request for distributed scraping, or pin them for login flows.
• Always add req_retry() and req_throttle() for production resilience.
• Use chromote with --proxy-server for JavaScript-rendered pages.
• Set realistic headers — a residential IP with a bot user agent still gets blocked.
• Respect robots.txt, ToS, and GDPR; prefer official APIs when available.

For teams running mixed-language pipelines, the ProxyHat SDK mirrors the same username-flag pattern in Python and Node.js, so you can share session IDs and geo-targeting logic across your entire data collection stack. Start with a small test — 10–20 requests through a residential session — and scale up once you've confirmed your success rate is above 95%.