Using Proxies in R: A Code-First Guide with httr2 and rvest

Why Using Proxies in R Matters for Data Collection

If you've ever run rvest::read_html() against a modern website and received an empty page, a 403, or a CAPTCHA wall, you've hit the same wall every R developer encounters: your request came from a datacenter IP that anti-bot systems flagged instantly. Using proxies in R is the practical answer — routing your requests through residential IPs so they look like real user traffic rather than a bot running on a cloud VM.

The good news is that R's modern HTTP stack — httr2 paired with rvest — makes proxy configuration clean and composable. Instead of fighting with environment variables or legacy httr workarounds, you chain a single req_proxy() call into your request pipeline. This guide walks through the full pattern: from a basic proxied request, through geo-targeting and sticky sessions, to a production-ready paginated scraper and JavaScript-rendered pages.

Residential vs Datacenter Proxies: The Technical Context

Anti-bot vendors like Cloudflare, Akamai, and PerimeterX classify incoming IPs using ASN databases. A request from AWS (us-east-1) or DigitalOcean gets a high risk score because those ASN ranges are overwhelmingly automated. A request from a UK household ISP ASN scores low — it looks like a real person. According to RFC 7230, the HTTP/1.1 message format carries no inherent identity, so these systems rely entirely on IP reputation, TLS fingerprinting, and behavioral signals.

Residential proxies solve this by issuing requests from real ISP-assigned IPs. Datacenter proxies are faster and cheaper — often 50–70% lower latency — but they fail against any site with serious bot detection. For SERP tracking, e-commerce price monitoring, or geo-restricted content, residential is the default choice. You can learn more about the distinction on our web scraping use case page.

The Modern R Stack: httr2 + rvest

httr2 is Hadley Wickham's rewrite of the classic httr package. It uses a pipeable request-builder pattern: you start with request("https://example.com"), chain modifiers like req_proxy(), req_headers(), req_retry(), and finally call req_perform(). For HTML parsing, rvest wraps xml2 and provides read_html(), html_elements(), html_table(), and html_text(). Together they replace the old httr::GET() + content() workflow.

Basic Proxied Request with req_proxy

Here's the minimal pattern — a single request routed through ProxyHat's HTTP gateway on port 8080:

library(httr2)
library(rvest)

# ProxyHat HTTP gateway credentials
proxy_user <- "your-username"
proxy_pass <- "your-password"

# Build and perform a proxied request
resp <- request("https://httpbin.org/ip") |>
  req_proxy(
    url = "http://gate.proxyhat.com",
    port = 8080,
    username = proxy_user,
    password = proxy_pass
  ) |>
  req_user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36") |>
  req_perform()

# Parse the JSON response
body <- resp |> resp_body_string()
cat(body)
# { "origin": "85.xxx.xxx.xxx" } — a residential IP, not your real one

The req_proxy() function sets the http_proxy environment for that specific request only, so you don't leak proxy settings into other parts of your script. This is a major improvement over httr, where set_config(use_proxy(...)) was global.

Parsing HTML with rvest

Once you have the response body, pass it to read_html() and use CSS selectors:

# Fetch a page through the proxy
page_resp <- request("https://books.toscrape.com/") |>
  req_proxy("http://gate.proxyhat.com", 8080, proxy_user, proxy_pass) |>
  req_user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64)") |>
  req_retry(max_tries = 3) |>
  req_perform()

# Parse with rvest
html <- page_resp |> resp_body_string() |> read_html()

titles <- html |>
  html_elements("article.product_pod h3 a") |>
  html_text2()

prices <- html |>
  html_elements(".price_color") |>
  html_text2()

df <- data.frame(title = titles, price = prices)
head(df)

If the target has a table, html_table() converts it directly to a data frame — no manual parsing required:

tables <- html |> html_table(fill = TRUE)
first_table <- tables[[1]]
str(first_table)

Geo-Targeting and Sticky Sessions in the Username

ProxyHat encodes routing instructions in the proxy username string. This means you can target a specific country, city, or maintain a sticky session — all without changing your R code structure. You just modify the username argument.

Country and City Targeting

# Target a UK residential IP in London
geo_user <- "your-username-country-GB-city-london"

resp_uk <- request("https://httpbin.org/ip") |>
  req_proxy(
    url = "http://gate.proxyhat.com",
    port = 8080,
    username = geo_user,
    password = proxy_pass
  ) |>
  req_perform()

cat(resp_uk |> resp_body_string())
# The origin IP will be a UK ISP address

This is essential for geo-restricted content. If you need to verify localized search results, ad placements, or pricing that varies by region, see our SERP tracking use case for the full workflow.

Sticky Sessions for Login Flows

Some sites require a consistent IP across multiple requests — for example, logging in, then fetching a dashboard. A rotating IP would invalidate the session. Use the session- flag:

# Sticky session: same IP for all requests sharing this session ID
sticky_user <- "your-username-session-mySession001"

# Request 1: login
login_resp <- request("https://quotes.toscrape.com/login") |>
  req_proxy("http://gate.proxyhat.com", 8080, sticky_user, proxy_pass) |>
  req_user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64)") |>
  req_perform()

# Request 2: authenticated page — same IP, session cookie still valid
page_resp <- request("https://quotes.toscrape.com/") |>
  req_proxy("http://gate.proxyhat.com", 8080, sticky_user, proxy_pass) |>
  req_perform()

SOCKS5 on Port 1080

For targets that block HTTP CONNECT proxying or when you need end-to-end TCP tunneling, use SOCKS5 on port 1080:

# SOCKS5 proxy — note the socks5:// scheme in the URL
socks_user <- "your-username-country-DE"

resp_socks <- request("https://httpbin.org/ip") |>
  req_proxy(
    url = "socks5://gate.proxyhat.com",
    port = 1080,
    username = socks_user,
    password = proxy_pass
  ) |>
  req_perform()

cat(resp_socks |> resp_body_string())

SOCKS5 adds a small overhead but is more reliable for certain CDN configurations. See our locations page for available country codes.

Worked Example: Paginated Table into a Tidy Data Frame

Now let's build a real pipeline. We'll scrape a paginated table — rotating the sticky session per page so each page request comes from a different residential IP, while still using req_retry() and req_throttle() for resilience.

library(httr2)
library(rvest)
library(purrr)
library(dplyr)
library(tibble)

proxy_user_base <- "your-username"
proxy_pass <- "your-password"

# Function to fetch one page with a unique session per page
fetch_page <- function(page_num) {
  # Rotate session ID per page for IP diversity
  session_id <- paste0("page-", page_num, "-", as.integer(runif(1, 1e6, 9e6)))
  username <- paste0(proxy_user_base, "-session-", session_id)

  url <- paste0("https://books.toscrape.com/catalogue/page-", page_num, ".html")

  resp <- request(url) |>
    req_proxy("http://gate.proxyhat.com", 8080, username, proxy_pass) |>
    req_user_agent(
      "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
    ) |>
    req_headers("Accept-Language" = "en-US,en;q=0.9") |>
    req_retry(
      max_tries = 4,
      backoff = ~ rnorm(1, 2, 0.5),  # jittered backoff
      is_transient = ~ resp_status(.x) %in% c(429, 502, 503, 504)
    ) |>
    req_throttle(delay = 1.5) |>  # max ~0.67 req/sec
    req_perform()

  if (resp_status(resp) != 200) {
    warning(paste("Page", page_num, "returned status", resp_status(resp)))
    return(NULL)
  }

  html <- resp |> resp_body_string() |> read_html()

  # Extract book data
  titles <- html |> html_elements("article.product_pod h3 a") |> html_text2()
  prices <- html |> html_elements(".price_color") |> html_text2()
  ratings <- html |> html_elements(".star-rating") |>
    html_attr("class") |>
    stringr::str_replace("star-rating ", "")
  availability <- html |> html_elements(".availability") |>
    html_text2() |> stringr::str_squish()

  tibble(
    page = page_num,
    title = titles,
    price = prices,
    rating = ratings,
    availability = availability
  )
}

# Scrape pages 1–10 with safe error handling
all_books <- 1:10 |>
  purrr::map_dfr(
    .f = possibly(fetch_page, otherwise = NULL),
    .progress = TRUE
  )

print(all_books)
# # A tibble: 200 × 5

Key patterns in this example:

• Per-page session rotation: each page gets a unique session-ID, so ProxyHat assigns a different residential IP. This spreads your footprint and avoids per-IP rate limits.
• Jittered exponential backoff: backoff = ~ rnorm(1, 2, 0.5) adds randomness so concurrent scrapers don't retry in lockstep.
• Transient detection: is_transient catches 429 (rate-limited) and 502/503/504 (server errors) — the statuses where retrying makes sense.
• Throttling: req_throttle(delay = 1.5) caps throughput at roughly 0.67 requests per second, keeping you under typical rate limits.
• Safe mapping: purrr::possibly() ensures one failed page doesn't abort the entire run.

JavaScript-Rendered Pages with read_html_live()

Many modern sites render content via JavaScript — read_html() on the raw response gives you an empty <div id="root"></div>. R's answer is rvest::read_html_live(), which uses the chromote package to drive a headless Chromium instance. As of rvest 1.0.4+, you can pass proxy configuration to the browser session.

library(rvest)
library(chromote)

# Configure a headless Chromium with the ProxyHat SOCKS5 proxy
# chromote respects --proxy-server Chrome flag
b <- chromote_session(
  args = c(
    "--headless=new",
    "--disable-gpu",
    "--proxy-server=socks5://gate.proxyhat.com:1080"
  )
)

# Navigate to a JS-rendered page
b |> chromote::ChrNavigate("https://quotes.toscrape.com/js/")
# Wait for dynamic content to load
Sys.sleep(3)

# Get the rendered HTML
html_content <- b |> chromote::ChrGetDocument()
html <- read_html(html_content$result$value)

# Now parse as usual
quotes <- html |>
  html_elements(".quote .text") |>
  html_text2()

authors <- html |>
  html_elements(".quote .author") |>
  html_text2()

df_js <- tibble(quote = quotes, author = authors)
print(df_js)

For read_html_live() specifically, you can pass proxy arguments directly:

# rvest 1.0.4+ read_html_live with proxy
live_html <- read_html_live(
  "https://quotes.toscrape.com/js/",
  options = list(
    args = c("--proxy-server=socks5://gate.proxyhat.com:1080")
  )
)

quotes <- live_html |> html_elements(".quote .text") |> html_text2()

Tip: Always set a realistic User-Agent and Accept-Language header. A residential IP with a default libcurl/8.x UA is still suspicious. Match the browser fingerprint to the IP's locale — a UK IP should send Accept-Language: en-GB,en;q=0.9.

Production Tips: Logging, Retries, and Circuit Breakers

For long-running collection jobs, add structured logging and a simple circuit breaker pattern:

library(httr2)
library(logger)

log_config(logger::log_layout(layout_glue(
  format = "[{level}] {timestamp} {msg}"
)))

fetch_with_circuit <- function(url, max_consecutive_failures = 5) {
  consecutive_failures <- 0

  attempt <- function() {
    tryCatch({
      resp <- request(url) |>
        req_proxy("http://gate.proxyhat.com", 8080, proxy_user, proxy_pass) |>
        req_user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64)") |>
        req_retry(max_tries = 3) |>
        req_perform()

      if (resp_status(resp) == 200) {
        consecutive_failures <- 0
        log_info("Success: {url}")
        return(resp)
      } else {
        consecutive_failures <- consecutive_failures + 1
        log_warn("Status {resp_status(resp)} on {url}")
      }
    }, error = function(e) {
      consecutive_failures <- consecutive_failures + 1
      log_error("Request failed: {conditionMessage(e)}")
    })

    if (consecutive_failures >= max_consecutive_failures) {
      log_error("Circuit breaker tripped after {max_consecutive_failures} failures")
      stop("Circuit breaker open — aborting")
    }
    NULL
  }

  attempt()
}

ProxyHat SDK for Mixed-Language Pipelines

If your data pipeline mixes R with Python or Node.js — common in analytics teams where R handles statistics and Python handles collection — the ProxyHat SDK mirrors the same gate.proxyhat.com:8080 pattern across languages. The credentials, geo-targeting flags, and session IDs are identical. You can find full SDK documentation at docs.proxyhat.com. For pricing details, see our pricing page.

Ethics, Compliance, and Responsible Scraping

Using proxies doesn't exempt you from legal and ethical obligations. Key considerations:

• robots.txt: Always check the target's robots.txt and respect its directives. The RFC 9309 specification defines the robots.txt protocol — it's a voluntary standard but widely expected.
• Terms of Service: Read the site's ToS. Some explicitly prohibit automated access; violating it can breach contract law.
• GDPR: If you collect data about EU data subjects (names, emails, user-generated content), the GDPR applies regardless of where your server is. Personal data must have a lawful basis under Article 6. See the EU GDPR reference site for guidance.
• Prefer official APIs: Many platforms (Twitter/X, Reddit, GitHub) offer APIs that are faster, more reliable, and contractually safe. Use scraping as a fallback, not the first resort.
• Rate limiting: Even with residential proxies, don't hammer a site. req_throttle() with 1–2 second delays is a reasonable default for most targets.

Key Takeaways

• httr2::req_proxy() is the clean, per-request way to route R HTTP traffic through a proxy — no global state, no environment variable hacks.
• Encode geo-targeting and sticky sessions in the username string: user-country-GB-city-london or user-session-abc123.
• Rotate session IDs per request for IP diversity; use sticky sessions only when the target requires IP continuity (logins, multi-step flows).
• Always pair proxies with req_retry() (jittered backoff), req_throttle(), and a realistic req_user_agent().
• For JS-rendered pages, use read_html_live() with --proxy-server=socks5://gate.proxyhat.com:1080 to drive headless Chromium through the same proxy.
• Residential proxies achieve 90%+ success rates against protected sites versus ~20% for datacenter IPs — the latency tradeoff (150–500 ms vs 20–80 ms) is worth it for serious targets.
• Check robots.txt, ToS, and GDPR before scraping. Prefer official APIs where available.

FAQ

What is using proxies in R?

Using proxies in R means routing your HTTP requests — typically via httr2 or curl — through an intermediary server so the target website sees the proxy's IP instead of your real IP. In practice, you call req_proxy("http://gate.proxyhat.com", 8080, username, password) in your httr2 request chain. This is essential for web scraping, SERP tracking, and geo-restricted content access where datacenter IPs get blocked.

Why does using proxies in R matter for proxy users?

Without a proxy, every request from your R session carries your real IP — often a cloud or office datacenter IP that anti-bot systems flag instantly. Residential proxies issue requests from real ISP-assigned household IPs, achieving 90%+ success rates against protected sites versus roughly 20% for datacenter IPs. This matters because a blocked request wastes compute time and can contaminate your dataset with partial or empty responses.

Which proxy type works best for using proxies in R?

Residential proxies are the best default for R-based web scraping because they bypass anti-bot detection effectively. Datacenter proxies are faster (20–80 ms vs 150–500 ms) but fail against any site with serious bot protection. Use datacenter only for fast, low-security targets. SOCKS5 (port 1080) is preferable over HTTP CONNECT (port 8080) when the target blocks HTTP proxying or you need full TCP tunneling.

How do you avoid blocks when implementing using proxies in R?

Combine four strategies: (1) rotate sticky session IDs per request so each call uses a different residential IP; (2) set a realistic User-Agent and Accept-Language header matching the IP's locale; (3) throttle with req_throttle(delay = 1.5) to stay under rate limits; (4) use req_retry() with jittered exponential backoff for transient failures (429, 502, 503, 504). Also respect robots.txt and prefer official APIs when available.

Can you use proxies with rvest's read_html_live for JavaScript pages?

Yes. rvest::read_html_live() drives a headless Chromium via the chromote package. You pass --proxy-server=socks5://gate.proxyhat.com:1080 as a Chrome argument, and all browser traffic routes through the proxy. This lets you scrape JavaScript-rendered pages with the same residential IP rotation strategy you use for static HTML.