[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"blog-post-en-got-scraping-nodejs-residential-proxies":3},{"success":4,"payload":5,"related":58,"errors":149},true,{"id":6,"slug":7,"title":8,"excerpt":9,"body_html":10,"meta_title":11,"meta_description":12,"canonical_url":13,"canonical_override":14,"og_title":11,"og_description":12,"og_image_url":15,"faq_json":16,"key_takeaways":14,"allow_index":4,"noindex_google_only":26,"allow_follow":4,"published_at":27,"updated_at":27,"content_updated_at":28,"featured_image_url":29,"featured_image_alt":8,"card_image_url":30,"reading_time_minutes":31,"author":32,"categories":36,"tags":43,"alternates":44,"locale_paths":52,"views":57},176,"got-scraping-nodejs-residential-proxies","got-scraping in Node.js: A Developer's Guide with Residential Proxies","A practical guide to using Apify's got-scraping HTTP client in Node.js with residential proxies — header generation, proxy rotation, retry hooks, and Crawlee scaling patterns for production scrapers.","\u003Cp>If you've ever fired off a few hundred requests with \u003Ccode>axios\u003C\u002Fcode> or raw \u003Ccode>got\u003C\u002Fcode> only to watch your scraper return 403s and CAPTCHA pages within minutes, you already know the problem: modern anti-bot systems don't just look at your IP. They fingerprint the entire TLS handshake, inspect header order, and flag incoherent browser signatures. got-scraping in Node.js — Apify's got-based HTTP client — was built specifically to solve this. It generates coherent browser header sets, supports HTTP\u002F2, and integrates cleanly with residential proxies. This guide walks through the idiomatic API surface, proxy rotation patterns, and production scaling strategies that keep your scraper alive.\u003C\u002Fp>\n\u003Cp>\u003Cem>Legal note: This guide covers scraping publicly accessible data. Web scraping may violate terms of service or laws like the CFAA (US) or GDPR (EU). Always review a site's ToS, honor \u003Ccode>robots.txt\u003C\u002Fcode>, respect rate limits, and prefer official APIs when available. Nothing here is legal advice.\u003C\u002Fem>\u003C\u002Fp>\n\u003Ch2>Why Raw got and axios Requests Get Flagged\u003C\u002Fh2>\n\u003Cp>Anti-bot vendors like Cloudflare, DataDome, and PerimeterX layer multiple signals. A \u003Ca href=\"\u002Fproxy\u002Fresidential\">residential IP\u003C\u002Fa> alone won't save you if your request looks like a bot at the HTTP layer. Here's what they inspect:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>TLS fingerprint (JA3\u002FJA4):\u003C\u002Fstrong> The cipher suites and extensions your client advertises in the TLS ClientHello are hashed into a fingerprint. Node.js's default TLS stack produces a fingerprint that no real browser uses, so even a perfect set of HTTP headers won't help if the TLS layer screams \"server-side runtime.\"\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Header order and coherence:\u003C\u002Fstrong> Browsers send headers in a specific order and include fields like \u003Ccode>sec-ch-ua\u003C\u002Fcode>, \u003Ccode>sec-fetch-dest\u003C\u002Fcode>, and \u003Ccode>accept-language\u003C\u002Fcode> that axios and raw got simply don't emit. Missing or misordered headers are a strong bot signal.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>HTTP\u002F2 support:\u003C\u002Fstrong> Modern browsers negotiate HTTP\u002F2 by default. If your scraper falls back to HTTP\u002F1.1 while claiming to be Chrome, that's an immediate inconsistency.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>got-scraping addresses all three layers. It bundles \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fapify\u002Fheader-generator\" rel=\"noopener noreferrer\">Apify's header-generator\u003C\u002Fa>, which produces realistic, version-aware header sets for specific browser\u002FOS\u002Fdevice combinations — including the correct order and the full set of \u003Ccode>sec-ch-ua\u003C\u002Fcode> client hints. It also supports HTTP\u002F2 out of the box, so the protocol your client uses matches what the browser you're impersonating would actually use.\u003C\u002Fp>\n\u003Ch2>The Idiomatic got-scraping Surface\u003C\u002Fh2>\n\u003Cp>got-scraping extends \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fsindresorhus\u002Fgot\" rel=\"noopener noreferrer\">sindresorhus\u002Fgot\u003C\u002Fa>, so the API is familiar if you've used got before. The key extension points are \u003Ccode>got.extend()\u003C\u002Fcode>, \u003Ccode>useHeaderGenerator\u003C\u002Fcode>, \u003Ccode>headerGeneratorOptions\u003C\u002Fcode>, and the \u003Ccode>proxyUrl\u003C\u002Fcode> option.\u003C\u002Fp>\n\u003Ch3>got.extend with useHeaderGenerator\u003C\u002Fh3>\n\u003Cp>The idiomatic way to create a configured instance is \u003Ccode>got.extend()\u003C\u002Fcode> with the \u003Ccode>useHeaderGenerator\u003C\u002Fcode> option. This tells got-scraping to generate a fresh, coherent header set for each request based on the options you provide:\u003C\u002Fp>\n\u003Cpre>\u003Ccode>import { gotScraping } from 'got-scraping';\n\nconst client = gotScraping.extend({\n  useHeaderGenerator: true,\n  headerGeneratorOptions: {\n    browsers: ['chrome'],\n    devices: ['desktop'],\n    operatingSystems: ['windows', 'macos'],\n    locales: ['en-US', 'en'],\n  },\n  http2: true,\n});\n\nconst response = await client('https:\u002F\u002Fexample.com');\nconsole.log(response.statusCode);\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>The \u003Ccode>headerGeneratorOptions\u003C\u002Fcode> object accepts arrays so you can randomize across combinations — Chrome on Windows, Chrome on macOS, Firefox on Linux, and so on. Each request gets a self-consistent set: if the generator picks Chrome 120 on Windows, it emits the matching \u003Ccode>sec-ch-ua\u003C\u002Fcode> string, \u003Ccode>user-agent\u003C\u002Fcode>, and \u003Ccode>accept\u003C\u002Fcode> headers in the order Chrome would send them. This is the \u003Cstrong>got scraping header generator\u003C\u002Fstrong> doing the heavy lifting that raw HTTP clients can't.\u003C\u002Fp>\n\u003Ch3>The proxyUrl Option\u003C\u002Fh3>\n\u003Cp>got-scraping accepts a \u003Ccode>proxyUrl\u003C\u002Fcode> option on both the extended instance and individual requests. It supports HTTP\u002FHTTPS proxies for HTTP\u002F1.1 connections and SOCKS proxies via the \u003Ccode>socks-proxy-agent\u003C\u002Fcode> integration. For HTTP\u002F2 upstream connections, the proxy tunnel uses HTTP CONNECT, which works with standard HTTP proxies.\u003C\u002Fp>\n\u003Cpre>\u003Ccode>const response = await client('https:\u002F\u002Fexample.com', {\n  proxyUrl: 'http:\u002F\u002Fuser-country-US:pass@gate.proxyhat.com:8080',\n});\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>You can also override the proxy per-request, which is essential for rotation:\u003C\u002Fp>\n\u003Cpre>\u003Ccode>const proxies = [\n  'http:\u002F\u002Fuser-country-US-sid-s1:pass@gate.proxyhat.com:8080',\n  'http:\u002F\u002Fuser-country-DE-sid-s2:pass@gate.proxyhat.com:8080',\n  'http:\u002F\u002Fuser-country-GB-sid-s3:pass@gate.proxyhat.com:8080',\n];\n\nfor (const url of urls) {\n  const proxy = proxies[Math.floor(Math.random() * proxies.length)];\n  const res = await client(url, { proxyUrl: proxy });\n  console.log(res.statusCode);\n}\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch2>Routing Through Residential Proxies\u003C\u002Fh2>\n\u003Cp>Once your headers and TLS fingerprint look like a real browser, the remaining signal anti-bot systems rely on is your IP address. Datacenter IPs are flagged in threat intelligence databases maintained by anti-bot vendors — a request from an AWS or DigitalOcean IP range claiming to be a residential Chrome browser is an immediate red flag.\u003C\u002Fp>\n\u003Cp>Residential proxies route your traffic through ISP-assigned IP addresses that look like real home users. This is why pairing got-scraping's header generation with a \u003Cstrong>got-scraping proxy\u003C\u002Fstrong> using residential IPs is the combination that actually works: the HTTP layer says \"Chrome on Windows\" and the IP layer says \"Comcast subscriber in Chicago.\"\u003C\u002Fp>\n\u003Cp>ProxyHat provides residential, mobile, and datacenter proxies through a single gateway at \u003Ccode>gate.proxyhat.com\u003C\u002Fcode>. The default HTTP port is \u003Ccode>8080\u003C\u002Fcode>, and SOCKS5 is available on port \u003Ccode>1080\u003C\u002Fcode>. Geo-targeting and session control are encoded in the username:\u003C\u002Fp>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\u003Cth>Feature\u003C\u002Fth>\u003Cth>HTTP Proxy (8080)\u003C\u002Fth>\u003Cth>SOCKS5 Proxy (1080)\u003C\u002Fth>\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\u003Ctd>Country targeting\u003C\u002Ftd>\u003Ctd>\u003Ccode>user-country-US:pass\u003C\u002Fcode>\u003C\u002Ftd>\u003Ctd>\u003Ccode>user-country-US:pass\u003C\u002Fcode>\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>City targeting\u003C\u002Ftd>\u003Ctd>\u003Ccode>user-country-DE-region-state_of_berlin-city-berlin:pass\u003C\u002Fcode>\u003C\u002Ftd>\u003Ctd>\u003Ccode>user-country-DE-region-state_of_berlin-city-berlin:pass\u003C\u002Fcode>\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>Sticky session\u003C\u002Ftd>\u003Ctd>\u003Ccode>user-sid-abc123:pass\u003C\u002Fcode>\u003C\u002Ftd>\u003Ctd>\u003Ccode>user-sid-abc123:pass\u003C\u002Fcode>\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>Combined\u003C\u002Ftd>\u003Ctd>\u003Ccode>user-country-US-sid-abc123:pass\u003C\u002Fcode>\u003C\u002Ftd>\u003Ctd>\u003Ccode>user-country-US-sid-abc123:pass\u003C\u002Fcode>\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>Protocol\u003C\u002Ftd>\u003Ctd>HTTP\u002FHTTPS (CONNECT)\u003C\u002Ftd>\u003Ctd>SOCKS5\u003C\u002Ftd>\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\u003Cp>SOCKS5 via port \u003Ccode>1080\u003C\u002Fcode> is useful when you need to tunnel non-HTTP traffic or when the target site's CDN interferes with HTTP proxy headers. For most nodejs scraping proxy use cases, the HTTP gateway on \u003Ccode>8080\u003C\u002Fcode> is sufficient and simpler to configure.\u003C\u002Fp>\n\u003Ch2>Runnable Example: Rotating Residential Endpoints with Retry Hooks\u003C\u002Fh2>\n\u003Cp>Here's a complete, runnable Node.js example that combines got-scraping's header generation with ProxyHat residential proxy rotation, retry logic, and per-request session IDs:\u003C\u002Fp>\n\u003Cpre>\u003Ccode>import { gotScraping } from 'got-scraping';\nimport crypto from 'crypto';\n\n\u002F\u002F ProxyHat credentials\nconst PROXYHAT_USER = 'your_username';\nconst PROXYHAT_PASS = 'your_password';\nconst GATEWAY = 'gate.proxyhat.com:8080';\n\n\u002F\u002F Generate a per-request proxy URL with country + sticky session\nfunction makeProxyUrl(country = 'US') {\n  const session = crypto.randomBytes(6).toString('hex');\n  const username = `${PROXYHAT_USER}-country-${country}-sid-${session}`;\n  return `http:\u002F\u002F${username}:${PROXYHAT_PASS}@${GATEWAY}`;\n}\n\n\u002F\u002F Create a got-scraping client with header generation\nconst client = gotScraping.extend({\n  useHeaderGenerator: true,\n  headerGeneratorOptions: {\n    browsers: ['chrome', 'firefox'],\n    devices: ['desktop'],\n    operatingSystems: ['windows', 'macos', 'linux'],\n    locales: ['en-US', 'en'],\n  },\n  http2: true,\n  timeout: { request: 30000 },\n  retry: {\n    limit: 3,\n    statusCodes: [403, 429, 500, 502, 503],\n  },\n  hooks: {\n    beforeRequest: [\n      (options) => {\n        \u002F\u002F Rotate proxy on every request\n        const countries = ['US', 'DE', 'GB', 'FR'];\n        const country = countries[Math.floor(Math.random() * countries.length)];\n        options.proxyUrl = makeProxyUrl(country);\n      },\n    ],\n    afterResponse: [\n      (response, retryWithMergedOptions) => {\n        \u002F\u002F If we get a 403, retry with a fresh proxy\n        if (response.statusCode === 403) {\n          return retryWithMergedOptions({\n            proxyUrl: makeProxyUrl('US'),\n          });\n        }\n        return response;\n      },\n    ],\n  },\n});\n\n\u002F\u002F Scrape a list of URLs\nconst urls = [\n  'https:\u002F\u002Fexample.com\u002Fpage-1',\n  'https:\u002F\u002Fexample.com\u002Fpage-2',\n  'https:\u002F\u002Fexample.com\u002Fpage-3',\n];\n\nfor (const url of urls) {\n  try {\n    const res = await client(url);\n    console.log(`[${res.statusCode}] ${url}`);\n  } catch (err) {\n    console.error(`[FAIL] ${url}: ${err.message}`);\n  }\n}\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>The \u003Ccode>beforeRequest\u003C\u002Fcode> hook rotates the proxy on every request, while \u003Ccode>afterResponse\u003C\u002Fcode> catches 403 responses and retries with a fresh IP. The \u003Ccode>retry\u003C\u002Fcode> configuration also handles 429 rate-limit responses automatically. This combination typically reduces block rates by over 80% compared to using a single datacenter IP, and the \u003Ccode>30000\u003C\u002Fcode> ms timeout prevents hung connections from blocking your queue.\u003C\u002Fp>\n\u003Ch2>Production Patterns for got-scraping at Scale\u003C\u002Fh2>\n\u003Ch3>Bounded Concurrency with p-limit\u003C\u002Fh3>\n\u003Cp>Hammering a target with 100 concurrent requests from a single IP — even a residential one — will get you blocked. Use \u003Ccode>p-limit\u003C\u002Fcode> to cap concurrency and add jitter between requests:\u003C\u002Fp>\n\u003Cpre>\u003Ccode>import pLimit from 'p-limit';\n\nconst limit = pLimit(10); \u002F\u002F max 10 concurrent requests\nconst results = await Promise.allSettled(\n  urls.map((url) => limit(() => client(url)))\n);\n\nfor (const r of results) {\n  if (r.status === 'fulfilled') {\n    console.log(r.value.statusCode);\n  } else {\n    console.error(r.reason.message);\n  }\n}\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>A concurrency of 5–10 per target domain is a reasonable starting point. Monitor your success rate and adjust downward if you see 429s or 403s spike.\u003C\u002Fp>\n\u003Ch3>Cookie Jars for Session Continuity\u003C\u002Fh3>\n\u003Cp>Many sites set cookies that affect subsequent responses (CSRF tokens, session IDs, A\u002FB test buckets). got-scraping supports got's built-in cookie jar via \u003Ccode>tough-cookie\u003C\u002Fcode>:\u003C\u002Fp>\n\u003Cpre>\u003Ccode>import { gotScraping } from 'got-scraping';\nimport { CookieJar } from 'tough-cookie';\n\nconst cookieJar = new CookieJar();\n\nconst client = gotScraping.extend({\n  useHeaderGenerator: true,\n  cookieJar,\n});\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>When using sticky sessions with ProxyHat (via \u003Ccode>-sid-abc123\u003C\u002Fcode>), the cookie jar ensures your session state persists across requests that share the same exit IP. This matters for sites that issue a session cookie on the first request and validate it on subsequent ones.\u003C\u002Fp>\n\u003Ch3>Graduating to Crawlee's CheerioCrawler\u003C\u002Fh3>\n\u003Cp>When your scraping project grows beyond a simple loop, \u003Ca href=\"https:\u002F\u002Fcrawlee.dev\" rel=\"noopener noreferrer\">Crawlee\u003C\u002Fa> — Apify's web scraping framework — builds on got-scraping and adds request queuing, automatic retry, proxy rotation, and structured data extraction. The \u003Ccode>CheerioCrawler\u003C\u002Fcode> uses got-scraping under the hood:\u003C\u002Fp>\n\u003Cpre>\u003Ccode>import { CheerioCrawler } from 'crawlee';\n\nconst crawler = new CheerioCrawler({\n  useHeaderGenerator: true,\n  headerGeneratorOptions: {\n    browsers: ['chrome'],\n    operatingSystems: ['windows', 'macos'],\n  },\n  proxyConfiguration: {\n    proxyUrls: [\n      'http:\u002F\u002Fuser-country-US-sid-s1:pass@gate.proxyhat.com:8080',\n      'http:\u002F\u002Fuser-country-DE-sid-s2:pass@gate.proxyhat.com:8080',\n    ],\n  },\n  maxConcurrency: 10,\n  requestHandler: async ({ $, request }) => {\n    const title = $('title').text();\n    console.log(`[${request.url}] ${title}`);\n  },\n});\n\nawait crawler.run(['https:\u002F\u002Fexample.com']);\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>Crawlee handles the queue, retries, and proxy rotation for you. It's the natural next step when you outgrow a raw got-scraping loop. You can explore ProxyHat's \u003Ca href=\"\u002Fpricing\">pricing\u003C\u002Fa> to find a plan that matches your concurrency needs, and check available \u003Ca href=\"\u002Flocations\">proxy locations\u003C\u002Fa> for geo-targeting.\u003C\u002Fp>\n\u003Ch2>When a Headless Browser Is Unavoidable\u003C\u002Fh2>\n\u003Cp>got-scraping handles the HTTP layer beautifully, but some sites require JavaScript execution to render content or to solve challenges. If the page you need is client-rendered (React, Vue, Next.js with client-side hydration), or if the anti-bot system serves a JS challenge that no HTTP client can solve, you need a headless browser.\u003C\u002Fp>\n\u003Cp>Signs you've hit this wall:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Responses contain a \u003Ccode><script>\u003C\u002Fcode> challenge but no content.\u003C\u002Fli>\n\u003Cli>The HTML references a JS bundle but has no server-rendered content.\u003C\u002Fli>\n\u003Cli>You get 200s but the body is under 2 KB of boilerplate.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>In that case, use Playwright or Puppeteer with ProxyHat's residential proxies. You can pass the proxy URL directly to the browser launch options. For most scraping tasks, though, got-scraping nodejs with proper headers and residential IPs is faster, cheaper, and more reliable than a headless fleet.\u003C\u002Fp>\n\u003Ch2>Ethics and Legal Considerations\u003C\u002Fh2>\n\u003Cp>Before you scrape anything, consider these principles:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Prefer official APIs. Many platforms offer APIs that are faster, more reliable, and legally safer than scraping. Check our \u003Ca href=\"\u002Fuse-cases\u002Fserp-tracking\">SERP tracking\u003C\u002Fa> and \u003Ca href=\"\u002Fuse-cases\u002Fweb-scraping\">web scraping\u003C\u002Fa> use case guides for compliant approaches.\u003C\u002Fli>\n\u003Cli>Honor robots.txt. The \u003Ca href=\"https:\u002F\u002Fwww.rfc-editor.org\u002Frfc\u002Frfc9309\" rel=\"noopener noreferrer\">RFC 9309\u003C\u002Fa> robots.txt specification defines the standard for crawler exclusion. Respect it.\u003C\u002Fli>\n\u003Cli>Rate-limit yourself. Even if a site doesn't enforce limits, aggressive scraping can degrade service for real users.\u003C\u002Fli>\n\u003Cli>Public data only. Don't scrape behind authentication unless you have explicit permission. The CFAA (US) and GDPR (EU) have significant penalties for unauthorized access.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch2>Key Takeaways\u003C\u002Fh2>\n\u003Cblockquote>\n\u003Cul>\n\u003Cli>got-scraping solves the HTTP fingerprint problem — coherent headers, correct ordering, HTTP\u002F2 — that raw got\u002Faxios can't.\u003C\u002Fli>\n\u003Cli>Residential proxies close the IP gap. Once headers and TLS look real, the IP is the remaining signal anti-bot systems check.\u003C\u002Fli>\n\u003Cli>Rotate per-request using ProxyHat's \u003Ccode>-country-XX-sid-XXX\u003C\u002Fcode> username format and got-scraping's \u003Ccode>beforeRequest\u003C\u002Fcode> hook.\u003C\u002Fli>\n\u003Cli>Scale with p-limit for bounded concurrency, then graduate to Crawlee's \u003Ccode>CheerioCrawler\u003C\u002Fcode> for queue management and structured extraction.\u003C\u002Fli>\n\u003Cli>Drop to a headless browser only when necessary — if the content is JS-rendered or anti-bot serves a challenge that no HTTP client can pass.\u003C\u002Fli>\n\u003Cli>Scrape ethically: honor robots.txt, respect rate limits, prefer APIs, and stick to public data.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003C\u002Fblockquote>\n\u003Cp>Ready to put this into practice? Review the \u003Ca href=\"https:\u002F\u002Fdocs.proxyhat.com\u002F\" rel=\"noopener noreferrer\">ProxyHat documentation\u003C\u002Fa> for full proxy configuration details, or head to the \u003Ca href=\"\u002Fpricing\">pricing page\u003C\u002Fa> to choose a residential proxy plan that fits your scale.\u003C\u002Fp>","got-scraping in Node.js: Residential Proxy Guide | ProxyHat","Complete guide to got-scraping in Node.js with residential proxies. Configure header generation, proxy rotation, retry hooks, and Crawlee scaling.","\u002Fblog\u002Fgot-scraping-nodejs-residential-proxies",null,"https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Fgot-scraping-nodejs-residential-proxies-og.webp",[17,20,23],{"question":18,"answer":19},"Why does got-scraping in Node.js matter for proxy users?","Anti-bot systems fingerprint requests at multiple layers: TLS handshake, header order, HTTP protocol version, and IP reputation. Even with a residential proxy, if your headers look like a server-side runtime (missing sec-ch-ua, wrong order, HTTP\u002F1.1 only), you'll get blocked. got-scraping solves the HTTP-layer fingerprinting so your proxy investment actually pays off — the combination of coherent browser headers with residential IPs is what makes scraping reliable at scale.",{"question":21,"answer":22},"Which proxy type works best for got-scraping in Node.js?","Residential proxies are the best match for got-scraping because they provide ISP-assigned IP addresses that anti-bot systems can't distinguish from real users. Datacenter IPs are often flagged in threat intelligence databases regardless of how good your headers look. With ProxyHat, you can route got-scraping through residential proxies at gate.proxyhat.com:8080 (HTTP) or port 1080 (SOCKS5), with geo-targeting and sticky sessions encoded in the username for per-request rotation.",{"question":24,"answer":25},"How do you avoid blocks when implementing got-scraping in Node.js?","Use got-scraping's useHeaderGenerator with headerGeneratorOptions to emit coherent browser headers on every request. Route through residential proxies and rotate them per-request using the beforeRequest hook. Configure retry hooks to handle 429 and 403 responses with fresh IPs. Cap concurrency to 5–10 per domain with p-limit. Use cookie jars for session continuity on sites that require it. If you still get blocked, consider dropping to a headless browser like Playwright for JS-rendered or challenge-protected pages.",false,"2026-08-04T16:05:06+00:00","2026-10-03T08:29:58+00:00","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Fgot-scraping-nodejs-residential-proxies-master.webp","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Fgot-scraping-nodejs-residential-proxies-card.webp",12,{"name":33,"slug":34,"bio":35,"avatar_url":14,"website_url":14,"social_links":14},"ProxyHat Team","proxyhat-team","The ProxyHat editorial team covers proxies, web scraping, and data extraction.",[37],{"id":38,"slug":39,"name":40,"description":14,"meta_title":41,"meta_description":42,"parent_id":14},4,"developer-guides","Developer Guides","Developer Guides | ProxyHat Blog","Code tutorials and SDK guides for integrating proxies into your applications.",[],{"ru":45,"en":13,"it":46,"zh":47,"fr":48,"pl":49,"ko":50,"ja":51,"x-default":13},"\u002Fru\u002Fblog\u002Fgot-scraping-nodejs-residential-proxies","\u002Fit\u002Fblog\u002Fguida-got-scraping-nodejs-proxy","\u002Fcn\u002Fblog\u002Fgot-scraping-nodejs-proxy-guide","\u002Ffr\u002Fblog\u002Fgot-scraping-nodejs-proxy-residentiel-1","\u002Fpl\u002Fblog\u002Fgot-scraping-nodejs-proxy","\u002Fko\u002Fblog\u002Fgot-scraping-nodejs-proxy-guide-1","\u002Fja\u002Fblog\u002Fgot-scraping-nodejs-purokishi",{"pt":53,"ru":45,"en":13,"it":46,"de":54,"es":55,"zh":47,"fr":48,"pl":49,"ko":50,"ja":51,"tr":56},"\u002Fpt\u002Fblog\u002Fguia-got-scraping-nodejs-proxies-residenciais","\u002Fde\u002Fblog\u002Fgot-scraping-nodejs-proxy-guide","\u002Fes\u002Fblog\u002Fguia-got-scraping-nodejs-proxy-residencial","\u002Ftr\u002Fblog\u002Fnodejs-got-scraping-proxy-rehberi",123,[59,92,118,133],{"id":60,"slug":61,"title":62,"excerpt":63,"featured_image_url":64,"featured_image_alt":62,"card_image_url":65,"og_image_url":66,"published_at":67,"updated_at":68,"content_updated_at":69,"reading_time_minutes":70,"is_featured":26,"indexable":4,"views":71,"author":72,"categories":73,"tags":75},210,"proxy-setup-for-ai-agents","Proxy Setup for AI Agents: browser-use, Crawl4AI and an MCP Server","Agents on cloud VMs exit from datacenter IPs. Here is current, working proxy configuration for browser-use, Crawl4AI and an MCP server, with a sticky-vs-rotating decision table and the cost traps to avoid.","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Fproxy-setup-for-ai-agents-master.webp","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Fproxy-setup-for-ai-agents-card.webp","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Fproxy-setup-for-ai-agents-og.webp","2026-10-24T09:00:00+00:00","2026-10-24T09:00:20+00:00","2026-10-03T09:31:26+00:00",14,18,{"name":33,"slug":34,"bio":35,"avatar_url":14,"website_url":14,"social_links":14},[74],{"id":38,"slug":39,"name":40,"description":14,"meta_title":41,"meta_description":42,"parent_id":14},[76,80,84,88],{"id":77,"slug":78,"name":79},1,"web-scraping","Web Scraping",{"id":81,"slug":82,"name":83},2,"residential-proxies","Residential Proxies",{"id":85,"slug":86,"name":87},9,"python","Python",{"id":89,"slug":90,"name":91},22,"playwright","Playwright",{"id":93,"slug":94,"title":95,"excerpt":96,"featured_image_url":97,"featured_image_alt":95,"card_image_url":98,"og_image_url":99,"published_at":100,"updated_at":101,"content_updated_at":69,"reading_time_minutes":70,"is_featured":26,"indexable":4,"views":102,"author":103,"categories":104,"tags":106},206,"camoufox-residential-proxies-python-guide","Camoufox with Residential Proxies: A Code-First Python Guide","How to run Camoufox, the C++-patched Firefox, behind residential proxies: current Python API, why geoip needs a sticky session, a fingerprint coherence checklist, and when Patchright or nodriver fits better.","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Fcamoufox-residential-proxies-python-guide-master.webp","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Fcamoufox-residential-proxies-python-guide-card.webp","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Fcamoufox-residential-proxies-python-guide-og.webp","2026-10-13T09:00:00+00:00","2026-10-13T09:00:22+00:00",33,{"name":33,"slug":34,"bio":35,"avatar_url":14,"website_url":14,"social_links":14},[105],{"id":38,"slug":39,"name":40,"description":14,"meta_title":41,"meta_description":42,"parent_id":14},[107,108,109,113,114],{"id":77,"slug":78,"name":79},{"id":81,"slug":82,"name":83},{"id":110,"slug":111,"name":112},6,"anti-detection","Anti-Detection",{"id":85,"slug":86,"name":87},{"id":115,"slug":116,"name":117},21,"headless-browser","Headless Browser",{"id":119,"slug":120,"title":121,"excerpt":122,"featured_image_url":123,"featured_image_alt":121,"card_image_url":124,"og_image_url":125,"published_at":126,"updated_at":126,"content_updated_at":127,"reading_time_minutes":31,"is_featured":26,"indexable":26,"views":128,"author":129,"categories":130,"tags":132},175,"rotating-proxies-in-colly-golang","Rotating Proxies in Colly: A Framework-Idiomatic Guide for Go Developers","A developer-first guide to implementing rotating proxies in Colly using residential IPs, custom proxy switchers, geo-targeting, and production patterns for high-throughput Go web scraping.","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Frotating-proxies-in-colly-golang-master.webp","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Frotating-proxies-in-colly-golang-card.webp","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Frotating-proxies-in-colly-golang-og.webp","2026-08-04T10:05:34+00:00","2026-10-03T08:29:56+00:00",73,{"name":33,"slug":34,"bio":35,"avatar_url":14,"website_url":14,"social_links":14},[131],{"id":38,"slug":39,"name":40,"description":14,"meta_title":41,"meta_description":42,"parent_id":14},[],{"id":134,"slug":135,"title":136,"excerpt":137,"featured_image_url":138,"featured_image_alt":139,"card_image_url":140,"og_image_url":141,"published_at":142,"updated_at":142,"content_updated_at":143,"reading_time_minutes":31,"is_featured":26,"indexable":4,"views":144,"author":145,"categories":146,"tags":148},195,"scrape-bybit-v5-market-api-rotating-proxies","Unified Crypto Market Data APIs with Rotating Proxies: Rate Limits, Geo-Blocks and Pagination (Bybit V5 Example)","A developer-first guide to scraping Bybit's unified V5 Market API endpoints with rotating residential proxies, rate-limit budgeting, retries, and sticky-session pagination.","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Fscrape-bybit-v5-market-api-rotating-proxies-master.webp","How to Scrape the Bybit V5 Market API with Rotating Proxies","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Fscrape-bybit-v5-market-api-rotating-proxies-card.webp","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Fscrape-bybit-v5-market-api-rotating-proxies-og.webp","2026-08-03T16:09:16+00:00","2026-10-04T01:29:51+00:00",88,{"name":33,"slug":34,"bio":35,"avatar_url":14,"website_url":14,"social_links":14},[147],{"id":38,"slug":39,"name":40,"description":14,"meta_title":41,"meta_description":42,"parent_id":14},[],[]]