got-scraping in Node.js: A Developer's Guide with Residential Proxies

If you've ever fired off a few hundred requests with axios or raw got only to watch your scraper return 403s and CAPTCHA pages within minutes, you already know the problem: modern anti-bot systems don't just look at your IP. They fingerprint the entire TLS handshake, inspect header order, and flag incoherent browser signatures. got-scraping in Node.js — Apify's got-based HTTP client — was built specifically to solve this. It generates coherent browser header sets, supports HTTP/2, and integrates cleanly with residential proxies. This guide walks through the idiomatic API surface, proxy rotation patterns, and production scaling strategies that keep your scraper alive.

Legal note: This guide covers scraping publicly accessible data. Web scraping may violate terms of service or laws like the CFAA (US) or GDPR (EU). Always review a site's ToS, honor robots.txt, respect rate limits, and prefer official APIs when available. Nothing here is legal advice.

Why Raw got and axios Requests Get Flagged

Anti-bot vendors like Cloudflare, DataDome, and PerimeterX layer multiple signals. A residential IP alone won't save you if your request looks like a bot at the HTTP layer. Here's what they inspect:

• TLS fingerprint (JA3/JA4): The cipher suites and extensions your client advertises in the TLS ClientHello are hashed into a fingerprint. Node.js's default TLS stack produces a fingerprint that no real browser uses, so even a perfect set of HTTP headers won't help if the TLS layer screams "server-side runtime."
• Header order and coherence: Browsers send headers in a specific order and include fields like sec-ch-ua, sec-fetch-dest, and accept-language that axios and raw got simply don't emit. Missing or misordered headers are a strong bot signal.
• HTTP/2 support: Modern browsers negotiate HTTP/2 by default. If your scraper falls back to HTTP/1.1 while claiming to be Chrome, that's an immediate inconsistency.

got-scraping addresses all three layers. It bundles Apify's header-generator, which produces realistic, version-aware header sets for specific browser/OS/device combinations — including the correct order and the full set of sec-ch-ua client hints. It also supports HTTP/2 out of the box, so the protocol your client uses matches what the browser you're impersonating would actually use.

The Idiomatic got-scraping Surface

got-scraping extends sindresorhus/got, so the API is familiar if you've used got before. The key extension points are got.extend(), useHeaderGenerator, headerGeneratorOptions, and the proxyUrl option.

got.extend with useHeaderGenerator

The idiomatic way to create a configured instance is got.extend() with the useHeaderGenerator option. This tells got-scraping to generate a fresh, coherent header set for each request based on the options you provide:

import { gotScraping } from 'got-scraping';

const client = gotScraping.extend({
  useHeaderGenerator: true,
  headerGeneratorOptions: {
    browsers: ['chrome'],
    devices: ['desktop'],
    operatingSystems: ['windows', 'macos'],
    locales: ['en-US', 'en'],
  },
  http2: true,
});

const response = await client('https://example.com');
console.log(response.statusCode);

The headerGeneratorOptions object accepts arrays so you can randomize across combinations — Chrome on Windows, Chrome on macOS, Firefox on Linux, and so on. Each request gets a self-consistent set: if the generator picks Chrome 120 on Windows, it emits the matching sec-ch-ua string, user-agent, and accept headers in the order Chrome would send them. This is the got scraping header generator doing the heavy lifting that raw HTTP clients can't.

The proxyUrl Option

got-scraping accepts a proxyUrl option on both the extended instance and individual requests. It supports HTTP/HTTPS proxies for HTTP/1.1 connections and SOCKS proxies via the socks-proxy-agent integration. For HTTP/2 upstream connections, the proxy tunnel uses HTTP CONNECT, which works with standard HTTP proxies.

const response = await client('https://example.com', {
  proxyUrl: 'http://user-country-US:pass@gate.proxyhat.com:8080',
});

You can also override the proxy per-request, which is essential for rotation:

const proxies = [
  'http://user-country-US-session-s1:pass@gate.proxyhat.com:8080',
  'http://user-country-DE-session-s2:pass@gate.proxyhat.com:8080',
  'http://user-country-GB-session-s3:pass@gate.proxyhat.com:8080',
];

for (const url of urls) {
  const proxy = proxies[Math.floor(Math.random() * proxies.length)];
  const res = await client(url, { proxyUrl: proxy });
  console.log(res.statusCode);
}

Routing Through Residential Proxies

Once your headers and TLS fingerprint look like a real browser, the remaining signal anti-bot systems rely on is your IP address. Datacenter IPs are flagged in threat intelligence databases maintained by anti-bot vendors — a request from an AWS or DigitalOcean IP range claiming to be a residential Chrome browser is an immediate red flag.

Residential proxies route your traffic through ISP-assigned IP addresses that look like real home users. This is why pairing got-scraping's header generation with a got-scraping proxy using residential IPs is the combination that actually works: the HTTP layer says "Chrome on Windows" and the IP layer says "Comcast subscriber in Chicago."

ProxyHat provides residential, mobile, and datacenter proxies through a single gateway at gate.proxyhat.com. The default HTTP port is 8080, and SOCKS5 is available on port 1080. Geo-targeting and session control are encoded in the username:

SOCKS5 via port 1080 is useful when you need to tunnel non-HTTP traffic or when the target site's CDN interferes with HTTP proxy headers. For most nodejs scraping proxy use cases, the HTTP gateway on 8080 is sufficient and simpler to configure.

Runnable Example: Rotating Residential Endpoints with Retry Hooks

Here's a complete, runnable Node.js example that combines got-scraping's header generation with ProxyHat residential proxy rotation, retry logic, and per-request session IDs:

import { gotScraping } from 'got-scraping';
import crypto from 'crypto';

// ProxyHat credentials
const PROXYHAT_USER = 'your_username';
const PROXYHAT_PASS = 'your_password';
const GATEWAY = 'gate.proxyhat.com:8080';

// Generate a per-request proxy URL with country + sticky session
function makeProxyUrl(country = 'US') {
  const session = crypto.randomBytes(6).toString('hex');
  const username = `${PROXYHAT_USER}-country-${country}-session-${session}`;
  return `http://${username}:${PROXYHAT_PASS}@${GATEWAY}`;
}

// Create a got-scraping client with header generation
const client = gotScraping.extend({
  useHeaderGenerator: true,
  headerGeneratorOptions: {
    browsers: ['chrome', 'firefox'],
    devices: ['desktop'],
    operatingSystems: ['windows', 'macos', 'linux'],
    locales: ['en-US', 'en'],
  },
  http2: true,
  timeout: { request: 30000 },
  retry: {
    limit: 3,
    statusCodes: [403, 429, 500, 502, 503],
  },
  hooks: {
    beforeRequest: [
      (options) => {
        // Rotate proxy on every request
        const countries = ['US', 'DE', 'GB', 'FR'];
        const country = countries[Math.floor(Math.random() * countries.length)];
        options.proxyUrl = makeProxyUrl(country);
      },
    ],
    afterResponse: [
      (response, retryWithMergedOptions) => {
        // If we get a 403, retry with a fresh proxy
        if (response.statusCode === 403) {
          return retryWithMergedOptions({
            proxyUrl: makeProxyUrl('US'),
          });
        }
        return response;
      },
    ],
  },
});

// Scrape a list of URLs
const urls = [
  'https://example.com/page-1',
  'https://example.com/page-2',
  'https://example.com/page-3',
];

for (const url of urls) {
  try {
    const res = await client(url);
    console.log(`[${res.statusCode}] ${url}`);
  } catch (err) {
    console.error(`[FAIL] ${url}: ${err.message}`);
  }
}

The beforeRequest hook rotates the proxy on every request, while afterResponse catches 403 responses and retries with a fresh IP. The retry configuration also handles 429 rate-limit responses automatically. This combination typically reduces block rates by over 80% compared to using a single datacenter IP, and the 30000 ms timeout prevents hung connections from blocking your queue.

Production Patterns for got-scraping at Scale

Bounded Concurrency with p-limit

Hammering a target with 100 concurrent requests from a single IP — even a residential one — will get you blocked. Use p-limit to cap concurrency and add jitter between requests:

import pLimit from 'p-limit';

const limit = pLimit(10); // max 10 concurrent requests
const results = await Promise.allSettled(
  urls.map((url) => limit(() => client(url)))
);

for (const r of results) {
  if (r.status === 'fulfilled') {
    console.log(r.value.statusCode);
  } else {
    console.error(r.reason.message);
  }
}

A concurrency of 5–10 per target domain is a reasonable starting point. Monitor your success rate and adjust downward if you see 429s or 403s spike.

Cookie Jars for Session Continuity

Many sites set cookies that affect subsequent responses (CSRF tokens, session IDs, A/B test buckets). got-scraping supports got's built-in cookie jar via tough-cookie:

import { gotScraping } from 'got-scraping';
import { CookieJar } from 'tough-cookie';

const cookieJar = new CookieJar();

const client = gotScraping.extend({
  useHeaderGenerator: true,
  cookieJar,
});

When using sticky sessions with ProxyHat (via -session-abc123), the cookie jar ensures your session state persists across requests that share the same exit IP. This matters for sites that issue a session cookie on the first request and validate it on subsequent ones.

Graduating to Crawlee's CheerioCrawler

When your scraping project grows beyond a simple loop, Crawlee — Apify's web scraping framework — builds on got-scraping and adds request queuing, automatic retry, proxy rotation, and structured data extraction. The CheerioCrawler uses got-scraping under the hood:

import { CheerioCrawler } from 'crawlee';

const crawler = new CheerioCrawler({
  useHeaderGenerator: true,
  headerGeneratorOptions: {
    browsers: ['chrome'],
    operatingSystems: ['windows', 'macos'],
  },
  proxyConfiguration: {
    proxyUrls: [
      'http://user-country-US-session-s1:pass@gate.proxyhat.com:8080',
      'http://user-country-DE-session-s2:pass@gate.proxyhat.com:8080',
    ],
  },
  maxConcurrency: 10,
  requestHandler: async ({ $, request }) => {
    const title = $('title').text();
    console.log(`[${request.url}] ${title}`);
  },
});

await crawler.run(['https://example.com']);

Crawlee handles the queue, retries, and proxy rotation for you. It's the natural next step when you outgrow a raw got-scraping loop. You can explore ProxyHat's pricing to find a plan that matches your concurrency needs, and check available proxy locations for geo-targeting.

When a Headless Browser Is Unavoidable

got-scraping handles the HTTP layer beautifully, but some sites require JavaScript execution to render content or to solve challenges. If the page you need is client-rendered (React, Vue, Next.js with client-side hydration), or if the anti-bot system serves a JS challenge that no HTTP client can solve, you need a headless browser.

Signs you've hit this wall:

• Responses contain a <script> challenge but no content.
• The HTML references a JS bundle but has no server-rendered content.
• You get 200s but the body is under 2 KB of boilerplate.

In that case, use Playwright or Puppeteer with ProxyHat's residential proxies. You can pass the proxy URL directly to the browser launch options. For most scraping tasks, though, got-scraping nodejs with proper headers and residential IPs is faster, cheaper, and more reliable than a headless fleet.

Ethics and Legal Considerations

Before you scrape anything, consider these principles:

• Prefer official APIs. Many platforms offer APIs that are faster, more reliable, and legally safer than scraping. Check our SERP tracking and web scraping use case guides for compliant approaches.
• Honor robots.txt. The RFC 9309 robots.txt specification defines the standard for crawler exclusion. Respect it.
• Rate-limit yourself. Even if a site doesn't enforce limits, aggressive scraping can degrade service for real users.
• Public data only. Don't scrape behind authentication unless you have explicit permission. The CFAA (US) and GDPR (EU) have significant penalties for unauthorized access.

Key Takeaways

• got-scraping solves the HTTP fingerprint problem — coherent headers, correct ordering, HTTP/2 — that raw got/axios can't.
• Residential proxies close the IP gap. Once headers and TLS look real, the IP is the remaining signal anti-bot systems check.
• Rotate per-request using ProxyHat's -country-XX-session-XXX username format and got-scraping's beforeRequest hook.
• Scale with p-limit for bounded concurrency, then graduate to Crawlee's CheerioCrawler for queue management and structured extraction.
• Drop to a headless browser only when necessary — if the content is JS-rendered or anti-bot serves a challenge that no HTTP client can pass.
• Scrape ethically: honor robots.txt, respect rate limits, prefer APIs, and stick to public data.

Ready to put this into practice? Review the ProxyHat documentation for full proxy configuration details, or head to the pricing page to choose a residential proxy plan that fits your scale.