got-scraping in Node.js: A Developer's Guide to Headers, HTTP/2 & Residential Proxies

If you've ever fired off a few thousand got or axios requests and watched your success rate crater from 95% to 12% overnight, you already know why got-scraping in Node.js exists. Raw HTTP clients send headers that no real browser would produce — missing sec-ch-ua, wrong accept ordering, no sec-fetch-* — and modern anti-bot stacks flag those mismatches in milliseconds. got-scraping, maintained by Apify, wraps got with a coherent header generator, HTTP/2 support, and a clean proxy API so your requests look like genuine browser traffic instead of script traffic.

Legal note: This guide covers scraping publicly accessible data only. Always honor robots.txt, site terms of service, rate limits, and applicable laws including the Computer Fraud and Abuse Act and GDPR. If a platform offers an official API, prefer it over scraping.

Why Raw got/axios Requests Get Flagged

Anti-bot systems like Cloudflare, Datadome, and Akamai Bot Manager don't just inspect a single header — they build a fingerprint from the full request envelope. When you call got('https://example.com'), the underlying stack sends a minimal set of headers in an order that no Chrome or Firefox instance would ever produce. The accept header might be */* instead of the browser-standard text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8. The accept-language might be absent entirely, or the sec-ch-ua client hints might be missing while user-agent claims to be Chrome 120.

That inconsistency is the real tell. A real Chrome browser sends roughly 15–20 headers in a specific order, including sec-ch-ua, sec-ch-ua-mobile, sec-ch-ua-platform, sec-fetch-site, sec-fetch-mode, sec-fetch-dest, and a user-agent that matches the sec-ch-ua version string. If your user-agent says Chrome 120 but there's no sec-ch-ua: "Not_A Brand";v="8.0", "Chromium";v="120.0", "Google Chrome";v="120.0", the WAF knows you're lying.

TLS fingerprinting (JA3/JA4) adds another layer. Node.js's TLS ClientHello differs from Chrome's in cipher suite ordering, extensions, and supported groups. got-scraping doesn't solve TLS fingerprinting directly — for that you'd need a TLS impersonation library or a headless browser — but it does solve the HTTP-layer fingerprint by generating header sets that are internally consistent and match real browser captures.

How got-scraping's Header Generator Works

The header-generator library bundles a dataset of real browser header profiles. When you request headers for chrome on desktop with windows, it returns a complete, coherent set — correct ordering, matching sec-ch-ua version, appropriate accept-language, and the full sec-fetch-* family. This is what makes got scraping header generator integration more effective than manually setting a user-agent string: you get the entire envelope, not just one field.

The Idiomatic got-scraping API Surface

got-scraping re-exports got with sensible scraping defaults applied via got.extend(). The key extension points are:

• useHeaderGenerator — a boolean that toggles header generation on every request.
• headerGeneratorOptions — an object specifying browsers, devices, operatingSystems, and locales to constrain the generated profiles.
• proxyUrl — a standard got option that accepts http://, https://, or socks5:// URLs. For HTTP/2 proxies, got-scraping handles the CONNECT tunnel correctly.
• http2 — enables HTTP/2, which many sites require for browser-like behavior (Chrome uses HTTP/2 by default for HTTPS).

Here's the minimal idiomatic setup:

import { gotScraping } from 'got-scraping';

const client = gotScraping.extend({
  useHeaderGenerator: true,
  headerGeneratorOptions: {
    browsers: ['chrome', 'firefox'],
    devices: ['desktop'],
    operatingSystems: ['windows', 'macos'],
    locales: ['en-US', 'en'],
  },
  http2: true,
  timeout: { request: 30000 },
});

const response = await client('https://example.com');
console.log(response.statusCode); // 200

Every call through client now gets a fresh, coherent header set. The headerGeneratorOptions constrain the generator to realistic combinations — you wouldn't want Safari on Linux, which would itself be a fingerprint anomaly.

Routing Through Residential Proxies

Headers get you past the HTTP-layer inspection. But once your TLS and headers look like a real browser, the next signal anti-bot systems check is your IP address. A datacenter IP from AWS or Hetzner is an immediate red flag — legitimate users don't browse from 54.x.x.x. This is where a got-scraping proxy setup becomes essential: you need residential IPs that belong to real ISPs.

ProxyHat's residential proxy gateway runs at gate.proxyhat.com:8080 for HTTP and gate.proxyhat.com:1080 for SOCKS5. Geo-targeting and session stickiness are encoded in the username string, so you can rotate countries, cities, and sessions per request without changing your client configuration.

HTTP vs SOCKS5: When to Use Each

Feature	HTTP Proxy (port 8080)	SOCKS5 Proxy (port 1080)
Protocol support	HTTP/HTTPS via CONNECT	Any TCP traffic
HTTP/2 support	Yes (tunneled via CONNECT)	Yes (transparent TCP tunnel)
Overhead	Lower (single hop)	Slightly higher
Use case	Standard web scraping	WebSocket, non-HTTP protocols
got-scraping compatibility	Full	Full (via socks-proxy-agent)

For most nodejs scraping proxy workflows, HTTP on port 8080 is the right default. Use SOCKS5 when you need to tunnel WebSocket connections or when a target site's WAF specifically inspects proxy-protocol metadata.

Runnable Example: Rotating Residential Proxies with Retry Hooks

This example shows a production-grade got-scraping client that rotates ProxyHat residential endpoints per request, uses sticky sessions for multi-step flows, and retries on 403/429 with exponential backoff.

import { gotScraping } from 'got-scraping';
import pLimit from 'p-limit';
import { randomUUID } from 'crypto';

const PROXYHAT_GATEWAY = 'gate.proxyhat.com';
const PROXYHAT_PORT = 8080;
const PROXYHAT_USER = 'your-account-username';
const PROXYHAT_PASS = 'your-account-password';

// Build a per-request proxy URL with geo + session flags
function buildProxyUrl({ country = 'US', session = randomUUID() } = {}) {
  const username = `${PROXYHAT_USER}-country-${country}-session-${session}`;
  return `http://${username}:${PROXYHAT_PASS}@${PROXYHAT_GATEWAY}:${PROXYHAT_PORT}`;
}

const client = gotScraping.extend({
  useHeaderGenerator: true,
  headerGeneratorOptions: {
    browsers: ['chrome'],
    devices: ['desktop'],
    operatingSystems: ['windows', 'macos'],
    locales: ['en-US'],
  },
  http2: true,
  timeout: { request: 30000 },
  retry: {
    limit: 3,
    statusCodes: [403, 429, 500, 502, 503],
    backoffLimit: 5000,
  },
  hooks: {
    beforeRequest: [(options) => {
      // Inject a fresh residential proxy per request
      options.proxyUrl = buildProxyUrl({
        country: 'US',
        session: randomUUID(),
      });
    }],
    afterResponse: [(response, retryWithMergedOptions) => {
      if (response.statusCode === 403 || response.statusCode === 429) {
        // Rotate proxy and retry
        retryWithMergedOptions({
          proxyUrl: buildProxyUrl({
            country: 'US',
            session: randomUUID(),
          }),
        });
      }
      return response;
    }],
  },
});

// Bounded concurrency: 10 parallel requests
const limit = pLimit(10);

async function scrapeUrls(urls) {
  const results = await Promise.allSettled(
    urls.map((url) => limit(() => client(url).text()))
  );
  return results;
}

The beforeRequest hook generates a unique session ID per request, so each request exits through a different residential IP. For multi-step flows (login → dashboard → data page), pass the same session value across requests to maintain IP stickiness — ProxyHat keeps the same residential IP alive for the session's lifetime.

Per-Request Username Generation Pattern

Because ProxyHat encodes targeting in the username, you can build a thin helper that generates usernames for any combination of country, city, and session:

class ProxyHatSessionManager {
  constructor(username, password, gateway = 'gate.proxyhat.com', port = 8080) {
    this.username = username;
    this.password = password;
    this.gateway = gateway;
    this.port = port;
  }

  proxyUrl({ country, city, session }) {
    let user = this.username;
    if (country) user += `-country-${country}`;
    if (city) user += `-city-${city.toLowerCase()}`;
    if (session) user += `-session-${session}`;
    return `http://${user}:${this.password}@${this.gateway}:${this.port}`;
  }

  socksUrl({ country, session }) {
    let user = this.username;
    if (country) user += `-country-${country}`;
    if (session) user += `-session-${session}`;
    return `socks5://${user}:${this.password}@${this.gateway}:1080`;
  }
}

const proxy = new ProxyHatSessionManager('myuser', 'mypass');
// http://myuser-country-DE-city-berlin-session-abc:mypass@gate.proxyhat.com:8080
console.log(proxy.proxyUrl({ country: 'DE', city: 'berlin', session: 'abc' }));

This pattern keeps proxy configuration declarative and testable. See the ProxyHat documentation for the full list of supported targeting flags.

Production Patterns for got-scraping at Scale

Bounded Concurrency with p-limit

Node.js is single-threaded but async I/O is concurrent. Unbounded Promise.all on 10,000 URLs will exhaust file descriptors, trigger proxy rate limits, and get your IPs banned. Use p-limit to cap concurrent in-flight requests. A good starting point is 5–15 concurrent connections per proxy session; scale up cautiously while monitoring success rates.

Cookie Jars for Session Continuity

Many sites set anti-bot cookies on the first response and expect them on subsequent requests. got supports tough-cookie jars natively via the cookieJar option:

import { gotScraping } from 'got-scraping';
import { CookieJar } from 'tough-cookie';

const jar = new CookieJar();

const sessionClient = gotScraping.extend({
  cookieJar: jar,
  useHeaderGenerator: true,
  headerGeneratorOptions: {
    browsers: ['chrome'],
    devices: ['desktop'],
    operatingSystems: ['macos'],
    locales: ['en-US'],
  },
  http2: true,
  hooks: {
    beforeRequest: [(opts) => {
      opts.proxyUrl = proxy.proxyUrl({
        country: 'US',
        session: 'sticky-session-001',
      });
    }],
  },
});

// Step 1: hit the landing page to collect cookies
await sessionClient('https://target-site.com/').text();
// Step 2: cookies + same IP → the API endpoint trusts you
const data = await sessionClient('https://target-site.com/api/data').json();

Pair the cookie jar with a sticky session (-session-abc123) so the same residential IP carries the same cookies across the flow.

Dropping into Crawlee's CheerioCrawler

When your scraping grows beyond a single script — when you need request queues, deduplication, automatic retries with exponential backoff, and structured data extraction — Crawlee is the natural next step. Crawlee's CheerioCrawler uses got-scraping under the hood, so your header generation and proxy configuration carry over directly:

import { CheerioCrawler, ProxyConfiguration } from 'crawlee';

const proxyConfig = new ProxyConfiguration({
  proxyUrls: [
    'http://myuser-country-US-session-s1:mypass@gate.proxyhat.com:8080',
    'http://myuser-country-US-session-s2:mypass@gate.proxyhat.com:8080',
    'http://myuser-country-DE-session-s3:mypass@gate.proxyhat.com:8080',
  ],
});

const crawler = new CheerioCrawler({
  proxyConfiguration: proxyConfig,
  requestHandlerTimeoutSecs: 30,
  maxConcurrency: 10,
  async requestHandler({ $, request }) {
    const title = $('title').text();
    const links = $('a[href]').map((_, el) => $(el).attr('href')).get();
    await Actor.pushData({ url: request.url, title, linkCount: links.length });
  },
});

await crawler.run(['https://example.com/page-1', 'https://example.com/page-2']);

Crawlee handles the retry logic, request queue, and concurrency for you. The ProxyConfiguration rotates through the provided URLs, and got-scraping's header generator runs automatically on every request. For more on scaling strategies, see our web scraping use case guide.

When a Headless Browser Is Unavoidable

got-scraping handles HTTP-layer fingerprinting well, but some sites require JavaScript execution to render content or to solve CAPTCHA challenges. If the page you need is populated client-side via React/Vue/Angular, or if the WAF serves a JavaScript challenge that no HTTP client can solve, you need a headless browser — Playwright or Puppeteer. In Crawlee, that means switching from CheerioCrawler to PlaywrightCrawler.

The proxy configuration is identical: pass the same gate.proxyhat.com:8080 URLs to the browser's proxy settings. The residential IP + real browser combination is the strongest fingerprint you can produce without a physical device. For SERP-specific patterns, see our SERP tracking guide.

Choosing the Right Proxy Type

Proxy Type	Best For	Stealth Level	Cost
Residential	General scraping, SERP, e-commerce	High (real ISP IPs)	Medium
Mobile	Mobile-optimized sites, app APIs	Highest (carrier IPs)	High
Datacenter	High-volume, low-stealth targets	Low (flagged ranges)	Low

For got-scraping proxy workflows, residential is the default choice. Datacenter proxies work for targets that don't inspect IP reputation, but once you've invested in coherent headers and HTTP/2, residential IPs complete the illusion. Browse available proxy locations and check pricing to plan your capacity.

Key Takeaways

• Headers are the first line of defense. got-scraping's header generator produces coherent, browser-accurate header sets — correct ordering, matching sec-ch-ua, full sec-fetch-* family. This alone defeats many naive WAF checks.
• Residential IPs close the loop. Once headers and TLS look real, IP reputation is the next signal. Route through gate.proxyhat.com:8080 with -country-US-session-abc123 targeting for per-request rotation.
• Use got.extend() idiomatic-ally. Configure useHeaderGenerator, headerGeneratorOptions, http2, and proxyUrl once on the extended client. Inject per-request proxies via beforeRequest hooks.
• Bound your concurrency. p-limit at 5–15 concurrent requests per session prevents file descriptor exhaustion and proxy rate-limit bans.
• Graduate to Crawlee when needed. CheerioCrawler wraps got-scraping with request queues, retries, and structured extraction. Switch to PlaywrightCrawler only when JavaScript execution is required.
• Scrape ethically. Public data only. Honor robots.txt, respect rate limits, and prefer official APIs when available.

FAQ

What is got-scraping in Node.js?

got-scraping is an npm package by Apify that wraps the got HTTP client with browser-like header generation, HTTP/2 support, and proxy integration. It generates coherent header sets — correct ordering, matching sec-ch-ua version strings, full sec-fetch-* family — so requests look like genuine browser traffic rather than script traffic. It's the HTTP client used internally by Crawlee's CheerioCrawler.

Why does got-scraping in Node.js matter for proxy users?

Anti-bot systems inspect the full request envelope: headers, TLS fingerprint, and IP reputation. Even with perfect headers, a datacenter IP gets flagged. got-scraping solves the header layer; residential proxies solve the IP layer. Together, they produce requests that pass both HTTP-level and network-level inspection. Without coherent headers, even the best residential proxy won't help — the mismatch between a browser user-agent and missing sec-ch-ua is an instant red flag.

Which proxy type works best for got-scraping in Node.js?

Residential proxies are the default choice for got-scraping workflows. They use real ISP-assigned IPs, so they pass IP reputation checks that immediately flag datacenter ranges from AWS, Hetzner, or DigitalOcean. Mobile proxies offer the highest stealth level (carrier IPs) but cost more. Use datacenter proxies only for targets that don't inspect IP reputation. Configure residential proxies via gate.proxyhat.com:8080 with country and session targeting in the username.

How do you avoid blocks when implementing got-scraping in Node.js?

Layer your defenses: enable useHeaderGenerator with constrained headerGeneratorOptions (specific browsers, devices, OS combinations), enable HTTP/2, route through residential proxies with per-request session rotation, use p-limit to bound concurrency to 5–15 parallel requests, implement retry hooks for 403/429 responses with exponential backoff, and use cookie jars for multi-step flows. Always honor robots.txt and site rate limits.

Can got-scraping use SOCKS5 proxies?

Yes. got-scraping supports SOCKS5 proxies via the proxyUrl option. Use the format socks5://username:password@gate.proxyhat.com:1080. SOCKS5 is useful for tunneling non-HTTP traffic or WebSocket connections. For standard web scraping, HTTP proxies on port 8080 are simpler and have lower overhead. Both protocols support the same username-based geo-targeting and session flags.