如何爬取JavaScript重度渲染的网站

JavaScript- 渲染内容的挑战

现代网站越来越依赖JavaScript来渲染内容. 单页应用程序(SPA)用React,Vue,或Angular装入一个最小的HTML shell,然后获取和渲染数据客户端侧. 当向这些站点提出简单的HTTP请求时,会得到一个空的或不完整的页面,因为内容只在JavaScript执行后才存在.

搜索 JavaScript 重置网站需要 无头浏览器 ——真实的浏览器引擎运行时没有可见的窗口,可以执行JavaScript,渲染DOM,并与页面元素交互. 结合代理,无头浏览器解锁甚至最动态网站的数据.

这个向导是我们的一部分 完整网页搜索代理指南为避免在使用无头浏览器时检测,请参见 反毒系统如何检测代理。 。 。

您何时需要无头浏览器 ?

总是在到达无头浏览器前检查网站是否有API或服务器侧渲染. 许多"JavaScript-havy"网站实际上都有API端点,可以返回干净的JSON——可以更快,更便宜的刮刮.

管道工+代理(节点.js)

Puppeteer在程序上控制着铬/铬. 它是Node.js最成熟的无头浏览器工具.

基本设置, 使用代理汉特

const puppeteer = require('puppeteer');
async function scrapeWithPuppeteer(url) {
  const browser = await puppeteer.launch({
    headless: 'new',
    args: [
      '--proxy-server=http://gate.proxyhat.com:8080',
      '--no-sandbox',
      '--disable-setuid-sandbox',
      '--disable-dev-shm-usage',
    ],
  });
  const page = await browser.newPage();
  // Authenticate with proxy
  await page.authenticate({
    username: 'USERNAME',
    password: 'PASSWORD',
  });
  // Set realistic viewport and user agent
  await page.setViewport({ width: 1920, height: 1080 });
  await page.setUserAgent(
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' +
    '(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
  );
  try {
    await page.goto(url, { waitUntil: 'networkidle2', timeout: 60000 });
    // Wait for specific content to render
    await page.waitForSelector('.product-list', { timeout: 10000 });
    const content = await page.content();
    const data = await page.evaluate(() => {
      return Array.from(document.querySelectorAll('.product-item')).map(el => ({
        name: el.querySelector('.product-name')?.textContent?.trim(),
        price: el.querySelector('.product-price')?.textContent?.trim(),
        url: el.querySelector('a')?.href,
      }));
    });
    return { html: content, data };
  } finally {
    await browser.close();
  }
}
// Usage
const result = await scrapeWithPuppeteer('https://example.com/products');
console.log(`Found ${result.data.length} products`);

优化多页搜索

const puppeteer = require('puppeteer');
class PuppeteerScraper {
  constructor(concurrency = 3) {
    this.concurrency = concurrency;
    this.browser = null;
  }
  async init() {
    this.browser = await puppeteer.launch({
      headless: 'new',
      args: [
        '--proxy-server=http://gate.proxyhat.com:8080',
        '--no-sandbox',
        '--disable-setuid-sandbox',
        '--disable-dev-shm-usage',
        '--disable-gpu',
        '--disable-extensions',
      ],
    });
  }
  async scrapePage(url) {
    const page = await this.browser.newPage();
    await page.authenticate({ username: 'USERNAME', password: 'PASSWORD' });
    await page.setViewport({ width: 1920, height: 1080 });
    // Block unnecessary resources to speed up loading
    await page.setRequestInterception(true);
    page.on('request', (req) => {
      const type = req.resourceType();
      if (['image', 'stylesheet', 'font', 'media'].includes(type)) {
        req.abort();
      } else {
        req.continue();
      }
    });
    try {
      await page.goto(url, { waitUntil: 'networkidle2', timeout: 30000 });
      const content = await page.content();
      return { url, status: 'success', html: content };
    } catch (err) {
      return { url, status: 'error', error: err.message };
    } finally {
      await page.close();
    }
  }
  async scrapeMany(urls) {
    const results = [];
    for (let i = 0; i < urls.length; i += this.concurrency) {
      const batch = urls.slice(i, i + this.concurrency);
      const batchResults = await Promise.all(
        batch.map(url => this.scrapePage(url))
      );
      results.push(...batchResults);
      console.log(`Progress: ${results.length}/${urls.length}`);
    }
    return results;
  }
  async close() {
    if (this.browser) await this.browser.close();
  }
}
// Usage
const scraper = new PuppeteerScraper(3);
await scraper.init();
const results = await scraper.scrapeMany(urls);
await scraper.close();

Playwright + 代理 (Python) (英语).

Playwright是一个支持Chrimium,Firefox,和WebKit的较新的替代品. 它的Python API是干净的,适合刮刮.

基本设置

from playwright.sync_api import sync_playwright
def scrape_with_playwright(url: str) -> dict:
    """Scrape a JavaScript-heavy page using Playwright with ProxyHat proxy."""
    with sync_playwright() as p:
        browser = p.chromium.launch(
            headless=True,
            proxy={
                "server": "http://gate.proxyhat.com:8080",
                "username": "USERNAME",
                "password": "PASSWORD",
            }
        )
        context = browser.new_context(
            viewport={"width": 1920, "height": 1080},
            user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                       "AppleWebKit/537.36 (KHTML, like Gecko) "
                       "Chrome/120.0.0.0 Safari/537.36",
        )
        page = context.new_page()
        try:
            page.goto(url, wait_until="networkidle", timeout=60000)
            # Wait for dynamic content
            page.wait_for_selector(".product-list", timeout=10000)
            # Extract data using page.evaluate
            products = page.evaluate("""() => {
                return Array.from(document.querySelectorAll('.product-item')).map(el => ({
                    name: el.querySelector('.product-name')?.textContent?.trim(),
                    price: el.querySelector('.product-price')?.textContent?.trim(),
                    url: el.querySelector('a')?.href,
                }));
            }""")
            return {"url": url, "products": products, "html": page.content()}
        finally:
            browser.close()

并行搜索的 Async Playwright

import asyncio
from playwright.async_api import async_playwright
async def scrape_batch(urls: list[str], concurrency: int = 3) -> list[dict]:
    """Scrape multiple JS-heavy pages in parallel using Playwright."""
    results = []
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            headless=True,
            proxy={
                "server": "http://gate.proxyhat.com:8080",
                "username": "USERNAME",
                "password": "PASSWORD",
            }
        )
        semaphore = asyncio.Semaphore(concurrency)
        async def scrape_one(url: str) -> dict:
            async with semaphore:
                context = await browser.new_context(
                    viewport={"width": 1920, "height": 1080},
                )
                page = await context.new_page()
                # Block heavy resources
                await page.route("**/*.{png,jpg,jpeg,gif,svg,css,woff,woff2}",
                                 lambda route: route.abort())
                try:
                    await page.goto(url, wait_until="networkidle", timeout=30000)
                    html = await page.content()
                    return {"url": url, "status": "success", "html": html}
                except Exception as e:
                    return {"url": url, "status": "error", "error": str(e)}
                finally:
                    await context.close()
        tasks = [scrape_one(url) for url in urls]
        results = await asyncio.gather(*tasks)
        await browser.close()
    return results
# Usage
urls = [f"https://example.com/product/{i}" for i in range(50)]
results = asyncio.run(scrape_batch(urls, concurrency=5))

跳: 与代理使用色调

package main
import (
    "context"
    "fmt"
    "log"
    "time"
    "github.com/chromedp/chromedp"
)
func scrapeJSPage(targetURL string) (string, error) {
    // Configure proxy
    opts := append(chromedp.DefaultExecAllocatorOptions[:],
        chromedp.ProxyServer("http://gate.proxyhat.com:8080"),
        chromedp.Flag("headless", true),
        chromedp.Flag("disable-gpu", true),
        chromedp.Flag("no-sandbox", true),
        chromedp.UserAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) "+
            "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"),
    )
    allocCtx, cancel := chromedp.NewExecAllocator(context.Background(), opts...)
    defer cancel()
    ctx, cancel := chromedp.NewContext(allocCtx)
    defer cancel()
    ctx, cancel = context.WithTimeout(ctx, 60*time.Second)
    defer cancel()
    var htmlContent string
    err := chromedp.Run(ctx,
        chromedp.Navigate(targetURL),
        chromedp.WaitVisible(".product-list", chromedp.ByQuery),
        chromedp.OuterHTML("html", &htmlContent),
    )
    if err != nil {
        return "", fmt.Errorf("scrape failed: %w", err)
    }
    return htmlContent, nil
}
func main() {
    html, err := scrapeJSPage("https://example.com/products")
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("Got %d bytes of rendered HTML\n", len(html))
}

业绩优化战略

无头浏览器比简单的HTTP请求慢10-50x. 以下是尽量减少业绩差距的战略:

1. 不必要的资源

数据提取不需要图像,CSS,字体,和媒体文件. 截断它们会加快页面载荷 :

# Playwright resource blocking
async def fast_scrape(page, url):
    # Block images, CSS, fonts, media
    await page.route("**/*.{png,jpg,jpeg,gif,svg,css,woff,woff2,mp4,webm}",
                     lambda route: route.abort())
    # Also block tracking scripts
    await page.route("**/*google-analytics*", lambda route: route.abort())
    await page.route("**/*facebook*", lambda route: route.abort())
    await page.goto(url, wait_until="domcontentloaded")  # Faster than networkidle
    return await page.content()

2. 使用 " 权利等待战略 "

3. 重新使用浏览器

推出浏览器需要1-3秒. 对于批量刮刮,一次发射,并为每个URL创建新的页面/文本:

from playwright.sync_api import sync_playwright
class BrowserPool:
    """Reusable browser pool for efficient headless scraping."""
    def __init__(self, pool_size: int = 3):
        self.pool_size = pool_size
        self.playwright = None
        self.browsers = []
    def start(self):
        self.playwright = sync_playwright().start()
        for _ in range(self.pool_size):
            browser = self.playwright.chromium.launch(
                headless=True,
                proxy={
                    "server": "http://gate.proxyhat.com:8080",
                    "username": "USERNAME",
                    "password": "PASSWORD",
                }
            )
            self.browsers.append(browser)
    def get_browser(self, index: int):
        return self.browsers[index % self.pool_size]
    def stop(self):
        for browser in self.browsers:
            browser.close()
        self.playwright.stop()
# Usage
pool = BrowserPool(pool_size=3)
pool.start()
for i, url in enumerate(urls):
    browser = pool.get_browser(i)
    context = browser.new_context()
    page = context.new_page()
    page.goto(url, wait_until="networkidle")
    html = page.content()
    context.close()
pool.stop()

4. 截取API 调用而非解析 DOM

许多SPA从API获取数据. 截取这些 API 直接调用—— 您可以在不解析 HTML 的情况下获得干净的 JSON :

const puppeteer = require('puppeteer');
async function interceptAPIData(url) {
  const browser = await puppeteer.launch({
    headless: 'new',
    args: ['--proxy-server=http://gate.proxyhat.com:8080'],
  });
  const page = await browser.newPage();
  await page.authenticate({ username: 'USERNAME', password: 'PASSWORD' });
  const apiResponses = [];
  // Intercept XHR/fetch responses
  page.on('response', async (response) => {
    const url = response.url();
    if (url.includes('/api/') || url.includes('/graphql')) {
      try {
        const json = await response.json();
        apiResponses.push({ url, data: json });
      } catch {
        // Not JSON, skip
      }
    }
  });
  await page.goto(url, { waitUntil: 'networkidle2' });
  await browser.close();
  return apiResponses;
}
// Get clean API data instead of scraping DOM
const data = await interceptAPIData('https://example.com/products');
console.log(`Intercepted ${data.length} API calls`);

无头浏览器与 HTTP 比较

最佳做法

• 总是先尝试HTTP. 在使用无头浏览器之前,先检查API的端点,服务器上的内容,或者JSON嵌入到HTML中.
• 封锁不必要的资源。 图像,CSS,和字体在不提供数据的情况下加载时间.
• 使用特定的选择器等待 。 networkidle 安全但缓慢。 等待您需要的特定元素 。
• 重用浏览器实例 。 发射一次,每页创建新的上下文.
• 拦截API呼叫. 许多SPA通过API加载数据——直接拦截JSON.
• 限制货币。 无头浏览器是内存密集型的. 每GB的RAM共3-5页是一个很好的规则.
• 使用住宅代理. 代理哈特住宅代理 提供最高的信任分数,减少运行无头浏览器时的检测.

对于无头浏览器遇到的处理 CAPTCHA ,请参见 处理CAPTCHA 当摇摆缩放无头浏览器 如何缩放基础设施。 。 。

开始吧 Python SDK 键盘, (中文). 节点 SDK,或 冲啊 SDK 用于代理集成,并探索 网页搜索的代理用户名。 。 。

经常被问到的问题

我总是需要JavaScript网站的无头浏览器吗?

没有 许多JavaScript-havy站点从API端点加载数据. 请检查access-date=中的日期值 (帮助) XHR/fach请求的浏览器的网络标签——如果数据来自API,可以通过一个代理程序直接使用简单的HTTP请求调用API,这要快得多.

Puppeteer或Playwright——哪一种更适合刮?

Playwright一般被推荐用于新项目. 它支持多个浏览器引擎(Chrimium,Firefox,WebKit),在Python中拥有更好的自动等待,本地的ASync支持,以及内置代理配置. 普普特尔更成熟,如果处于节点世界,拥有更大的生态系统.

我能同时运行多少个无头浏览器页面?

每页消耗RAM的200-500 MB. 在一台拥有8GB内存的机器上,3-10页并存是现实的. 使用资源屏蔽(images, CSS)来减少内存. 对于更高的货币,使用基于队列的架构在多个机器之间分布.

为什么使用无头浏览器的代理?

即使有了真正的浏览器,同一IP的重复请求也会被封杀. 代理旋转您的IP, 所以每个页面负荷似乎来自不同的用户 。 通过代理Hat(英语:ProxyHat)的居家代理提供最高的信任分数,最小化块和CAPTCHA.