Build a Google Rank Tracker in Python with Residential Proxies

Why You Need to Build a Google Rank Tracker in Python with Residential Proxies

If you've ever tried to build a Google rank tracker in Python with residential proxies, you already know the hardest part isn't parsing HTML — it's getting the HTML in the first place. Google's anti-bot defenses have evolved far beyond simple rate limiting. TLS fingerprinting, IP-reputation scoring, and behavioral analysis all work together to block automated requests before they ever reach the results page.

Commercial rank trackers like Ahrefs, Semrush, and SE Ranking charge anywhere from $50 to $500+ per month for rank data, and most cap you at daily or weekly granularity. Building your own tracker gives you full control over keywords, frequency, countries, and device types — at a fraction of the cost when you're already running infrastructure.

This guide walks through a complete, production-ready rank tracker in Python. We cover the data model, SERP pagination, proxy rotation with ProxyHat, HTML parsing, SQLite storage, retries, CAPTCHA detection, and concurrency. Every code block is runnable.

The Data Model: Keywords, Positions, and Daily Snapshots

A rank tracker is fundamentally a time-series database: for each keyword, you record where your domain appeared in Google's results on a given day. The minimal schema looks like this:

CREATE TABLE IF NOT EXISTS keywords (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    keyword TEXT NOT NULL UNIQUE,
    country TEXT DEFAULT 'US',
    device TEXT DEFAULT 'desktop',
    target_domain TEXT NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE IF NOT EXISTS rankings (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    keyword_id INTEGER NOT NULL,
    position INTEGER,
    page INTEGER,
    result_url TEXT,
    result_title TEXT,
    captured_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY (keyword_id) REFERENCES keywords(id)
);

CREATE INDEX idx_rankings_keyword_date ON rankings(keyword_id, captured_at);

Each row in rankings is a snapshot. If your domain ranks #3 on Monday and #7 on Tuesday, you get two rows — not an overwrite. This lets you calculate rank volatility, spot algorithm-update impacts, and generate trend charts.

Why daily snapshots beat one-off checks: A single rank check tells you nothing about volatility. Daily snapshots let you compute 7-day moving averages, detect sudden drops within 24 hours, and correlate movements with Google algorithm updates or your own content changes.

The device field matters because Google serves different results on mobile vs. desktop. If your audience is primarily mobile, track mobile rankings separately. The country field drives proxy geo-targeting, which we'll cover below.

Fetching SERPs After Google Removed num=100

For years, scrapers used num=100 to fetch 100 results in a single request. Google deprecated this parameter in September 2025, meaning you now need to paginate across multiple pages using start=0, start=10, start=20, and so on up to start=90 to cover the top 100 organic results.

Each page returns approximately 10 organic results (the count varies with SERP features). You need to handle this in your fetcher:

import asyncio
from curl_cffi import requests

class RateLimitError(Exception):
    pass

class SERPError(Exception):
    pass

async def fetch_serp_page(keyword: str, start: int, proxy_url: str,
                         country: str = "US") -> str:
    """Fetch one SERP page (10 results) using curl_cffi with Chrome TLS fingerprint."""
    url = "https://www.google.com/search"
    params = {
        "q": keyword,
        "start": start,
        "num": 10,
        "hl": "en",
        "gl": country.lower(),
    }
    headers = {
        "Accept": "text/html,application/xhtml+xml",
        "Accept-Language": "en-US,en;q=0.9",
        "Cache-Control": "no-cache",
    }
    try:
        response = await requests.async_get(
            url,
            params=params,
            headers=headers,
            proxies={"http": proxy_url, "https": proxy_url},
            impersonate="chrome",
            timeout=20,
        )
        if response.status_code == 200:
            return response.text
        elif response.status_code == 429:
            raise RateLimitError(f"Rate limited at start={start}")
        else:
            raise SERPError(f"Unexpected status {response.status_code}")
    except Exception as e:
        raise SERPError(f"Fetch failed at start={start}: {e}")

The key insight: start increments by 10 for each page. To cover the top 100, you make 10 requests per keyword. With 50 keywords, that's 500 requests per daily run — well within ProxyHat's concurrency limits if you batch properly.

Parsing Organic Results

Google's HTML is not a stable API. CSS class names change frequently, and SERP features (Featured Snippets, People Also Ask, Knowledge Panels, Shopping ads) are interleaved with organic results. The safest approach is to target the structural pattern of organic result blocks rather than specific class names.

from bs4 import BeautifulSoup
import re

def parse_organic_results(html: str) -> list[dict]:
    """Extract organic results, skipping ads and SERP features."""
    soup = BeautifulSoup(html, "html.parser")
    results = []

    # Google organic results live in div elements whose class starts with 'g'
    for div in soup.find_all("div", attrs={"class": re.compile(r"\bg\b")}):
        anchor = div.find("a", href=True)
        if not anchor:
            continue
        href = anchor.get("href", "")
        # Skip internal Google links and sponsored results
        if href.startswith("/search") or href.startswith("/url?q=/"):
            continue
        # Extract the actual URL from /url?q= redirects
        if "/url?q=" in href:
            href = href.split("/url?q=")[1].split("&")[0]
        # Skip if it's still a google.com link (maps, shopping, etc.)
        if "google.com" in href:
            continue
        title_tag = div.find("h3")
        title = title_tag.get_text(strip=True) if title_tag else ""
        if title and href.startswith("http"):
            results.append({"url": href, "title": title})

    return results

This parser skips sponsored ads (which use different DOM structures), Google's own properties (Maps, Shopping), and SERP features. It returns a clean list of {"url", "title"} dicts. You then match target_domain against each result's URL to find the position.

from urllib.parse import urlparse

def find_position(results: list[dict], target_domain: str) -> int | None:
    """Find the 1-based position of target_domain in organic results."""
    target = target_domain.lower().replace("www.", "")
    for i, result in enumerate(results, start=1):
        result_host = urlparse(result["url"]).netloc.lower().replace("www.", "")
        if result_host == target or result_host.endswith("." + target):
            return i
    return None

Why Residential Proxies Beat Datacenter for SERP Scraping

Google doesn't just look at your IP address — it looks at your TLS handshake fingerprint (JA3/JA4 hashes), your HTTP/2 settings, your IP's ASN (Autonomous System Number), and your request patterns. Datacenter IPs are flagged because they come from hosting providers (AWS, DigitalOcean, Hetzner) that Google associates with bot traffic. According to Google's Search Central documentation, automated requests that violate their terms are actively blocked.

Residential proxies route your traffic through real ISP-assigned IP addresses. Google sees a request from a Comcast or AT&T subscriber in Chicago, not from a DigitalOcean VPS. This dramatically reduces the chance of CAPTCHAs and IP bans.

For rank tracking, residential proxies with city-level geo-targeting are the sweet spot. You get high success rates, precise country/city targeting, and enough concurrency for daily runs. Mobile proxies work but cost 3–5x more per GB and aren't necessary for SERP scraping.

TLS Fingerprinting and Why impersonate='chrome' Matters

Even with a residential IP, if your TLS handshake looks like Python's requests library (which uses OpenSSL's default cipher ordering), Google can still flag you. The curl_cffi library solves this by using the same TLS library as Chrome (BoringSSL) and replicating Chrome's exact JA3/JA4 fingerprint, HTTP/2 settings, and header ordering.

# Raw proxy URL format (no SDK needed):
# HTTP:  http://user-country-US-city-chicago-session-kw123:pass@gate.proxyhat.com:8080
# SOCKS5: socks5://user-country-US-city-chicago-session-kw123:pass@gate.proxyhat.com:1080

import asyncio
from curl_cffi import requests

async def fetch_with_proxyhat(keyword: str, country: str, session_id: str):
    """Fetch a SERP page through ProxyHat residential proxy with Chrome TLS."""
    proxy_url = (
        f"http://user-country-{country}-city-chicago-session-{session_id}"
        f":pass@gate.proxyhat.com:8080"
    )
    response = await requests.async_get(
        "https://www.google.com/search",
        params={"q": keyword, "num": 10, "gl": country.lower(), "hl": "en"},
        proxies={"http": proxy_url, "https": proxy_url},
        impersonate="chrome",
        timeout=20,
    )
    return response

The -session-{session_id} flag in the username keeps the same IP for all 10 paginated requests for a single keyword. This is critical: if your IP changes mid-keyword, Google may serve different results from a different datacenter, corrupting your position data. Use a unique session ID per keyword per daily run — for example session-kw42-20260115.

You can also test the proxy from the command line before writing any Python:

# Test ProxyHat residential proxy with curl (HTTP)
curl -x "http://user-country-US-city-chicago-session-test1:pass@gate.proxyhat.com:8080" \
  "https://www.google.com/search?q=python+rank+tracker&num=10&gl=us&hl=en" \
  -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" \
  -H "Accept-Language: en-US,en;q=0.9" \
  -o serp_page_0.html

Production Hardening: Retries, CAPTCHAs, and Concurrency

A rank tracker that works on your laptop will fail in production. Here's what you need to handle.

Retries with Exponential Backoff

import asyncio
import random
from functools import wraps

def with_retries(max_retries=3, base_delay=2.0, backoff_factor=2.0):
    """Async retry decorator with exponential backoff and jitter."""
    def decorator(func):
        @wraps(func)
        async def wrapper(*args, **kwargs):
            last_error = None
            for attempt in range(max_retries + 1):
                try:
                    return await func(*args, **kwargs)
                except (RateLimitError, SERPError) as e:
                    last_error = e
                    if attempt < max_retries:
                        delay = base_delay * (backoff_factor ** attempt)
                        delay += random.uniform(0, 1)  # jitter
                        print(f"Attempt {attempt+1} failed: {e}. Retrying in {delay:.1f}s")
                        await asyncio.sleep(delay)
            raise last_error
        return wrapper
    return decorator

@with_retries(max_retries=3, base_delay=2.0)
async def fetch_serp_safe(keyword, start, proxy_url, country):
    return await fetch_serp_page(keyword, start, proxy_url, country)

CAPTCHA Detection

Google serves CAPTCHAs as HTTP 429 responses or as HTML pages containing captcha in the URL or body. Detect these early so you don't waste parsing time:

def detect_captcha(html: str, status_code: int) -> bool:
    """Check if the response is a CAPTCHA or block page."""
    if status_code == 429:
        return True
    captcha_signals = [
        "recaptcha",
        "g-recaptcha",
        "sorry/index",
        "unusual traffic",
        "Our systems have detected",
    ]
    html_lower = html.lower()
    return any(signal.lower() in html_lower for signal in captcha_signals)

When you detect a CAPTCHA, rotate to a new session ID (new IP) and retry. If you see CAPTCHAs on more than 5% of requests, reduce your concurrency or add longer delays between batches.

Concurrency Control

import asyncio
from datetime import datetime

async def track_keyword(keyword_id: int, keyword: str, country: str,
                        target_domain: str, semaphore: asyncio.Semaphore):
    """Track a single keyword across the top 100 results (10 pages)."""
    session_id = f"kw{keyword_id}-{datetime.now().strftime('%Y%m%d')}"
    proxy_url = (
        f"http://user-country-{country}-session-{session_id}"
        f":pass@gate.proxyhat.com:8080"
    )
    all_results = []
    for start in range(0, 100, 10):
        async with semaphore:
            html = await fetch_serp_safe(keyword, start, proxy_url, country)
            if detect_captcha(html, 200):
                # Rotate IP by changing session ID
                session_id += f"-r{start}"
                proxy_url = (
                    f"http://user-country-{country}-session-{session_id}"
                    f":pass@gate.proxyhat.com:8080"
                )
                html = await fetch_serp_safe(keyword, start, proxy_url, country)
            results = parse_organic_results(html)
            all_results.extend(results)
            await asyncio.sleep(1.5)  # polite delay between pages

    position = find_position(all_results, target_domain)
    return {
        "keyword_id": keyword_id,
        "position": position,
        "result_count": len(all_results),
        "captured_at": datetime.now().isoformat(),
    }

async def run_daily_track(keywords: list[dict], max_concurrent: int = 10):
    """Run the daily rank tracking job for all keywords."""
    semaphore = asyncio.Semaphore(max_concurrent)
    tasks = [
        track_keyword(
            kw["id"], kw["keyword"], kw["country"], kw["target_domain"], semaphore
        )
        for kw in keywords
    ]
    results = await asyncio.gather(*tasks, return_exceptions=True)
    return results

The asyncio.Semaphore limits concurrent requests to 10 by default. For 50 keywords × 10 pages = 500 requests, this completes in roughly 75 seconds with 1.5-second delays between pages. Adjust max_concurrent based on your ProxyHat plan — check ProxyHat pricing for concurrency limits.

Rank Volatility Smoothing and CSV Export

Single-day position changes can be noisy. Google tests results, personalization kicks in, and datacenter variance causes fluctuations. A 7-day moving average smooths this:

import sqlite3
import csv

def export_rank_history(db_path: str, keyword_id: int, output_csv: str):
    """Export rank history with 7-day moving average to CSV."""
    conn = sqlite3.connect(db_path)
    cursor = conn.execute(
        """SELECT captured_at, position FROM rankings
           WHERE keyword_id = ? ORDER BY captured_at""",
        (keyword_id,),
    )
    rows = cursor.fetchall()
    conn.close()

    with open(output_csv, "w", newline="") as f:
        writer = csv.writer(f)
        writer.writerow(["date", "position", "7day_avg"])
        positions = []
        for date, pos in rows:
            positions.append(pos)
            avg = sum(positions[-7:]) / min(len(positions), 7)
            writer.writerow([date, pos, round(avg, 1)])

Ethics, Rate Limits, and Compliance

Rank tracking sits in a gray area. Google's Terms of Service technically prohibit automated scraping, but millions of SEO professionals do it daily. Here are practical guardrails:

• Track your own domains first. If you're monitoring your own rankings, you have a stronger ethical justification than scraping competitors.
• Respect rate limits. Keep requests under 1 request per second per IP. Use delays between pages and between keywords.
• Prefer official APIs at low volume. If you track fewer than 100 keywords, consider using the Google Custom Search JSON API, which provides 100 queries per day for free and up to 10,000 per day at $5 per 1,000 queries.
• Check robots.txt. While Google's robots.txt doesn't explicitly block /search, respect the spirit of crawl-delay directives.
• Don't redistribute raw SERP data. Use rankings internally — don't resell Google's HTML to third parties.

For high-volume tracking (500+ keywords daily), a residential proxy service like ProxyHat's SERP tracking solution is the most reliable approach. The combination of real ISP IPs, city-level geo-targeting, and sticky sessions per keyword minimizes CAPTCHAs and ensures consistent results. Explore ProxyHat's proxy locations for available geo-targeting options, or read our broader web scraping guide for additional patterns.

Key Takeaways

• Daily snapshots over one-off checks. Time-series data lets you detect volatility, algorithm updates, and trend reversals.
• Paginate with start=0,10,20…90. Google removed num=100, so you need 10 requests per keyword to cover the top 100.
• Residential proxies with city targeting. Datacenter IPs get blocked; residential IPs from real ISPs have 90–98% success rates on SERPs.
• Use impersonate='chrome' with curl_cffi. TLS fingerprinting (JA3/JA4) is the first line of defense — match Chrome's handshake exactly.
• Sticky sessions per keyword. Use -session-{keyword_hash}-{date} in your ProxyHat username to keep the same IP across all 10 pages.
• Retries, CAPTCHA detection, and concurrency limits. Production rank tracking fails without exponential backoff, captcha detection, and semaphore-based concurrency control.

Ready to start tracking? Check out the ProxyHat documentation for advanced session and rotation configuration, or head to ProxyHat pricing to pick a plan that fits your keyword volume.