[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"blog-post-en-geo-targeted-serp-scraping-best-practices":3},{"success":4,"payload":5,"related":64,"errors":128},true,{"id":6,"slug":7,"title":8,"excerpt":9,"body_html":10,"meta_title":11,"meta_description":12,"canonical_url":13,"canonical_override":14,"og_title":11,"og_description":12,"og_image_url":15,"faq_json":16,"key_takeaways":14,"allow_index":4,"noindex_google_only":32,"allow_follow":4,"published_at":33,"updated_at":33,"content_updated_at":34,"featured_image_url":35,"featured_image_alt":8,"card_image_url":36,"reading_time_minutes":37,"author":38,"categories":42,"tags":49,"alternates":50,"locale_paths":51,"views":63},198,"geo-targeted-serp-scraping-best-practices","Geo-Targeted SERP Scraping Best Practices for Local SEO Data","Learn how to build city-precise SERP datasets using uule, gl, hl, and residential proxies. Includes Python code for uule construction, result parsing, sticky sessions, and retry logic.","\u003Cp>Geo-targeted SERP scraping best practices are essential for any SEO engineer building localized rank-tracking datasets. If you've ever collected Google results for the same keyword from two different cities and seen completely different organic rankings, local packs, and featured snippets, you've already encountered the core problem: Google personalizes SERPs by geography, and country-level targeting alone is insufficient for accurate local SEO intelligence.\u003C\u002Fp>\n\n\u003Ch2>Why the Same Query Returns Different Results by Location\u003C\u002Fh2>\n\n\u003Cp>Google's ranking pipeline applies a geographic filter long before results reach the user. The search engine infers location from several signals: the requester's IP address (resolved to city-level via \u003Ca href=\"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FGeolocation_software\" rel=\"noopener noreferrer\">IP geolocation databases\u003C\u002Fa>), explicit URL parameters like \u003Ccode>gl\u003C\u002Fcode> and \u003Ccode>uule\u003C\u002Fcode>, browser language headers, and the user's signed-in account location. When these signals conflict, Google tends to trust the IP address most heavily for local intent queries.\u003C\u002Fp>\n\n\u003Cp>This is why country-level targeting (e.g., \u003Ccode>gl=us\u003C\u002Fcode>) is insufficient for local SEO. A search for \u003Cstrong>\"plumber\"\u003C\u002Fstrong> in Austin, Texas returns a different local pack than the same query in Brooklyn, New York — even though both are in the United States. The local pack, map results, and even organic rankings shift based on proximity to the searcher. For rank-tracking systems that need to report on a client's visibility in specific markets, you need city-level or even district-level precision.\u003C\u002Fp>\n\n\u003Cp>According to Google's Custom Search documentation, the \u003Ccode>gl\u003C\u002Fcode> parameter controls the country of results, while \u003Ccode>hl\u003C\u002Fcode> sets the interface language. But neither parameter pins results to a specific city. That's where \u003Ccode>uule\u003C\u002Fcode> comes in.\u003C\u002Fp>\n\n\u003Ch2>Core Localization Parameters for SERP Scraping\u003C\u002Fh2>\n\n\u003Cp>Five parameters control the geographic and linguistic context of a Google SERP request:\u003C\u002Fp>\n\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\u003Cth>Parameter\u003C\u002Fth>\u003Cth>Purpose\u003C\u002Fth>\u003Cth>Example\u003C\u002Fth>\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\u003Ctd>\u003Ccode>gl\u003C\u002Fcode>\u003C\u002Ftd>\u003Ctd>Country code (ISO 3166-1 alpha-2)\u003C\u002Ftd>\u003Ctd>\u003Ccode>gl=us\u003C\u002Fcode>\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>\u003Ccode>hl\u003C\u002Fcode>\u003C\u002Ftd>\u003Ctd>Interface language\u003C\u002Ftd>\u003Ctd>\u003Ccode>hl=en\u003C\u002Fcode>\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>\u003Ccode>google_domain\u003C\u002Fcode>\u003C\u002Ftd>\u003Ctd>Google domain to query\u003C\u002Ftd>\u003Ctd>\u003Ccode>google_domain=google.com\u003C\u002Fcode>\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>\u003Ccode>lr\u003C\u002Fcode>\u003C\u002Ftd>\u003Ctd>Language restrict for results\u003C\u002Ftd>\u003Ctd>\u003Ccode>lr=lang_en\u003C\u002Fcode>\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>\u003Ccode>uule\u003C\u002Fcode>\u003C\u002Ftd>\u003Ctd>Exact geographic location (city\u002Fdistrict)\u003C\u002Ftd>\u003Ctd>\u003Ccode>uule=w+CAIQICI...\u003C\u002Fcode>\u003C\u002Ftd>\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\n\u003Cp>Always add \u003Ccode>pws=0\u003C\u002Fcode> to disable personalization. This strips signed-in user history and provides a non-personalized baseline that's reproducible across runs — critical for rank tracking where you need consistent, comparable data over time.\u003C\u002Fp>\n\n\u003Ch3>Constructing the uule Value\u003C\u002Fh3>\n\n\u003Cp>The \u003Ccode>uule\u003C\u002Fcode> parameter encodes a canonical place name into a compact string. The format is a fixed prefix, a length byte, and a \u003Ca href=\"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FBase64\" rel=\"noopener noreferrer\">base64-encoded\u003C\u002Fa> canonical place name:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode>uule = "w+CAIQICI" + chr(len(canonical_name_bytes) + 63) + base64(canonical_name)\n\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>The canonical name follows the pattern \u003Ccode>City,State,Country\u003C\u002Fcode> — for example, \u003Ccode>Austin,Texas,United States\u003C\u002Fcode> or \u003Ccode>Brooklyn,New York,United States\u003C\u002Fcode>. The length byte uses a shifted ASCII encoding where \u003Ccode>chr(byte_length + 63)\u003C\u002Fcode> represents the raw byte count of the canonical name string.\u003C\u002Fp>\n\n\u003Cp>Here's a Python function that builds valid uule values:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode>import base64\n\ndef build_uule(canonical_name: str) -> str:\n    """Build a Google uule parameter from a canonical place name.\n\n    Canonical name format: 'City,State,Country'\n    Examples:\n      'Austin,Texas,United States'\n      'Brooklyn,New York,United States'\n      'Munich,Bavaria,Germany'\n    """\n    prefix = "w+CAIQICI"\n    name_bytes = canonical_name.encode("utf-8")\n    length_char = chr(len(name_bytes) + 63)\n    encoded = base64.b64encode(name_bytes).decode("utf-8")\n    return f"{prefix}{length_char}{encoded}"\n\n# Build uule for several cities\ncities = [\n    "Austin,Texas,United States",\n    "Brooklyn,New York,United States",\n    "Munich,Bavaria,Germany",\n]\nfor city in cities:\n    uule = build_uule(city)\n    print(f"{city:40s} -> uule={uule}")\n\n# Output:\n# Austin,Texas,United States               -> uule=w+CAIQICIYQXVsdGluLFRleGFzLFVuaXRlZCBTdGF0ZXM=\n# Brooklyn,New York,United States          -> uule=w+CAIQICIcYnJvb2tseW4sTmV3IFlvcmssVW5pdGVkIFN0YXRlcw==\n# Munich,Bavaria,Germany                    -> uule=w+CAIQICIXbXVuaWNoLEJhdmFyaWEsR2VybWFueQ==\n\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>When you pass this \u003Ccode>uule\u003C\u002Fcode> alongside \u003Ccode>gl\u003C\u002Fcode> and \u003Ccode>hl\u003C\u002Fcode>, Google applies the city-level geographic filter to the SERP. But there's a catch: Google cross-checks the uule location against the requester's IP geolocation. If your uule says \"Austin, Texas\" but your request comes from a datacenter IP in Frankfurt, Google may ignore the uule or trigger anti-bot verification.\u003C\u002Fp>\n\n\u003Ch2>Why Parameters Must Agree with IP Geography\u003C\u002Fh2>\n\n\u003Cp>Google's anti-abuse systems compare the geographic signals in your request. If \u003Ccode>gl=us\u003C\u002Fcode> and \u003Ccode>uule\u003C\u002Fcode> point to a US city, but the IP address resolves to a non-US location or a known datacenter ASN, the request is flagged as suspicious. This manifests as CAPTCHA challenges, 429 rate limits, or silently de-personalized results that don't reflect the target location.\u003C\u002Fp>\n\n\u003Cp>The solution is to route requests through \u003Ca href=\"\u002Fproxy\u002Fresidential\">residential proxies\u003C\u002Fa> whose exit IP geographically matches your target market. With ProxyHat, you can specify country and city in the proxy username, ensuring the IP geolocation aligns with your SERP parameters.\u003C\u002Fp>\n\n\u003Ch3>ProxyHat Proxy URL Formats\u003C\u002Fh3>\n\n\u003Cpre>\u003Ccode># HTTP proxy — US, New York\nhttp:\u002F\u002Fuser-country-US-region-new_york-city-new_york_city:pass@gate.proxyhat.com:8080\n\n# HTTP proxy — Germany, Berlin\nhttp:\u002F\u002Fuser-country-DE-region-state_of_berlin-city-berlin:pass@gate.proxyhat.com:8080\n\n# SOCKS5 proxy — US, New York\nsocks5:\u002F\u002Fuser-country-US-region-new_york-city-new_york_city:pass@gate.proxyhat.com:1080\n\n# Sticky session (same IP across requests)\nhttp:\u002F\u002Fuser-country-US-region-new_york-city-new_york_city-sid-abc123:pass@gate.proxyhat.com:8080\n\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>The raw proxy URL goes directly into your HTTP client configuration. For a more structured approach, you can wrap proxy URL construction in a helper class — a lightweight client pattern that keeps your scraping code clean:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode>from urllib.parse import quote\n\nclass ProxyHatClient:\n    """Helper for building ProxyHat residential proxy URLs."""\n\n    GATEWAY = "gate.proxyhat.com"\n    HTTP_PORT = 8080\n    SOCKS5_PORT = 1080\n\n    def __init__(self, username: str, password: str):\n        self.username = username\n        self.password = password\n\n    def http_url(self, country: str, region: str = None, city: str = None, session: str = None) -> str:\n        """Build an HTTP proxy URL with geo-targeting."""\n        user_part = f"{self.username}-country-{country.upper()}"\n        if region and city:\n            user_part += f"-region-{region}-city-{city.lower().replace(' ', '_')}"\n        elif region:\n            user_part += f"-region-{region}"\n        if session:\n            user_part += f"-sid-{session}"\n        return f"http:\u002F\u002F{user_part}:{self.password}@{self.GATEWAY}:{self.HTTP_PORT}"\n\n    def socks5_url(self, country: str, region: str = None, city: str = None, session: str = None) -> str:\n        """Build a SOCKS5 proxy URL with geo-targeting."""\n        user_part = f"{self.username}-country-{country.upper()}"\n        if region and city:\n            user_part += f"-region-{region}-city-{city.lower().replace(' ', '_')}"\n        elif region:\n            user_part += f"-region-{region}"\n        if session:\n            user_part += f"-sid-{session}"\n        return f"socks5:\u002F\u002F{user_part}:{self.password}@{self.GATEWAY}:{self.SOCKS5_PORT}"\n\n\n# --- Usage side by side: raw proxy URL vs helper ---\n\n# Raw proxy URL (direct string)\nraw_proxy = "http:\u002F\u002Fuser-country-US-region-new_york-city-new_york_city:pass@gate.proxyhat.com:8080"\n\n# ProxyHat helper (structured)\nclient = ProxyHatClient("user", "pass")\nhelper_proxy = client.http_url("US", "new_york", "new york city")\n\nprint("Raw:  ", raw_proxy)\nprint("Helper:", helper_proxy)\n# Both produce equivalent proxy URLs\n\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Ch2>Python Example: Rotating Residential IPs per Locale with curl_cffi\u003C\u002Fh2>\n\n\u003Cp>For SERP scraping, \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Flexiforest\u002Fcurl_cffi\" rel=\"noopener noreferrer\">curl_cffi\u003C\u002Fa> is a strong choice because it impersonates real browser TLS fingerprints, which helps avoid detection. Here's a complete example that builds uule values, rotates residential IPs per locale, and fetches SERPs with Chrome impersonation:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode>import base64\nimport time\nimport logging\nfrom curl_cffi import requests as cffi_requests\n\nlogging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")\nlogger = logging.getLogger(__name__)\n\ndef build_uule(canonical_name: str) -> str:\n    prefix = "w+CAIQICI"\n    name_bytes = canonical_name.encode("utf-8")\n    length_char = chr(len(name_bytes) + 63)\n    encoded = base64.b64encode(name_bytes).decode("utf-8")\n    return f"{prefix}{length_char}{encoded}"\n\ndef build_proxy_url(country: str, region: str, city: str, username: str, password: str) -> str:\n    """Build a ProxyHat residential proxy URL for the target locale."""\n    city_slug = city.lower().replace(" ", "_")\n    return (\n        f"http:\u002F\u002F{username}-country-{country.upper()}-region-{region}-city-{city_slug}"\n        f":{password}@gate.proxyhat.com:8080"\n    )\n\ndef fetch_serp(\n    keyword: str,\n    country: str,\n    region: str,\n    city: str,\n    canonical_name: str,\n    username: str,\n    password: str,\n    num: int = 100,\n) -> str:\n    """Fetch a geo-targeted Google SERP page."""\n    uule = build_uule(canonical_name)\n    proxy = build_proxy_url(country, region, city, username, password)\n\n    params = {\n        "q": keyword,\n        "gl": country.lower(),\n        "hl": "en",\n        "uule": uule,\n        "pws": "0",\n        "num": str(num),\n    }\n\n    headers = {\n        "Accept-Language": f"en-{country.upper()}",\n        "User-Agent": (\n            "Mozilla\u002F5.0 (Windows NT 10.0; Win64; x64) "\n            "AppleWebKit\u002F537.36 (KHTML, like Gecko) "\n            "Chrome\u002F120.0.0.0 Safari\u002F537.36"\n        ),\n    }\n\n    logger.info("Fetching SERP: keyword='%s' city=%s proxy=%s", keyword, city, proxy)\n\n    try:\n        response = cffi_requests.get(\n            "https:\u002F\u002Fwww.google.com\u002Fsearch",\n            params=params,\n            headers=headers,\n            proxies={"http": proxy, "https": proxy},\n            impersonate="chrome",\n            timeout=30,\n        )\n        response.raise_for_status()\n        logger.info("SERP fetched: %d bytes, status=%d", len(response.text), response.status_code)\n        return response.text\n    except Exception as exc:\n        logger.error("SERP fetch failed for %s\u002F%s: %s", country, city, exc)\n        raise\n\n# --- Run for multiple US cities ---\ntargets = [\n    ("US", "new_york", "new york city", "New York,New York,United States"),\n    ("US", "texas", "austin", "Austin,Texas,United States"),\n    ("US", "illinois", "chicago", "Chicago,Illinois,United States"),\n    ("US", "florida", "miami", "Miami,Florida,United States"),\n]\n\nUSERNAME = "user"\nPASSWORD = "pass"\n\nfor country, region, city, canonical in targets:\n    html = fetch_serp("plumber", country, region, city, canonical, USERNAME, PASSWORD)\n    # Save or parse the HTML\n    with open(f"serp_{city.replace(' ', '_')}.html", "w", encoding="utf-8") as f:\n        f.write(html)\n    time.sleep(2)  # Be polite between requests\n\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>This example uses \u003Ccode>impersonate=\"chrome\"\u003C\u002Fcode> to send a TLS fingerprint that matches a real Chrome browser, significantly reducing the chance of bot detection compared to a standard \u003Ccode>requests\u003C\u002Fcode> call. Each city gets its own residential proxy exit IP via the \u003Ccode>-country-US-region-...-city-...\u003C\u002Fcode> flag, ensuring Google sees a consistent geographic story.\u003C\u002Fp>\n\n\u003Ch2>Parsing Organic Results and the Local Pack with selectolax\u003C\u002Fh2>\n\n\u003Cp>Once you have the HTML, you need to extract structured data. \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Frushter\u002Fselectolax\" rel=\"noopener noreferrer\">selectolax\u003C\u002Fa> is a fast HTML parser that's ideal for SERP extraction. Here's a parser that pulls organic results, the local pack, and related searches:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode>from selectolax.parser import HTMLParser\nimport json\n\ndef parse_serp(html: str) -> dict:\n    """Parse a Google SERP HTML page into structured data."""\n    tree = HTMLParser(html)\n    results = {"organic": [], "local_pack": [], "related": []}\n\n    # --- Organic results ---\n    for node in tree.css("div.g"):\n        title_node = node.css_first("h3")\n        link_node = node.css_first("a[href]")\n        snippet_node = node.css_first("div[data-sncf], span.aCOpRe, div.VwiC3b")\n\n        if title_node and link_node:\n            results["organic"].append({\n                "title": title_node.text(strip=True),\n                "url": link_node.attributes.get("href", ""),\n                "snippet": snippet_node.text(strip=True) if snippet_node else "",\n            })\n\n    # --- Local pack (map results) ---\n    for node in tree.css("div[jsmodel] div[data-cid]"):\n        name_node = node.css_first("span.OSrXXb, div.dbg0pd")\n        rating_node = node.css_first("span.YDINWc, span.BTpUQb")\n        if name_node:\n            entry = {\n                "name": name_node.text(strip=True),\n                "rating": rating_node.text(strip=True) if rating_node else "",\n            }\n            results["local_pack"].append(entry)\n\n    # --- Related searches ---\n    for node in tree.css("a.k8XOCe"):\n        text = node.text(strip=True)\n        if text:\n            results["related"].append(text)\n\n    return results\n\n# --- Parse a saved SERP ---\nwith open("serp_austin.html", "r", encoding="utf-8") as f:\n    html = f.read()\n\nparsed = parse_serp(html)\nprint(json.dumps(parsed, indent=2, ensure_ascii=False))\n\n# Example output:\n# {\n#   "organic": [\n#     {"title": "Austin Plumbing - 24\u002F7 Emergency Service", "url": "https:\u002F\u002F...", "snippet": "..."},\n#     ...\n#   ],\n#   "local_pack": [\n#     {"name": "ABC Plumbing Austin", "rating": "4.8"},\n#     ...\n#   ],\n#   "related": ["plumber near me", "emergency plumber austin", ...]\n# }\n\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>Google's HTML structure changes frequently, so CSS selectors may need periodic adjustment. The key is to target structural attributes (like \u003Ccode>data-cid\u003C\u002Fcode> for local pack entries) rather than class names that change with each layout update.\u003C\u002Fp>\n\n\u003Ch2>Sticky Sessions for Multi-Page SERP Collection\u003C\u002Fh2>\n\n\u003Cp>When paginating through SERPs (e.g., collecting results beyond the first 100), you need the same IP across all page requests. If the IP rotates mid-pagination, Google may serve inconsistent results or trigger a CAPTCHA. ProxyHat sticky sessions solve this — append \u003Ccode>-sid-{id}\u003C\u002Fcode> to the proxy username to pin a specific exit IP:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode>import uuid\nimport time\nimport logging\nfrom curl_cffi import requests as cffi_requests\n\nlogger = logging.getLogger(__name__)\n\ndef fetch_serp_paginated(\n    keyword: str,\n    country: str,\n    region: str,\n    city: str,\n    canonical_name: str,\n    username: str,\n    password: str,\n    max_pages: int = 3,\n) -> list[str]:\n    """Fetch multiple SERP pages using a sticky session for IP consistency."""\n    session_id = uuid.uuid4().hex[:12]\n    city_slug = city.lower().replace(" ", "_")\n\n    proxy = (\n        f"http:\u002F\u002F{username}-country-{country.upper()}-region-{region}-city-{city_slug}"\n        f"-sid-{session_id}:{password}@gate.proxyhat.com:8080"\n    )\n\n    pages = []\n    for page in range(max_pages):\n        start = page * 100  # num=100 per page\n        params = {\n            "q": keyword,\n            "gl": country.lower(),\n            "hl": "en",\n            "uule": build_uule(canonical_name),\n            "pws": "0",\n            "num": "100",\n            "start": str(start),\n        }\n        headers = {\n            "Accept-Language": f"en-{country.upper()}",\n            "User-Agent": (\n                "Mozilla\u002F5.0 (Windows NT 10.0; Win64; x64) "\n                "AppleWebKit\u002F537.36 (KHTML, like Gecko) "\n                "Chrome\u002F120.0.0.0 Safari\u002F537.36"\n            ),\n        }\n\n        logger.info("Page %d: start=%d session=%s", page + 1, start, session_id)\n        try:\n            resp = cffi_requests.get(\n                "https:\u002F\u002Fwww.google.com\u002Fsearch",\n                params=params,\n                headers=headers,\n                proxies={"http": proxy, "https": proxy},\n                impersonate="chrome",\n                timeout=30,\n            )\n            resp.raise_for_status()\n            pages.append(resp.text)\n            logger.info("Page %d fetched: %d bytes", page + 1, len(resp.text))\n        except Exception as exc:\n            logger.error("Page %d failed: %s", page + 1, exc)\n            break\n\n        time.sleep(3)  # Delay between pages\n\n    return pages\n\ndef build_uule(canonical_name: str) -> str:\n    import base64\n    prefix = "w+CAIQICI"\n    name_bytes = canonical_name.encode("utf-8")\n    length_char = chr(len(name_bytes) + 63)\n    encoded = base64.b64encode(name_bytes).decode("utf-8")\n    return f"{prefix}{length_char}{encoded}"\n\n# Fetch 3 pages of results for "plumber" in Austin\npages = fetch_serp_paginated(\n    keyword="plumber",\n    country="US",\n    region="texas",\n    city="austin",\n    canonical_name="Austin,Texas,United States",\n    username="user",\n    password="pass",\n    max_pages=3,\n)\nprint(f"Collected {len(pages)} SERP pages")\n\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>The session ID persists the same residential IP across all page requests. Generate a new session ID for each new keyword or locale to get a fresh IP, preventing pattern detection from repeated use of the same IP across many queries.\u003C\u002Fp>\n\n\u003Ch2>CAPTCHA, 429 Backoff, and Retry Logic\u003C\u002Fh2>\n\n\u003Cp>Even with residential proxies and browser impersonation, Google may still serve CAPTCHAs or return HTTP 429 (Too Many Requests) under aggressive scraping. A robust pipeline needs exponential backoff with jitter and per-country proxy pools to distribute load:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode>import random\nimport time\nimport logging\nfrom curl_cffi import requests as cffi_requests\n\nlogger = logging.getLogger(__name__)\n\nclass SERPScraper:\n    """SERP scraper with retry, backoff, and per-country proxy rotation."""\n\n    MAX_RETRIES = 4\n    BASE_DELAY = 5  # seconds\n    MAX_DELAY = 120  # seconds\n\n    def __init__(self, username: str, password: str):\n        self.username = username\n        self.password = password\n        # Per-country proxy pools (rotate IPs within a country)\n        self._proxy_counter: dict[str, int] = {}\n\n    def _next_proxy(self, country: str, region: str, city: str, session: str = None) -> str:\n        """Rotate to the next proxy in the country pool."""\n        count = self._proxy_counter.get(country, 0)\n        self._proxy_counter[country] = count + 1\n        city_slug = city.lower().replace(" ", "_")\n        user_part = f"{self.username}-country-{country.upper()}-region-{region}-city-{city_slug}"\n        if session:\n            user_part += f"-sid-{session}"\n        else:\n            # Use counter as session to get different IPs\n            user_part += f"-sid-rot{count}"\n        return f"http:\u002F\u002F{user_part}:{self.password}@gate.proxyhat.com:8080"\n\n    def fetch_with_retry(\n        self,\n        url: str,\n        params: dict,\n        country: str,\n        region: str,\n        city: str,\n        session: str = None,\n    ) -> str | None:\n        """Fetch with exponential backoff and jitter."""\n        headers = {\n            "Accept-Language": f"en-{country.upper()}",\n            "User-Agent": (\n                "Mozilla\u002F5.0 (Windows NT 10.0; Win64; x64) "\n                "AppleWebKit\u002F537.36 (KHTML, like Gecko) "\n                "Chrome\u002F120.0.0.0 Safari\u002F537.36"\n            ),\n        }\n\n        for attempt in range(self.MAX_RETRIES):\n            proxy = self._next_proxy(country, region, city, session)\n            try:\n                resp = cffi_requests.get(\n                    url,\n                    params=params,\n                    headers=headers,\n                    proxies={"http": proxy, "https": proxy},\n                    impersonate="chrome",\n                    timeout=30,\n                )\n\n                if resp.status_code == 200:\n                    # Check for CAPTCHA in HTML\n                    if "captcha" in resp.text.lower() or "unusual traffic" in resp.text.lower():\n                        logger.warning("CAPTCHA detected on attempt %d", attempt + 1)\n                        delay = self._backoff_delay(attempt)\n                        logger.info("Backing off %.1fs", delay)\n                        time.sleep(delay)\n                        continue\n                    logger.info("Success on attempt %d via %s", attempt + 1, country)\n                    return resp.text\n\n                elif resp.status_code == 429:\n                    delay = self._backoff_delay(attempt)\n                    logger.warning("429 rate limited. Backing off %.1fs", delay)\n                    time.sleep(delay)\n                    continue\n\n                else:\n                    logger.error("HTTP %d on attempt %d", resp.status_code, attempt + 1)\n                    if resp.status_code >= 500:\n                        delay = self._backoff_delay(attempt)\n                        time.sleep(delay)\n                        continue\n                    return None  # Client error, don't retry\n\n            except Exception as exc:\n                delay = self._backoff_delay(attempt)\n                logger.error("Request failed (attempt %d): %s. Backoff %.1fs", attempt + 1, exc, delay)\n                time.sleep(delay)\n\n        logger.error("All %d retries exhausted for %s", self.MAX_RETRIES, country)\n        return None\n\n    def _backoff_delay(self, attempt: int) -> float:\n        """Exponential backoff with jitter."""\n        delay = min(self.BASE_DELAY * (2 ** attempt), self.MAX_DELAY)\n        jitter = random.uniform(0, delay * 0.3)\n        return delay + jitter\n\n# --- Usage ---\nscraper = SERPScraper("user", "pass")\nimport base64\n\ndef build_uule(canonical_name: str) -> str:\n    prefix = "w+CAIQICI"\n    name_bytes = canonical_name.encode("utf-8")\n    length_char = chr(len(name_bytes) + 63)\n    encoded = base64.b64encode(name_bytes).decode("utf-8")\n    return f"{prefix}{length_char}{encoded}"\n\nhtml = scraper.fetch_with_retry(\n    url="https:\u002F\u002Fwww.google.com\u002Fsearch",\n    params={\n        "q": "coffee shop",\n        "gl": "us",\n        "hl": "en",\n        "uule": build_uule("Seattle,Washington,United States"),\n        "pws": "0",\n        "num": "100",\n    },\n    country="US",\n    region="washington",\n    city="seattle",\n)\nif html:\n    print(f"Got SERP: {len(html)} bytes")\nelse:\n    print("Failed to fetch SERP after retries")\n\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>Key design decisions in this retry logic:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>CAPTCHA detection:\u003C\u002Fstrong> Check the response body for \u003Ccode>captcha\u003C\u002Fcode> or \u003Ccode>unusual traffic\u003C\u002Fcode> strings — Google sometimes returns HTTP 200 with a CAPTCHA page instead of a proper error code.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Exponential backoff with jitter:\u003C\u002Fstrong> Start at 5 seconds, double each attempt, cap at 120 seconds, add 0–30% random jitter to avoid thundering herd effects.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Per-country proxy rotation:\u003C\u002Fstrong> Each retry uses a different session ID, forcing ProxyHat to assign a new residential IP within the same geographic pool.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>429 handling:\u003C\u002Fstrong> Rate limit responses trigger backoff but don't consume the full retry budget aggressively — the jitter helps avoid synchronized retries across workers.\u003C\u002Fli>\n\u003C\u002Ful>\n\n\u003Ch2>Per-Country Proxy Pools and Concurrency\u003C\u002Fh2>\n\n\u003Cp>For production SERP collection across multiple markets, organize your proxy pools by country. This prevents one market's rate limits from cascading into others and lets you tune concurrency per locale. A practical setup:\u003C\u002Fp>\n\n\u003Cul>\n\u003Cli>US markets: 50–100 concurrent sessions across cities, 2–3 second delay between requests per session.\u003C\u002Fli>\n\u003Cli>EU markets: 30–50 concurrent sessions, respect local business hours for fresher results.\u003C\u002Fli>\n\u003Cli>Asia-Pacific: 20–40 concurrent sessions, account for higher latency (200ms+ RTT to some regions).\u003C\u002Fli>\n\u003C\u002Ful>\n\n\u003Cp>Monitor success rates per country pool. If a pool's success rate drops below 90%, reduce concurrency or increase delays for that market. ProxyHat residential proxies typically achieve 99.9% uptime, but individual IPs may get temporarily flagged — rotation handles this naturally.\u003C\u002Fp>\n\n\u003Ch2>curl Example: Raw Proxy with uule\u003C\u002Fh2>\n\n\u003Cp>For quick testing or shell-based pipelines, here's a curl command using a ProxyHat residential proxy with full geo-targeting:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode># Fetch SERP for "plumber" in Austin, TX via US residential proxy\n# uule value pre-computed: w+CAIQICIYQXVsdGluLFRleGFzLFVuaXRlZCBTdGF0ZXM=\n\ncurl -s \\\n  --proxy "http:\u002F\u002Fuser-country-US-region-texas-city-austin:pass@gate.proxyhat.com:8080" \\\n  -H "Accept-Language: en-US" \\\n  -H "User-Agent: Mozilla\u002F5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\u002F537.36 (KHTML, like Gecko) Chrome\u002F120.0.0.0 Safari\u002F537.36" \\\n  "https:\u002F\u002Fwww.google.com\u002Fsearch?q=plumber&gl=us&hl=en&uule=w+CAIQICIYQXVsdGluLFRleGFzLFVuaXRlZCBTdGF0ZXM%3D&pws=0&num=100" \\\n  -o serp_austin.html\n\necho "Saved $(wc -c < serp_austin.html) bytes"\n\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>Note the \u003Ccode>%3D\u003C\u002Fcode> URL-encoding of the \u003Ccode>=\u003C\u002Fcode> character in the uule value. Always URL-encode uule when passing it in a query string.\u003C\u002Fp>\n\n\u003Ch2>Public Data and Terms of Service Considerations\u003C\u002Fh2>\n\n\u003Cp>Scraping Google SERPs operates in a legally gray area. Google's Terms of Service prohibit automated querying, and the \u003Ca href=\"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FHiQ_Labs_v._LinkedIn\" rel=\"noopener noreferrer\">hiQ Labs v. LinkedIn\u003C\u002Fa> precedent in the US suggests that scraping publicly accessible data may be permissible, but this varies by jurisdiction. In the EU, GDPR may apply to any personal data inadvertently collected. Practical guidelines:\u003C\u002Fp>\n\n\u003Cul>\n\u003Cli>Respect \u003Ccode>robots.txt\u003C\u002Fcode> directives where technically feasible.\u003C\u002Fli>\n\u003Cli>Rate-limit your requests to avoid degrading Google's service (2–5 seconds between requests per session).\u003C\u002Fli>\n\u003Cli>Collect only the data you need — organic titles, URLs, snippets, and local pack entries. Avoid storing user-specific personalization data.\u003C\u002Fli>\n\u003Cli>Consider using the \u003Ca href=\"https:\u002F\u002Fdevelopers.google.com\u002Fcustom-search\u002Fdocs\u002Foverview\" rel=\"noopener noreferrer\">Google Custom Search JSON API\u003C\u002Fa> for lower-volume, ToS-compliant data collection (limited to 100 queries\u002Fday on the free tier).\u003C\u002Fli>\n\u003Cli>Consult legal counsel before deploying large-scale scraping infrastructure, especially across multiple jurisdictions.\u003C\u002Fli>\n\u003C\u002Ful>\n\n\u003Ch2>ProxyHat Setup and Resources\u003C\u002Fh2>\n\n\u003Cp>ProxyHat residential proxies provide city-level geo-targeting across 190+ countries. To get started:\u003C\u002Fp>\n\n\u003Col>\n\u003Cli>Sign up at \u003Ca href=\"\u002Fpricing\">ProxyHat pricing\u003C\u002Fa> and choose a residential proxy plan.\u003C\u002Fli>\n\u003Cli>Check available \u003Ca href=\"\u002Flocations\">proxy locations\u003C\u002Fa> to confirm coverage for your target markets.\u003C\u002Fli>\n\u003Cli>Review the \u003Ca href=\"https:\u002F\u002Fdocs.proxyhat.com\u002F\" rel=\"noopener noreferrer\">ProxyHat documentation\u003C\u002Fa> for advanced session and rotation options.\u003C\u002Fli>\n\u003Cli>Explore \u003Ca href=\"\u002Fuse-cases\u002Fweb-scraping\">web scraping use cases\u003C\u002Fa> and \u003Ca href=\"\u002Fuse-cases\u002Fserp-tracking\">SERP tracking\u003C\u002Fa> for implementation patterns.\u003C\u002Fli>\n\u003C\u002Fol>\n\n\u003Ch2>Key Takeaways\u003C\u002Fh2>\n\n\u003Cblockquote>\n\u003Cul>\n\u003Cli>Country-level targeting is insufficient for local SEO. Use \u003Ccode>uule\u003C\u002Fcode> with city-level canonical names to pin results to specific markets.\u003C\u002Fli>\n\u003Cli>Parameters must agree with IP geography. Route requests through residential proxies matching your target locale — Google cross-checks IP geolocation against \u003Ccode>gl\u003C\u002Fcode> and \u003Ccode>uule\u003C\u002Fcode>.\u003C\u002Fli>\n\u003Cli>Always add \u003Ccode>pws=0\u003C\u002Fcode> to disable personalization and get reproducible baseline SERPs for rank tracking.\u003C\u002Fli>\n\u003Cli>Use sticky sessions for pagination to maintain IP consistency across multi-page SERP collection.\u003C\u002Fli>\n\u003Cli>Implement exponential backoff with jitter and CAPTCHA detection — Google may return 200 with a CAPTCHA page rather than an error code.\u003C\u002Fli>\n\u003Cli>Organize proxy pools by country to isolate rate limits and tune concurrency per market.\u003C\u002Fli>\n\u003Cli>Use \u003Ccode>curl_cffi\u003C\u002Fcode> with \u003Ccode>impersonate=\"chrome\"\u003C\u002Fcode> to send realistic TLS fingerprints that reduce detection rates.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003C\u002Fblockquote>","Geo-Targeted SERP Scraping Best Practices | ProxyHat","Master geo-targeted SERP scraping with uule, gl, hl parameters and city-level residential proxies. Python code, retry logic, and production tips included.","\u002Fblog\u002Fgeo-targeted-serp-scraping-best-practices",null,"https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Fgeo-targeted-serp-scraping-best-practices-og.webp",[17,20,23,26,29],{"question":18,"answer":19},"What is geo-targeted SERP scraping?","Geo-targeted SERP scraping is the practice of collecting Google search results pinned to a specific geographic location — typically a city or district — rather than just a country. It uses parameters like uule, gl, and hl alongside geographically matching residential proxies to ensure the SERP reflects what a local user would see. This is essential for local SEO rank tracking, competitive analysis, and market research where results vary by location.",{"question":21,"answer":22},"Why does geo-targeted SERP scraping matter for proxy users?","Google personalizes search results based on the requester's IP geolocation. If your proxy IP doesn't match your target market, Google may ignore your uule and gl parameters or serve de-personalized results. Proxy users need residential proxies with city-level geo-targeting to ensure the IP geolocation aligns with the SERP parameters, producing accurate local results.",{"question":24,"answer":25},"Which proxy type works best for geo-targeted SERP scraping?","Residential proxies are the best choice for geo-targeted SERP scraping because they use real ISP-assigned IP addresses that Google trusts. Datacenter IPs are frequently flagged by Google's anti-bot systems, leading to CAPTCHAs and blocked requests. Mobile proxies also work well but are more expensive. Residential proxies with city-level targeting provide the best balance of reliability, geographic precision, and cost.",{"question":27,"answer":28},"How do you avoid blocks when implementing geo-targeted SERP scraping?","To avoid blocks: use residential proxies with geo-targeting that matches your SERP parameters, impersonate a real browser TLS fingerprint with curl_cffi, add pws=0 for non-personalized baselines, implement exponential backoff with jitter for 429s and CAPTCHAs, use sticky sessions for multi-page pagination, maintain 2-5 second delays between requests per session, and rotate IPs across per-country proxy pools.",{"question":30,"answer":31},"How do you construct a uule parameter for city-level targeting?","The uule parameter is built from a fixed prefix (w+CAIQICI), a length byte (chr(len(canonical_name_bytes) + 63)), and the base64-encoded canonical place name (e.g., Austin,Texas,United States). Pass the resulting string as the uule query parameter alongside gl and hl to pin SERP results to that specific city.",false,"2026-07-31T16:11:16+00:00","2026-10-03T23:52:59+00:00","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Fgeo-targeted-serp-scraping-best-practices-master.webp","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Fgeo-targeted-serp-scraping-best-practices-card.webp",12,{"name":39,"slug":40,"bio":41,"avatar_url":14,"website_url":14,"social_links":14},"ProxyHat Team","proxyhat-team","The ProxyHat editorial team covers proxies, web scraping, and data extraction.",[43],{"id":44,"slug":45,"name":46,"description":14,"meta_title":47,"meta_description":48,"parent_id":14},3,"serp-seo-tracking","SERP & SEO Tracking","SERP & SEO Tracking Guides | ProxyHat Blog","Learn how to track search engine rankings and scrape SERPs with proxies.",[],{"en":13,"x-default":13},{"zh":52,"ja":53,"pl":54,"ko":55,"es":56,"it":57,"tr":58,"en":13,"ru":59,"de":60,"fr":61,"pt":62},"\u002Fcn\u002Fblog\u002Fserp-dili-dingwei-zhuaqu-zhinan","\u002Fja\u002Fblog\u002Fgeo-targeted-serp-scraping-guide","\u002Fpl\u002Fblog\u002Fgeotargetowane-scrapowanie-serp-najlepsze-praktyki","\u002Fko\u002Fblog\u002Fgeo-targeted-serp-scraping-guide","\u002Fes\u002Fblog\u002Fscraping-serp-geo-segmentado-mejores-practicas","\u002Fit\u002Fblog\u002Fscraping-serp-geo-targetato-best-practices","\u002Ftr\u002Fblog\u002Fcografi-hedefli-serp-kazima-rehberi","\u002Fru\u002Fblog\u002Fgeo-targeted-serp-scraping-best-practices","\u002Fde\u002Fblog\u002Fgeo-targeted-serp-scraping-leitfaden","\u002Ffr\u002Fblog\u002Fscraping-serp-geocible-meilleures-pratiques","\u002Fpt\u002Fblog\u002Fpraticas-serp-scraping-geo-targeted",59,[65,82,97,113],{"id":66,"slug":67,"title":68,"excerpt":69,"featured_image_url":70,"featured_image_alt":71,"card_image_url":72,"og_image_url":73,"published_at":74,"updated_at":74,"content_updated_at":75,"reading_time_minutes":76,"is_featured":32,"indexable":4,"views":77,"author":78,"categories":79,"tags":81},155,"amazon-keyword-rank-tracking-proxies-1","Marketplace Keyword Rank Tracking with Proxies: A Developer's Guide (Amazon Example)","Learn how to build an Amazon keyword rank tracker with Python, residential proxies, and geo-targeting. Parse search results, separate organic from Sponsored, and track ASIN positions over time.","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Famazon-keyword-rank-tracking-proxies-1-master.webp","Amazon Keyword Rank Tracking with Proxies: A Developer's Guide","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Famazon-keyword-rank-tracking-proxies-1-card.webp","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Famazon-keyword-rank-tracking-proxies-1-og.webp","2026-07-16T16:05:31+00:00","2026-10-04T01:29:51+00:00",13,98,{"name":39,"slug":40,"bio":41,"avatar_url":14,"website_url":14,"social_links":14},[80],{"id":44,"slug":45,"name":46,"description":14,"meta_title":47,"meta_description":48,"parent_id":14},[],{"id":83,"slug":84,"title":85,"excerpt":86,"featured_image_url":87,"featured_image_alt":85,"card_image_url":88,"og_image_url":89,"published_at":90,"updated_at":90,"content_updated_at":91,"reading_time_minutes":37,"is_featured":32,"indexable":32,"views":92,"author":93,"categories":94,"tags":96},153,"scrape-google-paa-autocomplete-keyword-research-1","Scrape Google People Also Ask and Autocomplete for Keyword Research (Python Guide)","A code-first guide to extracting Google Autocomplete suggestions, People Also Ask questions, and Related Searches at scale using Python, Playwright, and residential proxies for clean localized keyword research.","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Fscrape-google-paa-autocomplete-keyword-research-1-master.webp","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Fscrape-google-paa-autocomplete-keyword-research-1-card.webp","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Fscrape-google-paa-autocomplete-keyword-research-1-og.webp","2026-07-14T16:06:25+00:00","2026-10-03T08:29:54+00:00",124,{"name":39,"slug":40,"bio":41,"avatar_url":14,"website_url":14,"social_links":14},[95],{"id":44,"slug":45,"name":46,"description":14,"meta_title":47,"meta_description":48,"parent_id":14},[],{"id":98,"slug":99,"title":100,"excerpt":101,"featured_image_url":102,"featured_image_alt":100,"card_image_url":103,"og_image_url":104,"published_at":105,"updated_at":105,"content_updated_at":106,"reading_time_minutes":107,"is_featured":32,"indexable":4,"views":108,"author":109,"categories":110,"tags":112},154,"track-brand-google-ai-overviews-1","How to Track Your Brand in Google AI Overviews: A Strategic Guide for SEO & Data Teams","Generative search has made citation share the new ranking KPI. Here's how to track your brand in Google AI Overviews with a proxy-powered, headless-browser pipeline—plus build-vs-buy economics and governance guardrails.","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Ftrack-brand-google-ai-overviews-1-master.webp","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Ftrack-brand-google-ai-overviews-1-card.webp","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Ftrack-brand-google-ai-overviews-1-og.webp","2026-07-14T10:05:37+00:00","2026-10-03T08:29:51+00:00",10,66,{"name":39,"slug":40,"bio":41,"avatar_url":14,"website_url":14,"social_links":14},[111],{"id":44,"slug":45,"name":46,"description":14,"meta_title":47,"meta_description":48,"parent_id":14},[],{"id":114,"slug":115,"title":116,"excerpt":117,"featured_image_url":118,"featured_image_alt":116,"card_image_url":119,"og_image_url":120,"published_at":121,"updated_at":121,"content_updated_at":122,"reading_time_minutes":37,"is_featured":32,"indexable":32,"views":123,"author":124,"categories":125,"tags":127},152,"build-rank-tracker-python-proxies","Build a Google Rank Tracker in Python with Residential Proxies","A complete, code-first guide to building a production Google rank tracker in Python using curl_cffi for TLS impersonation and ProxyHat residential proxies with city-level geo-targeting, sticky sessions, and SQLite storage.","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Fbuild-rank-tracker-python-proxies-master.webp","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Fbuild-rank-tracker-python-proxies-card.webp","https:\u002F\u002Fstatic.proxyhat.com\u002Fstorage\u002Fblog-covers\u002Fen\u002Fbuild-rank-tracker-python-proxies-og.webp","2026-07-02T16:04:58+00:00","2026-10-03T23:53:00+00:00",56,{"name":39,"slug":40,"bio":41,"avatar_url":14,"website_url":14,"social_links":14},[126],{"id":44,"slug":45,"name":46,"description":14,"meta_title":47,"meta_description":48,"parent_id":14},[],[]]