Scrape Google People Also Ask and Autocomplete for Keyword Research (Python Guide)

Keyword research tools are expensive, but the raw data is sitting right in Google's search interface — free for anyone who knows how to collect it responsibly. If you want to scrape People Also Ask boxes, harvest Google Autocomplete suggestions, and pull Related Searches at scale, this guide walks through the full stack: from a single HTTP call to a recursive, proxy-backed pipeline that turns one seed keyword into hundreds of long-tail opportunities.

How to Scrape Google People Also Ask and Autocomplete for Keyword Research

Google exposes three free keyword goldmines that every SEO tool under the hood relies on to some degree:

• Autocomplete (Suggest) — the dropdown predictions you see as you type. These are query-level, intent-rich suggestions ranked by popularity and freshness. They map directly to search intent because Google only suggests queries that real users actually search.
• People Also Ask (PAA) — the expandable question boxes on the SERP. Each PAA question reveals a sub-intent, and clicking one expands it to reveal an answer plus a cited source URL — and critically, it spawns 2–4 new nested questions. This recursion is a goldmine for FAQ-page and content-cluster planning.
• Related Searches — the list at the bottom of the SERP. These are broader, co-occurring queries that help you map topical neighborhoods and find adjacent keywords you might not have considered.

Each source maps to a different layer of intent. Autocomplete gives you long-tail variations; PAA gives you question-format intent (informational, often mid-funnel); Related Searches gives you topical adjacency. Combined, they form a lightweight keyword-research dataset that rivals what you'd pay $50–$200/month for in a commercial SaaS tool.

Why Google Blocks Rapid Autocomplete and PAA Scraping

The Autocomplete endpoint — suggestqueries.google.com/complete/search?client=chrome&q= — is fast and returns clean JSON, which makes it tempting to hammer. But Google enforces per-IP rate limits that trigger after roughly 100–200 rapid requests from the same IP within a short window. When you hit the limit, you'll see empty responses, HTTP 429s, or CAPTCHA challenges. The same applies to repeated SERP fetches for PAA extraction.

There's a second, subtler problem: locale bias. Google personalizes Autocomplete and PAA results based on the IP's geolocation. If your server is in Virginia, you'll get US-centric suggestions even when you're researching the German or Japanese market. This makes raw datacenter IPs unreliable for localized keyword research. Google's own documentation on structured data and search features confirms that results are influenced by location and language settings, and the broader behavior of suggestion algorithms is discussed in public resources like the Google Suggest Wikipedia article.

This is why residential proxies with country and city geo-targeting are essential. With ProxyHat, you can route requests through real ISP-assigned IPs in a specific location — for example, user-country-DE-city-berlin — so Google returns genuinely localized suggestions as if a real Berlin resident typed them.

ProxyHat Setup: Residential Proxies for Localized Suggestions

ProxyHat provides a single gateway endpoint that accepts geo-targeting and session flags directly in the username. The gateway is gate.proxyhat.com, HTTP on port 8080, SOCKS5 on port 1080.

For keyword research, you want rotating residential IPs with sticky sessions so that a multi-step PAA expansion (which requires several page loads) stays on the same IP, while different keywords rotate to fresh IPs to avoid rate limits.

For this guide, residential proxies are the sweet spot. Check ProxyHat pricing for current rates, and browse available locations to confirm your target country/city is supported. More context on proxy-aware scraping is in our web scraping use case and SERP tracking use case.

Code Block 1: Fetch Autocomplete JSON with httpx + ProxyHat

The Chrome client endpoint returns a lightweight JSON array. Here's a production-ready function with retries, error handling, and proxy integration:

import httpx
import time
import json
from typing import Optional

def fetch_autocomplete(
    query: str,
    country: str = "US",
    city: Optional[str] = None,
    session_id: Optional[str] = None,
    max_retries: int = 3,
) -> list[str]:
    """Fetch Google Autocomplete suggestions for a query.

    Args:
        query: The seed keyword.
        country: ISO country code for geo-targeting.
        city: Optional city name (lowercase, no spaces).
        session_id: Optional sticky session identifier.

    Returns:
        List of suggestion strings.
    """
    # Build ProxyHat username with geo + session flags
    username_parts = [f"user-country-{country}"]
    if city:
        username_parts.append(f"city-{city}")
    if session_id:
        username_parts.append(f"session-{session_id}")
    username = "-".join(username_parts)
    password = "YOUR_PROXYHAT_PASSWORD"

    proxy_url = f"http://{username}:{password}@gate.proxyhat.com:8080"

    # Chrome client returns clean JSON: [query, [suggestions], ...]
    url = "https://suggestqueries.google.com/complete/search"
    params = {"client": "chrome", "q": query, "hl": "en"}

    for attempt in range(max_retries):
        try:
            with httpx.Client(proxy=proxy_url, timeout=15.0) as client:
                resp = client.get(url, params=params)
                resp.raise_for_status()
                data = resp.json()
                # data[1] is the list of suggestion strings
                suggestions = data[1] if len(data) > 1 else []
                return suggestions
        except (httpx.HTTPStatusError, httpx.ProxyError, json.JSONDecodeError) as exc:
            if attempt == max_retries - 1:
                raise
            backoff = 2 ** attempt
            print(f"Retry {attempt + 1}/{max_retries} after {backoff}s: {exc}")
            time.sleep(backoff)
    return []

# --- Usage ---
if __name__ == "__main__":
    suggestions = fetch_autocomplete("best running shoes", country="DE", city="berlin")
    for s in suggestions:
        print(s)

This returns something like ['best running shoes for men', 'best running shoes for flat feet', 'best running shoes 2024', ...]. The client=chrome parameter is key — it returns a compact JSON array instead of the XML or HTML that other clients produce.

Code Block 2: Seed-to-Longtail Expander with a–z and Modifiers

One keyword becomes hundreds when you append every letter of the alphabet and common modifier prefixes. This is the same technique commercial keyword tools use internally:

import itertools
import concurrent.futures
from fetch_autocomplete import fetch_autocomplete  # from Code Block 1

ALPHABET = list("abcdefghijklmnopqrstuvwxyz")
MODIFIERS = [
    "how to", "what is", "best", "top", "cheap", "buy",
    "vs", "review", "near me", "for beginners",
    "2024", "2025", "alternative", "free", "vs",
]

def expand_seed(seed: str, country: str = "US", max_workers: int = 10) -> set[str]:
    """Expand a single seed keyword into hundreds of long-tail variants.

    Strategy:
    1. Append each letter a-z to the seed (e.g., 'seo a', 'seo b', ...)
    2. Prepend each modifier (e.g., 'how to seo', 'best seo', ...)
    3. Deduplicate all results.
    """
    queries = set()

    # Alphabet expansion: 'seed a', 'seed b', ...
    alpha_queries = [f"{seed} {ch}" for ch in ALPHABET]

    # Modifier expansion: 'how to seed', 'best seed', ...
    mod_queries = [f"{m} {seed}" for m in MODIFIERS]

    all_queries = alpha_queries + mod_queries

    def safe_fetch(q: str) -> list[str]:
        try:
            return fetch_autocomplete(q, country=country)
        except Exception:
            return []

    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as pool:
        results = pool.map(safe_fetch, all_queries)
        for q, suggestions in zip(all_queries, results):
            queries.add(q)
            queries.update(suggestions)

    # Filter: keep only suggestions containing the seed keyword
    filtered = {q for q in queries if seed.lower() in q.lower()}
    return filtered

if __name__ == "__main__":
    longtail = expand_seed("running shoes", country="US")
    print(f"Generated {len(longtail)} long-tail keywords")
    for kw in sorted(longtail)[:20]:
        print(f"  {kw}")

With a 26-letter expansion plus 14 modifiers, a single seed like running shoes typically yields 200–400 unique long-tail keywords. Run this with a residential proxy pool so each thread can rotate IPs — ProxyHat's rotating residential network handles this transparently through the gateway.

Code Block 3: Recursive PAA Expansion with Playwright

PAA questions are nested: expanding one question reveals new ones. To capture the full tree, you need a browser automation tool like Playwright that can click accordions and wait for new questions to load. Here's a recursive extractor:

from playwright.sync_api import sync_playwright
import time
import json
from typing import Optional

def build_proxyhat_proxy(country: str, city: Optional[str] = None, session_id: Optional[str] = None) -> dict:
    """Build a Playwright-compatible proxy config for ProxyHat."""
    username_parts = [f"user-country-{country}"]
    if city:
        username_parts.append(f"city-{city}")
    if session_id:
        username_parts.append(f"session-{session_id}")
    return {
        "server": "http://gate.proxyhat.com:8080",
        "username": "-".join(username_parts),
        "password": "YOUR_PROXYHAT_PASSWORD",
    }

def scrape_paa(seed: str, country: str = "US", city: Optional[str] = None, max_depth: int = 3) -> list[dict]:
    """Scrape People Also Ask questions recursively from a Google SERP.

    Returns a list of dicts: {question, answer_snippet, source_url, depth}
    """
    proxy = build_proxyhat_proxy(country, city, session_id=f"paa-{seed[:10]}")
    all_questions = []
    seen_questions = set()

    with sync_playwright() as p:
        browser = p.chromium.launch(proxy=proxy, headless=True)
        page = browser.new_page()

        # Navigate to Google search
        page.goto(f"https://www.google.com/search?q={seed}&hl=en&gl={country.lower()}",
                  wait_until="domcontentloaded", timeout=30000)
        time.sleep(2)  # Let PAA box render

        def extract_paa_block(depth: int):
            """Find all PAA question buttons and click each to reveal nested ones."""
            # PAA questions are typically in elements with role or specific classes
            # Google changes these frequently, so we use a text-based heuristic
            question_selectors = page.query_selector_all("div[jsname], div[role='button']")
            # Filter to likely PAA questions by checking for the expand arrow
            paa_questions = []
            for el in question_selectors:
                text = el.inner_text().strip()
                if text.endswith("?") and len(text) < 150 and text not in seen_questions:
                    paa_questions.append((el, text))

            for el, question in paa_questions[:5]:  # Limit per depth to avoid runaway
                if question in seen_questions:
                    continue
                seen_questions.add(question)

                try:
                    el.click()
                    page.wait_for_timeout(1500)  # Wait for expansion

                    # Extract answer snippet and source
                    answer_el = page.query_selector("div[data-attrid]")
                    answer_snippet = answer_el.inner_text().strip() if answer_el else ""

                    source_el = page.query_selector("a[data-ved] cite, a[href] cite")
                    source_url = source_el.inner_text().strip() if source_el else ""

                    all_questions.append({
                        "question": question,
                        "answer_snippet": answer_snippet[:500],
                        "source_url": source_url,
                        "depth": depth,
                    })

                    # Recurse into newly revealed questions
                    if depth < max_depth:
                        extract_paa_block(depth + 1)
                except Exception as exc:
                    print(f"Error expanding '{question}': {exc}")
                    continue

        extract_paa_block(depth=0)
        browser.close()

    return all_questions

if __name__ == "__main__":
    results = scrape_paa("what is seo", country="US", max_depth=2)
    print(json.dumps(results[:5], indent=2))

Google's DOM structure changes frequently, so the selectors above are heuristic. The key pattern is: find question-like text ending in ?, click it, wait for expansion, extract the answer snippet and cited source URL, then recurse into newly revealed questions. A depth of 2–3 typically yields 15–40 unique questions per seed.

Code Block 4: Async Pipeline with Concurrency and Circuit Breaker

For production keyword research, you need async I/O, bounded concurrency, and a circuit breaker to stop if Google starts blocking. Here's an httpx.AsyncClient pipeline:

import asyncio
import httpx
import json
from collections import defaultdict
from datetime import datetime

class CircuitBreaker:
    """Simple circuit breaker: opens after N consecutive failures."""
    def __init__(self, threshold: int = 10, reset_after: int = 60):
        self.threshold = threshold
        self.reset_after = reset_after
        self.failures = 0
        self.opened_at: float | None = None

    def record_failure(self):
        self.failures += 1
        if self.failures >= self.threshold:
            self.opened_at = asyncio.get_event_loop().time()

    def record_success(self):
        self.failures = 0
        self.opened_at = None

    @property
    def is_open(self) -> bool:
        if self.opened_at is None:
            return False
        if asyncio.get_event_loop().time() - self.opened_at > self.reset_after:
            self.failures = 0
            self.opened_at = None
            return False
        return True

async def fetch_autocomplete_async(
    client: httpx.AsyncClient,
    query: str,
    country: str,
    session_id: str,
    cb: CircuitBreaker,
) -> list[str]:
    if cb.is_open:
        raise RuntimeError("Circuit breaker is open — pausing requests")

    username = f"user-country-{country}-session-{session_id}"
    proxy = f"http://{username}:YOUR_PROXYHAT_PASSWORD@gate.proxyhat.com:8080"

    url = "https://suggestqueries.google.com/complete/search"
    params = {"client": "chrome", "q": query, "hl": "en"}

    try:
        # Note: for per-request proxy rotation, use the ProxyHat gateway
        # with a fresh session_id each call, or use a proxy pool manager
        resp = await client.get(url, params=params, proxy=proxy, timeout=15.0)
        resp.raise_for_status()
        data = resp.json()
        cb.record_success()
        return data[1] if len(data) > 1 else []
    except Exception as exc:
        cb.record_failure()
        print(f"Failed for '{query}': {exc}")
        return []

async def batch_expand(seeds: list[str], country: str = "US") -> dict[str, list[str]]:
    """Expand multiple seeds concurrently with circuit-breaker protection."""
    cb = CircuitBreaker(threshold=15, reset_after=90)
    results: dict[str, list[str]] = defaultdict(list)

    async with httpx.AsyncClient() as client:
        semaphore = asyncio.Semaphore(10)  # Max 10 concurrent requests

        async def bounded_fetch(seed: str):
            async with semaphore:
                session_id = f"kw-{hash(seed) % 10000}"
                suggestions = await fetch_autocomplete_async(
                    client, seed, country, session_id, cb
                )
                results[seed].extend(suggestions)

        tasks = [bounded_fetch(seed) for seed in seeds]
        await asyncio.gather(*tasks, return_exceptions=True)

    return dict(results)

if __name__ == "__main__":
    seeds = ["seo tools", "keyword research", "backlink analysis", "technical seo"]
    results = asyncio.run(batch_expand(seeds, country="US"))
    total = sum(len(v) for v in results.values())
    print(f"Collected {total} suggestions across {len(seeds)} seeds")
    for seed, sugs in results.items():
        print(f"\n{seed}: {len(sugs)} suggestions")

The circuit breaker stops the pipeline after 15 consecutive failures, preventing a runaway loop that burns proxy bandwidth against a blocking endpoint. The semaphore caps concurrency at 10 parallel requests — a reasonable default for residential proxies that balances speed against rate-limit risk.

Code Block 5: Dedup, Cluster by Intent, and Export to CSV

Raw keyword lists are noisy. You need to deduplicate, cluster by intent category, and export to a format your content team can use:

import csv
import re
from collections import defaultdict
from typing import Optional

# Intent classification patterns
INTENT_PATTERNS = {
    "informational": [
        r"\bhow\b", r"\bwhat\b", r"\bwhy\b", r"\bwhen\b",
        r"\bguide\b", r"\btutorial\b", r"\bexamples?\b",
    ],
    "transactional": [
        r"\bbuy\b", r"\bprice\b", r"\bcost\b", r"\bcheap\b",
        r"\bdeal\b", r"\bdiscount\b", r"\bfor sale\b",
    ],
    "navigational": [
        r"\blogin\b", r"\bsign up\b", r"\bdashboard\b", r"\baccount\b",
    ],
    "comparison": [
        r"\bvs\b", r"\balternative\b", r"\bcompare\b",
        r"\bor\b", r"\bdifference\b",
    ],
}

def classify_intent(keyword: str) -> str:
    """Classify a keyword into a search-intent bucket."""
    kw_lower = keyword.lower()
    for intent, patterns in INTENT_PATTERNS.items():
        for pattern in patterns:
            if re.search(pattern, kw_lower):
                return intent
    return "commercial"  # Default for product-ish queries

def dedup_and_cluster(keywords: list[str]) -> dict[str, list[str]]:
    """Deduplicate keywords and cluster by intent."""
    # Normalize: lowercase, strip whitespace
    normalized = {kw.strip().lower() for kw in keywords if kw.strip()}

    clusters: dict[str, list[str]] = defaultdict(list)
    for kw in sorted(normalized):
        intent = classify_intent(kw)
        clusters[intent].append(kw)

    # Sort each cluster alphabetically
    for intent in clusters:
        clusters[intent].sort()

    return clusters

def export_to_csv(clusters: dict[str, list[str]], filepath: str = "keywords.csv"):
    """Export clustered keywords to CSV for content/FAQ planning."""
    with open(filepath, "w", newline="", encoding="utf-8") as f:
        writer = csv.writer(f)
        writer.writerow(["keyword", "intent", "char_count", "word_count"])
        for intent, keywords in sorted(clusters.items()):
            for kw in keywords:
                writer.writerow([
                    kw,
                    intent,
                    len(kw),
                    len(kw.split()),
                ])
    print(f"Exported {sum(len(v) for v in clusters.values())} keywords to {filepath}")

# --- Full pipeline ---
if __name__ == "__main__":
    # Simulated collected keywords from Autocomplete + PAA
    raw_keywords = [
        "how to do seo", "what is seo", "seo vs sem", "best seo tools",
        "buy seo software", "seo guide for beginners", "cheap seo services",
        "seo tutorial 2024", "seo alternative", "why seo matters",
        "seo cost small business", "technical seo checklist",
        "seo for ecommerce", "local seo tips", "how long does seo take",
    ]

    clusters = dedup_and_cluster(raw_keywords)
    for intent, kws in clusters.items():
        print(f"\n{intent} ({len(kws)}):")
        for kw in kws:
            print(f"  - {kw}")

    export_to_csv(clusters, "keyword_research.csv")

The CSV includes character and word counts so you can filter for long-tail (typically 4+ words) versus head terms. This output feeds directly into content calendars, FAQ page generation, and topic-cluster mapping.

Code Block 6: Extracting Related Searches from the SERP

Related Searches round out the dataset with topical adjacency. Here's a lightweight httpx fetcher that parses the SERP HTML:

import httpx
from bs4 import BeautifulSoup
from typing import Optional

def fetch_related_searches(
    query: str,
    country: str = "US",
    city: Optional[str] = None,
) -> list[str]:
    """Extract Related Searches from Google SERP HTML."""
    username_parts = [f"user-country-{country}"]
    if city:
        username_parts.append(f"city-{city}")
    username = "-".join(username_parts)
    proxy = f"http://{username}:YOUR_PROXYHAT_PASSWORD@gate.proxyhat.com:8080"

    url = "https://www.google.com/search"
    params = {"q": query, "hl": "en", "gl": country.lower()}

    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
        "Accept-Language": "en-US,en;q=0.9",
    }

    with httpx.Client(proxy=proxy, timeout=20.0, headers=headers) as client:
        resp = client.get(url, params=params)
        resp.raise_for_status()

    soup = BeautifulSoup(resp.text, "html.parser")

    # Related searches are typically in <a> tags within a specific section
    # Look for links containing '/search?q=' that appear in the related section
    related = []
    for a in soup.find_all("a", href=True):
        href = a["href"]
        if "/search?q=" in href and a.get_text(strip=True):
            text = a.get_text(strip=True)
            # Filter out navigation links (usually short or contain Google UI text)
            if len(text) > 5 and text != query and text not in related:
                related.append(text)

    return related[:20]  # Google typically shows 8–20 related searches

if __name__ == "__main__":
    related = fetch_related_searches("content marketing", country="US")
    print(f"Found {len(related)} related searches:")
    for r in related:
        print(f"  {r}")

Common Mistakes and Edge Cases

• Using datacenter IPs for localized research — you'll get US-biased suggestions regardless of the gl parameter. Residential proxies with city-level targeting are the only reliable way to get authentic local suggestions.
• Ignoring the client=chrome parameter — without it, the Autocomplete endpoint returns XML or HTML, not clean JSON. Always use client=chrome.
• Scraping PAA without a real browser — PAA questions are loaded dynamically via JavaScript. httpx alone can't expand accordions; you need Playwright or Selenium.
• No deduplication — Autocomplete and PAA overlap heavily. Without dedup, you'll inflate your keyword count and waste time analyzing duplicates.
• Forgetting hl and gl parameters — these control language and country for the SERP. Even with a German proxy, omitting hl=de may return English results.
• Not throttling — even with proxies, aggressive concurrency triggers blocks. Keep it under 10 concurrent requests and add jitter between batches.

Ethics and Responsible Scraping

Autocomplete suggestions and PAA questions are public data visible to any search user. However, scraping at scale consumes Google's resources, and Google's Terms of Service prohibit automated access without permission. For small-scale research (a few hundred queries), the impact is negligible. For large-scale production use, consider Google's Custom Search JSON API or the official Search APIs, which provide structured data with explicit permission.

Best practices:

• Throttle requests to 1–2 per second per IP.
• Respect robots.txt — check https://suggestqueries.google.com/robots.txt and https://www.google.com/robots.txt.
• Cache results — Autocomplete suggestions change slowly; cache for 24–48 hours to avoid redundant requests.
• Use the data for research, not for republishing Google's content verbatim.

Key Takeaways

• Google Autocomplete, People Also Ask, and Related Searches are three free keyword goldmines that map to different layers of search intent.
• The client=chrome Autocomplete endpoint returns clean JSON — use it with httpx and residential proxies for localized suggestions.
• A seed-to-longtail expander using a–z prefixes and modifiers turns one keyword into 200–400 long-tail variants.
• PAA questions are recursive — use Playwright to click accordions and capture nested questions plus their cited answer sources.
• Residential proxies with country/city geo-targeting (ProxyHat's user-country-DE-city-berlin format) are essential for accurate localized data and avoiding per-IP rate limits.
• Dedup, cluster by intent, and export to CSV for immediate use in content and FAQ planning.
• For production scale, prefer official Google APIs; for research-scale collection, throttle politely and cache aggressively.

Ready to build your keyword-research pipeline? Start with ProxyHat residential proxies, review our web scraping guide, and check the ProxyHat documentation for advanced session and rotation settings.