Scrape Google People Also Ask and Autocomplete for Keyword Research (Python Guide)

If you want to build a keyword research pipeline without paying $99/month for a SaaS tool, you can scrape Google People Also Ask and Autocomplete for keyword research directly. Google exposes three free suggestion surfaces — Autocomplete, PAA boxes, and Related Searches — that collectively cover informational, commercial, and long-tail intent. This guide shows you how to extract all three with Python, expand a single seed into hundreds of long-tail variants, and use residential proxies to avoid rate limits and locale bias.

Why Scrape Google People Also Ask and Autocomplete for Keyword Research (Python Guide)

Traditional keyword tools like Ahrefs and SEMrush aggregate clickstream and Google Keyword Planner data, but they miss the real-time, hyper-local suggestions that Google itself surfaces. Autocomplete reflects what people are actively typing right now. PAA questions capture informational intent that content teams need for FAQ schema. Related Searches show bottom-of-funnel queries. Together, they form a zero-cost dataset that updates daily.

The problem: Google rate-lights aggressive automated requests within roughly 50–100 queries per IP per hour on the autocomplete endpoint, and PAA extraction requires JavaScript rendering. That's where residential proxies come in — they distribute requests across real ISP-assigned IPs and let you geo-target suggestions by country and city.

The Three Keyword Goldmines

1. Google Autocomplete (Suggest API)

Google's Chrome autocomplete endpoint returns JSON suggestions for any partial query. The endpoint is:

https://suggestqueries.google.com/complete/search?client=chrome&q=YOUR_QUERY

It returns a JSON array where index 1 is a list of suggestion strings. This is the same data you see in the Chrome address bar and Google Search box. Each suggestion maps to active search demand — Google only suggests queries that real users type frequently enough to cross an internal threshold.

You can test it immediately with curl:

curl -s "https://suggestqueries.google.com/complete/search?client=chrome&q=best+running+shoes" \
  -x http://user-country-US:pass@gate.proxyhat.com:8080

The response looks like ["best running shoes", ["best running shoes 2025", "best running shoes for flat feet", ...], ...]. No API key, no OAuth, no billing — just a GET request.

2. People Also Ask (PAA)

PAA boxes appear on roughly 60–70% of Google search results pages for informational queries, according to Moz's SERP feature analysis. Each PAA box contains 3–5 initial questions, and clicking a question expands it to reveal an answer snippet plus a source URL — and critically, it injects new questions below. This recursive expansion means a single PAA box can yield 20–50+ questions if you keep expanding accordions.

3. Related Searches

At the bottom of every SERP, Google shows 8–10 related queries. These tend to be broader and more commercial than PAA questions. They're excellent for identifying sibling topics and building topic clusters.

Building a Seed-to-Longtail Expander

One seed keyword like running shoes yields ~10 autocomplete suggestions. But if you append every letter a–z and common modifiers (best, how to, vs, for, near me), you get 26 + modifiers = 40+ prefixes, each returning up to 10 suggestions. That's 400+ long-tail variants from a single seed.

Here's a Python implementation using httpx with ProxyHat residential proxies:

import httpx
import string
import time
import logging
from typing import List

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)

PROXY_URL = "http://user-country-US:pass@gate.proxyhat.com:8080"
AUTOCOMPLETE_URL = "https://suggestqueries.google.com/complete/search"

MODIFIERS = ["best", "how to", "what is", "why", "vs", "for", "near me", "cheap", "review", "2025"]

def fetch_suggestions(query: str, client: httpx.Client) -> List[str]:
    """Fetch autocomplete suggestions for a single query."""
    params = {"client": "chrome", "q": query}
    try:
        resp = client.get(AUTOCOMPLETE_URL, params=params, timeout=10)
        resp.raise_for_status()
        data = resp.json()
        if len(data) > 1 and isinstance(data[1], list):
            return data[1]
    except (httpx.HTTPError, ValueError, IndexError) as e:
        logger.warning(f"Failed for '{query}': {e}")
    return []

def expand_seed(seed: str, client: httpx.Client) -> List[str]:
    """Expand a seed keyword into long-tail variants using a-z and modifiers."""
    prefixes = list(string.ascii_lowercase) + MODIFIERS
    all_suggestions = set()
    
    for prefix in prefixes:
        expanded_query = f"{seed} {prefix}"
        suggestions = fetch_suggestions(expanded_query, client)
        all_suggestions.update(suggestions)
        logger.info(f"{expanded_query}: {len(suggestions)} suggestions")
        time.sleep(0.5)  # polite throttle
    
    return sorted(all_suggestions)

# Usage
with httpx.Client(proxy=PROXY_URL, headers={"User-Agent": "Mozilla/5.0"}) as client:
    keywords = expand_seed("running shoes", client)
    print(f"Collected {len(keywords)} keywords")

This produces 300–500 unique long-tail keywords per seed. With 10 seeds, you're at 3,000–5,000 keywords — more than enough for a content calendar.

Why You Need Residential Proxies

Google's autocomplete endpoint is deceptively easy to call, which means it's also aggressively rate-limited. If you hammer it from a single datacenter IP, you'll start receiving empty responses or HTTP 429 errors within 50–100 requests. Datacenter IPs are also flagged by Google's anti-abuse systems — they know AWS, DigitalOcean, and GCP IP ranges.

Residential proxies solve two problems:

• Rate limit distribution: Rotating residential IPs spread requests across thousands of real ISP-assigned addresses, so no single IP triggers rate limits.
• Locale bias: Google personalizes autocomplete suggestions by IP geolocation. A datacenter IP in Virginia returns US-centric suggestions. With ProxyHat's geo-targeting (user-country-DE-region-state_of_berlin-city-berlin), you get authentic German-localized suggestions for international keyword research.

According to Google's Search Console documentation, search personalization depends on location, language, and search history — meaning your proxy's IP location directly affects the suggestions you receive.

ProxyHat Setup for Localized Suggestions

ProxyHat lets you geo-target by country and city via the username parameter. Here's how to use it for multi-locale keyword research:

import httpx

LOCALES = {
    "US": "user-country-US:pass",
    "DE": "user-country-DE-region-state_of_berlin-city-berlin:pass",
    "UK": "user-country-GB-region-england-city-london:pass",
    "JP": "user-country-JP-region-tokyo-city-tokyo:pass",
}

def get_proxy_for_locale(locale_code: str) -> str:
    """Build a ProxyHat proxy URL for a given locale."""
    username = LOCALES.get(locale_code, LOCALES["US"])
    return f"http://{username}@gate.proxyhat.com:8080"

def fetch_localized_suggestions(query: str, locale_code: str) -> list:
    proxy = get_proxy_for_locale(locale_code)
    with httpx.Client(proxy=proxy, timeout=15) as client:
        resp = client.get(
            "https://suggestqueries.google.com/complete/search",
            params={"client": "chrome", "q": query, "gl": locale_code.lower()},
            headers={"User-Agent": "Mozilla/5.0"},
        )
        data = resp.json()
        return data[1] if len(data) > 1 else []

# Compare suggestions across locales
for locale in ["US", "DE", "UK"]:
    suggestions = fetch_localized_suggestions("buy laptop", locale)
    print(f"{locale}: {suggestions[:5]}")

For SOCKS5 connections (useful when HTTP proxies are blocked), switch to port 1080:

SOCKS5_PROXY = "socks5://user-country-US:pass@gate.proxyhat.com:1080"

with httpx.Client(proxy=SOCKS5_PROXY, timeout=15) as client:
    resp = client.get(AUTOCOMPLETE_URL, params={"client": "chrome", "q": "test"})
    print(resp.json()[1])

See all available proxy locations and pricing plans on the ProxyHat dashboard.

Extracting People Also Ask with Playwright

PAA boxes require JavaScript rendering — the questions load dynamically and expand on click. Playwright is the cleanest way to handle this. Here's a production-ready scraper that extracts PAA questions and their cited answer sources:

import asyncio
from playwright.async_api import async_playwright
import logging

logger = logging.getLogger(__name__)

PROXY_SERVER = "gate.proxyhat.com:8080"
PROXY_USER = "user-country-US"
PROXY_PASS = "pass"

async def scrape_paa(query: str, max_expansions: int = 20) -> list:
    """Scrape People Also Ask questions with recursive expansion."""
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            headless=True,
            proxy={
                "server": f"http://{PROXY_SERVER}",
                "username": PROXY_USER,
                "password": PROXY_PASS,
            },
        )
        page = await browser.new_page()
        await page.goto(f"https://www.google.com/search?q={query}", wait_until="networkidle")
        
        paa_questions = []
        seen = set()
        
        for i in range(max_expansions):
            # Find unclicked PAA questions
            paa_elements = await page.query_selector_all("div[jsname='x5fWsd'] .wQiwMc")
            
            for el in paa_elements:
                text = (await el.inner_text()).strip()
                if text and text not in seen:
                    seen.add(text)
                    paa_questions.append({"question": text, "source": None})
                    # Click to expand and reveal source + new questions
                    try:
                        await el.click()
                        await page.wait_for_timeout(1500)
                        # Extract source URL from the expanded answer
                        source_el = await page.query_selector("div[jsname='x5fWsd'] a")
                        if source_el:
                            href = await source_el.get_attribute("href")
                            paa_questions[-1]["source"] = href
                    except Exception as e:
                        logger.warning(f"Expansion failed: {e}")
                    break  # process one at a time
            
            if len(paa_questions) >= max_expansions:
                break
        
        await browser.close()
    return paa_questions

# Usage
results = asyncio.run(scrape_paa("how to start a podcast", max_expansions=30))
for r in results:
    print(f"Q: {r['question']} | Source: {r['source']}")

This script clicks each PAA question, waits for the accordion to expand, captures the answer source URL, and repeats. A single SERP can yield 30–50 questions with full recursive expansion.

Deduping, Clustering, and Exporting

Once you've collected autocomplete suggestions and PAA questions from multiple seeds, you'll have thousands of raw keywords. Here's how to clean and cluster them:

import csv
import re
from collections import defaultdict
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans

def dedupe_keywords(keywords: list) -> list:
    """Remove duplicates and normalize."""
    seen = set()
    unique = []
    for kw in keywords:
        normalized = re.sub(r"\s+", " ", kw.strip().lower())
        if normalized not in seen and len(normalized) > 3:
            seen.add(normalized)
            unique.append(normalized)
    return unique

def cluster_by_intent(keywords: list, n_clusters: int = 10) -> dict:
    """Cluster keywords using TF-IDF + K-Means."""
    vectorizer = TfidfVectorizer(ngram_range=(1, 2), max_features=5000)
    X = vectorizer.fit_transform(keywords)
    
    kmeans = KMeans(n_clusters=n_clusters, random_state=42, n_init=10)
    labels = kmeans.fit_predict(X)
    
    clusters = defaultdict(list)
    for kw, label in zip(keywords, labels):
        clusters[label].append(kw)
    return dict(clusters)

def export_to_csv(keywords: list, clusters: dict, filename: str = "keywords.csv"):
    """Export keywords with cluster assignments to CSV."""
    with open(filename, "w", newline="", encoding="utf-8") as f:
        writer = csv.writer(f)
        writer.writerow(["keyword", "cluster_id", "cluster_size"])
        for cluster_id, cluster_kws in clusters.items():
            for kw in keywords:
                if kw in cluster_kws:
                    writer.writerow([kw, cluster_id, len(cluster_kws)])
    print(f"Exported {len(keywords)} keywords to {filename}")

# Full pipeline
all_keywords = []  # combine autocomplete + PAA results
unique = dedupe_keywords(all_keywords)
clusters = cluster_by_intent(unique, n_clusters=15)
export_to_csv(unique, clusters, "keyword_research.csv")

The TF-IDF + K-Means clustering groups keywords by shared terms, which roughly maps to search intent clusters. Export to CSV for content planning, or feed the clusters directly into a FAQ schema generator.

Common Mistakes and Edge Cases

• Forgetting gl and hl parameters: Even with geo-targeted proxies, you should pass gl=de and hl=de in autocomplete requests to force German locale. Proxy geo alone isn't always sufficient.
• Not handling empty autocomplete responses: Google returns ["query", []] when rate-limited. Always check that data[1] is a non-empty list before processing.
• PAA selectors change frequently: Google updates its DOM structure regularly. The jsname='x5fWsd' selector may break — always have a fallback selector and log when zero PAA elements are found.
• Using datacenter IPs for PAA: PAA boxes don't appear on all SERPs, and Google may suppress them for flagged IPs. Residential proxies have a much higher PAA detection rate.
• Ignoring robots.txt: Google's robots.txt doesn't block the autocomplete endpoint, but you should still throttle requests. A 500ms delay between requests is a reasonable minimum.

Ethics and Compliance

Autocomplete and PAA data is public — it's what Google shows to every user. Scraping it for keyword research is widely practiced and legally defensible in most jurisdictions. However, at scale you should:

• Throttle requests to avoid degrading Google's services (0.5s minimum delay).
• Prefer the Google Custom Search API if you need programmatic SERP data at high volume — it's $5 per 1,000 queries and comes with a proper SLA.
• Respect Google's Terms of Service for automated access; they technically prohibit scraping, but enforcement focuses on abusive patterns, not research-grade tooling.
• Don't redistribute raw scraped data commercially — use it for your own research and content planning.

For more on scraping best practices, see our web scraping use case guide and SERP tracking documentation.

Ready to build your keyword research pipeline? Get started with ProxyHat residential proxies and run the code examples above in minutes. Full connection details are in our ProxyHat documentation.