How to Scrape eBay Listings at Scale: API vs HTML, Proxies & Python Code

API Endpoints vs HTML: The eBay Trade-Off

eBay offers two official REST APIs — Finding API and Browse API — that return structured JSON for items, categories, and search results. For small projects, they're clean and reliable. But the moment you need scale, full data coverage, or seller-level analytics, the APIs hit hard limits. That's when you need to scrape eBay listings directly — and that's when proxies become non-negotiable.

This guide walks through the exact decision points, HTML structures you'll parse, proxy strategies that actually work against eBay's anti-bot stack, and production-ready Python code you can ship today.

eBay APIs: What You Get and Where You Hit the Wall

Finding API

eBay's Finding API (https://svcs.ebay.com/services/search/FindingService/v1) covers keyword search, category browsing, and item filters. It returns up to 100 items per call with pagination. The standard quota is 5,000 API calls per day for individual developers. Enterprise partners can negotiate up to 1.5 million daily calls — but that requires a commercial agreement and often weeks of onboarding.

• Rate limit: 5,000 calls/day (standard tier)
• Items per call: 100 max, paginated
• Auth: App token (OAuth client_credentials grant)
• Data gaps: No seller feedback detail, no real-time auction bid counts, no shipping cost breakdown for all options

Browse API

The Browse API (https://api.ebay.com/buy/browse/v1/item_summary/search) is newer and returns richer item data including images, condition, and price conversions. But it shares similar quota constraints and requires a user token for some endpoints.

• Rate limit: Similar tiered structure — ~5,000 calls/day standard
• Auth: OAuth with user consent for some endpoints
• Data gaps: Seller analytics, historical pricing, and cross-listing patterns are absent

When Scraping Becomes the Better Option

You should switch to HTML scraping when:

• You need to monitor more than ~500,000 items per day — API quotas can't keep up
• You need seller-level data (feedback scores, store categories, listing patterns)
• You need auction timing precision — the API returns end timestamps but not real-time bid counts or time-left strings as shown on the page
• You need regional pricing from eBay.de, eBay.co.uk, etc. — the API's geo handling is limited
• You need shipping cost details for every available option, not just the default

Dimension	Finding API	Browse API	HTML Scraping
Daily call limit	5,000 (std)	5,000 (std)	~100–200 req/hr per IP
Authentication	App token	User token	None (browser-like)
Items per request	100	200	~48–60
Auction bid count	No	Limited	Yes
Time-left string	No	End date only	Yes
Seller feedback	No	No	Yes
Shipping options	Primary only	Primary only	All displayed
Setup effort	Medium	Medium	Low
Scale cost	Enterprise deal	Enterprise deal	Proxy costs

Target HTML Structures on eBay

Search Results Grid (.s-item)

eBay's search results page renders items inside .s-item divs. Each card contains predictable child selectors:

• .s-item__title — listing title
• .s-item__price — current price (or price range)
• .s-item__shipping — shipping cost text
• .s-item__bids — bid count for auctions (absent for BIN-only listings)
• .s-item__time-left — time remaining for active auctions
• .s-item__bin-btn — Buy It Now button (present means BIN is available)
• .s-item__link — URL to the listing detail page
• .s-item__seller-info — seller name and rating (appears on some layouts)

A truncated sample of the search results HTML:

<div class="s-item">
  <div class="s-item__info">
    <a class="s-item__link" href="https://www.ebay.com/itm/1234567890">
      <h3 class="s-item__title">Vintage Omega Seamaster 1960</h3>
    </a>
    <div class="s-item__details">
      <span class="s-item__price">$1,250.00</span>
      <span class="s-item__shipping">Free shipping</span>
      <span class="s-item__bids">12 bids</span>
      <span class="s-item__time-left">2h 15m</span>
      <div class="s-item__bin-btn">Buy It Now</div>
    </div>
  </div>
</div>

Listing Detail Pages

Individual item pages at /itm/{id} contain the richest data per listing:

• #vi-desc-main — full item description (may be in an iframe — handle accordingly)
• #prcIsum or #mm-saleDscPrc — price elements
• .vi-VR-cav — condition and variation selectors
• #vi-time-wrapper — auction countdown timer
• #bidBtn — indicates an active auction
• #binBtn_btn — Buy It Now button

Key gotcha: eBay loads the description inside an iframe (#desc_ifr). You'll need a second request to the iframe's src URL to get the full description HTML. The item description often contains condition notes, authenticity details, and seller-specific return policies that aren't in the API.

Seller Profile Pages

Seller stores live at https://www.ebay.com/str/{seller_name} and feedback pages at https://www.ebay.com/fdbk/feedback_profile/{seller_name}:

• .seller-persona-card__feedback-score — cumulative feedback score
• .seller-persona-card__positive-feedback — positive feedback percentage
• .seller-persona-card__category — store categories (reveals niche focus)
• .feedback-item — individual feedback entries with item titles, ratings, and dates

Proxy Selection for eBay Scraping

eBay blocks datacenter IPs aggressively. A single datacenter IP making browser-like requests will typically get CAPTCHA'd within 20–50 requests. Residential proxies are the baseline for any serious eBay scraping operation.

Residential Proxies: The Default Choice

Residential IPs come from real ISP allocations. eBay's anti-bot system can't distinguish them from genuine shoppers without behavioral analysis. For bulk listing collection, residential proxies with per-request rotation are the standard approach.

First, verify your proxy connection works against eBay:

curl -x http://user-country-US:YOUR_PASSWORD@gate.proxyhat.com:8080 \
  "https://www.ebay.com/sch/i.html?_nkw=vintage+watch&_pgn=1" \
  -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" \
  -H "Accept: text/html" \
  -o ebay_test.html

# Check for CAPTCHA or block pages
grep -c "s-item" ebay_test.html

If the grep returns a count above zero, your proxy is getting real search results. If it returns zero, you're likely hitting a CAPTCHA page — switch to residential proxies or adjust your headers.

Configure ProxyHat residential proxies in Python:

import requests

PROXY = "http://user-country-US:YOUR_PASSWORD@gate.proxyhat.com:8080"
proxies = {"http": PROXY, "https": PROXY}

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Connection": "keep-alive",
}

def fetch_search(keyword, page=1):
    url = "https://www.ebay.com/sch/i.html"
    params = {"_nkw": keyword, "_pgn": page}
    resp = requests.get(url, params=params, headers=headers,
                       proxies=proxies, timeout=30)
    resp.raise_for_status()
    return resp.text

Geo-Targeting for Regional eBay Domains

eBay shows different pricing, availability, and shipping options depending on the visitor's country. If you're monitoring eBay.de or eBay.co.uk, you need proxies in those regions — otherwise you'll see redirected or filtered results.

With ProxyHat, geo-targeting goes in the username string:

# Germany (eBay.de)
DE_PROXY = "http://user-country-DE:YOUR_PASSWORD@gate.proxyhat.com:8080"

# United Kingdom (eBay.co.uk)
UK_PROXY = "http://user-country-GB:YOUR_PASSWORD@gate.proxyhat.com:8080"

# City-level targeting for localized results
BERLIN_PROXY = "http://user-country-DE-city-berlin:YOUR_PASSWORD@gate.proxyhat.com:8080"

When scraping eBay.de, always set the Accept-Language header to de-DE,de;q=0.9 and hit https://www.ebay.de/sch/i.html with a German proxy. The combination of IP + headers + domain determines what you see.

Rotation Strategy: Per-Request vs Sticky Sessions

For search result collection, use per-request rotation — a new IP for every page fetch. This spreads your footprint across thousands of IPs and avoids per-IP rate limits.

For listing detail pages where you need to follow pagination or load multiple tabs under one session, use sticky sessions:

# Sticky session — same IP for up to 30 minutes
STICKY_PROXY = "http://user-session-mylist42:YOUR_PASSWORD@gate.proxyhat.com:8080"

Sticky sessions are essential when eBay's anti-bot system checks for session continuity (same cart, same recently viewed items) across requests. A rotating IP mid-session can trigger a re-authentication prompt.

Working Python Scraper: Search Results to Structured Records

Here's a complete, production-ready parser that turns eBay search result HTML into structured records:

from bs4 import BeautifulSoup
import requests, time, random

PROXY = "http://user-country-US:YOUR_PASSWORD@gate.proxyhat.com:8080"
proxies = {"http": PROXY, "https": PROXY}

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9",
    "Accept-Language": "en-US,en;q=0.9",
}

def parse_search_results(html):
    soup = BeautifulSoup(html, "html.parser")
    items = []
    for card in soup.select(".s-item"):
        title_el = card.select_one(".s-item__title")
        price_el = card.select_one(".s-item__price")
        shipping_el = card.select_one(".s-item__shipping")
        url_el = card.select_one(".s-item__link")
        bids_el = card.select_one(".s-item__bids")
        time_el = card.select_one(".s-item__time-left")
        bin_el = card.select_one(".s-item__bin-btn")

        # Skip placeholder / ad items that eBay injects
        title = title_el.get_text(strip=True) if title_el else ""
        if title in ("", "Shop on eBay"):
            continue

        items.append({
            "title": title,
            "price": price_el.get_text(strip=True) if price_el else None,
            "shipping": shipping_el.get_text(strip=True) if shipping_el else None,
            "url": url_el["href"] if url_el else None,
            "bids": bids_el.get_text(strip=True) if bids_el else None,
            "time_left": time_el.get_text(strip=True) if time_el else None,
            "is_buy_it_now": bin_el is not None,
        })
    return items

def scrape_keyword(keyword, max_pages=5):
    all_items = []
    for page in range(1, max_pages + 1):
        url = "https://www.ebay.com/sch/i.html"
        params = {"_nkw": keyword, "_pgn": page}
        resp = requests.get(url, params=params, headers=HEADERS,
                           proxies=proxies, timeout=30)
        items = parse_search_results(resp.text)
        all_items.extend(items)
        print(f"Page {page}: {len(items)} items")
        time.sleep(random.uniform(2, 5))
    return all_items

# Run it
results = scrape_keyword("vintage omega watch", max_pages=3)
print(f"Total: {len(results)} listings")

Watch out for the first .s-item card — eBay injects a promotional "Shop on eBay" item as the first result. The filter in the code above skips it. If you don't filter it, you'll get a phantom listing with no URL or price in every result set.

Handling Auctions: Time-Left, Bids, and Buy-It-Now

Auction data is where HTML scraping delivers the most value over the API. The API gives you an end timestamp — but the page shows you real-time bid counts, time-left strings, and whether a Buy It Now option is still available alongside the auction.

Here's how to extract and normalize auction fields:

import re

def parse_auction_timing(time_str):
    """Convert eBay time-left string to minutes remaining.
    Examples: '2h 15m', '1d 3h', '5d 12h 30m', '15m'
    """
    if not time_str:
        return None
    total_minutes = 0
    days = re.search(r"(\d+)\s*d", time_str)
    hours = re.search(r"(\d+)\s*h", time_str)
    mins = re.search(r"(\d+)\s*m", time_str)
    if days:
        total_minutes += int(days.group(1)) * 1440
    if hours:
        total_minutes += int(hours.group(1)) * 60
    if mins:
        total_minutes += int(mins.group(1))
    return total_minutes if total_minutes > 0 else None

def parse_bid_count(bids_str):
    """Extract numeric bid count from strings like '12 bids'."""
    if not bids_str:
        return 0
    match = re.search(r"(\d+)", bids_str)
    return int(match.group(1)) if match else 0

def enrich_auction_data(item):
    """Add normalized auction fields to a parsed item dict."""
    item["minutes_remaining"] = parse_auction_timing(item.get("time_left"))
    item["bid_count"] = parse_bid_count(item.get("bids"))
    item["is_auction"] = (
        item.get("bids") is not None or item.get("time_left") is not None
    )
    return item

# Usage
for item in results:
    enrich_auction_data(item)
    if item["is_auction"]:
        print(f"{item['title']}: {item['bid_count']} bids, "
              f"{item['minutes_remaining']} min left, "
              f"BIN: {item['is_buy_it_now']}")

Pro tip for snipe monitoring: Filter for minutes_remaining < 60 and bid_count > 5 to surface auctions entering their final hour with competitive bidding — these are the listings where last-minute price shifts happen and where reseller-intel teams find pricing edge.

Seller Analytics at Scale

For reseller-intel teams, individual listings are only half the picture. Understanding who is selling, what categories they dominate, and whether they're cross-listing across niches is where the real competitive insight lives.

Scraping Seller Profiles

def fetch_seller_profile(seller_name):
    url = f"https://www.ebay.com/str/{seller_name}"
    resp = requests.get(url, headers=HEADERS, proxies=proxies, timeout=30)
    soup = BeautifulSoup(resp.text, "html.parser")

    feedback_el = soup.select_one(".seller-persona-card__feedback-score")
    positive_el = soup.select_one(".seller-persona-card__positive-feedback")
    categories = [
        el.get_text(strip=True)
        for el in soup.select(".seller-persona-card__category")
    ]

    return {
        "seller": seller_name,
        "feedback_score": feedback_el.get_text(strip=True) if feedback_el else None,
        "positive_pct": positive_el.get_text(strip=True) if positive_el else None,
        "categories": categories,
        "niche_count": len(categories),
    }

def detect_cross_listing(seller_name, niches):
    """Check if a seller lists across multiple product niches.
    Returns (unique_titles, total_titles) for overlap analysis.
    """
    all_titles = []
    for niche in niches:
        html = fetch_search(f"{niche} seller:{seller_name}")
        items = parse_search_results(html)
        all_titles.extend([i["title"] for i in items if i.get("title")])
        time.sleep(random.uniform(2, 4))
    return len(set(all_titles)), len(all_titles)

# Example: profile a seller
profile = fetch_seller_profile("vintage_watches_nyc")
print(profile)
# {'seller': 'vintage_watches_nyc', 'feedback_score': '12,450',
#  'positive_pct': '99.7%', 'categories': ['Watches', 'Parts'],
#  'niche_count': 2}

# Detect cross-listing
unique, total = detect_cross_listing(
    "vintage_watches_nyc",
    ["vintage watch", "camera lens", "leather bag"]
)
print(f"Unique: {unique}, Total: {total}")

Seller Intelligence Patterns

Once you're collecting seller data at scale, look for these patterns:

• Category expansion: A seller adding new store categories over time signals market entry — useful for early competitive detection.
• Feedback velocity: Track feedback score changes week-over-week. A spike in new feedback correlates with a sales surge.
• Cross-listing detection: Sellers operating across seemingly unrelated niches (e.g., watches and camera lenses) may be drop-shipping or arbitraging — prime candidates for pricing pressure analysis.
• Listing density: If a seller has 500+ active listings in one category, they're a dominant player. Track their average price as a market benchmark.

Anti-Bot Defenses on eBay

eBay runs a multi-layered anti-bot stack. Here's what you're up against:

• reCAPTCHA v3: Runs silently on every page. A low score triggers a CAPTCHA challenge on subsequent requests. Residential IPs and natural request cadence keep scores high.
• TLS fingerprinting: eBay's CDN checks TLS handshake patterns. Python's default requests library has a distinctive TLS fingerprint. Use curl_cffi or tls_client for impersonation if you see connection resets.
• Device fingerprinting: JavaScript collects browser characteristics. If you're running headless browsers, use undetected-chromedriver or Playwright with stealth plugins.
• Datacenter IP blocking: This is the hardest filter. eBay maintains blocklists of major hosting providers (AWS, GCP, DigitalOcean, etc.). Datacenter proxies will fail within 20–50 requests.
• Request pattern analysis: Hitting the same keyword at 1-second intervals from different IPs still looks robotic. Add jitter (2–5 second random delays) and vary your search patterns.

Rule of thumb: If your scraping success rate drops below 90%, you're either rotating too fast, using datacenter IPs, or sending identical headers across thousands of requests. Fix the proxy type first, then tune request cadence.

Ethical and Legal Considerations

eBay's Terms of Use prohibit automated data collection. In practice, this means:

• Respect robots.txt — eBay's allows some crawling but restricts certain paths.
• Don't hammer a single endpoint. Rate-limit your requests to a human-like cadence.
• Don't scrape personal data (buyer identities, private messages) — this runs into GDPR and CCPA territory.
• Public listing data (titles, prices, seller names) is generally considered fair game for market research, but consult legal counsel if you're building a commercial product on top of scraped eBay data.

Key Takeaways

• Use the API for small-scale, structured queries. eBay's Finding and Browse APIs work well under 5,000 calls/day but lack seller analytics, real-time auction data, and full shipping details.
• HTML scraping fills the gaps — auction timing, bid counts, Buy-It-Now flags, seller feedback, and cross-listing patterns are only reliably available on the page.
• Residential proxies are mandatory for eBay. Datacenter IPs get blocked in under 50 requests. Use rotating residential proxies with geo-targeting for regional domains.
• Parse .s-item cards with the selectors listed above. Skip the placeholder "Shop on eBay" ad item that appears as the first card.
• Sticky sessions for multi-page scraping on one listing; per-request rotation for bulk search collection.
• Mind the anti-bot stack: reCAPTCHA v3, TLS fingerprinting, and datacenter blocklists. Use proper headers, request jitter, and residential IPs to stay under the radar.

Ready to start scraping eBay at scale? Check out ProxyHat pricing for residential proxy plans, or explore our proxy locations to find IPs in the exact markets you need to monitor. For more scraping strategies, see our web scraping use case and SERP tracking pages.