If you're building a PropTech product — whether it's an investor deal-finder, an iBuyer pricing model, or a market-analytics dashboard — you need listing data from the platforms where properties actually get listed. The problem? Sites like Zillow, Rightmove, and ImmoScout24 treat data extraction as an existential threat. They invest heavily in anti-bot infrastructure, and they're very good at blocking requests that look automated.
This guide walks through how real-estate scraping works at scale: which sites to target, what data you can actually access, why real estate scraping proxies are non-negotiable, and how to architect a system that doesn't get shut down on day one.
Why PropTech Teams Need to Scrape Zillow and Beyond
Public listing data is the lifeblood of real-estate analytics. MLS databases are powerful but access is restricted — you need a broker license or a syndication agreement that comes with usage constraints. Meanwhile, the data on Zillow, Realtor.com, Rightmove, and their international counterparts is publicly visible. The question isn't whether this data is valuable; it's whether you can collect it reliably enough to build a business on top of it.
The answer is yes — but only with the right infrastructure. Datacenter IPs get blocked within minutes on most major listing platforms. Residential and mobile proxies are the difference between a pipeline that runs for weeks and one that dies on the first crawl.
Target Sites by Region
Different markets have different dominant platforms. Here's the landscape you need to understand:
United States
- Zillow — The dominant US listing platform. ~135M monthly visitors. Aggressive anti-bot measures including CAPTCHAs, rate limiting by IP range, and device fingerprinting. When you scrape Zillow, expect immediate pushback on datacenter IPs.
- Realtor.com — Backed by the National Association of Realtors. Slightly less aggressive bot detection than Zillow, but still blocks bulk requests from known datacenter ranges.
- Redfin — Tech-forward brokerage with rich listing data. Strong bot detection; they actively monitor for scraping patterns and will serve CAPTCHAs or 403 responses.
United Kingdom
- Rightmove — The UK's dominant portal. Rightmove data extraction is notoriously difficult — they employ sophisticated rate limiting, browser fingerprinting, and legal enforcement. Residential proxies are essential.
- Zoopla — Second-largest UK portal. Slightly easier to crawl but still implements IP-based throttling and CAPTCHA challenges under load.
Germany
- ImmoScout24 — Germany's market leader. Implements Cloudflare protection and rate limiting. German data protection law (BDSG + GDPR) adds a compliance layer you must account for.
France
- LeBonCoin — France's largest classifieds site with a significant real-estate section. Less specialized than the others, which means data quality varies, but volume is enormous.
| Site | Region | Anti-Bot Level | Key Data Strength | Proxy Requirement |
|---|---|---|---|---|
| Zillow | US | High | Price history, Zestimate | Residential / Mobile |
| Realtor.com | US | Medium-High | MLS-syndicated listings | Residential |
| Redfin | US | High | Time-on-market, tours | Residential / Mobile |
| Rightmove | UK | Very High | Listing lifecycle data | Residential / Mobile |
| Zoopla | UK | Medium | Price estimates | Residential |
| ImmoScout24 | DE | Medium-High | German market coverage | Residential |
| LeBonCoin | FR | Low-Medium | Volume + diversity | Residential or Datacenter |
What Data Is Accessible
Not all listing data is created equal. Here's what you can realistically extract from these platforms, and what you'll need to build yourself:
Listing Metadata
The core data: address, property type, bedrooms, bathrooms, square footage, lot size, year built. This is the easiest data to extract and is typically available in structured HTML or embedded JSON on the listing page.
Price and Price History
Current listing price is always available. Price history — listing price changes, previous sale prices, tax assessments — is available on Zillow and Redfin but harder to get from Rightmove and Zoopla. This data is gold for iBuyer pricing models.
School Ratings
Zillow and Realtor.com embed GreatSchools ratings. Redfin includes school district boundaries. This data is a major factor in US home valuations and is relatively straightforward to extract alongside listing metadata.
Photos and Media
Listing images are publicly accessible via CDN URLs, but downloading them at scale requires careful bandwidth management and respect for the source site's resources. A typical listing has 20–40 photos; at 2–5 MB each, you're looking at 50–200 MB per listing.
Agent and Brokerage Data
Listing agent name, brokerage, and contact info are typically displayed. Zillow also shows agent reviews and past sales. This data is useful for building agent-performance analytics or lead-generation tools.
Time-on-Market
Redfin excels here — they show days on market, price drops, and listing status changes. Zillow shows listing history but with less granularity. This metric is critical for market-health analytics and investor timing strategies.
Why Residential Proxies Are Essential
Here's the fundamental problem: datacenter IP addresses are trivially easy to identify. Zillow, Rightmove, and most major listing platforms maintain blocklists of known datacenter IP ranges. When your crawler sends a request from an AWS or DigitalOcean IP, the response is either a CAPTCHA page, a 403 Forbidden, or a redirect to a "verify you're human" page.
Residential proxies solve this by routing your requests through real ISP-assigned IP addresses. Your crawler looks like a regular user browsing from a home connection. The target site sees a Comcast, BT, or Deutsche Telekom IP — not a cloud provider.
Mobile proxies go further, routing through 4G/5G connections. These are the hardest for anti-bot systems to flag because mobile IPs rotate naturally and are shared across many real users. For the most aggressive platforms — Zillow and Rightmove — mobile proxies deliver the highest success rates.
Rule of thumb: If you're doing real-estate scraping at any meaningful scale, datacenter proxies will fail within hours. Budget for residential proxies from the start — it's not an optimization, it's a prerequisite.
Geo-Targeting Matters
Listing sites often serve different content based on the requester's location. Zillow shows different "suggested" listings. Rightmove tailors results to UK regions. If you're crawling US listings from a UK IP, you may get unexpected results or redirects.
With ProxyHat, you can geo-target at the country and city level:
# US residential IP for Zillow
curl -x http://user-country-US:pass@gate.proxyhat.com:8080 https://www.zillow.com/zestimate/
# UK residential IP for Rightmove
curl -x http://user-country-GB:pass@gate.proxyhat.com:8080 https://www.rightmove.co.uk/property-for-sale.html
# German residential IP for ImmoScout24
curl -x http://user-country-DE:pass@gate.proxyhat.com:8080 https://www.immoscout24.de/Architecture: Building a Production Scraping Pipeline
A real-estate data pipeline isn't just a crawler. It's a system that needs to handle deduplication, incremental updates, historical tracking, and storage for both structured data and binary assets. Here's a worked architecture:
Layer 1: Geo-Distributed Crawling
Deploy separate crawl jobs per region, each using proxies from the target country. This isn't just about avoiding blocks — it ensures you see the same content a local user sees. Each crawl job should:
- Use sticky sessions (10–30 minute TTL) to maintain cookies and avoid re-solving CAPTCHAs
- Rotate to a new residential IP when a session expires or encounters a block
- Respect rate limits: 1–3 requests per second per IP, with randomized delays
- Run headless browsers only when necessary — prefer structured JSON endpoints where available
Layer 2: Listing Deduplication
Multiple sites list the same property. A Zillow listing, a Realtor.com listing, and a Redfin listing may all reference the same MLS entry. You need a deduplication strategy:
- Primary key: Normalize the address + zip/postcode into a canonical form
- Secondary matching: Use coordinates (lat/lng) within a small radius
- MLS ID: Where available, this is the most reliable dedup key
Store a canonical property record that links to all source listings. This lets you merge data — price from Zillow, school ratings from Realtor.com, time-on-market from Redfin — into a single enriched record.
Layer 3: Price-History Tracking
Static snapshots miss the story. You need to track price changes over time:
- Crawl listings on a schedule (daily for active listings, weekly for off-market)
- Store each price observation with a timestamp
- Flag listings with price reductions — these are high-signal events for investor deal-finding
- Track status transitions: active → pending → sold, with dates
Layer 4: Photo-Asset Storage
Photos are the largest data volume component. A practical approach:
- Extract CDN URLs from listing pages — don't download immediately
- Batch-download images asynchronously from a separate worker pool
- Store in object storage (S3/GCS) with the listing ID as the partition key
- Deduplicate images across sources — the same photo often appears on multiple sites
Layer 5: Orchestration and Monitoring
Use a task queue (Celery, Bull, or cloud equivalents) to manage crawl jobs. Monitor:
- Success rate per site and per proxy type
- CAPTCHA encounter rate (if this spikes, your IPs are getting flagged)
- Average response time (latency spikes indicate throttling)
- Data freshness: how many listings were updated in the last crawl cycle
Legal Framing: What You Need to Consider
Legal risk in real-estate scraping is real and varies by jurisdiction. Here's a practical framing — not legal advice, but the landscape you need to understand.
Public MLS Data vs. Syndicated Feeds
MLS data is the authoritative source for US listings, but it's not publicly accessible. You need a broker license or a syndication agreement. The listing sites (Zillow, Realtor.com) receive this data via syndication agreements with specific terms of use. Scraping the same data from their public-facing pages is technically possible but may violate their terms of service.
Terms of Service Considerations
- Zillow: Terms explicitly prohibit scraping. They've sent cease-and-desist letters to known scrapers. They also offer an API through the Bridge Interactive group, but it's limited.
- Realtor.com: Similar anti-scraping terms. They're backed by NAR, which has significant legal resources.
- Rightmove: Very aggressive on enforcement. They've pursued legal action against scrapers in the UK courts.
- ImmoScout24: German law (BDSG + GDPR) applies. Scraping personal data (agent contact info) without consent is particularly risky.
- LeBonCoin: French law applies. CNIL has taken enforcement actions against scrapers that collected personal data without a legal basis.
Practical Risk Mitigation
- Scrape only public listing data — prices, addresses, features. Avoid personal data like agent phone numbers unless you have a legitimate basis.
- Respect robots.txt — it's not legally binding in most jurisdictions, but ignoring it is a factor courts consider.
- Rate-limit your requests — this reduces both legal risk and the likelihood of getting blocked.
- Consider data licensing — Zillow's Bridge API, Rightmove's data feeds, and Trestle (US MLS data) offer legal alternatives, though they're more expensive and less flexible.
- Consult a lawyer — especially before scraping in GDPR jurisdictions (EU/UK).
The CFAA (US), Computer Misuse Act (UK), and GDPR (EU) all have provisions that could apply to web scraping. The legal landscape is evolving — what was acceptable yesterday may not be tomorrow. Build your data strategy with legal counsel, not just technical capability.
Use Cases: Who Needs This Data and Why
Investor Deal-Finding
A real-estate investment platform scrapes Zillow and Realtor.com daily, looking for properties where the listing price has dropped more than 5% in the last 30 days. They cross-reference with Redfin's time-on-market data to find properties that have been sitting unsold — these are the most negotiable deals. With residential proxies from ProxyHat, they maintain a 94% success rate across 50,000 daily listing checks. The resulting deal alerts generate an estimated $2.3M in annual commission revenue for their investor clients.
Market Analytics Dashboards
A PropTech startup builds neighborhood-level analytics by aggregating listing data from Zillow, Redfin, and Realtor.com. They track median asking prices, days on market, inventory levels, and price-per-square-foot trends. The data feeds interactive dashboards used by 3,000+ real-estate professionals. Residential proxies ensure consistent data collection across all US metros.
iBuyer Price Modeling
iBuyer companies like Opendoor and OfferPad rely on accurate, current listing data to train their automated valuation models (AVMs). The model needs price history, comparable sales, time-on-market, and neighborhood trends — all data that must be refreshed daily. A 1% improvement in valuation accuracy translates to millions in avoided losses. These teams need real estate scraping proxies that can deliver data reliably at scale, with minimal latency and high success rates.
Key Takeaways
- Datacenter proxies won't work for Zillow, Rightmove, or any major listing platform with serious anti-bot defenses. Residential proxies are the minimum; mobile proxies deliver the highest reliability.
- Geo-targeting is essential — crawl each market with IPs from the same country to see accurate local content and avoid suspicion.
- Deduplication across sources is where most of the engineering complexity lives. Invest in a canonical property model early.
- Price-history tracking is the highest-value data layer. Static snapshots miss the most interesting signals.
- Legal risk is real — especially in the EU. Scrape only public listing data, respect rate limits, and consult legal counsel before scaling.
- Build vs. buy — if your core business isn't data collection, consider whether licensing data feeds is more cost-effective than maintaining a scraping infrastructure. For most PropTech startups, the answer is: start with scraping for speed, migrate to licensed feeds for stability.
Ready to build your real-estate data pipeline? ProxyHat's residential and mobile proxies give you the IP diversity you need to collect listing data at scale, with geo-targeting across 190+ countries. Check available locations or learn more about our web scraping use case.
