- Data analytics
- Proxy infrastructure
Real estate scraping: bypassing the toughest portals
Technical guide on bypassing enterprise WAFs like Akamai and Datadome in real estate scraping. Learn how to extract property data consistently at scale.
Table of contents
Real estate platforms protect their data aggressively. High-value targets like Zillow, Rightmove, and Idealista deploy enterprise anti-bot systems to block automated requests within seconds. This guide details the detection mechanisms behind the most heavily defended property portals and the technical infrastructure required to extract data without triggering blocks.
Enterprise WAFs in the property sector
Property sites face massive volumes of automated traffic from competitors, aggregators, and investment firms. To protect their bandwidth and proprietary datasets, these portals rely on Web Application Firewalls (WAFs) like Akamai Bot Manager, Datadome, and Cloudflare Turnstile.
Besides checking your IP address, these systems perform deep packet inspection and client-side behavioral analysis. When you send a standard HTTP request using libraries like Python Requests or Axios, the WAF analyzes your TLS fingerprint. A mismatch between your stated User-Agent and your TLS negotiation parameters results in an immediate HTTP 403 Forbidden or a redirect to a CAPTCHA challenge.
Successful real estate scraping depends on your automation presenting a consistent identity across the network layer, the HTTP headers, and the browser execution environment.
Zillow scraping constraints
Zillow is notoriously difficult to scrape at scale. The platform relies heavily on Akamai to filter incoming traffic. Akamai maintains a scoring system that evaluates every request based on hundreds of signals before allowing access to the origin server.
Standard datacenter IPs fail almost instantly on Zillow. The platform automatically flags traffic originating from known cloud providers like AWS, DigitalOcean, or Google Cloud. Furthermore, Zillow scraping requires careful attention to header ordering. Akamai expects headers to arrive in the exact order generated by modern browsers (like Chrome or Firefox). Sending custom headers or omitting standard ones like Sec-Fetch-Dest will spike your risk score.
Many teams attempt to bypass this by rotating generic proxies and brute-forcing requests. This approach results in shadow-banning. Zillow will return a HTTP 200 OK status but serve a cached, empty, or structurally altered HTML response instead of the actual listings.
European portals and regional blocking
European property portals present different challenges. Platforms like Rightmove (UK) and Idealista (Spain, Italy) enforce strict geographic boundaries. Rightmove assumes that legitimate users searching for UK properties are located in the UK. Any traffic originating from US or Asian data centers is subjected to extreme rate limits or blocked entirely.
Idealista relies on Datadome to protect its endpoints. Datadome injects a JavaScript payload into the browser that tracks mouse movements, keystroke dynamics, and canvas rendering. If your headless browser fails to execute this script or returns anomalous hardware concurrency values, Datadome blocks the session.
Extracting data from these sites requires localized request origins and solid JavaScript rendering capabilities. You cannot rely on simple HTML parsing. You must often load the page in a fully emulated browser environment to pass the initial security checks before property listings scraping can begin.
Infrastructure for property listings scraping
Relying on standard servers for your scraping pipeline will bottleneck your data collection. When portals track your origin to a server farm, they apply zero-trust rules to your session. Overcoming this requires masking your traffic through devices that carry high inherent trust.
The most reliable method to achieve this is routing your requests through a rotating residential proxy network. By using IPs assigned by consumer ISPs to real home networks, your requests blend in with regular user traffic. The target portal sees a residential broadband connection instead of an automated server. This significantly reduces your block rate and minimizes CAPTCHA triggers.
A properly configured real estate proxy setup must also handle connection persistence. When you need to scrape pagination links or load dynamically rendered map clusters, your IP must remain static for the duration of the workflow. Rotating on every single request during a continuous session will alert the WAF to suspicious behavior.
The importance of localized IPs
Because real estate markets are highly regional, your scraper must adopt the appropriate regional identity. If you are scraping a portal based in Germany, your requests should ideally originate from German ISPs. Mismatched geolocation data is a primary trigger for modern anti-bot systems.
Avoiding regional blocks takes infrastructure that allows you to target precise city-level IPs. This allows your scraper to mimic localized user behavior accurately. For example, scraping a localized sub-domain of a property portal using an IP from that exact metropolitan area drastically lowers the probability of triggering rate limits.
Localized targeting is particularly important when dealing with pricing data. Some portals alter the properties shown or the associated fees based on the user's apparent location to comply with regional regulations.
Managing vertical-specific WAF rules
Each property portal maintains custom rate limits and block thresholds. A request frequency that works perfectly on Redfin might trigger a permanent IP ban on Realtor.com within seconds. Engineering teams often spend weeks tuning concurrent connection limits, delay randomization, and rotation algorithms for each specific domain.
To reduce this engineering overhead, many data teams utilize proxy configurations designed for real estate portals. These specialized pools group IPs that have a proven success rate against major property WAFs. Using infrastructure optimized for the real estate vertical means you spend less time reverse-engineering Datadome or Akamai rule sets and more time parsing the actual listing data.
Client-side execution and fingerprinting
Even with high-reputation IPs, your client software must look human. When using headless browsers like Puppeteer or Playwright, you must patch the default configuration. Out of the box, headless Chrome exposes the navigator.webdriver = true property, which immediately identifies your session as a bot.
- Use stealth plugins to strip automation variables from the browser environment.
- Ensure your WebGL and Canvas fingerprints match the operating system specified in your User-Agent.
- Randomize your viewport sizes to match standard monitor resolutions rather than the default headless dimensions.
- Limit concurrent headless instances. Spawning 100 parallel browser sessions from a single IP will fail regardless of your proxy quality.
When you only need raw data, investigate the platform's internal APIs. Many portals load listing details via XHR requests returning JSON payloads. If you can replicate the exact headers, cookies, and tokens required by these endpoints, you can bypass the heavy browser rendering entirely. This significantly reduces your infrastructure costs and increases extraction speed.
Production scraping rests on trusted IPs and clean fingerprints
Scaling a real estate scraper from a local script to a production-grade data pipeline requires resilient infrastructure. Enterprise portals will continuously update their anti-bot logic. Your success depends on using high-trust IP networks, maintaining precise localized targeting, and managing your client-side fingerprints effectively.
If you need help sizing the right proxy stack for your specific use case, talk to our team.
