A retail intelligence team needed to see competitor grocery pricing the way a local shopper sees it: refreshed every four hours, across eight chains and six markets, complete before the morning pricing review. This document sets out what they were trying to achieve, why their existing setup could not deliver it, how the proxy layer was restructured, and what changed as a result.
DoubleData Proxy is a newly separated product line. Rather than publish a client story we cannot yet substantiate, we have reconstructed a deployment from patterns that recur across grocery price intelligence work.
Everything describing our network (pool types, targeting granularity, session behavior, published pricing) is factual. Everything describing the client, their volumes and their results is illustrative. Page 09 sets out the split line by line.
What is a competitor actually charging in a given delivery zone, at a given hour, rather than what their national page shows to a datacenter IP?
Is a competitor's promoted SKU actually available in this delivery zone, or is the promotion running against stock that ran out days ago?
Which products entered or left a category this week, and which of those movements should trigger an assortment response?
The deployment was led jointly by three stakeholders with different definitions of success. Reconciling them mattered more than any single technical decision: a feed that satisfied engineering's uptime target but arrived at 10:30 was, to pricing, a failed feed.
| Owner | What they owned | What success looked like | What they feared |
|---|---|---|---|
| Manager, Marketing Intelligence |
Target list, refresh cadence, category scope, competitive briefing | Every tracked chain represented in every cycle, no silent category drop-outs | Presenting a competitive view built on partial coverage |
| Head of Pricing |
Pricing decisions the feed informed; margin outcomes | A complete, zone-accurate price picture on the desk before the 09:00 review | Repricing against a figure that was stale, or never offered to a real shopper |
| VP, Data Engineering |
Pipeline, delivery SLA, proxy spend, infrastructure stability | Predictable success rates and predictable cost per cycle | Volatile block rates turning a fixed budget into a variable one |
Volume was never the difficulty. Twenty-five thousand SKUs is a modest crawl by any measure.
The difficulty was that the same crawl was supposed to complete six times a day, across eight independently defended estates, with no gaps, and that one of the three jobs required holding a session open while the other two required discarding identity as fast as possible.
It never did. At a 5h 10m sweep the existing setup could fit four cycles into a day running back to back. The four-hour cadence was an intention, not a schedule.
Read as a business requirement, this is a single project: monitor competitor grocery pricing. Read as a network requirement, it is three unrelated workloads that happen to share a scheduler. This distinction is where most single-pool deployments quietly fail.
Grocery chains price by delivery zone. The same SKU carries a different figure in Munich and in Hamburg, and both differ from what the national landing page renders to an unrecognized visitor. To capture the price a shopper was actually offered, the request has to originate inside that zone, which means city-level targeting, not country-level. High request count, minimal state, and the highest tolerance for identity churn of the three workloads.
Availability as the retailer publishes it (in stock, low stock, unavailable), read from the product page exactly as rendered to an ordinary visitor. Grocery estates resolve this per delivery zone, so the zone has to be selected once and then hold across several hundred subsequent page reads. Rotate the IP mid-traverse and the zone resets: the request returns 200, the page renders against a different zone, and the pipeline logs a success it should have logged as a fault. Low volume, high state, zero tolerance for identity change.
A weekly sweep of every tracked category to establish what entered and what left. Broad, shallow, and repetitive, with no session state to preserve, but a very large surface area and a strong preference for IPs that have not been seen on the same estate recently. The workload furthest in character from availability signals, despite feeding the same dashboard.
The team arrived running a single rotating datacenter pool across all three workloads. It had been adequate eighteen months earlier. By the time of this deployment it was returning roughly a third of what it requested and, more damaging, it was doing so without the pipeline registering a fault.
Modeled rates against defended grocery estates, as usable records per request issued. Actual rates are target-specific.
ASN reputation and TLS fingerprinting resolve during the handshake. A datacenter IP is classified before it asks for a page, so header tuning cannot recover an outcome already decided at the network layer.
Not a contested point: our own datacenter documentation states that the product is not recommended for targets with advanced anti-bot protection, major e-commerce platforms included.
At a 34% success rate the crawler issues close to three requests per usable record. Cycle duration stretches past the delivery window, and capacity goes to attempts rather than data.
Leasing more datacenter IPs buys concurrency to retry faster. It does not move the 34% ceiling, and it does nothing at all for zone resolution: the problem is which page the server returns, not how quickly it is asked.
The availability workload breaks silently. The session resets, the request completes with a 200, the record is empty, and no monitor fires, because nothing technically failed.
Most mornings the feed arrived incomplete or late, so pricing fell back on the previous day's figure, which in a market adjusting intraday meant repricing against a number no shopper was ever shown. The availability gap compounded it: promotions were benchmarked against competitor SKUs unavailable for days.
Datacenter pools are typically sold on a flat per-IP basis. Under that model a collapsing success rate never appears on the invoice at all. It appears as throughput: the same leased IPs spending two thirds of their capacity on responses that yield nothing.
That is why the sweep took 5h 10m and the cadence was never met: capacity went to retries, not records. A flat per-IP rate hides a failing success rate rather than surfacing it.
The restructure added no capacity. It split one undifferentiated stream into two routed paths, each matched to the workload it carries, with a defined escalation rule when a target pushes back. Requests per usable record fell by two thirds; usable records per cycle rose.
Carries price sweeps and assortment crawls. Rotation per request, so no identity persists long enough to build a history on the estate. City-level targeting per market is what makes a captured price the price that zone was actually offered.
Billed per gigabyte on a monthly tier: a blocked request still returns bytes, and those bytes are billed, so success rate translates directly into spend. A deployment at this volume sits in the enterprise band.
Carries zone-scoped availability reads. A dedicated IP whose session persists for as long as the target's session does, with no expiry window to design around, so a traverse taking four minutes and one taking forty behave identically. Dedicated allocation keeps reputation under the client's control rather than shared across tenants.
Billed per IP: flat per identity regardless of how hard the estate pushes back. Against page 05, though: a flat rate is only predictable once the success rate is high and monitored. On a failing pool it makes the invoice predictable, not the delivery.
Four weeks from first call to full cadence, deliberately sequenced so that the highest-risk workload (zone-scoped availability reads) was proven on one estate before the remaining seven were switched over. No estate was cut over until its replacement path had run clean for a full cycle.
Existing pipeline measured per estate rather than in aggregate. Silent failures identified by comparing 200-responses against records carrying the requested zone: the gap the previous monitoring had missed entirely.
Residential and ISP paths configured against a single German estate. Price and assortment moved first; zone-scoped reads held back until zone persistence was confirmed across full category traverses.
403 handling rewritten to switch pool type rather than retry in place. Consecutive-failure pausing added so a defended estate raises an alert instead of quietly draining budget.
Remaining seven estates migrated. Six-cycle daily schedule enabled across all markets, with the 09:00 delivery deadline monitored as a first-class SLA rather than an aspiration.
The client's scrapers were not rewritten. Parsing logic, scheduling and storage stayed as they were.
The entire intervention sat at the network layer and in the retry logic above it, which is why four weeks was realistic rather than optimistic.
The previous pool stayed provisioned and reachable throughout. Reverting an estate was a routing change, not a migration, which is what made it reasonable to move seven estates in a single week.
Usable records per estate, zone-resolution rate, and the count of 200-responses returning nothing. The third metric had not existed before week 1 and became the one that mattered most.
Any estate whose zone-resolution rate failed to clear the baseline within one cycle of cutover. None did, but the criterion was set before the work started, not after the results arrived.
Measured across the twelve weeks after full rollout, against the four weeks preceding week 1. All figures are modeled, and the definitions below let the arithmetic be checked rather than taken on trust. Sweep duration falls in proportion to requests per record at unchanged concurrency.
A usable record is one SKU observation carrying every required field (price, identifier, availability status and the zone it was read from), parsed from a response that returned the page actually requested. An HTTP 200 rendering a challenge, an empty template or a different zone is not a usable record. Success rate here therefore means usable records ÷ requests issued, not HTTP 200s ÷ requests issued, a gap that is the whole of failure mode 03.
| Metric | Baseline | After restructure | Change |
|---|---|---|---|
| Usable records per request issued (blended) | 34% | 94% | +60 pts |
| Requests issued per usable record | 2.94 | 1.06 | −64% |
| Full sweep duration, concurrency unchanged | 5h 10m | 1h 52m | −64% |
| Cycles completed per day (target: 6) | 4 | 6 | Target met |
| Zone-scoped reads resolving to the requested zone | 22% | 96% | +74 pts |
| Cycle-to-cycle variance in requests issued | High | Low | Stabilised |
| Estates missing from a daily cycle | 3 of 8 | 0 of 8 | Eliminated |
A zone-accurate picture on the desk before the review means repricing against what competitors charge this morning, in this zone, not a national average captured yesterday.
And matching a competitor's headline price on a SKU unavailable in that zone is margin given away for nothing.
Requests per usable record fell from 2.94 to 1.06. On the residential path, billed per GB, that is bandwidth no longer spent retrieving pages that yield nothing. On the ISP path, billed per IP, it is capacity returned to useful work rather than to retries.
And silent failures became visible ones: a pipeline that alerts is cheaper to run than one that quietly reports success.
Vendor case studies routinely present unverifiable numbers as measurement. We would rather state the line explicitly. Everything below is separated into what we can substantiate about our own network and what has been constructed to illustrate a deployment pattern.
Success rates reflect the published and commonly reported spread between proxy types against defended commercial estates, not a measurement at a named client. Durations follow from those rates applied to the stated SKU count and cadence; requests-per-record is arithmetic on the success rate, not an independent observation. Pricing is quoted at published entry rates and is not converted into a total cost figure, since that would require volume tiers and request weights specific to each deployment. Where a number could have been rounded flatteringly it has been left where the arithmetic put it: a 94% blended rate rather than a headline 99%, because a blend weighted toward the residential path cannot exceed that path's own 94%.
Price intelligence sits close enough to a target's commercial interests that the boundaries matter as much as the method. These are the limits applied throughout the deployment described here.
Public product and category pages only, read as rendered to an ordinary visitor. No authentication, no account creation, no interaction with checkout or payment flows, and no attempt to reach content behind an access control. A delivery zone is selected the same way any shopper selects one. Fields collected are commercial (price, identifier, availability, category, zone), and no personal data is collected at any point. Request rates are shaped so collection stays a marginal share of an estate's ordinary traffic.
DoubleData Proxy supplies network infrastructure and consent-based IP sourcing. What is collected, from which targets, and under which lawful basis remains the customer's decision and responsibility.
A DPA is available; our compliance documentation covers sub-processor transparency, audit rights and data residency. EU teams should expect to run their own assessment against the terms of the estates they intend to collect from. That assessment is not something an infrastructure provider can make on their behalf.
Residential, ISP, mobile, datacenter and dedicated pools, routed to the workload rather than sold as a single undifferentiated stream. Consent-based sourcing, city-level targeting across 190+ countries, and engineers on call when an estate changes its defenses.
Coverage, quality and infrastructure reviewed against your live target list. No obligation.
A scoping call with someone who has unblocked your category before, not a sales qualifier.
Routing configured against your own estates, measured per target rather than blended.