An illustrative deployment of residential and ISP proxy routing across eight European grocery chains
Deployment Case Study
DoubleData Proxy
Infrastructure Series 01
Grocery Price
Intelligence at
Four-Hour Cadence
What this
case covers
A retail intelligence team needed to see competitor grocery pricing the way a local shopper sees it — refreshed every four hours, across eight chains and six markets, complete before the morning pricing review. This document sets out what they were trying to achieve, why their existing setup could not deliver it, how the proxy layer was restructured, and what changed as a result.
Deployment snapshot
- Sector
- Online grocery / q-commerce
- Markets
- DE, FR, PL, NL, ES, IT
- Target chains
- 8 national retailers
- SKUs in rotation
- ~25,000
- Refresh cadence (target)
- Every 4 hours · 6 cycles/day
- Delivery deadline
- 09:00 local, daily
- Proxy types deployed
- Residential + ISP
- Observation window
- 12 weeks
DoubleData Proxy is a newly separated product line. Rather than publish a client story we cannot yet substantiate, we have reconstructed a deployment from patterns that recur across grocery price intelligence work.
Everything describing our network — pool types, targeting granularity, session behaviour, published pricing — is factual. Everything describing the client, their volumes and their results is illustrative. Page 09 sets out the split line by line.
Data points collected
The three questions this deployment had to answer
What is a competitor actually charging in a given delivery zone, at a given hour — not what their national page shows to a datacenter IP?
Is a competitor's promoted SKU actually available in this delivery zone, or is the promotion running against stock that ran out days ago?
Which products entered or left a category this week, and which of those movements should trigger an assortment response?
Three owners,
one shared feed
The deployment was led jointly by three stakeholders with different definitions of success. Reconciling them mattered more than any single technical decision: a feed that satisfied engineering's uptime target but arrived at 10:30 was, to pricing, a failed feed.
| Owner | What they owned | What success looked like | What they feared |
|---|---|---|---|
|
Manager, Marketing Intelligence |
Target list, refresh cadence, category scope, competitive briefing | Every tracked chain represented in every cycle, no silent category drop-outs | Presenting a competitive view built on partial coverage |
|
Head of Pricing |
Pricing decisions the feed informed; margin outcomes | A complete, zone-accurate price picture on the desk before the 09:00 review | Repricing against a figure that was stale, or never offered to a real shopper |
|
VP, Data Engineering |
Pipeline, delivery SLA, proxy spend, infrastructure stability | Predictable success rates and predictable cost per cycle | Volatile block rates turning a fixed budget into a variable one |
Success criteria agreed before work started
- Completeness over volume. A cycle counted as successful only if every chain and every tracked category returned data — not if the total record count looked healthy.
- Zone accuracy. Prices had to be captured from inside the delivery zone they belonged to. A national average was explicitly out of scope.
- Zone-resolved availability, not a national flag. A country-level in-stock indicator was already available more cheaply; the value sat in knowing availability inside each delivery zone.
- Predictable spend. Engineering needed cost per cycle to stay within a stated band rather than fluctuating with block rates.
- Delivery before 09:00. Any feed arriving after the pricing review was treated as missing, regardless of quality.
Volume was never the difficulty. Twenty-five thousand SKUs is a modest crawl by any measure.
The difficulty was that the same crawl was supposed to complete six times a day, across eight independently defended estates, with no gaps — and that one of the three jobs required holding a session open while the other two required discarding identity as fast as possible.
It never did. At a 5h 10m sweep the existing setup could fit four cycles into a day running back to back. The four-hour cadence was an intention, not a schedule.
One feed,
three traffic
patterns
Read as a business requirement, this is a single project: monitor competitor grocery pricing. Read as a network requirement, it is three unrelated workloads that happen to share a scheduler. This distinction is where most single-pool deployments quietly fail.
01 · Price monitoring
Grocery chains price by delivery zone. The same SKU carries a different figure in Munich and in Hamburg, and both differ from what the national landing page renders to an unrecognised visitor. To capture the price a shopper was actually offered, the request has to originate inside that zone — which means city-level targeting, not country-level. High request count, minimal state, and the highest tolerance for identity churn of the three workloads.
02 · Availability signals
Availability as the retailer publishes it — in stock, low stock, unavailable — read from the product page exactly as rendered to an ordinary visitor. Grocery estates resolve this per delivery zone, so the zone has to be selected once and then hold across several hundred subsequent page reads. Rotate the IP mid-traverse and the zone resets: the request returns 200, the page renders against a different zone, and the pipeline logs a success it should have logged as a fault. Low volume, high state, zero tolerance for identity change.
03 · Assortment mapping
A weekly sweep of every tracked category to establish what entered and what left. Broad, shallow, and repetitive — no session state to preserve, but a very large surface area and a strong preference for IPs that have not been seen on the same estate recently. The workload furthest in character from availability signals, despite feeding the same dashboard.
Why the
existing setup
could not hold
The team arrived running a single rotating datacenter pool across all three workloads. It had been adequate eighteen months earlier. By the time of this deployment it was returning roughly a third of what it requested — and, more damaging, it was doing so without the pipeline registering a fault.
Modelled rates against defended grocery estates, as usable records per request issued. Actual rates are target-specific.
Rejected before the request
ASN reputation and TLS fingerprinting resolve during the handshake. A datacenter IP is classified before it asks for a page, so header tuning cannot recover an outcome already decided at the network layer.
Not a contested point: our own datacenter documentation states that the product is not recommended for targets with advanced anti-bot protection, major e-commerce platforms included.
Retries multiply real volume
At a 34% success rate the crawler issues close to three requests per usable record. Cycle duration stretches past the delivery window, and capacity goes to attempts rather than data.
Leasing more datacenter IPs buys concurrency to retry faster. It does not move the 34% ceiling, and it does nothing at all for zone resolution — the problem is which page the server returns, not how quickly it is asked.
Rotation destroys session state
The availability workload breaks silently. The session resets, the request completes with a 200, the record is empty — and no monitor fires, because nothing technically failed.
What this cost the business
For the Pricing Department
Most mornings the feed arrived incomplete or late, so pricing fell back on the previous day's figure — in a market adjusting intraday, repricing against a number no shopper was ever shown. The availability gap compounded it: promotions were benchmarked against competitor SKUs unavailable for days.
For Data Engineering
Datacenter pools are typically sold on a flat per-IP basis. Under that model a collapsing success rate never appears on the invoice at all — it appears as throughput: the same leased IPs spending two thirds of their capacity on responses that yield nothing.
That is why the sweep took 5h 10m and the cadence was never met: capacity went to retries, not records. A flat per-IP rate hides a failing success rate rather than surfacing it.
Routing,
not a pool
The restructure added no capacity. It split one undifferentiated stream into two routed paths, each matched to the workload it carries, with a defined escalation rule when a target pushes back. Requests per usable record fell by two thirds; usable records per cycle rose.
Carries price sweeps and assortment crawls. Rotation per request, so no identity persists long enough to build a history on the estate. City-level targeting per market is what makes a captured price the price that zone was actually offered.
Billed per gigabyte on a monthly tier — a blocked request still returns bytes, and those bytes are billed, so success rate translates directly into spend. A deployment at this volume sits in the enterprise band.
Carries zone-scoped availability reads. A dedicated IP whose session persists for as long as the target's session does — no expiry window to design around, so a traverse taking four minutes and one taking forty behave identically. Dedicated allocation keeps reputation under the client's control rather than shared across tenants.
Billed per IP — flat per identity regardless of how hard the estate pushes back. Against page 05, though: a flat rate is only predictable once the success rate is high and monitored. On a failing pool it makes the invoice predictable, not the delivery.
Routing rules in force
How the
rollout ran
Four weeks from first call to full cadence, deliberately sequenced so that the highest-risk workload — zone-scoped availability reads — was proven on one estate before the remaining seven were switched over. No estate was cut over until its replacement path had run clean for a full cycle.
Audit & baseline
Existing pipeline measured per estate rather than in aggregate. Silent failures identified by comparing 200-responses against records carrying the requested zone — the gap the previous monitoring had missed entirely.
Path split, one market
Residential and ISP paths configured against a single German estate. Price and assortment moved first; zone-scoped reads held back until zone persistence was confirmed across full category traverses.
Escalation logic
403 handling rewritten to switch pool type rather than retry in place. Consecutive-failure pausing added so a defended estate raises an alert instead of quietly draining budget.
Full cadence
Remaining seven estates migrated. Six-cycle daily schedule enabled across all markets, with the 09:00 delivery deadline monitored as a first-class SLA rather than an aspiration.
Decisions that mattered more than they looked
- Measuring per estate, not in aggregate. The aggregate success rate had been masking two estates performing far below the mean. Averages hide exactly the failures that matter.
- Treating an empty 200 as a failure. Until the pipeline counted silent session collapses as errors, no amount of proxy tuning would have shown up as an improvement.
- Escalating across pool types rather than retrying. Retrying a blocked request on the same pool type reproduces the block and pays for it twice.
- Pausing rather than pushing through. An estate that has started challenging is best left alone briefly; continuing hardens its response and costs the following cycles.
The client's scrapers were not rewritten. Parsing logic, scheduling and storage stayed as they were.
The entire intervention sat at the network layer and in the retry logic above it — which is why four weeks was realistic rather than optimistic.
The previous pool stayed provisioned and reachable throughout. Reverting an estate was a routing change, not a migration — which is what made it reasonable to move seven estates in a single week.
Usable records per estate, zone-resolution rate, and the count of 200-responses returning nothing. The third metric had not existed before week 1 and became the one that mattered most.
Any estate whose zone-resolution rate failed to clear the baseline within one cycle of cutover. None did — but the criterion was set before the work started, not after the results arrived.
What
changed
Measured across the twelve weeks after full rollout, against the four weeks preceding week 1. All figures are modelled, and the definitions below let the arithmetic be checked rather than taken on trust. Sweep duration falls in proportion to requests per record at unchanged concurrency.
A usable record is one SKU observation carrying every required field — price, identifier, availability status and the zone it was read from — parsed from a response that returned the page actually requested. An HTTP 200 rendering a challenge, an empty template or a different zone is not a usable record. Success rate here therefore means usable records ÷ requests issued, not HTTP 200s ÷ requests issued — a gap that is the whole of failure mode 03.
| Metric | Baseline | After restructure | Change |
|---|---|---|---|
| Usable records per request issued (blended) | 34% | 94% | +60 pts |
| Requests issued per usable record | 2.94 | 1.06 | −64% |
| Full sweep duration, concurrency unchanged | 5h 10m | 1h 52m | −64% |
| Cycles completed per day (target: 6) | 4 | 6 | Target met |
| Zone-scoped reads resolving to the requested zone | 22% | 96% | +74 pts |
| Cycle-to-cycle variance in requests issued | High | Low | Stabilised |
| Estates missing from a daily cycle | 3 of 8 | 0 of 8 | Eliminated |
What this means for your business
For Pricing & Category
A zone-accurate picture on the desk before the review means repricing against what competitors charge this morning, in this zone — not a national average captured yesterday.
And matching a competitor's headline price on a SKU unavailable in that zone is margin given away for nothing.
For Data Engineering
Requests per usable record fell from 2.94 to 1.06. On the residential path, billed per GB, that is bandwidth no longer spent retrieving pages that yield nothing. On the ISP path, billed per IP, it is capacity returned to useful work rather than to retries.
And silent failures became visible ones — a pipeline that alerts is cheaper to run than one that quietly reports success.
What is real
and what is
modelled
Vendor case studies routinely present unverifiable numbers as measurement. We would rather state the line explicitly. Everything below is separated into what we can substantiate about our own network and what has been constructed to illustrate a deployment pattern.
- Pool types offered: residential, ISP, datacenter, dedicated datacenter, mobile
- Network scale of 150M+ IPs across 195 countries
- Targeting granularity to city, region and carrier
- Residential sticky sessions configurable at 1, 10 or 30 minutes
- ISP proxies are permanently static, from a pool of 700,000+, available dedicated or shared
- City and ZIP-level geo-targeting included at no extra cost; ASN and carrier targeting supported
- 99.9% uptime SLA with penalty credits
- Consent-based IP sourcing
- Published pricing. Residential and mobile run on monthly plans, with the per-GB rate falling as the tier rises: residential $4.20 / GB pay-as-you-go down to $2.93 / GB at the $1,999 enterprise tier; mobile $8.40 down to $5.87 / GB. ISP from $1.30 / IP (min. 10 IPs), datacenter from $0.75 / IP (min. 5 IPs) or $0.44 / GB (min. 80 GB), dedicated datacenter from $1.20 / IP. No billing period is published for the per-IP products, and none is assumed here.
- Datacenter pool of 770,000+ IPs; standard datacenter plans are shared, with exclusive IPs available on request
- API and dashboard management, 24/7 engineering support
- The client, their identity, sector position and internal structure
- Eight chains, six markets, ~25,000 SKUs, six cycles daily
- All success rates, durations and percentages on pages 5 and 8
- The four-week rollout sequence and its weekly milestones
- Baseline figures attributed to the previous single-pool setup
- The relationship drawn between success rate and throughput, which applies published entry rates to modelled success rates. No total cost, cost per record or monthly figure is calculated anywhere in this document.
How the modelled figures were constructed
Success rates reflect the published and commonly reported spread between proxy types against defended commercial estates, not a measurement at a named client. Durations follow from those rates applied to the stated SKU count and cadence; requests-per-record is arithmetic on the success rate, not an independent observation. Pricing is quoted at published entry rates and is not converted into a total cost figure, since that would require volume tiers and request weights specific to each deployment. Where a number could have been rounded flatteringly it has been left where the arithmetic put it — a 94% blended rate rather than a headline 99%, because a blend weighted toward the residential path cannot exceed that path's own 94%.
What was
collected —
and what
was not
Price intelligence sits close enough to a target's commercial interests that the boundaries matter as much as the method. These are the limits applied throughout the deployment described here.
Public product and category pages only, read as rendered to an ordinary visitor. No authentication, no account creation, no interaction with checkout or payment flows, and no attempt to reach content behind an access control. A delivery zone is selected the same way any shopper selects one. Fields collected are commercial — price, identifier, availability, category, zone — and no personal data is collected at any point. Request rates are shaped so collection stays a marginal share of an estate's ordinary traffic.
- Publicly rendered pages only; no logged-in state at any point
- No circumvention of authentication or access controls
- No checkout, payment or order flows touched
- Commercial fields only — no personal data collected
- Request rates shaped to avoid burdening target infrastructure
DoubleData Proxy supplies network infrastructure and consent-based IP sourcing. What is collected, from which targets, and under which lawful basis remains the customer's decision and responsibility.
A DPA is available; our compliance documentation covers sub-processor transparency, audit rights and data residency. EU teams should expect to run their own assessment against the terms of the estates they intend to collect from — not something an infrastructure provider can make on their behalf.
How we would establish these numbers on your targets
- Free data audit against your live target list
- No obligation
- Success rate measured per target, not blended
- Included
- Proxy configuration and rotation policy review
- Included
- Silent-failure detection across your existing pipeline
- Included
- Coverage gaps and optimisation roadmap, in writing
- Included
- Commitment required before results are delivered
- None
Proxy infrastructure for teams whose data has a deadline.
Residential, ISP, mobile, datacenter and dedicated pools — routed to the workload rather than sold as a single undifferentiated stream. Consent-based sourcing, city-level targeting across 195 countries, and engineers on call when an estate changes its defences.
Where to start
Coverage, quality and infrastructure reviewed against your live target list. No obligation.
A scoping call with someone who has unblocked your category before — not a sales qualifier.
Routing configured against your own estates, measured per target rather than blended.
