Building a rotating proxy pool in Python with asyncio

Synchronous requests will bottleneck any serious data extraction pipeline. A concurrent Python proxy pool built on asyncio and aiohttp lets you scale scraping throughput while handling IP rotations, retries, and connection limits.

Why standard aiohttp proxy handling breaks at scale

When engineers move from synchronous libraries like requests to asynchronous frameworks, the typical approach is to pass a single proxy string into an aiohttp.ClientSession. This works for a handful of requests. When you scale to thousands of concurrent connections, this pattern breaks down rapidly.

Firing massive bursts of concurrent requests through a single proxy node or an unmanaged list leads to socket exhaustion, localized rate limits, and immediate target bans. A reliable async proxy scraping pipeline requires strict concurrency control. You must limit how many active requests run simultaneously and distribute those requests intelligently across your available IP addresses.

Designing the Python proxy pool

A production-ready rotating proxy Python implementation needs three distinct components. First, it requires a dependable queue system to cycle through available IP addresses. Second, it needs a concurrency limiter, typically an asyncio.Semaphore, to prevent the script from opening too many sockets at once. Third, it needs intelligent error handling to distinguish between network timeouts, proxy authentication failures, and target-level bans like HTTP 429.

We will construct a ProxyPool class that initializes an asyncio.Queue. Every available proxy URL is loaded into this queue. When a worker task requires a connection, it pulls a proxy from the queue, executes the request, and returns the proxy to the back of the queue. This creates an infinite, non-blocking round-robin rotation.

The core asyncio implementation

Start with the infrastructure for our proxy manager. We use a context manager pattern to ensure proxies are always returned to the queue, even if the underlying request raises an unhandled exception.

import asyncio
import aiohttp
import logging

logging.basicConfig(level=logging.INFO)

class ProxyPool:
    def __init__(self, proxies: list):
        self.queue = asyncio.Queue()
        for proxy in proxies:
            self.queue.put_nowait(proxy)
        self.total_proxies = len(proxies)

    async def get_proxy(self):
        return await self.queue.get()

    def return_proxy(self, proxy):
        self.queue.put_nowait(proxy)

class ProxyContext:
    def __init__(self, pool: ProxyPool):
        self.pool = pool
        self.proxy = None

    async def __aenter__(self):
        self.proxy = await self.pool.get_proxy()
        return self.proxy

    async def __aexit__(self, exc_type, exc_val, exc_tb):
        self.pool.return_proxy(self.proxy)

Matching infrastructure to the target

Code alone cannot bypass anti-bot systems. If you route 500 concurrent requests through a small list of flagged IPs, the target server will block your entire pool. The success of your aiohttp proxy implementation heavily depends on the underlying network.

For scraping public APIs, non-protected internal tools, or generating high-volume market research data, deploying our datacenter IPs provides the highest throughput at the lowest cost. Datacenter infrastructure supports rapid concurrent connections without the latency overhead typical of peer-to-peer networks.

However, if your target uses aggressive bot mitigation or restricts access based on IP reputation, you must adapt. Sneaker sites, major social media platforms, and prominent e-commerce marketplaces inspect connection signatures closely. In these scenarios, routing your traffic through real residential proxies is mandatory. Residential nodes appear as standard consumer devices to the target server, drastically reducing the rate of blocked requests and CAPTCHA triggers.

Handling HTTP 429 and proxy bans

Even with high-quality IPs, network instability and rate limits occur. Your worker function must intercept specific status codes. If a target returns HTTP 429 Too Many Requests or HTTP 403 Forbidden, the worker needs to delay execution or cycle to a different IP.

Implement the worker function using our ProxyPool. We include a retry mechanism that catches common connection errors and respects the queue structure.

async def fetch_data(url: str, session: aiohttp.ClientSession, pool: ProxyPool, semaphore: asyncio.Semaphore):
    max_retries = 3
    
    async with semaphore:
        for attempt in range(max_retries):
            async with ProxyContext(pool) as proxy:
                try:
                    async with session.get(url, proxy=proxy, timeout=10) as response:
                        if response.status == 200:
                            return await response.text()
                        elif response.status in [429, 403]:
                            logging.warning(f"Target blocked proxy {proxy}. Retrying.")
                            await asyncio.sleep(2)
                            continue
                        else:
                            response.raise_for_status()
                            
                except (aiohttp.ClientError, asyncio.TimeoutError) as e:
                    logging.error(f"Connection error on {proxy}: {str(e)}")
                    await asyncio.sleep(1)
        
        logging.error(f"Failed to fetch {url} after {max_retries} attempts.")
        return None

Executing the extraction pipeline

To run the pipeline, we instantiate the pool, define the concurrency limit, and gather our tasks. Managing concurrency is critical. If your server allows too many open file descriptors, the operating system will forcefully terminate the script. An asyncio.Semaphore prevents this by capping the number of active fetch_data coroutines.

async def main():
    proxy_list = [
        "http://user:pass@proxy1.example.com:8000",
        "http://user:pass@proxy2.example.com:8000",
        "http://user:pass@proxy3.example.com:8000"
    ]
    
    target_urls = [f"https://example.com/data/page/{i}" for i in range(1, 101)]
    
    pool = ProxyPool(proxy_list)
    semaphore = asyncio.Semaphore(20) 
    
    connector = aiohttp.TCPConnector(limit=0)
    async with aiohttp.ClientSession(connector=connector) as session:
        tasks = [fetch_data(url, session, pool, semaphore) for url in target_urls]
        results = await asyncio.gather(*tasks)
        
    successful_fetches = [r for r in results if r is not None]
    logging.info(f"Successfully scraped {len(successful_fetches)} out of {len(target_urls)} pages.")

if __name__ == "__main__":
    asyncio.run(main())

By passing limit=0 to the TCPConnector, we disable aiohttp internal connection limiting. We do this because our asyncio.Semaphore is already handling concurrency control at the task level, providing more predictable execution for our rotation logic.

Prune dead proxies as your volume grows

Building a custom Python proxy pool gives you fine-grained control over request timing, retry logic, and concurrent connections. The implementation above provides a solid foundation for most mid-scale data extraction tasks. As your daily request volume scales into the millions, you will need to add more sophisticated logic to automatically prune dead proxies from the queue, monitor success rates per IP, and rotate headers.

Enterprise workloads often pull engineering resources away from actual data processing and into maintaining custom rotation infrastructure. In these cases, moving to enterprise-grade custom infrastructure allows you to offload the rotation logic entirely. Our backconnect gateways handle the queueing, load balancing, and automated ban recovery at the network level, letting your code remain simple and focused strictly on extraction.

If you need help sizing the right proxy stack for your use case, talk to our team.

Custom proxy solutions

Proxies tailored to your use case

Our team designs custom proxy infrastructure for B2B data teams scraping at scale. Residential, datacenter, ISP, and mobile pools with SLA, GDPR-ready sourcing, and dedicated account management.

Talk to our team