Building a Resilient Async Web Crawler in Python: Rate Limiting, Retries, and Concurrency Control

I recently needed to crawl ~50,000 pages from a documentation site for a search index rebuild. The naive approach — requests.get() in a loop — would have taken roughly 14 hours. With asyncio and aiohttp, I got it down to 22 minutes.

But speed isn’t the only challenge. Polite crawlers need rate limiting, retry logic, and error handling. Here is the production-grade pattern I landed on.

The Core Pattern: Semaphore-Controlled Concurrency

The fundamental building block is an asyncio.Semaphore to cap concurrent requests:

import asyncio
import aiohttp
from aiohttp import ClientTimeout, ClientError

class AsyncCrawler:
    def __init__(self, max_concurrent: int = 10, rate_limit: float = 0.1):
        self.semaphore = asyncio.Semaphore(max_concurrent)
        self.rate_limit = rate_limit  # seconds between requests
        self.last_request_time = 0.0
        self.session: aiohttp.ClientSession | None = None

    async def __aenter__(self):
        timeout = ClientTimeout(total=30, connect=10)
        self.session = aiohttp.ClientSession(timeout=timeout)
        return self

    async def __aexit__(self, *args):
        await self.session.close()

The semaphore acts as a gate: at most max_concurrent tasks can hold the semaphore at once. The rest queue up automatically.

Adding Polite Rate Limiting

A semaphore alone doesn’t prevent you from hammering the server when all concurrent tasks fire simultaneously. You need a token bucket or at minimum a time-gated delay:

async def _rate_limit(self):
    """Ensure minimum delay between requests."""
    now = asyncio.get_event_loop().time()
    wait = self.rate_limit - (now - self.last_request_time)
    if wait > 0:
        await asyncio.sleep(wait)
    self.last_request_time = asyncio.get_event_loop().time()

async def fetch(self, url: str) -> str | None:
    async with self.semaphore:
        await self._rate_limit()
        try:
            async with self.session.get(url) as response:
                if response.status == 429:
                    retry_after = int(response.headers.get("Retry-After", 5))
                    print(f"Rate limited on {url}, waiting {retry_after}s")
                    await asyncio.sleep(retry_after)
                    return await self.fetch(url)  # retry once
                response.raise_for_status()
                return await response.text()
        except (ClientError, asyncio.TimeoutError) as e:
            print(f"Failed to fetch {url}: {e}")
            return None

Exponential Backoff with Retries

For transient failures, exponential backoff prevents cascading load:

async def fetch_with_retry(self, url, max_retries=3):
    for attempt in range(max_retries):
        result = await self.fetch(url)
        if result is not None:
            return result
        wait = 2 ** attempt
        print(f"Retry {attempt + 1}/{max_retries} for {url} in {wait}s")
        await asyncio.sleep(wait)
    return None

Managing the Work Queue

Use an asyncio.Queue to stream URLs and avoid memory blow-up:

async def crawl(self, urls):
    queue = asyncio.Queue()
    for url in urls:
        await queue.put(url)

    results = {}

    async def worker():
        while not queue.empty():
            url = await queue.get()
            results[url] = await self.fetch_with_retry(url)
            queue.task_done()

    workers = [asyncio.create_task(worker()) for _ in range(10)]
    await queue.join()
    for w in workers:
        w.cancel()
    return results

Benchmarks from My Project

On a standard VPS with 4 vCPUs:

Sequential (requests.get): ~500 pages/hour
Semaphore (10 concurrent): ~4,500 pages/hour
Semaphore (20 concurrent) + connection pooling: ~8,200 pages/hour

The law of diminishing returns kicks in around 30-50 concurrent connections for most servers. At that point you’re limited by the server’s capacity, not your client.

Full Example: Putting It All Together

async with AsyncCrawler(max_concurrent=20, rate_limit=0.05) as crawler:
    urls = [f"https://docs.example.com/page/{i}" for i in range(1000)]
    results = await crawler.crawl(urls)

# results is a dict: url -> HTML content or None
for url, html in results.items():
    if html:
        process_page(url, html)

This pattern — semaphore + rate limiting + retry + queue — has been handling 50,000-page crawls reliably for months in production. It won the battle between “fast enough” and “polite enough.”

Originally posted on my blog at AIXHDD. Check out more Python development tools and content there.