Building a Robust Python Web Crawler (2026 Guide)
Want to build a python web crawler that is fast, polite and reliable? This guide shows you how to write a robust python web crawler with rate limiting, error handling and clean data extraction.
I recently needed to crawl ~50,000 pages from a documentation site for a search index rebuild. The naive approach — requests.get() in a loop — would have taken roughly 14 hours. With asyncio and aiohttp, I got it down to 22 minutes.
But speed isn’t the only challenge. Polite crawlers need rate limiting, retry logic, and error handling. Here is the production-grade pattern I landed on.
The Core Pattern: Semaphore-Controlled Concurrency
The fundamental building block is an asyncio.Semaphore to cap concurrent requests:
import asyncio
import aiohttp
from aiohttp import ClientTimeout, ClientError
class AsyncCrawler:
def __init__(self, max_concurrent: int = 10, rate_limit: float = 0.1):
self.semaphore = asyncio.Semaphore(max_concurrent)
self.rate_limit = rate_limit # seconds between requests
self.last_request_time = 0.0
self.session: aiohttp.ClientSession | None = None
async def __aenter__(self):
timeout = ClientTimeout(total=30, connect=10)
self.session = aiohttp.ClientSession(timeout=timeout)
return self
async def __aexit__(self, *args):
await self.session.close()The semaphore acts as a gate: at most max_concurrent tasks can hold the semaphore at once. The rest queue up automatically.
Adding Polite Rate Limiting
A semaphore alone doesn’t prevent you from hammering the server when all concurrent tasks fire simultaneously. You need a token bucket or at minimum a time-gated delay:
async def _rate_limit(self):
"""Ensure minimum delay between requests."""
now = asyncio.get_event_loop().time()
wait = self.rate_limit - (now - self.last_request_time)
if wait > 0:
await asyncio.sleep(wait)
self.last_request_time = asyncio.get_event_loop().time()
async def fetch(self, url: str) -> str | None:
async with self.semaphore:
await self._rate_limit()
try:
async with self.session.get(url) as response:
if response.status == 429:
retry_after = int(response.headers.get("Retry-After", 5))
print(f"Rate limited on {url}, waiting {retry_after}s")
await asyncio.sleep(retry_after)
return await self.fetch(url) # retry once
response.raise_for_status()
return await response.text()
except (ClientError, asyncio.TimeoutError) as e:
print(f"Failed to fetch {url}: {e}")
return NoneExponential Backoff with Retries
For transient failures, exponential backoff prevents cascading load:
async def fetch_with_retry(self, url, max_retries=3):
for attempt in range(max_retries):
result = await self.fetch(url)
if result is not None:
return result
wait = 2 ** attempt
print(f"Retry {attempt + 1}/{max_retries} for {url} in {wait}s")
await asyncio.sleep(wait)
return NoneManaging the Work Queue
Use an asyncio.Queue to stream URLs and avoid memory blow-up:
async def crawl(self, urls):
queue = asyncio.Queue()
for url in urls:
await queue.put(url)
results = {}
async def worker():
while not queue.empty():
url = await queue.get()
results[url] = await self.fetch_with_retry(url)
queue.task_done()
workers = [asyncio.create_task(worker()) for _ in range(10)]
await queue.join()
for w in workers:
w.cancel()
return resultsBenchmarks from My Project
On a standard VPS with 4 vCPUs:
- Sequential (requests.get): ~500 pages/hour
- Semaphore (10 concurrent): ~4,500 pages/hour
- Semaphore (20 concurrent) + connection pooling: ~8,200 pages/hour
The law of diminishing returns kicks in around 30-50 concurrent connections for most servers. At that point you’re limited by the server’s capacity, not your client.
Full Example: Putting It All Together
async with AsyncCrawler(max_concurrent=20, rate_limit=0.05) as crawler:
urls = [f"https://docs.example.com/page/{i}" for i in range(1000)]
results = await crawler.crawl(urls)
# results is a dict: url -> HTML content or None
for url, html in results.items():
if html:
process_page(url, html)This pattern — semaphore + rate limiting + retry + queue — has been handling 50,000-page crawls reliably for months in production. It won the battle between “fast enough” and “polite enough.”
Originally posted on my blog at AIXHDD. Check out more Python development tools and content there.
