Advanced web scraping: rate limits, sessions, retries and rotation without overloading sites

Sites communicate their limits in different ways, and a robust scraper recognises all of them:

• 429 Too Many Requests: you've exceeded the accepted pace. It often comes with a Retry-After header saying how long to wait: a number of seconds or a date. See also 429 error.
• 503 Service Unavailable: the server is overloaded or under maintenance. Retry-After may be present here too.
• 403 Forbidden: access is denied. Repeating the same request usually won't help: check headers, session and pacing.
• Quota headers: some APIs and sites report remaining requests with headers such as X-RateLimit-Remaining. If they're there, use them to slow down before you hit the limit.
• Silent blocks: the response is 200 but contains a captcha, a consent page or an empty list. You can only spot them by checking the content.

The load a site feels depends on how many requests you have open to it at the same time, not on your total over an hour. That's why the first setting to tune is per-domain concurrency: how many parallel requests go to the same site.

• Start low: one or two parallel requests per domain, with varied pauses between them.
• Adapt to response times: if pages start answering more slowly, the server is under load and it's time to slow down.
• Respect robots.txt: access rules are standardised (RFC 9309). The Crawl-delay directive isn't part of the standard and Google ignores it, but when a site sets it, it's a clear hint of the pace it welcomes.
• Spread over time: a slow, steady overnight run beats a burst at the site's peak hour.

In Scrapy you get this behaviour with the AutoThrottle extension (AUTOTHROTTLE_ENABLED), which adapts the delay to the site's latency, together with CONCURRENT_REQUESTS_PER_DOMAIN, DOWNLOAD_DELAY and ROBOTSTXT_OBEY.

To a site, a session is a consistent whole: the same IP, the same cookies, the same user agent and the same language. If one of them changes halfway, the session looks like it has passed to someone else.

• Keep the same IP when the site uses a login, a cart, saved filters or token-based pagination: that's a sticky session.
• Change IP between sessions, not during one: close the session, rotate the IP, start again with fresh cookies.
• One proxy per workflow: if you collect data from several sites in parallel, give each workflow its own proxy, so a rotation doesn't interrupt the others.

The full reasoning on when to rotate and when not to is in IP rotation.

1. Retry only what makes sense: timeouts, network errors, 429, 502, 503 and 504. A 404 or a 403 will stay that way.
2. Respect Retry-After when it's there: it's the site telling you how long to wait.
3. Otherwise use exponential backoff with jitter: growing waits (5, 10, 20, 40 seconds) with a random component, so several processes don't all retry at the same instant.
4. Set a cap: a maximum number of attempts and a maximum wait. Beyond that, save the URL and move on.
5. Pause per domain: if errors on a site go above a threshold (say 10% of the last hundred requests), stop all work on that site for a few minutes instead of insisting.

With ProxyHub you change IP from the rotation link, the API or with automatic rotation. The link and the API reply once the change is complete, with the new IP, and a proxy can change IP at most once every 60 seconds.

• Rotate between rounds, not on every request: each new session starts from a new IP.
• For high volumes, use several proxies in parallel: each has its own dedicated IP, and while one changes the others keep going.
• After a limit, slow down before rotating: if a site limits you, wait as long as it says and resume at a slower pace. Changing IP and carrying on at the same speed leads to the same result.

• Conditional requests: store ETag and Last-Modified and send If-None-Match or If-Modified-Since. If the page hasn't changed the server answers 304, with no body.
• Sitemaps: to discover pages, use the site's sitemap instead of following every link.
• Compression: accept gzip or br in the Accept-Encoding header.
• Only what you need: in automated browsers, block images, fonts and video. The data often comes from a JSON call you can read directly.
• Incremental collection: fetch only new or changed pages, not the whole site on every round.

With Playwright, blocking heavy resources takes one line:

page.route("**/*.{png,jpg,jpeg,gif,webp,woff,woff2,mp4}", lambda route: route.abort())

Proxy setup in Playwright is covered in the Playwright guide.

A skeleton that respects Retry-After, uses backoff with jitter and changes IP only after a repeated limit, allowing for the 60 seconds between changes:

import random, time, requests
from email.utils import parsedate_to_datetime

PROXY = "http://USER:PASSWORD@HOST:PORT"
ROTATE = "ROTATION_LINK"
EXPECTED = 'class="price"'                # element the page must contain
session = requests.Session()
session.proxies = {"http": PROXY, "https": PROXY}
last_rotation = 0.0

def retry_after(r):
    v = r.headers.get("Retry-After", "")
    if v.isdigit():
        return int(v)
    try:
        return max(0, parsedate_to_datetime(v).timestamp() - time.time())
    except (TypeError, ValueError):
        return None

def rotate():
    global last_rotation
    if time.time() - last_rotation >= 60:     # one change a minute per proxy
        requests.get(ROTATE, timeout=30)        # replies with the new IP
        last_rotation = time.time()
        session.cookies.clear()                 # new session, new cookies

def fetch(url, tries=5):
    for attempt in range(tries):
        try:
            r = session.get(url, timeout=30)
        except (requests.ConnectionError, requests.Timeout):
            time.sleep(min(60, 2 ** attempt) + random.random())
            continue
        if r.status_code == 200 and EXPECTED in r.text:
            return r.text
        if r.status_code == 200:                  # silent block (captcha, empty page)
            wait = min(120, 10 * 2 ** attempt)
        elif r.status_code in (429, 502, 503, 504):
            wait = retry_after(r) or min(120, 5 * 2 ** attempt)
        else:
            r.raise_for_status()                  # 403, 404…: no point retrying
            return r.text
        if attempt >= 1:
            rotate()                              # repeated limit: new session
        time.sleep(wait + random.uniform(0, 3))   # jitter
    raise RuntimeError(f"Too many attempts: {url}")

The same pattern works in Node.js and in no-code tools: the library and syntax change, the rules don't. If your work runs automatically, you'll find more tips on the Proxies for automation and bots page.