Building a Resilient Client

A first scraper is often a loop around get() with a bare except. At the first 429 it retries at once from the same IP and turns a rate limit into a ban. This chapter replaces that loop with a client that knows why a request failed.

The base client

The six code blocks in this chapter form one file. Pasted in order, they make a small async crawler. The first block defines the headers and a factory that builds a client for one proxy session.

import httpx

GATEWAY = "gate.example-proxy.com:7000"
HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:140.0) "
    "Gecko/20100101 Firefox/140.0",
    "Accept": "text/html,application/xhtml+xml,"
    "application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.5",
    "Upgrade-Insecure-Requests": "1",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
}


def make_client(session_id: str) -> httpx.AsyncClient:
    # Illustrative username syntax. The provider's docs define it.
    user = f"USERNAME-country-us-session-{session_id}"
    return httpx.AsyncClient(
        proxy=f"http://{user}:PASSWORD@{GATEWAY}",
        headers=HEADERS,
        http2=True,
        timeout=httpx.Timeout(20.0, connect=10.0),
        follow_redirects=True,
    )

Four settings carry most of the weight.

• Timeouts. The httpx default is 5 seconds per phase, which is tight for a residential exit. Each timeout covers one phase, such as connecting or waiting for the next chunk. It is not a deadline for the whole request. That deadline is added in the retry loop.
• HTTP/2. It needs the extra package, installed with pip install "httpx[http2]". Browsers use HTTP/2 or HTTP/3 on most large sites, so an HTTP/1.1 client stands out. The http_version attribute on a response shows what was negotiated.
• Redirects. httpx does not follow them by default. The factory turns that on, and the validator later checks where each response ended up.
• TLS verification. It is on by default and nothing here turns it off. A password with special characters must be percent-encoded in the proxy URL.

Headers that agree with each other

The default User-Agent is python-httpx plus a version. It is honest, and many sites refuse it on sight. There are two defensible replacements. One is a declared bot string with a contact URL, which suits tier 1 targets and sites that welcome crawlers. The other is a complete header set copied from one real browser release.

The example follows Firefox because Firefox sends a short set. Chrome adds a family of sec-ch-ua headers that would have to match as well. The Sec-Fetch values describe a page opened directly. A browser changes them, and adds a Referer, when a link is followed. The version in the example will age, so copy the set from a current release and refresh it on a schedule. The user-agent parser shows what a string claims. Accept-Encoding is left to httpx, which advertises only the encodings it can decode.

One identity per session ID

In httpx the proxy belongs to the client, not to the request. A new session ID therefore means a new client. That is useful, because a new client also starts with an empty cookie jar and no open connections. Nothing from a flagged session follows the scraper to its next exit IP.

import secrets


class Identity:
    """One session ID: one exit IP, one cookie jar, one pool."""

    def __init__(self) -> None:
        self.client = make_client(secrets.token_hex(4))

    async def rotate(self) -> None:
        # A new client drops old cookies and connections too.
        await self.client.aclose()
        self.client = make_client(secrets.token_hex(4))

    async def close(self) -> None:
        await self.client.aclose()

How long the exit IP holds for one session ID depends on the provider's sticky session rules. A call to an IP echo endpoint through each client confirms that two session IDs produce two different exits, and the proxy checker shows the country and network of each one.

Different failures, different responses

A status code says which layer objected, so each one deserves its own reaction. Retrying a 403 from the same identity repeats the request that was just refused. Rotating on a 429 dodges a limit the site set on purpose.

The next block turns that table into code. The wait uses exponential backoff with full jitter: a random delay below a ceiling that doubles on every attempt. Jitter keeps a pool of workers from retrying in step.

import random
from enum import Enum


class Verdict(Enum):
    OK = "ok"
    BACKOFF = "backoff"   # same identity, wait longer
    ROTATE = "rotate"     # new session ID, then wait
    FATAL = "fatal"       # retrying cannot help


def classify(status: int) -> Verdict:
    if 200 <= status < 300:
        return Verdict.OK
    if status == 403:
        return Verdict.ROTATE
    if status == 429 or status >= 500:
        return Verdict.BACKOFF
    return Verdict.FATAL


def wait_for(attempt: int, resp: httpx.Response | None) -> float:
    # Full jitter: a random wait below an exponential ceiling.
    wait = random.uniform(0, min(60.0, 2.0 ** attempt))
    hint = resp.headers.get("retry-after", "") if resp is not None else ""
    return max(wait, float(hint)) if hint.isdigit() else wait

Retry-After can also carry an HTTP date, which this function ignores. Production code should cap the hint and requeue the job instead of sleeping for an hour. A 502 or 503 may also come from the proxy gateway instead of the origin. A common cause is that no exit is free in the requested country. The provider's docs list those gateway errors.

The retry loop

The loop wraps each attempt in asyncio.timeout, so a response that drips in slowly cannot hold a worker forever. Network errors and missed deadlines count as a dead exit and trigger a rotation.

import asyncio

MAX_ATTEMPTS = 4
DEADLINE = 45  # seconds for one whole attempt


async def fetch(ident: Identity, url: str) -> httpx.Response | None:
    for attempt in range(MAX_ATTEMPTS):
        resp = None
        try:
            async with asyncio.timeout(DEADLINE):
                resp = await ident.client.get(url)
            verdict = classify(resp.status_code)
        except (httpx.RequestError, TimeoutError):
            verdict = Verdict.ROTATE
        if verdict is Verdict.OK and not looks_valid(resp):
            verdict = Verdict.ROTATE
        if verdict is Verdict.OK:
            return resp
        if verdict is Verdict.FATAL:
            return None
        if verdict is Verdict.ROTATE:
            await ident.rotate()
        await asyncio.sleep(wait_for(attempt, resp))
    return None

The attempt cap matters more than any other line. Four refusals from four exit IPs are not an IP problem. Looping harder only burns traffic, so the job should stop and move up the routing ladder from chapter 2. A rejected proxy password surfaces as httpx.ProxyError on HTTPS targets and fails all four attempts quickly. Logging the exception text makes that obvious.

Catching soft blocks in the body

A 200 proves that something was returned, not that it was the page. The validator checks three things. The final URL must not be an interstitial. The size must be plausible. A marker found on every good page must be present.

REQUIRED = 'id="product-title"'    # found on every good page
MIN_BYTES = 5_000                  # well below any real page
CHALLENGE_HINTS = ("captcha", "challenge", "blocked")


def looks_valid(resp: httpx.Response) -> bool:
    path = resp.url.path.lower()
    if any(hint in path for hint in CHALLENGE_HINTS):
        return False   # redirected to an interstitial
    if "text/html" not in resp.headers.get("content-type", ""):
        return False
    if len(resp.content) < MIN_BYTES:
        return False
    return REQUIRED in resp.text

All three constants are placeholders to set per target. The positive check does most of the work. Searching the body for a word like "captcha" misfires on healthy pages that embed a CAPTCHA widget in a form. For a JSON endpoint, the same idea applies to keys: parse the body and require the expected fields.

A bounded worker pool

A semaphore caps the number of requests in flight. Each task owns its identity, so a rotation in one task never closes a client that another task is using.

Result = tuple[str, str | None]


async def scrape(url: str, gate: asyncio.Semaphore) -> Result:
    async with gate:
        ident = Identity()
        try:
            resp = await fetch(ident, url)
        finally:
            await ident.close()
    return url, resp.text if resp is not None else None


async def crawl(urls: list[str], limit: int = 8) -> list[Result]:
    gate = asyncio.Semaphore(limit)
    return await asyncio.gather(*(scrape(u, gate) for u in urls))


if __name__ == "__main__":
    for url, body in asyncio.run(crawl(["https://example.com/"])):
        print(url, "ok" if body else "failed")

Run against example.com, the script prints "failed", because the placeholder marker is absent. That is the validator doing its job. One identity per URL matches per-request rotation. For a paginated sequence, pass the whole list to one task and reuse its identity.

Two limits remain. gather creates every task up front, which is fine for thousands of URLs and wasteful for millions. A concurrency cap is also not a rate limit: eight workers can still hit one small site eight at a time. Chapter 6 adds a queue and per-domain pacing.