Running in Production

A crawler runs cleanly on a laptop for an afternoon and gets scheduled nightly. On the ninth night the process is killed halfway, restarts from the top and writes every row a second time. Production scraping is mostly about what happens when work stops, fails and repeats.

Queues and idempotent jobs

A production crawl is a pipeline, not a loop. A scheduler puts jobs on a queue. Workers take jobs, fetch pages and hand results to a validator. Only validated records reach the store. A job carries the URL, the route, an attempt count and the run date.

Queues redeliver. A worker that dies mid-job leaves that job to be handed out again, so every job must be safe to run twice. The usual method is a deterministic key and an upsert. The key below hashes a normalized URL with the run date, so two spellings of one URL collapse into one row.

import hashlib
from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit


def job_key(url: str, run_date: str) -> str:
    parts = urlsplit(url)
    pairs = parse_qsl(parts.query, keep_blank_values=True)
    query = urlencode(sorted(pairs))
    canonical = urlunsplit(
        (parts.scheme, parts.netloc.lower(), parts.path, query, "")
    )
    raw = f"{run_date}|{canonical}".encode()
    return hashlib.sha256(raw).hexdigest()


print(job_key("https://Example.com/p?b=2&a=1", "2026-01-15"))
print(job_key("https://example.com/p?a=1&b=2", "2026-01-15"))

Both lines print the same digest. Two more habits pay off early. Jobs that exhaust their retries go to a dead-letter queue for inspection instead of vanishing. Raw pages are stored compressed beside the parsed records, so a parser fix does not require a second crawl.

Per-domain rate limits

The semaphore in chapter 3 caps total concurrency. It does not stop every worker from landing on one small site at once. Pacing has to be kept per domain, whatever the number of exit IPs. A rotating pool hides the crawler's volume from per-IP counters, but the origin server still carries the full load.

import asyncio
import time
from collections import defaultdict


class DomainLimiter:
    """Space out requests to each domain by a fixed interval."""

    def __init__(self, interval: float = 1.0) -> None:
        self.interval = interval
        self._locks: dict[str, asyncio.Lock] = defaultdict(asyncio.Lock)
        self._next: dict[str, float] = defaultdict(float)

    async def wait(self, domain: str) -> None:
        async with self._locks[domain]:
            now = time.monotonic()
            delay = self._next[domain] - now
            if delay > 0:
                await asyncio.sleep(delay)
            start = max(now, self._next[domain])
            self._next[domain] = start + self.interval

A worker calls await limiter.wait(domain) before each request. A sensible refinement is an adaptive interval. Double it for a domain after a 429 or 503, then shrink it slowly once responses recover. If robots.txt states a crawl delay, that value is the floor.

Metrics, budgets and alerts

Four numbers describe the health of a crawl. Each should be tracked per domain and per route, because an average across domains hides the one that broke.

"Valid" means the record passed validation, not that the status was 200. Counting bare 200s makes a soft block look like a good day.

Budgets need their own alarm. Blocked pages still cost bytes on a per-GB plan, so a retry storm overnight can spend a month of traffic. Count bytes in the client, set a daily budget per domain, and alert at fixed shares of it. At the full budget, pause that domain's queue and leave the decision to a person. The provider's dashboard or usage API gives a second reading to reconcile against. The cost calculator helps set the first budget, and the pricing page has current rates.

Data-quality checks that catch soft blocks

Per-page validation catches a challenge page. It can miss a page that is well formed and wrong. Batch checks compare each run with earlier ones.

• Record count. Compare with the last several runs. A listing that returned 4,000 items and now returns 900 needs a look.
• Null rate per field. A price field that is suddenly empty in a third of rows points to a degraded page.
• Duplicate ratio. Many URLs returning one identical body suggests a cached or decoy response.
• Value sanity. Prices above zero, the expected currency, dates in range.
• Canary URLs. A few pages with stable, known values, checked on every run.

A batch that fails a check goes to quarantine, not to the published dataset. Stale data that is known to be stale beats fresh data that is silently wrong.

Logging enough to debug a block

When success drops, the symptom table in chapter 1 is only useful if the evidence was kept. One structured log line per attempt is enough.

With those lines, yesterday's failures can be grouped by status, route and country in one query. Session IDs belong in logs. Proxy passwords and full proxy URLs do not.

Etiquette, law and launch

A site's robots.txt states what its operator asks of crawlers. Python's standard library can read it, and the fetch below keeps its timeout and TLS checks.

from urllib.robotparser import RobotFileParser

import httpx

AGENT = "example-crawler"


def load_robots(origin: str) -> RobotFileParser:
    parser = RobotFileParser()
    resp = httpx.get(
        f"{origin}/robots.txt", timeout=10.0, follow_redirects=True
    )
    if resp.status_code == 200:
        parser.parse(resp.text.splitlines())
    elif resp.status_code in (401, 403):
        parser.disallow_all = True
    else:
        parser.allow_all = True
    return parser


robots = load_robots("https://example.com")
print(robots.can_fetch(AGENT, "https://example.com/products"))
print(robots.crawl_delay(AGENT))

Good manners go further than the file. Crawl small sites off-peak. Back off at the first 429 or 503. Skip pages that have not changed, using sitemaps or conditional requests where the site supports them. Take only the fields the project needs. Where a declared user agent is practical, include a contact address. An operator can then write to the crawler's owner instead of blocking a whole range.

The launch checklist gathers the chapter into eight lines. For the proxy side of the setup, the ProxyLook web scraping page lists providers aimed at this work.

• Every job has a deterministic key and writes with an upsert.
• Failed jobs land in a dead-letter queue.
• Each domain has its own rate limit and daily traffic budget.
• Success rate, cost per 1,000 records, p95 latency and bytes per record are on a dashboard.
• Batch quality checks run before data is published.
• Each attempt writes one structured log line with no credentials in it.
• robots.txt is read, and its crawl delay is respected.
• Someone has reviewed the data fields for personal data and the target's terms.