Scaling and Cost Control

A scraper that retries instantly, downloads unchanged pages again and loads every product photo can burn a month of residential traffic in a week. None of those three habits improves the data. The biggest savings usually come from engineering, not from a cheaper plan.

Concurrency versus politeness

Concurrency is the number of requests in flight at once. Politeness is the load placed on one website. The two get confused because a rotating pool hides the problem. Take 500 parallel requests through 500 different IPs. Each address looks gentle, yet the target's servers still answer all 500.

Set limits per domain, not per job. A crawl across 2,000 small shops can run hundreds of connections in total while holding each shop to two or three. A crawl of one large marketplace can go higher, but it should rise in steps.

1. Start low. Begin with two to five concurrent requests per domain and a short random delay between them.
2. Raise in steps. Double the limit, run for several minutes, and compare success rate and latency with the previous step.
3. Stop at the first warning. Rising response times, 429 responses or new CAPTCHA pages mean the last step was too far. Drop back one level.
4. Write the limit down. Store the working number per domain in configuration so the next run starts from it.

Scrapy has this built in: CONCURRENT_REQUESTS_PER_DOMAIN, DOWNLOAD_DELAY and the AutoThrottle extension, which adjusts delay from observed latency. Provider plans add their own ceiling. Some bill per port or thread, so the concurrency limit is part of the price.

Retries with backoff and jitter

A failed request is worth retrying, but not immediately and not forever. An instant retry hits the same rate limit again and pays for the bandwidth twice. Exponential backoff doubles the wait after each failure. Jitter makes each wait random, so a hundred workers that failed together do not all return in the same second.

import random
import time
import requests

PROXY = "http://USERNAME:[email protected]:7000"
PROXIES = {"http": PROXY, "https": PROXY}
RETRY_ON = {408, 429, 500, 502, 503, 504}

def fetch(url, attempts=5, base=1.0, cap=30.0):
    for attempt in range(attempts):
        if attempt:
            ceiling = min(cap, base * 2 ** attempt)
            time.sleep(random.uniform(0, ceiling))
        try:
            r = requests.get(url, proxies=PROXIES, timeout=30)
        except requests.RequestException:
            continue
        if r.status_code not in RETRY_ON:
            return r
    raise RuntimeError(f"gave up on {url}")

print(fetch("https://example.com").status_code)

The function waits up to 2 seconds before the second attempt, then up to 4, 8 and 16, and gives up after five tries. Three rules keep retries cheap:

• Retry only what can recover. Timeouts, 429 and 5xx responses qualify. A 404 will still be a 404, and a 407 means the proxy rejected the credentials.
• Honor Retry-After. When a 429 or 503 carries that header, wait at least that long. The example leaves this out to stay short.
• Cap the total. Track retries as a share of all requests. A rising share usually means a block in progress, and backoff alone will not fix it.

A 403 is a special case. Repeating it from the same IP rarely helps. With a rotating gateway each retry leaves from a new address, so one or two more attempts are reasonable before escalating.

Not paying for the same bytes twice

The cheapest request is the one never sent. Chapter 6 showed how page weight drives a per-GB bill and named the habits that cut it. This section covers the mechanics.

Deduplicate and cache

Normalize URLs before queueing them, so that ?ref=home and ?ref=search versions of one product count once. Store every fetched page on disk with a timestamp, and give each page type a lifetime. A price may need a daily refresh. A product description can wait a month. During development, work from cached pages instead of the live site. Scrapy's HTTPCACHE_ENABLED setting does this in one line.

Ask whether the page changed

Many servers return an ETag or Last-Modified header. Sending it back as If-None-Match or If-Modified-Since lets the server answer 304 with no body when nothing changed. Sitemaps with modification dates serve the same purpose for whole sections of a site.

Block heavy resources in headless browsers

A rendered page pulls in images, fonts and video that the scraper never reads. In Playwright, route interception aborts those requests before they use any proxy bandwidth.

from playwright.sync_api import sync_playwright

BLOCKED = {"image", "media", "font"}

def handle(route):
    if route.request.resource_type in BLOCKED:
        route.abort()
    else:
        route.continue_()

with sync_playwright() as p:
    browser = p.chromium.launch(proxy={
        "server": "http://gate.example-proxy.com:7000",
        "username": "USERNAME",
        "password": "PASSWORD",
    })
    page = browser.new_page()
    page.route("**/*", handle)
    page.goto("https://example.com", timeout=30000)
    print(page.title())
    browser.close()

Check the output after adding the filter, because some pages break when resources are missing. A browser that fetches no images can also look unusual to a strict target. Leave scripts and XHR calls alone, since they usually carry the data.

Tiered routing: cheap first, expensive on failure

Datacenter traffic lists at roughly $0.10 to $1.40 per GB and rotating residential at roughly $1 to $5 per GB at entry level. Live figures are on the pricing page. Sending every request through the expensive pool pays the premium even on pages that a datacenter IP would have fetched without trouble.

Chapter 5 applied "start cheap, escalate on evidence" to a whole job. Tiered routing applies it to each request. The router tries the cheap pool first and moves up only when a request fails for a block-related reason.

Four rules make the router work in practice:

• Escalate on block signals only. A 403, a 429 after backoff, a challenge page or a failed content check moves the request up. A 404 does not.
• Remember what worked. Keep a per-domain record of the lowest tier that succeeds, and start there next time.
• Probe downward. Send a small share of traffic to the cheaper tier now and then. Targets relax as well as tighten.
• Limit the climb. A block moves a request up one tier, once. A URL that fails on every tier goes to a review queue.

The directory pages for datacenter proxies and residential proxies list networks for the first two tiers. Buying both from one provider is convenient. Buying from two gives a fallback when one has an outage.

The four metrics worth tracking

Dashboards fill up fast. Four numbers, recorded per target domain and per proxy tier, answer most cost questions.

Chapter 6 introduced cost per successful request as the way to compare plans. In production it becomes a daily reading, because it folds price, page weight and failure rate into one figure. When it rises, one of the other three metrics will show why.

All four come from one log line per request. It needs the timestamp, target domain, proxy tier, status code, bytes received, elapsed time and content-check result. Scripts that log only failures cannot calculate any of them.

The proxy cost calculator estimates monthly spend from bandwidth and price per GB before a job exists to measure.

Budgets and alerts

Usage-based billing has no natural ceiling. A loop that requests the same URL all night is billed like real work. A few guardrails turn a surprise invoice into a short alert.

• Give every project its own sub-user or API key so traffic can be traced to its source.
• Set a daily GB cap per job, and stop the job when it is reached.
• Alert when GB per day climbs well above the trailing weekly average.
• Alert when success rate falls below the normal range for a target.
• Pause a job automatically when success rate stays low for several minutes.
• Review cost per 1,000 successful requests each month, per target.

The automatic pause matters most. Failed requests still use bandwidth, so a blocked scraper left running pays full price for nothing. Many provider dashboards offer per-user traffic limits. Check for them during the trial, along with what happens when a prepaid balance runs out mid-job.