Staying Unblocked

A price scraper runs clean for three weeks. On a Monday every request still returns 200, and every price field in the output is empty. Nothing failed loudly. The site stopped trusting the visitor and began serving a page without the data.

What a website checks

Bot detection is a stack of separate checks, and a request has to pass every one a site runs. Some happen before the server reads a single header. Others build up over days of account activity.

A proxy changes layer 1 and helps with layer 2 by spreading requests across addresses. It does nothing for the other five. That explains a familiar complaint: premium residential IPs that still get blocked on the first request.

A proxy changes the IP address and nothing else about the request.

Layer 3 catches many scripts. Python's requests library opens a TLS connection that looks nothing like Chrome's. If the User-Agent header then claims to be Chrome, the two layers disagree, and that mismatch is easy to flag. Detection vendors hash the handshake into fingerprints with names such as JA3 and JA4. HTTP/2 settings and header order give a second fingerprint of the same kind.

Layers 5 and 6 only apply when the site runs JavaScript checks, which a plain HTTP client never executes. On those sites the missing script result is itself the signal. Layer 7 matters for anything logged in, and it is where the profile hygiene from Chapter 9 pays off.

How a block actually shows up

Blocks rarely announce themselves. A scraper that only counts 403s will miss most of the forms below.

Soft blocks do the most damage because the pipeline keeps running and bad data flows into reports. Chapter 6 made the point for billing: a 200 is not a success. The same check belongs in the code. Every response should be tested for something a good page always contains, such as a product title or a price element.

import requests

PROXY = "http://USERNAME:[email protected]:7000"
MARKERS = ("captcha", "unusual traffic", "access denied")

def classify(r):
    body = r.text.lower()
    if r.status_code in (403, 429):
        return "blocked"
    if r.history and "login" in r.url:
        return "redirected"
    if any(m in body for m in MARKERS):
        return "challenge"
    if 'class="price"' not in body:
        return "soft-block"
    return "ok"

r = requests.get("https://example.com/item/42",
                 proxies={"http": PROXY, "https": PROXY}, timeout=30)
print(r.status_code, len(r.content), classify(r))

The marker strings and the price selector are placeholders. Each target needs its own, taken from a real good page and a real blocked page. Log the label with every request so that success rate means valid data.

Symptom, likely cause, first fix

Most blocks fall into a small set of patterns. Chapter 8 covered errors from the proxy side. This table covers what the target sends back.

When the cause is unclear, separate the IP from the client. Load the target in a normal browser through the same proxy, then compare. The Proxy Checker also shows whether an address is registered as datacenter, residential or mobile.

The escalation ladder

Chapter 5 gave a ladder of proxy types. This one is wider, because the IP is only one of the things that can be changed. Fixes have very different prices. Slowing down is free. Residential traffic runs roughly $1 to $5 per GB at entry level. Datacenter traffic runs roughly $0.10 to $1.40. Live figures are on the pricing page. A headless browser multiplies both bandwidth and compute. Climb one rung at a time, rerun the same URL sample, and stop at the first rung that works.

For rung 2, open-source clients such as curl_cffi copy a real browser's TLS and HTTP/2 handshake. They stay as light as a plain HTTP library. On sites that run no JavaScript checks, that one change can remove the need for rungs 3 and 4.

The order of rungs 3 and 4 can swap. A target that needs JavaScript to show any content needs a browser whatever the IP. For the IP rung, the web scraping page in the directory lists providers for that job.

When a scraper API or unblocker makes sense

A scraper API or unblocker takes a URL and returns the page. The vendor picks the IP, sets the fingerprint, renders JavaScript where needed and retries on failure. These products are priced per successful request or per monthly plan. Where billing is per success, failed attempts are the vendor's cost.

Many teams end up with both: raw proxies for the easy majority of targets and an unblocker for the stubborn few. The proxy API providers page lists vendors that sell this kind of product. ProxyLook's companion book, Scraping Without Blocks, goes deeper on each detection layer.

No setup is permanent

Detection systems change without notice. A configuration that returned clean data last month can start failing this week, and no proxy type, browser or vendor is immune to that. Mobile IPs get challenged. Unblockers return errors on hard targets.

The practical answer is monitoring. Track the share of valid pages per target every day and alert on a drop. Keep the fixed URL sample ready for a quick diagnosis. The next chapter turns those numbers into a cost model.