Anatomy of a Block

A script sends GET /products?page=7 and gets back a 403 with a 4 KB page that explains nothing. Somewhere between the socket opening and that response, a system decided the client was not a person in a browser. This chapter follows the request through each point where that decision can be made.

One request, several observers

A scraper rarely talks to the origin server directly. Most commercial sites sit behind an edge layer: a CDN, a web application firewall or a bot-management service. The edge sees every connection first.

The edge collects evidence in the order the protocol reveals it. The source IP arrives with the TCP handshake. The TLS ClientHello comes next, then the HTTP/2 settings, then the request headers. If a page is served, JavaScript can report on the runtime.

Two details matter later. The proxy forwards the TLS handshake unchanged, so the edge still sees the client's own handshake. The edge can also answer on the origin's behalf, which is why a block page often looks nothing like the site.

The detection stack

Detection products differ in detail, but their signals fall into six layers. Lower layers are cheap and run on every request. Higher layers cost more and tend to run when the lower ones leave doubt.

Layers 1 and 2: the address and the tempo

Every IP belongs to an autonomous system, and the ASN says what kind of network that is. A cloud hosting ASN carries very little human browsing, so some sites refuse or challenge it on the first request. Reputation data adds history from earlier abuse.

Rate checks count requests per IP, per subnet and per session. Pattern checks look at shape. Examples are pages 1 to 500 in strict order, identical gaps between requests, or HTML fetched without its assets.

Layers 3 and 4: the software signature

A TLS ClientHello lists cipher suites, extensions and supported groups in an order chosen by the TLS library. Methods such as JA3 and JA4 hash those fields into a short fingerprint. Python's ssl module, Go's standard library, curl and Chrome each produce a different one. HTTP/2 adds more: the SETTINGS values a client sends and the order of its pseudo-headers also vary by implementation.

Headers are checked for agreement. A User-Agent that claims Chrome should arrive with Chrome's headers, in Chrome's order, over a handshake that looks like Chrome. A Chrome string on a Python handshake is a contradiction, and contradictions are cheap to detect.

Layers 5 and 6: the runtime and the session

When a page loads, a script can inspect the environment. It reads canvas and WebGL output, fonts, screen size, time zone and language. It also reads automation flags such as navigator.webdriver. A plain HTTP client never runs that script, and the missing result is a signal too.

Behavior is judged across a session. Real visitors pause, scroll and wander. A session that requests 300 product pages in a minute reads differently.

How a block presents

Blocks do not share one format, even on a single site.

Soft blocks deserve the most attention because nothing fails. Status monitoring stays green while the dataset degrades. Chapters 3 and 6 cover the checks that catch them.

A short probe makes the presentation visible. The script fetches one URL directly and through a proxy, then prints the status, redirect chain, final URL and body size.

import httpx

PROXY = "http://USERNAME:[email protected]:7000"
URL = "https://example.com/"


def probe(label: str, proxy: str | None) -> None:
    with httpx.Client(
        proxy=proxy, timeout=20.0, follow_redirects=True
    ) as client:
        resp = client.get(URL)
    hops = [hop.status_code for hop in resp.history]
    print(label, resp.status_code, hops, resp.url, len(resp.content))
    for name in ("server", "content-type", "retry-after"):
        print("   ", name, "=", resp.headers.get(name))


probe("direct", None)
probe("proxied", PROXY)

Different results on the two routes point at layer 1. The same failure on both points higher up the stack. A body far smaller than the page in a desktop browser suggests a challenge or a soft block.

Reading the symptom

The table maps common symptoms to the most likely layer. It is a starting point, not a verdict: one symptom can have two causes.

For the last row, the ProxyLook proxy checker reports the country and network of an exit IP. The user-agent parser shows what a header string claims to be.

Why a proxy fixes only the first two layers

A proxy changes one thing: the address the target sees. That answers layer 1 directly. A datacenter range with a poor record can be swapped for a residential, ISP or mobile address. The Proxy Playbook covers those types in detail.

A rotating pool also helps with layer 2. It spreads requests over many addresses, so no single IP crosses a per-IP limit. That help is partial. If the site groups requests by fingerprint instead of by IP, a thousand addresses with one identical handshake still look like one client.

Layers 3 to 6 are produced by the client and travel through the proxy tunnel untouched. The TLS handshake belongs to the client library. No JavaScript runs unless the client is a browser. Better IPs for a layer 3 problem cost money and change nothing.

Scope and responsibility

This book is about collecting publicly available data at a rate the target can absorb. It explains how detection works so that a client can behave well and fail gracefully. It does not explain how to break CAPTCHAs or get past logins and paywalls. No technique in it is promised to get through any particular anti-bot product.

A sound default: decide which fields are needed, take only those, respect robots.txt and keep the rate modest. The ProxyLook web scraping page lists providers aimed at this work.