Picking Proxies for the Target

A parts catalog on a small shop's own server answers plain datacenter requests all day. A search page on a large marketplace may refuse the same request before any HTML is built. Proxy choice starts with knowing which kind of target is on the list.

Sorting targets by what can be observed

Target difficulty does not need guessing. A short probe at a polite rate shows it. Fetch one page with a plain HTTP client from a datacenter IP, then from a residential IP, then in a real browser. Note where the full page first appears.

Tiers belong to endpoints, not to whole sites. Product pages may sit in tier 2 while search on the same domain sits in tier 4. Tiers also drift, so the probe is worth repeating whenever success rates move.

Which proxy type fits which tier

Each tier fails at a different detection layer, so each needs a different fix. The rule is to buy the cheapest type that clears the layer in question.

Datacenter proxies are fast and cheap, and their ASNs are easy to recognize. ISP proxies are hosted in datacenters but registered to consumer providers. They stay fixed, which suits sticky work. Residential proxies exit through home connections, so they are slower and less stable but hard to tell from visitors.

Mobile proxies exit through carrier networks, where many subscribers share one address. Sites tend to be cautious about refusing those addresses. Mobile is the most expensive type and fits mobile-only content, or tier 3 targets where residential still fails.

These ranges are rough guidance only. Live figures are on the ProxyLook pricing page, and the cost calculator turns monthly traffic in GB into an estimated spend. Pool sizes on provider pages are vendor-stated and say little about success on one specific target.

Rotation per request or sticky sessions

With per-request rotation, every request leaves from a new exit IP. That fits independent URLs, such as 50,000 product pages where each response stands alone. No single address builds up a history.

A sticky session holds one exit IP for a window of time. Providers usually expose it through a session ID in the proxy username, in a form like USERNAME-country-us-session-a1b2c3. That syntax is illustrative. The exact parameters come from the provider's docs.

Sticky sessions fit paginated and stateful crawls. Some searches hand out a cursor token. Some listings keep their filters in a cookie. Both expect page 3 to come from the visitor who loaded page 2. A new IP mid-sequence can reset the state or draw a challenge.

Residential sticky sessions end when the provider's window closes or the home device goes offline. A paginated crawl should checkpoint its position and restart the sequence under a new session. Static ISP addresses hold for as long as they are rented.

Geo-targeting for localized content

Prices, currency, stock and search rankings change with the visitor's country, and sometimes with the city. A crawl for German prices needs German exits. Country targeting is set at the gateway, usually by another username parameter.

Three checks keep localized data honest. Verify the exit country with the proxy checker, because geolocation databases disagree. Send an Accept-Language header that matches the country. Confirm the currency or region marker in the response body. City-level targeting shrinks the usable pool, so it is worth using only when the data varies by city.

A routing ladder that starts cheap

A crawler with one proxy type overpays on easy targets or fails on hard ones. A ladder sends each domain through the cheapest route that works and moves up only on evidence.

1. Start on datacenter. Every new domain begins on the cheapest rung with a plain HTTP client.
2. Widen the pool. If 429s dominate, slow down first, then add addresses or move to ISP.
3. Switch to residential. If 403s arrive on the first request, datacenter ranges are being refused.
4. Add a browser. If residential IPs still fail with an HTTP client, the client is the problem.
5. Hand off the remainder. Send the hardest endpoints to a scraper API, covered in chapter 5.

The class below keeps a rung per domain. It moves a domain up one rung when the success share over a recent window drops under a floor. The floor of 0.8 is an example value to tune, not a recommendation.

from collections import defaultdict, deque

LADDER = ("datacenter", "isp", "residential", "browser", "api")
WINDOW = 200   # recent results remembered per domain
FLOOR = 0.8    # example value: escalate below this success share


class Router:
    def __init__(self) -> None:
        self.rung: dict[str, int] = defaultdict(int)
        self.recent: dict[str, deque[bool]] = defaultdict(
            lambda: deque(maxlen=WINDOW)
        )

    def route(self, domain: str) -> str:
        return LADDER[self.rung[domain]]

    def record(self, domain: str, ok: bool) -> None:
        window = self.recent[domain]
        window.append(ok)
        if len(window) < WINDOW:
            return
        top = len(LADDER) - 1
        if sum(window) / WINDOW < FLOOR and self.rung[domain] < top:
            self.rung[domain] += 1
            window.clear()

Two habits make the ladder pay off. Store the rung with every result, so cost can be reported per route. Send a small sample back down a rung now and then, because defenses change in both directions.

Quick decisions

The table compresses the chapter into first moves. Each row assumes the probe from the first section has already been run.