Scraper APIs and Unblockers

A team spends three weeks tuning a headless crawler for one marketplace, and the success rate still swings from day to day. A scraper API offers to take the URL and return the HTML for a fee per successful request. Whether that is a bargain is a matter of arithmetic.

What these products do

A managed scraping product bundles the work of chapters 2 to 4 behind one endpoint. The vendor picks the proxy, builds the headers, runs a browser when needed, retries failures and returns the result. The products come in three shapes.

Proxy-style unblockers usually decrypt and re-encrypt traffic so they can rewrite requests. The client then has to trust the vendor's CA certificate. Install that certificate as the vendor documents. Switching verification off hides real TLS errors along with the expected one.

Two limits apply to every product in this category. None can promise success on every target, so a trial on the real URL list is the only useful test. A vendor also does not change the buyer's legal position. Responsibility for what is collected stays with the team that asked for it.

Build or buy

The choice turns on four things: target tier, volume, engineering time and the need for control. Tier 1 and tier 2 targets rarely justify a managed service, because a plain client already succeeds. Tier 4 targets are where the vendor's pooled effort is hard to match. A team of two may not have a week each month to spend on fingerprint upkeep.

Control cuts the other way. A self-built client decides its own headers, cookies, session length and pacing. An API exposes only the options its vendor chose to offer. Crawls that depend on a long sticky sequence need that support confirmed before any contract is signed.

How per-request pricing changes the math

Per-GB proxy billing charges for every byte, including failed attempts and blocked pages. Per-request billing usually charges only for successes, so failures and page weight become the vendor's problem. Rendering often carries a higher price or a credit multiplier, so the vendor's own table needs a careful read.

The fair comparison is cost per 1,000 valid pages, with engineering time included. For a self-built route, the function below does the sum.

def cost_per_1000(
    mb_per_attempt: float,
    success_share: float,
    usd_per_gb: float,
    upkeep_usd: float,
    pages: int,
) -> float:
    gb = pages * mb_per_attempt / success_share / 1000
    total = gb * usd_per_gb + upkeep_usd
    return round(total / pages * 1000, 2)


# Invented round numbers, matching the table below.
print(cost_per_1000(2.0, 0.5, 3.0, 500.0, 100_000))   # 17.0
print(cost_per_1000(0.2, 1.0, 0.5, 50.0, 100_000))    # 0.6

The table runs that sum for 100,000 valid pages on a hard target and on an easy one. The hard target needs a browser at 2 MB per attempt, and half the attempts fail. The easy target needs 0.2 MB per attempt and almost never fails. An hour of engineering is priced at $50.

Rerun the sum with real inputs before deciding. Measured bytes per attempt and success share come from the crawler's own logs. Live prices are on the proxy API providers page and the pricing page. The cost calculator handles the per-GB side.

Where an API fits in the ladder

The comparison points to a mixed setup. Cheap routes take everything they can handle, and the API takes only what they cannot. In the routing ladder from chapter 2, that makes the API the top rung.

A thin interface keeps the rungs interchangeable. Each rung implements one method and returns the same record, whether it wraps httpx, Playwright or a vendor's API.

from dataclasses import dataclass
from typing import Protocol


@dataclass
class Page:
    url: str
    body: str
    route: str   # the rung that produced it, for cost reports


class Fetcher(Protocol):
    name: str

    async def fetch(self, url: str) -> Page | None: ...


async def fetch_with_ladder(url: str, rungs: list[Fetcher]) -> Page | None:
    for rung in rungs:
        page = await rung.fetch(url)
        if page is not None:
            return page
    return None

In practice the router from chapter 2 remembers which rung works for each domain, so most URLs skip straight to it. The fall-through loop handles the exceptions. For a small team with a handful of hard targets, a ladder with one rung, the API, is a reasonable place to start.

Lock-in and observability

Every vendor has its own parameters, response envelope, error codes and credit rules. Code written against them directly is expensive to move. The interface above limits that cost to one adapter class per vendor. Storing raw HTML and keeping parsers in-house helps as well, since a vendor's parsed JSON has a schema no other vendor shares.

Observability is the larger trade. A self-built client can log every header, status and exit IP, so a block can be traced to a layer. An API reports that a request failed and little else. When its success rate drifts, the buyer can only open a ticket.

• Log the vendor's status, the final URL, the response time and whether the request was billed.
• Find out how the vendor defines success, since a soft block with status 200 may still be charged.
• Keep the body validation from chapter 3 in front of every stored page.
• Check the plan's concurrency cap against the crawl's peak rate.
• Keep a second vendor tested behind the same interface.

Decision table

The table condenses the chapter into first moves. Once the route is clear, the ProxyLook compare page sets providers side by side.