Headless Browsers Through Proxies

A product page shows a price in the browser. The same URL fetched with httpx returns an empty <div id="root"> and 300 KB of JavaScript. That is the honest case for a headless browser, and it comes up less often than it first appears.

When a browser is needed, and when it is waste

A browser is the most expensive way to fetch a page, so it should be the last tool tried. Three checks come first, in this order.

1. Read the raw HTML. Fetch the page with an HTTP client and search for a known value, such as a price. Server-rendered pages already contain it.
2. Look for embedded JSON. Many JavaScript sites ship their data inside a script tag, as JSON-LD or as framework state. A parser can read it without running anything.
3. Watch the network tab. If the page fills itself in from a public JSON endpoint, the HTTP client from chapter 3 can often call it directly.

The third check has a boundary. An endpoint that wants a login token or signed parameters is not public. It falls outside the scope of this book.

The last three rows are tier 4 targets from chapter 2. For those, a real browser sends real signals at layers 3 to 5: a browser handshake, browser headers and a JavaScript runtime.

What a browser costs

An HTTP client downloads one document. A browser downloads that document and then everything it references: scripts, stylesheets, fonts, images, video and third-party trackers. On a per-GB plan, every one of those bytes is billed, whether or not the scraper reads it.

The real ratio depends on the site, so measure it on the target before budgeting. Multiply bytes per page by pages per month, then put the total in GB into the cost calculator. Traffic is not the only cost. A machine that runs hundreds of concurrent HTTP requests may hold only a handful of browser pages in memory. Each page also waits on rendering.

A proxy per browser context

Playwright can launch one browser and open many contexts inside it. A context is an isolated profile with its own cookies, storage and cache. It can also have its own proxy. That makes a context the browser version of the identity from chapter 3: one session ID, one exit IP, one cookie jar.

import asyncio

from playwright.async_api import Browser, BrowserContext, async_playwright

GATEWAY = "http://gate.example-proxy.com:7000"
GEO = {
    "us": ("en-US", "America/New_York"),
    "de": ("de-DE", "Europe/Berlin"),
}


async def new_identity(
    browser: Browser, country: str, session: str
) -> BrowserContext:
    locale, zone = GEO[country]
    context = await browser.new_context(
        proxy={
            "server": GATEWAY,
            "username": f"USERNAME-country-{country}-session-{session}",
            "password": "PASSWORD",
        },
        locale=locale,
        timezone_id=zone,
    )
    context.set_default_timeout(30_000)
    return context

The username syntax is illustrative, and the provider's docs define the real parameters. Playwright takes the credentials as separate fields, not inside the server URL. The default timeout is set in milliseconds and covers navigation and other calls that wait. Older Playwright releases needed a placeholder proxy at launch before Chromium accepted a proxy per context. The documentation for the installed version says whether that still applies.

A proxy can also be set once at launch with the same dictionary. That fits a crawl where every page shares one exit. Per-context proxies fit rotation: ten contexts in one browser process give ten exits without starting ten browsers. Each open page still holds memory, so the semaphore pattern from chapter 3 applies here as well.

Keeping locale and time zone consistent

A scraper on a cloud server usually runs in UTC with an English locale. Put a German residential exit in front of it and the page sees a visitor in Germany whose clock and language say otherwise. That is a layer 5 contradiction, and a script can read it in two lines.

The locale option sets navigator.language and the Accept-Language header together. The timezone_id option sets the zone that JavaScript dates report. Both should follow the proxy's country, which is why the example keeps them in one table beside the country code.

No setting removes every difference between automation and a person at a keyboard. Projects such as Camoufox, an open-source Firefox fork built for Playwright automation, narrow some of them. None can promise to pass a given detection product, and a crawler should be built to notice failure instead of assuming success.

Blocking heavy resources

Route interception lets the scraper cancel requests before they leave the machine. Images, fonts and media are the usual candidates. They make up much of a page's weight and rarely contain the data.

import random
import secrets

from playwright.async_api import Route

HEAVY = {"image", "media", "font"}


async def skip_heavy(route: Route) -> None:
    if route.request.resource_type in HEAVY:
        await route.abort()
    else:
        await route.continue_()

Three cautions apply. Blocking scripts or stylesheets breaks many pages, so those types stay out of the set. Playwright's documentation notes that enabling routing disables the HTTP cache, so assets that are not blocked download again on every navigation. A browser that never loads an image is also an unusual visitor. Blocking trades bytes for a less ordinary profile, and the right balance depends on the target.

Reusing a context for sticky work

A paginated listing should run inside one context from first page to last. Cookies set on page 1 are still there on page 2, and the session ID keeps the exit IP steady. The block below attaches the route handler and walks the pages with a pause. It stops at the first response that is not a 200.

async def crawl_list(browser: Browser, base: str, pages: int) -> list[str]:
    context = await new_identity(browser, "de", secrets.token_hex(4))
    await context.route("**/*", skip_heavy)
    page = await context.new_page()
    bodies = []
    try:
        for number in range(1, pages + 1):
            url = f"{base}?page={number}"
            resp = await page.goto(url, wait_until="domcontentloaded")
            if resp is None or resp.status != 200:
                break
            bodies.append(await page.content())
            await page.wait_for_timeout(random.uniform(1500, 4000))
    finally:
        await context.close()
    return bodies


async def main() -> None:
    async with async_playwright() as pw:
        browser = await pw.chromium.launch(headless=True)
        html = await crawl_list(browser, "https://example.com/list", 3)
        print([len(body) for body in html])
        await browser.close()


asyncio.run(main())

The three blocks form one file. Closing the context in finally matters, because a leaked context keeps its memory and its proxy connections. A navigation timeout or a proxy failure raises Playwright's Error. The caller should catch it and requeue the sequence. The next sequence gets a new context and a new session ID, so no state crosses between visitors. The bodies still need the validation from chapter 3, since a browser can receive a soft block as easily as an HTTP client.