Setup in Code

A proxy setup works when a request to https://api.ipify.org returns an address that is not the machine's own. Every example in this chapter ends with that check. Each one reads its credentials from the environment, sets a timeout and leaves TLS verification on.

Credentials and a first request

A proxy URL carries a username and a password. Pasted into a script, it ends up in version control, logs and screenshots. Environment variables keep the secret in one place, outside the code.

# proxy.env: add this file name to .gitignore
export PROXY_HOST="gate.example-proxy.com:7000"
export PROXY_USER="USERNAME"
export PROXY_PASS="PASSWORD"
export PROXY_URL="http://$PROXY_USER:$PROXY_PASS@$PROXY_HOST"

# Load the variables into the current shell:
#   source proxy.env

HTTP clients take the full URL. Browsers want the host and the credentials as separate fields, so the file defines both forms. A password containing @, : or / must be percent-encoded before it goes into a URL.

Many providers read targeting options from the username, in a form such as USERNAME-country-us-session-a1b2c3. The exact syntax differs by vendor and comes from the provider's documentation.

Chapter 3 introduced curl as the quickest test of a proxy URL. The same commands read from the environment here.

# HTTP proxy. Give up after 30 seconds in total.
curl -x "$PROXY_URL" --max-time 30 https://api.ipify.org

# SOCKS5, with DNS resolved by the proxy instead of locally.
curl --socks5-hostname "$PROXY_HOST" \
     --proxy-user "$PROXY_USER:$PROXY_PASS" \
     --max-time 30 https://api.ipify.org

# Print the proxy handshake when something fails.
curl -v -x "$PROXY_URL" --max-time 30 https://api.ipify.org

The SOCKS5 port is often different from the HTTP port, so check the dashboard first. The -v output includes the encoded credentials. Trim it before pasting it into a support ticket.

Python: requests and httpx

A production requests script differs from a demo in three ways: a reused Session, a timeout on every call, and retries with backoff.

import os

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

proxy_url = os.environ["PROXY_URL"]
proxies = {"http": proxy_url, "https": proxy_url}

retry = Retry(
    total=3,
    backoff_factor=1,
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET", "HEAD"],
)
session = requests.Session()
session.mount("http://", HTTPAdapter(max_retries=retry))
session.mount("https://", HTTPAdapter(max_retries=retry))

resp = session.get(
    "https://api.ipify.org", proxies=proxies, timeout=(10, 30)
)
resp.raise_for_status()
print(resp.text)

The timeout tuple allows 10 seconds to connect and 30 seconds to read. The retry policy waits longer after each failure and honors a Retry-After header from the target. Chapter 11 adds jitter for larger jobs. Passing proxies on the call is deliberate. Proxies stored on the Session object can be overridden by an HTTPS_PROXY variable in the environment.

For concurrent work, httpx offers an async client. Current versions take a single proxy argument, and the older proxies argument was removed in 0.28. A semaphore caps how many requests run at once.

import asyncio
import os

import httpx

PROXY_URL = os.environ["PROXY_URL"]
TIMEOUT = httpx.Timeout(30.0, connect=10.0)


async def exit_ip(client, limit):
    async with limit:
        resp = await client.get("https://api.ipify.org")
        resp.raise_for_status()
        return resp.text


async def main():
    limit = asyncio.Semaphore(3)
    async with httpx.AsyncClient(proxy=PROXY_URL, timeout=TIMEOUT) as c:
        ips = await asyncio.gather(*(exit_ip(c, limit) for _ in range(6)))
    print(ips)


asyncio.run(main())

The six results may show fewer than six addresses. As Chapter 4 explained, a rotating gateway assigns the exit IP when a connection opens, and the client reuses open connections. Here three connections serve six requests.

Node.js with undici

Node's built-in fetch does not take a proxy URL directly. The undici package supplies a ProxyAgent that is passed as the dispatcher. Importing fetch from the same package keeps the two versions matched.

// npm install undici, then save as check.mjs
import { fetch, ProxyAgent } from "undici";

const dispatcher = new ProxyAgent(process.env.PROXY_URL);

try {
  const res = await fetch("https://api.ipify.org", {
    dispatcher,
    signal: AbortSignal.timeout(30_000),
  });
  if (!res.ok) throw new Error(`Target answered ${res.status}`);
  console.log(await res.text());
} finally {
  await dispatcher.close();
}

The .mjs extension matters because the file uses top-level await. AbortSignal.timeout cancels the request after 30 seconds. Closing the dispatcher lets the process exit cleanly.

Browsers: Playwright and Puppeteer

Automated browsers split the proxy into a server address and a pair of credentials. Playwright accepts all three at launch.

import os

from playwright.sync_api import sync_playwright

proxy = {
    "server": "http://" + os.environ["PROXY_HOST"],
    "username": os.environ["PROXY_USER"],
    "password": os.environ["PROXY_PASS"],
}

with sync_playwright() as p:
    browser = p.chromium.launch(proxy=proxy)
    page = browser.new_page()
    resp = page.goto("https://api.ipify.org", timeout=30_000)
    print(resp.text())
    browser.close()

Playwright takes the same proxy object on browser.new_context(), which gives each context its own exit. Puppeteer works differently. The server goes in a Chromium launch flag, and the credentials are supplied per page.

// npm install puppeteer, then save as check.mjs
import puppeteer from "puppeteer";

const browser = await puppeteer.launch({
  args: [`--proxy-server=${process.env.PROXY_HOST}`],
});
try {
  const page = await browser.newPage();
  await page.authenticate({
    username: process.env.PROXY_USER,
    password: process.env.PROXY_PASS,
  });
  const res = await page.goto("https://api.ipify.org", {
    timeout: 30_000,
  });
  console.log(await res.text());
} finally {
  await browser.close();
}

Chromium does not support SOCKS5 proxies that need a password, so browser automation should use the HTTP endpoint. A full page load also uses far more traffic than a plain HTTP call, as Chapter 6 showed.

Scrapy

Scrapy reads the proxy from each request's meta. A small downloader middleware sets it in one place for the whole project.

# myproject/middlewares.py
import os


class ProxyMiddleware:
    def process_request(self, request, spider=None):
        request.meta["proxy"] = os.environ["PROXY_URL"]


# myproject/settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ProxyMiddleware": 350,
}
DOWNLOAD_TIMEOUT = 30
RETRY_TIMES = 3
CONCURRENT_REQUESTS_PER_DOMAIN = 4

Priority 350 runs the middleware ahead of Scrapy's built-in HttpProxyMiddleware, which then moves the credentials into a Proxy-Authorization header. Replace myproject with the real project name. Start with low per-domain concurrency and raise it while watching the error rate.

Verifying the exit and fixing errors

A script can finish without errors and still send traffic from the machine's own IP. A misspelled key in the proxies dictionary is silently ignored, for example. The check below compares the direct address with the proxied one.

import os

import requests

CHECK = "https://api.ipify.org"
proxy_url = os.environ["PROXY_URL"]
proxies = {"http": proxy_url, "https": proxy_url}

direct = requests.get(CHECK, timeout=10).text
exits = {
    requests.get(CHECK, proxies=proxies, timeout=30).text
    for _ in range(5)
}

print("direct :", direct)
print("exits  :", sorted(exits))
assert direct not in exits, "traffic is not using the proxy"

A rotating plan should print several exit addresses. A sticky or static plan should print one. If the assertion fails, the proxy settings are not being applied. ProxyLook's free proxy checker and IP lookup give a second opinion from the browser.

When a request fails, the first task is to find where. A proxied request passes four points, and each one fails in its own way.

For HTTPS targets, a proxy-side failure arrives as a failed tunnel. curl reports that the CONNECT tunnel failed, and requests raises ProxyError. A status code inside a normal response came from the target.

One TLS case needs a note. Some unblocker products re-sign traffic and require the vendor's CA certificate to be installed. Install the certificate as the vendor documents. Turning verification off hides the error and removes the protection. Blocks from the target are the subject of Chapter 10, and ProxyLook's companion book, Scraping Without Blocks, goes further.