Security, Law and Ethics

A free proxy list promises thousands of working IPs at no cost. Someone pays for those servers, and the list does not say who or why. Every request sent through one passes across a stranger's machine first.

Why free public proxies are risky

A proxy sits in the middle of the connection by design. Chapter 3 set out what that position lets an operator see. With a known company, a contract and a reputation at stake, the exposure is acceptable. With an anonymous open proxy, it is the whole problem.

A free trial from a paid provider is a different thing: a named company, published terms and a support desk. The ProxyLook directory lists more than 200 proxy providers, and entry-level datacenter traffic costs little enough that a free list saves almost nothing.

Credentials and TLS

Proxy credentials are spending power. Anyone who holds them can run traffic on the account's bill, and whatever they do traces back to the account holder. Chapter 8 moved the proxy URL into an environment variable. The next place it tends to leak is the logs, where a full URL prints the password in clear text.

import os
from urllib.parse import urlsplit

def masked(proxy_url):
    """Return the proxy URL with the password hidden, for logs."""
    password = urlsplit(proxy_url).password
    if not password:
        return proxy_url
    return proxy_url.replace(f":{password}@", ":***@", 1)

proxy = os.environ["PROXY_URL"]
print("using proxy", masked(proxy))

Masking is one habit among several.

• Keep credentials in environment variables or a secret manager, never in source files.
• Add .env files to .gitignore and turn on secret scanning in the repository.
• Create one sub-user per project or person, so that one leak exposes one budget.
• Rotate passwords on a schedule, and at once when someone leaves or a password shows up in a log.
• Use IP allowlisting for servers with fixed addresses, and remove old entries.
• Mask the password before logging a proxy URL or pasting an error into a ticket.

Allowlisting has a second benefit. A username and password sent to an http:// gateway travel unencrypted between the client and the gateway. On a shared network anyone on the path can read them. An allowlisted IP sends no password at all.

Always verify TLS

A certificate error through a proxy is a warning, and the tempting fix is to switch verification off. Doing so removes the one protection that stops a proxy operator reading passwords and session cookies. With verification on, a proxy can carry HTTPS traffic but cannot open it.

Chapter 8 noted the one exception: unblocker products that re-sign traffic and need the vendor's certificate installed. That is a deliberate trade, and it means the vendor can read what passes through. Trust the certificate in that one client only, and never send logins or payment details through such a product.

The legal questions to ask first

Proxies are legal tools in most places. The legal questions attach to what is collected and how, and they are not one question. Four kinds of data call for very different levels of care.

The table turns those categories into questions to answer in writing before a project starts.

A few points deserve more than a table cell. The GDPR covers information about identifiable people in the EU, and it can reach operators based elsewhere. California's CCPA and a growing list of national laws set their own rules, and they do not all treat public data the same way. Under the GDPR, storing personal data creates duties: a lawful reason, limited retention, and a way to answer deletion requests.

Terms of service matter most where they were clearly accepted, as with an account signup. Multi-accounting usually falls here. It can breach a platform's terms, and the operator carries the risk of bans and claims.

Copyright generally protects expression, not facts. A price is a fact. The review text next to it is someone's writing. Some jurisdictions also protect databases as such. The rules for training AI models on collected content are still being argued in courts and legislatures.

Robots.txt and rate etiquette

The file at /robots.txt states which paths a site asks crawlers to stay out of. The format is an internet standard, RFC 9309. It is a request, not a lock, and its legal weight varies. Ignoring it still looks bad in any later dispute, and following it costs little. Python's standard library reads it with urllib.robotparser.

Etiquette goes beyond that file:

• Keep the load small. A crawl should never slow the site for real visitors. Small sites need gentler limits than large ones.
• Back off when asked. A 429 or 503 is a request to slow down, not a puzzle to route around.
• Prefer the official route. Where a site offers an API, a data feed or a sitemap, use it.
• Take only what the project needs. Fewer pages and fewer fields mean less cost, less risk and less to secure.

Residential IPs: sourcing and responsible use

Chapters 2 and 7 covered how residential and mobile pools are built and which sourcing questions to ask. The short version: buy only from networks whose device owners opted in, are compensated and can leave. The other half of the responsibility sits with the buyer. Every request leaves through a real person's home connection or phone, and the target sees that person's address.

• Send nothing through a residential IP that would be a problem if traced to its owner.
• Keep volume per IP modest. The bandwidth belongs to someone, and on mobile it may be metered.
• Stay inside the provider's acceptable use policy, including its list of restricted targets.
• Use datacenter IPs where they work, and keep residential IPs for the jobs that need them.
• Keep the provider's written sourcing answers on file and ask again each year.

The residential proxies page in the directory is a starting point for a shortlist. The sourcing questions still have to go to each vendor directly.

A 30-day action plan

The plan below turns the book into four weeks of work for one team and one project.

1. Week 1: define and pilot. Write down targets, countries, pages per day and the fields needed. Answer the five legal questions. Shortlist two or three providers with the compare page or the ProxyLook assistant, then run the seven-day pilot from Chapter 7.
2. Week 2: build and verify. Move credentials into environment variables with one sub-user per project. Confirm the exit IP with What Is My IP and run the WebRTC Leak Test in every browser profile. Add a content check to every response.
3. Week 3: harden and cut waste. Add per-domain concurrency limits, backoff with jitter, caching and resource blocking. Start recording success rate, cost per 1,000 successful requests, p95 latency and GB per day.
4. Week 4: govern. Set daily caps and alerts. Rotate passwords and clean the allowlist. Write one page on what is collected, why, and how long it is kept. File the provider's sourcing answers, and set a monthly review date.

The appendix holds a scorecard worksheet for week 1 and a pre-launch checklist for week 2.