Appendix

A crawl that returned 40,000 records yesterday returns 6,000 today, and the dashboard is red. These pages are for that moment. They condense the book into tables and lists that can be read in a minute.

HTTP status quick reference

The table gives the usual meaning of each result for a scraper and the first reaction. Chapter 3 turns it into code.

Result	Usual meaning	First reaction
200	Served, or a soft block	Validate the body before storing
301, 302	Moved, or sent to a challenge or login	Check the final URL
304	Unchanged since the last fetch	Keep the stored copy
403	Identity refused	New session; escalate if it repeats
404, 410	Page is gone	Record it; do not retry
407	Proxy rejected the credentials	Fix the configuration
429	Rate limit reached	Wait as told; slow the domain
500	Origin error	Back off and retry
502, 503, 504	Overloaded origin, gateway fault or challenge	Back off; inspect the body
Timeout or reset	Dead exit or a deliberate stall	New session, then retry

Pre-flight checklist

Run through this list before pointing a crawler at a new target.

• Confirm the data is public, and review personal data and the site's terms.
• Read robots.txt and note any crawl delay.
• Probe one page with an HTTP client, then a browser, and assign a tier.
• Look for the value in raw HTML or embedded JSON before using a browser.
• Pick the cheapest proxy type that clears the tier.
• Verify the exit country and match the language header.
• Set timeouts, an attempt cap and a per-domain rate limit.
• Define the validation marker and minimum size for a good page.
• Set a daily traffic budget with an alert.
• Check that logs record status, final URL, route and session ID.

Debugging order

When success drops, work from the cheapest check to the most expensive. The order follows the detection layers from chapter 1, with configuration first and data last.

1 · ConfigurationCredentials, gateway address, country parameter, remaining balance.

2 · Exit IPType, country and network of the address the target saw.

3 · RateRequests per domain per minute, and any Retry-After header.

4 · Client signatureHTTP version and headers compared with a current browser.

5 · Browser runtimeLocale, time zone, automation flags, blocked resources.

6 · Session behaviorPacing, page order and session length.

7 · DataBody validation and batch checks for soft blocks.

Glossary

Short definitions of terms used in this book. The ProxyLook glossary covers proxy vocabulary more widely.

Term	Meaning
ASN	The number of the network that owns an IP range
Backoff	Waiting longer after each failed attempt
Browser context	An isolated browser profile with its own cookies and proxy
Challenge page	An interstitial asking for proof of a browser or person
Dead-letter queue	A holding area for jobs that used up their retries
Exit IP	The address the target sees
Fingerprint	Client traits that together identify software or a device
Idempotent job	A job that stores the same result when run twice
JA3, JA4	Methods that hash a TLS ClientHello into a fingerprint
Jitter	Randomness added to waits so workers do not retry together
p95 latency	The time under which 95% of attempts finish
Rotating proxy	A gateway that assigns a new exit IP per request
Routing ladder	Routes ordered from cheapest to most capable
Scraper API	A service that fetches a URL and returns the page
Soft block	A 200 response with altered or missing data
Sticky session	One exit IP held for a sequence of requests
Unblocker	A scraper API exposed as a proxy endpoint

ProxyLook tools and further reading

ProxyLook is an independent directory that lists more than 200 proxy providers. Its free tools cover three of the checks above.

• The proxy checker shows the country and network of an exit IP.
• The user-agent parser shows what a header string claims to be.
• The cost calculator turns traffic estimates into a budget.

Two companion books sit beside this one. The Proxy Playbook is the flagship. It covers proxy types, protocols and basic setup. The Multi-Account Handbook is for teams that run several accounts. Multi-accounting can breach a platform's terms of service, and the operator carries that risk.

Key takeaways

• Read the status code as a clue to the layer, then validate the body anyway.
• Run the pre-flight list before the first request to a new target.
• Debug from configuration upward and change one variable per test.
• Keep the legal questions about personal data, logins and terms separate from the technical ones.