Scraping Without Blocks
A free developer guide to collecting public web data reliably: detection, proxies, resilient clients and production habits.
- The detection stack, layer by layer, and what each block looks like
- Python code for retries, rotation, soft-block checks and concurrency
- A routing ladder that starts cheap and escalates only when needed
About the eBook
A developer’s guide to proxies, fingerprints and reliable data collection
Scrapers rarely fail loudly. They slow down, return half a page or collect the wrong country’s prices while every status code says 200. Scraping Without Blocks explains what a website checks before it answers, and how to build a client that passes the checks it can and reports clearly when it cannot.
The code is Python with httpx and Playwright. It covers retries with backoff, session rotation, soft-block detection, bandwidth control and the metrics that show what a successful record costs. It is about public data, collected responsibly.
A scraper rarely fails loudly. It returns the wrong data with a 200.
By the last page, a reader can
- Read a block and tell which detection layer caused it
- Choose proxy types and rotation by target difficulty
- Build an httpx client with retries, backoff and session rotation
- Run Playwright through proxies without wasting bandwidth
- Decide when a scraper API is cheaper than building in-house
- Track the metrics that show cost per successful record
Look inside
Real pages from the eBook
Diagrams that show how things work, tables to decide from and code to copy. Select a page to see it full size.
What is inside
6 chapters and an appendix
Every chapter ends with key takeaways and is also a web page. Open a chapter to see its sections.
Part I · Diagnose
01Anatomy of a BlockFollows one request from script to block page, names the six detection layers it passes, and shows which symptom points to which layer.p. 5
- One request, several observers
- The detection stack
- How a block presents
- Reading the symptom
- Why a proxy fixes only the first two layers
- Scope and responsibility
02Picking Proxies for the TargetSorts targets into four difficulty tiers by observable signs, matches a proxy type to each, and builds a routing ladder that starts cheap and escalates.p. 11
- Sorting targets by what can be observed
- Which proxy type fits which tier
- Rotation per request or sticky sessions
- Geo-targeting for localized content
- A routing ladder that starts cheap
- Quick decisions
Part II · Build
03Building a Resilient ClientBuilds a small async httpx crawler step by step: proxy sessions, honest headers, status-aware retries with jitter, soft-block validation and a bounded worker pool.p. 17
- The base client
- One identity per session ID
- Different failures, different responses
- The retry loop
- Catching soft blocks in the body
- A bounded worker pool
04Headless Browsers Through ProxiesShows when a real browser earns its cost, then sets up Playwright with a proxy per context, matching locale and time zone, and resource blocking.p. 25
- When a browser is needed, and when it is waste
- What a browser costs
- A proxy per browser context
- Keeping locale and time zone consistent
- Blocking heavy resources
- Reusing a context for sticky work
Part III · Operate
05Scraper APIs and UnblockersExplains what scraper APIs and unblockers do, how per-request pricing changes the cost math, and where a managed service belongs in a routing ladder.p. 31
- What these products do
- Build or buy
- How per-request pricing changes the math
- Where an API fits in the ladder
- Lock-in and observability
- Decision table
06Running in ProductionTurns a working scraper into a scheduled service: idempotent jobs, per-domain pacing, the four metrics that matter, data-quality checks, useful logs and a launch checklist.p. 37
- Queues and idempotent jobs
- Per-domain rate limits
- Metrics, budgets and alerts
- Data-quality checks that catch soft blocks
- Logging enough to debug a block
- Etiquette, law and launch
Reference
AAppendixReference pages for the crawl desk: a status code table, a pre-flight checklist, a debugging order, a glossary and links to ProxyLook tools.p. 44
- HTTP status quick reference
- Pre-flight checklist
- Debugging order
- Glossary
- ProxyLook tools and further reading
Who it is for
Written for the people doing the work
Developers and data engineers
Patterns for clients that survive real targets, with code to start from.
Data and analytics teams
A way to budget collection work by cost per successful record.
Technical founders
A clear build-or-buy view of proxies, browsers and scraper APIs.
Live from the directory
Proxy networks to test, with current figures
The book prints no prices, because they change every month. This table reads the ProxyLook directory each time the page loads, so the numbers stay current.
| Network | Proxy types | Pool (vendor-stated) | Listed from |
|---|---|---|---|
| Bright Data | Residential, Datacenter, Mobile, ISP | 150M+ IPs | $5.04/GB |
| Decodo (formerly Smartproxy) | Residential, Datacenter, Mobile | 125M+ IPs (residential + mobile + ISP) | $3.75/GB |
| IPRoyal | Residential, Datacenter, Mobile, ISP | 32M+ IPs | $3.50/GB |
| NetNut | — | 85M+ residential + 5M+ mobile IPs across 195 countries | $3.45/GB |
| NodeMaven | Residential, Mobile, ISP | 30M+ residential + 250K+ mobile IPs, ~190 countries, 1,400+ locations | $4.25/GB |
| Oxylabs | Residential, Datacenter, Mobile, ISP | 177M+ IPs | $4.00/GB |
| Proxy-Seller | IPv4, IPv6, ISP, Residential, Mobile | 20M+ residential + 1M+ ISP/DC/IPv6 across 220+ countries | $1.77/GB |
| ProxyEmpire | Residential, Unlimited residential, ISP, Mobile, Dedicated mobile, Datacenter | 30M+ residential + 4M+ mobile IPs | $3.50/GB |
| SOAX | Residential, Mobile, ISP | 155M+ IPs | $4.00/GB |
| Webshare | — | 80M+ residential + 30M+ datacenter IPs across 195+ countries | $0.99/GB |
Alphabetical order, no ranking implied. "Listed from" is the entry price shown on each provider page. Pool sizes are the vendors' own figures. See the full directory or compare pricing.
FAQ
Questions, answered
Is the eBook really free?
Yes. There is no sign-up and no email gate. The PDF downloads directly and every chapter can be read online.
What language is the code in?
Python 3.11 or later, using httpx and Playwright. The patterns carry over to other languages.
Does it show how to bypass a specific anti-bot product?
No. It explains how detection works and how to build a client that behaves well and fails clearly. It makes no promise about defeating any particular product.
Is web scraping legal?
It depends on the data, the method and the country. Public data, personal data, logged-in content and terms of service raise different questions. The book flags them and is not legal advice.
Does a reader need The Proxy Playbook first?
It helps for the basics of proxy types and rotation. This book goes deeper on engineering and can be read alone by developers who already use proxies.
What format is it in?
A designed PDF with clickable links and bookmarks, plus the same chapters as web pages.
More from the library
Other free eBooks
The Proxy Playbook
The free handbook for choosing, setting up and running proxies, from the first request to a production pipeline.
The Multi-Account Handbook
A free guide to keeping accounts cleanly separate with antidetect browsers, cloud phones and the right proxies.
Get Scraping Without Blocks, free
49 pages, no sign-up and no email gate. Keep the PDF, or read any chapter in the browser.