Full definition
Rate limiting comes in two flavors. Per-IP rate limiting is what makes proxies useful — if a site allows 100 requests per minute per IP, you need a pool of 100 IPs to make 10,000 requests per minute. Per-account or per-API-key limiting is harder to bypass; it requires creating multiple accounts (often against ToS) or finding the unauthenticated public endpoints.
Symptoms of being rate-limited: HTTP 429 Too Many Requests, HTTP 403 Forbidden after a burst, response times suddenly increasing 10×, or the destination site silently returning empty results. Always log response status codes when scraping — silent failures are the worst kind.
Architecture: design your scraper to back off when you see 429s. Implement exponential backoff with jitter. Don't just rotate to a fresh proxy IP and retry instantly — that pattern is itself a bot signal.