Scraping only scales when it is engineered around how the modern web actually behaves. That means grounding decisions in protocol details, traffic composition, and site defense patterns, not rules of thumb. The data is clear enough to guide pragmatic choices that reduce blocks, bandwidth waste, and noisy retry storms.
Blend with the traffic you are entering
Automated activity is not an edge case. Nearly half of global traffic is generated by bots, and about one third is classified as malicious. Your crawler will be judged in that crowded context, which is why precision on headers, timing, and error handling matters more than ever.
Mobile now drives roughly 60 percent of visits worldwide. If your collector only imitates desktop, you will miss mobile-specific content, variant templates, and lazy loading behaviors that are served conditionally. Matching device class is not cosmetic, it affects what the server renders and the number of requests your job triggers.
Speak the protocols the web expects
HTTPS is the default. More than 95 percent of page loads in mainstream browsers use encrypted transport. Scrapers that fall back to older ciphers, TLS 1.0 or 1.1, or lack Server Name Indication will look dated and are more likely to be throttled or terminated.
HTTP/2 carries the bulk of requests on the open web, and HTTP/3 adoption keeps growing. Using HTTP/2’s multiplexing correctly reduces connection churn, shrinks head-of-line blocking, and lowers your IP’s behavioral footprint compared with opening many parallel HTTP/1.1 sockets.
Do the request math before you scale
The median page triggers about 70 to 80 network requests. JavaScript payloads alone often land near 450 KB, with images dominating the remaining bytes. Multiply that by your target set and the concurrency you intend to run, and you get bandwidth, time, and error budget needs that are orders of magnitude larger than a simple URL count suggests.
If you plan to collect 100,000 product pages and each page averages 75 requests, the job is about 7.5 million requests. A one percent block rate at that scale is 75,000 failures to triage, which is why proactive pacing and route diversity pay for themselves.
Fingerprint discipline beats retries
User agent distribution is not uniform across browsers. Chrome-based agents account for around two thirds of usage, with Safari a distant second. That should inform your UA rotation rather than randomizing across obscure clients that sites rarely see. Accept-Language, Accept, and encoding headers should be internally consistent with your chosen UA and device class.
Headless browser execution is often necessary for client-side rendering, but default fingerprints still get flagged. Stabilize canvas, WebGL, timezone, and media device signals, and keep navigation timing within realistic jitter windows. Avoid batchy, perfectly spaced request patterns that scream automation.
Pick the right IP strategy for the job
IP reputation and network topology drive a large share of blocks. Data center ranges are cost efficient and deliver consistent throughput, which makes them ideal for large static asset pulls, public API endpoints that do not gate by ASN, and sites with light defenses. When you need scale at a tight unit cost, a cheap data center proxy can push massive volumes with predictable latencies.
Residential or mobile routes tend to win on hardened targets that profile ASN, connection reuse, and session longevity, but they are pricier and more variable. A mixed pool that defaults to data center and escalates by rule when signals indicate bot management is often the most economical pattern.
Measure what operators actually block on
Track 429 and 403 rates separately, plus TLS errors, connection resets, and content-level soft blocks like empty shells or challenge pages. Attribute failures by ASN, region, protocol version, and device class to see where to pivot. With HTTP/2, monitor concurrent streams per origin, not just total sockets, since exceeding server limits invites throttling.
Pacing matters. Token-bucket rate limits on the server side will tolerate short bursts if your long-term average is modest. Uniformly spaced requests look robotic. Mild randomness within sane bounds is easier to accept than strict intervals or flood patterns.
