Every AI team I’ve talked to this year plans carefully for compute and labeling. Almost none of them budget for the layer underneath: actually getting the data. Then the crawler that ran beautifully against ten thousand pages falls over at a million, someone spends a week rotating user agents by hand, and a project that was “two sprints” quietly becomes a quarter.
The awkward truth is that web scraping at training-set scale is a network engineering problem before it’s a data problem. Here’s what actually breaks, and what the teams who ship datasets on time do differently.
Where collection actually breaks
It rarely breaks where you’d guess. HTTP 429s are the polite failure; you see them, you back off, you carry on. The expensive failures are the quiet ones. Cloudflare Turnstile and DataDome sit in front of a huge share of commercially interesting sites now, and they don’t just block, they score. A request from a known datacenter range, with a TLS setup that doesn’t match its claimed browser, arriving at machine-gun intervals, gets served a challenge page. Your parser doesn’t crash. It ingests the challenge page. Congratulations, your dataset now contains four hundred thousand copies of “checking your browser.”
Worse than the block is the decoy. Some sites detect scrapers and feed them plausible junk: shifted prices, reordered listings, truncated text. If that lands in a training set, no eval catches it, because the poison looks exactly like signal. This is the strongest argument for getting the access layer right before scale, not after. You can rerun a crawl. You can’t easily un-train a model.
Matching the IP type to the target
The standard mistake is buying one kind of proxy for everything. The teams that do this well run a mix, and the split is boring and practical.
Datacenter proxies handle the tolerant majority: public APIs, government portals, academic mirrors, most long-tail sites with no bot wall. They’re fast and they’re cheap, unlimited-bandwidth plans start around $1.39 per proxy, so you point them at everything that doesn’t fight back and save the expensive addresses for targets that do.
For the protected minority, the requests need to come from addresses that look like real households, which is what residential proxies are. They’re billed by traffic rather than by IP, from about $1.75 per gigabyte at the cheaper end of the market, which changes how you design the crawl: you deduplicate URLs before fetching, you skip binary assets you don’t need, and you stop re-downloading pages you already have. I’ve seen residential bills drop by half from hygiene alone, with no change to the target list.
Geography matters more than people expect, too. If your model needs to understand German e-commerce or Japanese news, collecting it through US exit points gets you redirects, consent walls and machine-translated variants. Country-level targeting isn’t a luxury feature for training data. It decides what distribution you actually sample.
Sessions and rotation
One more design decision that separates working pipelines from flaky ones: session handling. Rotating the IP on every request is right for breadth, wrong for depth. Anything with pagination, login state or a multi-step flow needs a sticky session, the same IP held for the whole sequence, or the site sees a user who teleports between countries mid-checkout and reacts accordingly. Decent providers let you choose per request. If yours doesn’t, that’s a real limitation, not a nitpick.
And instrument the thing. The single most useful metric in a collection pipeline is success rate per target domain, where “success” means the page you wanted, not any page with a 200 status. Fingerprint the challenge pages of the big anti-bot vendors once, alert when their share of responses climbs, and you’ll catch a silent block in an hour instead of discovering it in the dataset a month later.
The packet-level tell
There’s a subtler failure mode worth knowing about. Modern detection doesn’t only look at the IP; it reads the TCP handshake itself. Every operating system assembles packets with its own quirks, and a proxy server running Linux leaves Linux fingerprints on traffic that claims to be Chrome on Windows. That mismatch is invisible in your logs and perfectly visible to the site. Most mid-market providers ignore this entirely. Proxya is the rare one at its price point that rotates the OS-level fingerprint on its datacenter and ISP lines, which is a large part of why the same target that blocks a generic datacenter IP will often accept theirs. Fair warning alongside the praise: it’s a younger provider, there’s no named account manager or signed SLA, and the tooling around it is thinner than what Bright Data or Oxylabs ship. Teams with enterprise procurement requirements will notice. Small teams mostly won’t, and the difference shows up as budget they didn’t spend.
The part your counsel should read
None of this is a license to collect everything reachable. Publicly visible is not the same as fair game: personal data under GDPR stays personal when scraped, terms of service exist whether or not your crawler reads them, and the copyright status of text used for training is being litigated right now in several jurisdictions. Respect robots.txt, keep provenance records for every source, and have someone who isn’t the engineer sign off on the target list. Ethically sourced applies to the proxy layer as well; the honest residential providers pay consenting users for bandwidth, the dishonest ones bundle it into free apps. Ask how the pool is built. A vague answer is an answer.
Budget it like infrastructure
The pattern across teams that ship: they treat data access as a line item, same as compute. A number per gigabyte, a mixed pool matched to the target list, sticky sessions where flows need them, and a compliance check before the first request goes out. It’s unglamorous work. It’s also the difference between a training set you trust and a very expensive folder of challenge pages.
