Rotating Proxies Safely in Browser Automation

The Default Advice Is Backwards

Search for "how to rotate proxies for web scraping" and you'll get the same answer everywhere: rotate on every request. Maintain a pool of 1,000 proxies. Pick a random one for each fetch. The implication is that more rotation equals more anonymity equals fewer blocks.

This is exactly wrong against any modern bot-protection system. Cloudflare, DataDome, PerimeterX, Akamai — they all build a session model that joins TLS fingerprint, browser fingerprint, cookie state, and source IP into one identity. When the IP jumps every request but the TLS fingerprint and browser fingerprint stay constant, the protection layer sees an identity teleporting around the planet. That's a louder bot signal than just sending all your traffic from one IP and letting it look like one user.

The right model is the opposite: one identity, one IP, until that identity gets blocked. Then move on, slowly, and never come back. This article walks through what that looks like in practice — the rules, the data structures, and the cooldown logic that make a small proxy pool last weeks instead of hours.

Why Per-Request Rotation Fails

To see why per-request rotation is bad, think about what the target site sees. From their perspective, the same browser fingerprint (canvas hash, font list, navigator properties) is making requests from 47 different IPs in 30 seconds — Mumbai, Atlanta, Berlin, São Paulo. No human does this. No real session does this. The only thing that does this is a scraper rotating proxies on every request.

The fingerprint is what links the requests together. Even if the IP changes, the TLS handshake (JA3 hash) is identical. The HTTP/2 stream prioritization is identical. The header order is identical. The browser canvas hash is identical. To a system that joins these signals, you've just made yourself easier to identify, not harder.

Worse, you've burned 47 proxies to fetch what could have been fetched with one. Modern proxy pricing — residential at $5–15/GB, ISP/static at $1–3/IP/month — makes that waste real money. A 1,000-proxy pool sounds like a lot until you realize a per-request rotator can churn through it in a single overnight run.

The Block-Only Rotation Model

The rule is simple: pair an identity (browser fingerprint, cookies, profile) with one outbound IP, and use that pairing until the target actually blocks it. Only then rotate.

"Actually blocks" means a clear protection signal:

• HTTP 403 with a known challenge body
• HTTP 429 rate-limit response
• HTTP 503 with a Cloudflare or DataDome marker
• A redirect to a challenge URL (/cdn-cgi/challenge-platform/)
• A page that loads but contains a known challenge marker ("Just a moment…", "Press & Hold")

What it doesn't mean:

• A 500 from the application (that's the site's own bug, not a block on you)
• A timeout (could be network, could be the site being slow)
• A 404 (the URL doesn't exist; rotating won't help)
• A successful response with unexpected content (could be a soft block, but verify before treating as one)

The detection logic is a small Python function that runs after every fetch and returns a BlockReason enum or None. None means proceed. Anything else triggers rotation.

Pool Structure and State

The proxy pool is split into tiers based on cost and recovery characteristics:

Mobile rotating proxies — services like iproxy, Soax, IPRoyal — are billed by bandwidth and rotate through cellular IPs on demand. These are "self-healing" in the sense that calling the rotation URL gives you a fresh IP. Best for high-stakes targets where you need rotation but want each rotation to be a controlled event.

Static residential / ISP proxies — Webshare and similar — are billed per IP per month. They don't rotate; once assigned to you, they stay yours. Best for long-running pairings (per-account workflows, customer service automation) where you need IP stability over weeks. When one gets blocked, it's out of the pool until cooldown expires.

Datacenter proxies are the cheapest tier. Many sites block datacenter ASNs entirely; on those, they're useless. On sites without ASN-level protection, they're fine and economical.

State for each proxy lives in a small JSON file (or a database table for larger pools):

{
  "proxies": {
    "5": {
      "host": "proxy-5.example.com",
      "port": 8080,
      "tier": "static_residential",
      "last_used_at": "2026-04-30T03:14:22Z",
      "in_use_by_worker": null,
      "blocked_at": null,
      "block_reason": null,
      "blocks_today": 0
    }
  }
}

Three flags drive the claim logic:

1. in_use_by_worker — set when a worker claims the proxy at startup, cleared when it releases. Prevents two workers from using the same proxy simultaneously.
2. blocked_at — set when the proxy hits a block. Cleared after the cooldown period expires (24 hours is my default for static; immediate for mobile via rotation URL).
3. blocks_today — counter. If a proxy gets blocked more than, say, 3 times in 24 hours, it goes into a longer quarantine (a week). Some IPs are permanently flagged and there's no point cycling them back into rotation.

Worker Claim Logic

At worker startup:

1. Read the proxy state file (with a file lock, or row-level lock in MySQL).
2. Filter to proxies in the worker's assigned slice (round-robin partitioning by ID).
3. Within that slice, exclude any with in_use_by_worker set or blocked_at within the cooldown window.
4. Pick the available proxy with the oldest last_used_at. (Spread usage across the pool rather than hammering the same one.)
5. Mark it in_use_by_worker = my_worker_id, update last_used_at = now.
6. Release the lock.

The worker now owns this proxy for its run. It pairs the proxy with its browser profile (Kameleo or otherwise), starts the session, and begins claiming work from the queue.

On block:

1. Mark the current proxy blocked_at = now, increment blocks_today, clear in_use_by_worker.
2. Stop and discard the current browser profile (or, for mobile rotating, just call the rotation URL and continue with the same profile + a new IP).
3. Claim a new proxy via the same logic above.
4. Start a new browser profile (or reuse with the rotated IP).
5. Retry the failed URL exactly once. On second block, mark URL error and move on.

Cooldown Timing

The cooldown window depends on the protection layer.

For Cloudflare and DataDome, 24 hours is the right default for a static residential IP. The protection systems' models age out signals on roughly that cycle, and an IP that was blocked yesterday usually has a clean slate the next day. For aggressive targets, push it to 48 or 72 hours.

For mobile rotating proxies, the cooldown is the time it takes the carrier to assign you a different IP — usually seconds via the rotation URL. That's the whole point of mobile pools; you're paying for elastic IP supply.

For datacenter proxies, cooldown rarely helps. If a datacenter ASN is being blocked, the entire ASN is suspect — rotating to another IP in the same /24 won't change anything. Move that proxy to a different target or retire it.

What This Looks Like at Scale

To make this concrete: a recent Cloudflare-fronted scrape ran six parallel workers against 580 static residential proxies, fetching ~25,000 URLs over an overnight run. Total proxies consumed: 5. Total blocks observed: 89. Total URLs that failed permanently: 14.

That ratio — 5 proxies consumed, 580 still healthy in the morning — is the point. With per-request rotation, the same job would have churned through 200+ proxies, generated more blocks (because the rotation pattern itself was the bot signal), and produced worse data. Block-only rotation is both more polite and more effective.

Wrap-Up

Proxy rotation is one of those areas where the conventional wisdom is actively counterproductive. The instinct to rotate more aggressively for safety is exactly backward against modern bot protection. The right move is fewer rotations, longer pairings, and disciplined block detection that drives rotation only when it's actually needed.

Combined with a real anti-detect browser (Kameleo or equivalent) and polite request pacing, this model produces scrapers that consume IPs in single digits per overnight run instead of hundreds. That's the operational difference between "we need to keep buying more proxies" and "the pool we have is way more than enough."

For more on the surrounding architecture — Kameleo integration, worker pool design, block detection signals — see the Undetectable Browser Automation, Kameleo Automation, and Browser Automation hubs.