Cloudflare
Cloudflare Bot Management
The most common case. Plain Python gets a 403 "Just a moment…" page; a real browser fingerprint walks through.
// Topic
Undetectable Browser Automation — Surviving Cloudflare Without Crossing Lines
"Undetectable" doesn't mean breaking the rules. It means not looking like a bot — using a real browser fingerprint, a real residential or datacenter IP, polite request timing, and block-aware recovery. This topic covers how I do that responsibly and at scale.
What This Covers — and What It Doesn't
"Undetectable" is a loaded word. Here's what it means in this work, and what it doesn't.
It means: presenting a real browser fingerprint (canvas, WebGL, fonts, timezone, audio context) instead of a headless-Chrome shape; routing traffic through a real residential or datacenter proxy instead of a server's outbound IP; pacing requests politely (5–7 seconds between fetches, with jitter); identifying yourself in the user-agent with project name and contact info; respecting robots.txt; and recovering gracefully when the site does block you, rather than hammering through.
It does not mean: solving CAPTCHAs, bypassing authentication, faking JavaScript challenges, scraping disallowed paths, or violating site terms. If a site puts you behind a CAPTCHA you can't legitimately solve, the right answer is to back off — or to license the data through an official API. The systems I build abort cleanly when blocked, log the block, and never escalate to evasion tactics that would cross legal or ethical lines.
When You'd Need This
DataDome
DataDome-Protected Sites
Common on retail, ticketing, and high-value e-commerce. Stricter than Cloudflare in many cases. Real fingerprint + clean proxy is still the right approach.
PerimeterX / HUMAN
PerimeterX (HUMAN) Protection
Slightly more sensitive to interaction patterns. Real-fingerprint browser + slower request pacing usually gets through.
Akamai Bot Manager
Akamai-Fronted Targets
Heavyweight protection on enterprise sites. Same toolkit applies; expect lower per-IP throughput and plan around it.
Multi-Account
Multi-Account Workflows
When the same automation needs to run across many accounts without cookie / fingerprint cross-contamination — a profile-per-account model with proxy isolation per profile.
Compliance
Sites Where You're Authorized
Vendor portals, partner sites, and your own systems where you have permission to automate but the site still has bot protection that you have to legitimately work with.
How I Approach It
The toolkit is layered, lightest tool first. Layer 1: plain HTTP with realistic headers and polite pacing. About 60% of sites with "bot protection" badges are actually defeated by sending a normal user-agent and waiting 5 seconds between requests.
Layer 2: headless Chrome via Playwright. Adds JavaScript execution and a more realistic TCP/TLS fingerprint. Defeats another large slice of protected sites.
Layer 3: headed Playwright with a residential or datacenter proxy. Adds a real outbound IP and a non-headless browser shape.
Layer 4: Kameleo profiles attached to Playwright via CDP, with proxy-per-profile, behind a real Kameleo fingerprint. This is what defeats Cloudflare and most DataDome-class targets. About 95% of "uncrackable" scraping problems collapse at this layer.
Above all four layers sits the same operational discipline: block-only rotation (never rotate proactively, only on observed blocks), one retry on a fresh IP (never more — no retry storms), and clean abort on real blocks (if Layer 4 plus a fresh proxy still gets a 403, the URL is marked error and the run continues without it). The system logs every block with the response signal that triggered it, so a post-run inspection can tell you exactly where the protection escalated and which strategy still worked.
The result, in practice, is a scraper that consumes 5 proxies per overnight run instead of 50, absorbs 89 blocks without aborting, and ships clean data with no manual intervention. That's the system shape this topic produces.
Typical Stack
- Kameleo (anti-detect browser)
- Playwright (CDP attach)
- Python 3.11+
- Residential / mobile rotating proxies
- Static datacenter proxies (Webshare)
- Block-only rotation
- One-retry-on-fresh-IP rule
- Headed mode (no headless flags)
- Polite pacing + jitter
- Per-worker proxy partitioning
- Structured block logging
- robots.txt compliance
Case Studies
Case Study
Tariff-Exposure Data Pipeline
Cloudflare-protected ImportYeti scraped responsibly via Kameleo + Playwright with block-only rotation. 89 blocks absorbed, 5 of 580 proxies consumed.
Read the case study →Case Study
AI Customer Service Automation
Multi-session browser automation using isolated Kameleo profiles for legitimate customer service workflows at high volume.
Read the case study →Related Topics
This area overlaps closely with Kameleo Automation, Playwright Automation, and Python Web Scraping.
// Let's Build
Need an Anti-Detect Scraping System Built?
I build production scraping systems for sites with aggressive bot protection — using real fingerprints, real proxies, and block-aware architecture. No CAPTCHA bypassing, no terms violations, just the lightest tooling that gets the data.