Scraping JavaScript Websites — Network Layer vs. Browser

Two Paths into the Same Data

Most modern e-commerce, marketplace, and SaaS sites are JavaScript-heavy. The HTML you get from requests.get() is a near-empty skeleton; the real data shows up after the browser executes a bundle of JS that fetches the actual content from one or more API endpoints. The conventional wisdom — "JavaScript site → use a headless browser" — is technically correct and often unnecessarily expensive.

There are two real options for these sites:

Network-layer scraping: figure out which API calls the page is making behind the scenes, replay them with plain Python, and parse the JSON directly. No browser, no rendering, no JavaScript execution.

Browser scraping: drive a real browser with Playwright (or Selenium), let the page render fully, then read the data out of the rendered DOM.

The browser approach always works. The network approach, when it works, is dramatically faster, cheaper, and more stable — but it doesn't always work, and figuring out which a target supports is a 10-minute investigation that pays for itself many times over. This article is about that investigation, the decision framework, and how to operationalize each approach once you've picked one.

The 10-Minute Investigation

Before writing any scraping code, open the target page in Chrome with DevTools open, Network tab selected, "Fetch/XHR" filter on. Reload the page. Look at the requests that come back.

You're looking for one of three patterns.

Pattern A: Clean JSON API. One or two XHR requests come back with structured JSON containing the data you want. Often /api/products?id=..., /_next/data/... (Next.js), /graphql, or similar. The JSON has clean, named fields. This is the jackpot — network-layer scraping is going to be 10x faster than browser scraping, and you can usually skip rendering entirely.

Pattern B: GraphQL or signed API. The data comes from an API, but the request is a POST with a complex query body, or includes a signed token, an HMAC, or a CSRF cookie that's hard to reproduce. Network-layer scraping is possible but requires reverse-engineering the auth pattern. Sometimes worth it for high-volume work; usually not worth it for one-time scrapes.

Pattern C: Server-rendered HTML in an XHR response. The XHR returns a chunk of HTML rather than JSON, and the page just inserts it into the DOM. You can fetch the XHR endpoint and parse the HTML yourself — same speed advantage as Pattern A, slightly more parsing work.

If none of these patterns are present and the page is rendered through opaque WebSocket streams, real-time client-side computation, or aggressive anti-debugging measures — that's the case for browser scraping.

The investigation takes 10 minutes per target site. Skipping it and defaulting to browser scraping is one of the most common avoidable mistakes in scraping work.

Network-Layer Scraping in Practice

Once you've identified the API call, the implementation is short. Copy the request from DevTools as cURL ("Copy as cURL" in the Network tab), paste it into your editor, translate to Python requests or httpx, and you're 80% done.

import httpx

def fetch_product(product_id: str) -> dict:
    response = httpx.get(
        f"https://target.example.com/api/products/{product_id}",
        headers={
            "Accept": "application/json",
            "User-Agent": "MyScraper/1.0 (contact@example.com)",
            "Referer": f"https://target.example.com/p/{product_id}",
        },
        timeout=10.0,
    )
    response.raise_for_status()
    return response.json()

The remaining 20% is figuring out what makes the API accept your request. Usually it's one of:

• A specific Referer header.
• A cookie set on the homepage. (Hit / first to populate, then call the API with the same cookie jar.)
• A header like X-Requested-With: XMLHttpRequest that the site uses to distinguish API calls from direct loads.
• An anti-CSRF token embedded in the homepage HTML. (Fetch the homepage, regex the token, include it in subsequent requests.)

None of these are hard once you've identified them. The DevTools "copy as cURL" output usually contains the working set of headers and cookies; reproducing them in Python is straightforward.

What you get from network-layer scraping is dramatically better unit economics. A request that takes a browser 4 seconds to complete (load page, run JS, render) takes 200ms over plain HTTP. Memory per worker drops from 200MB+ to under 20MB. Concurrency goes from "6 browsers per machine" to "200 simultaneous HTTP requests per machine." For high-volume work, the difference is the difference between "this is feasible" and "this needs its own server farm."

When Browser Scraping Is the Right Answer

Browser scraping is the right call in several specific situations:

The site has aggressive bot protection that checks JS execution. Cloudflare's "Just a moment…" challenge, DataDome's interaction signals, and PerimeterX's challenge pages all require a real browser to pass. No amount of header manipulation will get plain HTTP through them. (Even then, it's usually browser plus Kameleo for fingerprint control — see the Kameleo article.)

The data is genuinely produced by client-side computation. Some pricing tools, financial dashboards, or interactive maps compute their displayed values in JS rather than fetching them from an API. There's no API call to intercept; the data only exists once the JS runs.

You need to perform actions, not just read. Logging in, filling forms, clicking through multi-step workflows, triggering downloads. These are browser-shaped tasks; the network layer doesn't help.

The API auth is too complex to be worth reverse-engineering. If the site uses signed requests with rotating client-side tokens generated by obfuscated JS, the engineering cost of reproducing the auth pattern in Python often exceeds the cost of just running a browser. Decide based on how often you need to scrape and how much volume.

The data is paginated through scroll-loaded JS. "Load more" buttons that fetch a new XHR are easy to script with Playwright; reproducing the same pagination in pure HTTP requires walking the same XHR sequence the browser would. Sometimes that's straightforward; sometimes the cursor-token logic is opaque enough that scripting the browser is faster.

Hybrid: Browser to Bootstrap, Network for Volume

For sites with auth complexity but large volumes, the right answer is often a hybrid. Use Playwright (or Playwright + Kameleo) to log in once and capture the resulting cookies / tokens. Then hand those cookies off to a plain httpx session and do the bulk work over the network layer.

def get_authenticated_session(username: str, password: str) -> httpx.Client:
    with sync_playwright() as pw:
        browser = pw.chromium.launch(headless=False)
        context = browser.new_context()
        page = context.new_page()
        page.goto("https://target.example.com/login")
        page.fill("[name=username]", username)
        page.fill("[name=password]", password)
        page.click("text=Sign in")
        page.wait_for_url("**/dashboard")

        cookies = {c["name"]: c["value"] for c in context.cookies()}
        browser.close()

    return httpx.Client(cookies=cookies, base_url="https://target.example.com")

session = get_authenticated_session(USER, PASS)
data = session.get("/api/items").json()  # plain HTTP, fast

You pay the browser cost once at startup, then run the bulk of the scrape over the network layer. This is the right shape for "logged-in but high-volume" workloads — the kind that come up constantly in marketplace, vendor-portal, and partner-API scraping.

Wrap-Up

"JavaScript site → use a headless browser" is the convenient default and the wrong default. The right default is "open DevTools first, find the API call, default to network-layer scraping unless you have a specific reason to render."

The investigation takes 10 minutes. The throughput difference is 10x or more. The operational simplicity of "no browser, no rendering, no Kameleo, just HTTP" is enormous for any project where it's available.

For the architectural pieces around either approach — worker pools, caching, anti-bot strategy — see the Python Web Scraping, Playwright Automation, and Browser Automation hubs.