Stop Spoofing User Agents

A modern fetch pipeline should stop treating the User-Agent string as the main switch for "browser-like" behavior. Chrome's privacy work has pushed more detail into opt-in Client Hints, MDN documents the same direction for HTTP negotiation, and bot defenses score more than headers. Builders need to test the whole request profile: headers, TLS fingerprint, cookies, region, and session continuity.

The old header trick is getting weaker

For years, the cheap scraping playbook was simple: copy a recent desktop Chrome User-Agent, add a few accept headers, and hope the page returned the same HTML a person would see. That was never a full browser contract, but it worked often enough to hide the gap.

Chrome's User-Agent Client Hints guidance explains the shift: instead of exposing increasingly detailed identity in one string, browsers can send lower-entropy defaults and let servers request more detail with Accept-CH. MDN's Client Hints guide frames the same mechanism as server-driven content negotiation that can replace parts of older user-agent sniffing.

That means a copied User-Agent is not just incomplete. It may be inconsistent with the rest of the request.

Consistency matters more than one perfect header

Real browsers send coherent bundles of signals. A Chrome session does not only have a Chrome-looking User-Agent; it has matching Sec-CH-UA headers when requested, a normal TLS handshake, cookies scoped to the right top-level context, realistic navigation timing, and JavaScript-visible properties that agree with the network layer.

Cloudflare's bot documentation makes the point from the defense side. Its bot scores are request-level confidence scores, not a single header check. Its JA3 and JA4 fingerprint documentation describes how TLS client fingerprints help group client behavior across requests.

If your scraper rotates only the User-Agent, you can create a worse signal: a header that claims modern Chrome while the TLS stack, Client Hints, cookie jar, or JavaScript environment says something else.

Build a header profile test, not a header list

Before scaling a browser-grade fetch job, add a small compatibility test for the target site:

1. Load the page with a real browser session.
2. Record the request headers after redirects and challenges.
3. Record which Client Hints the server asks for with Accept-CH.
4. Replay the flow through your fetch pipeline with the same region and session.
5. Compare status, redirects, cookies, rendered content, and API calls.

The goal is not to clone every byte. The goal is to catch contradictions before they become production failures: a mobile user-agent with desktop viewport assumptions, a Chrome string without expected Client Hints, a fresh cookie jar on every request, or a TLS profile that never matches the browser you claim to be.

What to log in production

For each target, log the signals that explain fetch behavior without storing unnecessary page data:

• Final URL and status after redirects.
• Whether the server returned Accept-CH or varied on Client Hints.
• The User-Agent family you intended to use.
• Session ID, country, and whether cookies were reused.
• Bot-wall, challenge, or clearance events.
• The first useful readiness signal: selector found, API response captured, or structured data extracted.

This makes failures diagnosable. A sudden 403 can be separated from a regional block, a missing Client Hint path, a lost session cookie, or a target-side change in the JavaScript API.

The practical rule

Use User-Agent as one field in a browser profile, not as a disguise. If the site depends on browser negotiation, fetch it with a real browser context, keep the session stable, and inspect the network behavior before deciding whether a lighter HTTP replay is safe.

That is slower than pasting a header string into curl. It is also the difference between a demo scraper and a production data pipeline that survives modern browser privacy changes and bot scoring.