Fetch Correctness Is Runtime Specific

A production fetch pipeline rarely has one HTTP client. It has a server-side client for cheap discovery, a browser for rendered pages, maybe a proxy tunnel for regional access, and a retry path that behaves slightly differently from the first attempt. Operators often describe those pieces as if they are interchangeable: fetch the URL, follow redirects, decode the body, cache what is cacheable, and extract the result.

This week is a useful reminder that they are not interchangeable. On July 2, the Undici project released Undici v8.6.0, with changes that touch real fetch behavior: updating the Accept-Encoding header in fetch, dropping response chunks after a destroyed response stream, keeping retry flow-control wired to the active connection across resumes, failing a request when a proxy CONNECT tunnel drops instead of looping, delivering early final responses for Expect: 100-continue over HTTP/2, and adding support for the HTTP QUERY method from RFC 10008. One day earlier, WebKit published Safari Technology Preview 247, including networking fixes for valid xn-- ASCII domains, using arbitrary Content-* headers from 304 responses to update cached entries, and honoring Cache-Control request directives including max-age, min-fresh, and no-store.

Those are implementation details, but implementation details are where many crawler incidents live. A browser and a Node runtime can disagree on cache revalidation, domain handling, compression negotiation, proxy tunnel failure, retry semantics, or stream teardown. If the pipeline treats those differences as noise, it will misclassify production failures as target-site changes, proxy problems, or extractor bugs.

For Better Fetch-style infrastructure, the lesson is simple: fetch correctness is runtime specific. You need to know which client produced each artifact, and you need canaries that prove your browser and non-browser fetch paths still agree on the behaviors your data quality depends on.

Why this matters beyond release notes

Most crawler reliability programs focus on high-level outcomes: status code, final URL, rendered DOM, screenshot, extracted fields, and maybe a network waterfall. Those are necessary, but they can hide the cause of drift. A failed extraction might start much earlier than the parser:

• The server-side client sent a different Accept-Encoding set than the browser path.
• A dropped proxy tunnel was retried in a loop instead of failing cleanly.
• A destroyed response stream still delivered chunks to downstream code.
• An HTTP/2 Expect: 100-continue exchange completed differently than a fixture expected.
• A cache revalidation response updated metadata in one runtime but not another.
• A punycode-looking hostname was accepted by one URL stack and rejected by another.
• A request marked Cache-Control: no-store was still served from an unexpected local or intermediary cache.

None of those are exotic scraping tricks. They are normal web platform and HTTP-client edge cases. They matter because production crawling operates at volume, across regions, through intermediaries, and against pages that vary by state. Rare runtime differences become common once they are multiplied across millions of requests.

The risk is especially high when a pipeline uses a cheap HTTP fetch to make decisions before launching a browser. For example, a discovery stage may decide whether a page is HTML, JSON, blocked, redirected, unchanged, or worth rendering. If that stage has subtly different cache, compression, redirect, or connection behavior from the browser stage, the browser never gets a chance to prove the decision wrong.

Separate transport failures from target failures

Undici v8.6.0 includes a proxy change that is easy to overlook: fail the request when the CONNECT tunnel drops instead of looping. That is not just an internal bug fix. It is an operational classification issue.

A proxy tunnel drop should be recorded as a transport failure. It should not become an infinite retry, a generic timeout, a target 5xx, or an extractor null. If a crawler collapses those categories, the remediation path becomes noisy. Operators may rotate proxies when the issue is a client retry bug, increase target backoff when the target was never reached, or blame selectors when no valid response body existed.

For every fetch attempt, log enough to classify the failure layer: DNS and TCP/TLS phase reached, HTTP version, proxy pool and tunnel result, whether headers arrived, whether the body stream completed or was canceled, retry attempt number, retry reason, and runtime version. Do not log secrets, full proxy credentials, or sensitive headers. Redacted metadata is enough to prevent a transport incident from masquerading as a site change.

Treat cache behavior as a data-quality input

The WebKit Technology Preview networking fixes are a good prompt to revisit cache assumptions. In a browser-grade fetch, cache behavior is not merely a performance optimization. It can change the content that extraction sees.

Consider a product page whose HTML is revalidated with 304 Not Modified while headers affecting rendering, content negotiation, or metadata change. If one runtime updates cached entries from Content-* headers in a 304 response and another does not, the final artifact can diverge without any obvious status-code difference. Likewise, if request directives such as max-age, min-fresh, and no-store are not honored consistently, a test that is supposed to prove freshness may instead prove only that one client used stale local state.

Production fetch records should include cache evidence, not just the final body: request cache mode, response validators, Age, ETag, Last-Modified, Vary, relevant Content-* headers, whether the response came from browser cache, service worker, intermediary cache, or network, and the runtime path that observed it. A mismatch between direct HTTP and rendered browser output may be a real page difference, but it may also be a cache-state difference.

Compression and decoding need canaries

The Undici release note about updating the Accept-Encoding header in fetch is another reminder that request headers are part of the client contract. Compression affects intermediary behavior, content negotiation, decompression errors, body-size accounting, and sometimes bot-management heuristics.

Canaries should cover plain text, gzip, Brotli, mismatched Content-Encoding, large streaming responses that are canceled, multi-byte UTF-8 split across chunks, and responses where the consumer destroys the stream early. Assert more than success: decoded text hash, raw byte count when available, reported content length, stream completion state, and error classification. If the crawler stores partial data intentionally, mark it as partial. If it discards partial data, prove that no downstream extractor sees it as complete.

Domain handling is not boring

Safari Technology Preview 247 also fixed a case where WebKit refused to load valid ASCII domains starting with xn-- that did not pass strict IDNA 2008 validation, aligning behavior with the WHATWG URL Standard. That sounds narrow until you operate a crawler that sees internationalized domains, merchant-created hostnames, affiliate redirects, CDN aliases, and malformed-but-live URLs at scale.

URL acceptance and normalization should be tested in the same way as redirect handling. For controlled fixtures and sampled production URLs, record:

• Original URL string exactly as discovered.
• Parsed URL components from each runtime.
• Normalized URL used for deduplication.
• Navigation or request result by runtime.
• Final URL after redirects.
• Rejection reason when a runtime refuses to parse or load the URL.

Avoid making the direct HTTP parser the only source of truth. If the browser is the environment that users experience, browser URL behavior matters. If the server-side discovery client is the gatekeeper that decides what gets rendered, its parser matters too. Disagreements should be visible and reviewed, not silently normalized away.

Build a runtime conformance matrix

A useful fetch conformance matrix does not need to be large. Start with the behaviors that affect data quality most often:

Run the matrix whenever you upgrade Node, Undici, Puppeteer, Playwright, Chrome, Safari Technology Preview, proxy software, container images, or retry code. Store the runtime versions beside the result. The point is not to freeze everything forever; it is to know which upgrade changed which observable behavior.

Operational checklist

Before the next runtime upgrade, make sure your fetch system can answer these questions:

• Which runtime produced this response, screenshot, DOM, or extracted field?
• Was the response fresh, revalidated, cached, or served through a service worker?
• Did the request use a proxy tunnel, and did that tunnel fail separately from the target request?
• Were compression and decoding outcomes logged separately from extraction outcomes?
• Can a failed URL be replayed through both the direct HTTP and browser paths with the same region, headers, cookies, and cache state?
• Do dashboards distinguish transport errors, runtime errors, target HTTP errors, browser render errors, and parser errors?
• Are runtime conformance fixtures part of the release process for browser and HTTP-client upgrades?

The web platform keeps moving, and HTTP clients keep fixing edge cases. That is good. It also means a fetch pipeline that was correct last week may become observably different this week. Treat runtime behavior as part of the data contract, validate it with fixtures, and keep direct HTTP and browser-grade fetching honest against each other.