Better Fetch

· Paul Crossland

Stale Service Workers Distort Fetch Evidence

Fresh Safari fixes show why crawler incidents need service-worker, partitioned-cookie, and storage cleanup evidence.

A browser-grade fetch can be wrong before your extractor sees a single selector. The URL is correct. The status code is 200. The browser finishes navigation. The rendered page even looks plausible. But the bytes and API calls were shaped by a service worker registration, a partitioned cookie that did not clear, or a browser storage state left behind by an earlier run.

That is a production data problem, not a browser trivia problem.

Two current WebKit releases make the issue concrete. On July 27, 2026, WebKit published WebKit Features for Safari 26.6. The release includes fixes for partitioned cookies that could not be deleted using WKHTTPCookieStore, service worker registrations with missing main scripts that were not automatically unregistered, service worker registrations with missing imported scripts that were not automatically unregistered, and extension service worker database files accumulating on launch. On July 22, WebKit published Safari Technology Preview 248 release notes, covering a broad set of Web API, CSS, JavaScript, rendering, accessibility, Web Inspector, and networking-adjacent changes.

Neither post says "change your crawler." The operational thesis is narrower and more useful: service-worker and storage cleanup state should be part of fetch evidence, because stale browser state can make a successful navigation represent the wrong application version, wrong cache path, or wrong session boundary.

Why this matters for production fetching

Many web-data systems treat a browser context as a convenience wrapper around HTTP. Launch browser, load page, wait for a condition, extract DOM, close browser. That model misses a major difference between browser-grade fetching and stateless HTTP: a browser has durable client-side machinery.

Service workers can intercept requests, populate caches, respond while offline, rewrite network timing, and keep serving an older asset graph after the origin has changed. Cookies can be partitioned by top-level site or embedded context. Storage can persist consent, feature flags, localization choices, authentication shells, A/B buckets, and prior navigation state. Extensions and automation harnesses can introduce their own background workers or databases.

For an ordinary user, much of that state is desirable. For a data pipeline, it is evidence that must be controlled and recorded. If the service worker is stale, the extractor may be sampling yesterday's application shell. If a cookie did not actually clear, a supposedly clean crawl may still be carrying a previous consent or region decision. If a broken registration is not unregistered, the page may fail to recover even after the site deploys a fixed script. If a browser update changes cleanup semantics, two workers running different browser versions can observe different web applications at the same URL.

The hard part is that these incidents rarely announce themselves as "service worker failure." They show up as ordinary symptoms:

  • fewer API calls discovered during rendering;
  • rendered HTML missing a section that is present in manual testing;
  • screenshots showing an old navigation layout;
  • inconsistent results between fresh containers and warm browser pools;
  • a spike in timeouts after a site deploy;
  • empty extraction records only in one browser family or OS version;
  • consent, language, or login state surviving a cleanup step that was assumed to be complete.

Retries do not fix that class of problem. They often make it worse by mixing more stateful attempts into the incident timeline.

The mechanism to watch

A service worker sits between the page and the network for its scope. Once registered, it can receive fetch events and decide whether to serve from cache, call the network, synthesize a response, or combine sources. That means the page you crawl is not only a function of the origin server. It is a function of the registration, script version, cache contents, update lifecycle, and browser implementation.

The Safari 26.6 fixes are important because they involve cleanup and recovery paths. A missing main script or imported script should not leave a page stuck behind a dead registration forever. A partitioned cookie deletion API should actually delete the intended state. An extension service worker database should not accumulate in a way that degrades performance across launches.

For fetch operators, the lesson is not that Safari is uniquely risky. The broader lesson is that browser state cleanup is a compatibility surface. Chrome, Safari, Firefox, WebKit-based embedded views, mobile browsers, and automation-pinned builds can differ in how quickly registrations update, how storage is partitioned, which deletion APIs are available, and what survives context reuse. If your pipeline only logs browser: safari or browser: chrome, you cannot replay the state boundary that shaped the result.

What to log on every browser-grade fetch

A practical logging contract does not need to dump private data or capture credentials. It should record the shape of the state that affected the fetch while preserving a safe boundary around sensitive values.

At minimum, log these fields for browser-backed jobs:

FieldWhy it matters
browser_name, browser_version, engine_version, os_versionCleanup behavior and storage partitioning can differ by runtime.
context_reuse_modeDistinguishes fresh profile, pooled profile, persistent profile, and replay profile.
top_level_site and target_originPartitioned cookies and storage depend on site relationships, not just URL.
service_worker_registrations_count before navigationShows whether the run started with client-side request interception state.
service_worker_scopes hashed or redactedIdentifies scope changes without logging sensitive paths verbatim.
service_worker_controller_present after navigationTells you whether the loaded page was controlled by a worker.
cache_storage_bucket_count and approximate byte sizeHelps separate fresh network rendering from cached application shells.
cookie_jar_partition_count and cookie names hashedConfirms whether cleanup changed the expected state surface.
storage_cleanup_resultRecords whether cookies, local storage, session storage, IndexedDB, Cache Storage, and service workers were cleared successfully.
network_response_source when availableSeparates browser cache, service worker, prefetch, and network responses.
first_party_vs_embedded_modeConsent, cookies, and service workers may differ for iframes and embedded flows.

Do not log cookie values, tokens, session secrets, or credential material. The goal is provenance, not exfiltration. Names can often be hashed with a stable salt per source so that operators can detect drift without exposing contents.

A cleanup test plan for crawler fleets

The safest time to learn that cleanup is broken is before an incident. Add a small canary that runs whenever browser versions, automation libraries, or profile-reuse settings change.

  1. Start with a fresh profile. Visit a controlled test page that registers a service worker, writes a partitioned cookie where supported, writes local storage, writes IndexedDB, and stores one Cache Storage entry.
  2. Record state evidence. Capture registration count, controller status, cache count, storage presence, and cookie-name hashes.
  3. Run the same cleanup path used in production. Do not use a special test-only deletion function unless production uses it too.
  4. Open a new page in the same reuse mode. Confirm that the service worker no longer controls the page, caches are gone, and cookies/storage expected to be cleared are absent.
  5. Repeat across browser families and channels. Include stable, pinned automation builds, and any mobile or embedded browser family that contributes production data.
  6. Test broken-registration recovery. Register a worker that imports a script, then make the imported script unavailable in the fixture and verify that your pipeline can classify the resulting state instead of treating it as a parser failure.
  7. Fail closed on ambiguous cleanup. If the harness cannot prove state was cleared, mark the fetch as state-contaminated and keep it out of high-confidence datasets.

This canary should produce a simple result: clean, not_clean, or unknown, with enough evidence to explain the classification. unknown is not a success state. It means the pipeline cannot prove which browser state shaped the fetch.

Incident diagnostics when data drifts

When extraction yield drops or rendered output changes, ask state questions before changing selectors or routing:

  • Did failures concentrate in warm browser pools rather than fresh profiles?
  • Did the affected runs have a service worker controller after navigation?
  • Did network logs show fewer origin requests but normal DOM completion?
  • Did screenshots resemble an older application shell?
  • Did the issue appear after a browser or WebView update rather than a site deploy?
  • Did cookie or storage cleanup report partial success?
  • Did the same URL behave differently in first-party and embedded contexts?
  • Did retrying in the same profile reproduce the failure, while a fresh profile did not?

Those questions turn a vague crawler incident into a testable state hypothesis. They also reduce unsafe pressure to "try another identity" or bypass a policy surface. If the page is different because state is stale, the responsible fix is to isolate, reset, or classify the run correctly. If the page is different because access is denied or authorization is required, the responsible fix is to stop and handle that boundary explicitly.

Trade-offs: fresh profiles versus pooled browsers

The simple answer is to use a fresh browser profile for every job. That is clean, expensive, and not always necessary. Persistent or pooled contexts can be reasonable when the state is intentional: authenticated internal crawls, consented first-party monitoring, repeat checks for a stable source, or high-volume rendering where startup cost dominates.

The mistake is using pooled state without evidence. A good policy is:

  • use fresh profiles for discovery, compliance-sensitive sources, and high-confidence extraction;
  • use pooled profiles only when the state contract is explicit;
  • attach state evidence to every extracted record batch;
  • quarantine results when cleanup is partial or unverified;
  • run browser-channel canaries before promoting a new runtime;
  • treat service-worker, cookie, and storage changes as incident timeline events.

That policy is durable because it does not depend on one Safari bug fix. It reflects how modern web applications work. The client is part of the application boundary.

The operator takeaway

Browser-grade fetching is valuable precisely because it observes the web as a browser does. That also means it inherits browser state, service workers, partitioned storage, cleanup semantics, and version-specific behavior. Fresh WebKit fixes are a timely reminder that successful navigation is not the same thing as clean evidence.

For production web-data systems, the next maturity step is to record and test the state boundary around every browser fetch. If the pipeline cannot say whether a service worker controlled the page, whether cleanup succeeded, and which storage partitions existed, it cannot confidently explain the data it produced.