The Sixty-Fifth Navigation Is Part of the Test

A browser worker can pass a cold-start smoke test and still be unfit for a production crawl. The first URL loads. The twentieth loads. Much later, a navigation waits forever without putting a request on the wire, page-side observers multiply until the renderer stops responding, or a close command is acknowledged while the target remains alive.

Those failures do not belong to the source website, and they are not described by average page latency. They depend on what the browser, automation client, page, and extensions have already done.

The practical thesis is: a reusable browser worker should be admitted and retired using cumulative-exposure evidence—navigation history, document churn, CPU seconds, request-start receipts, and teardown settlement—not one successful page or a fixed wall-clock age. A canary that launches a fresh browser for every URL removes the history most likely to break a pooled crawler.

This is reliability guidance for owned or properly authorized fetching. A login, challenge, denial, payment boundary, rate limit, or publisher restriction remains a policy outcome. Worker recycling must not become a way to change identity or repeatedly approach an access boundary.

Fresh evidence from long and stateful runs

Current reports expose four different cumulative failure paths:

Older Better Fetch posts already cover browser version pinning, release-channel canaries, cancellation, quiescence, and orphan cleanup. The repeated article would say to log the browser revision and restart leaked workers. The new information surplus is a cumulative-exposure contract: promotion tests and retirement decisions must preserve realistic operation order, then identify the first browser milestone that stopped making progress.

A browser worker has a history, not just a version

A worker's effective state includes more than cookies and storage. It also includes process generations, pages created and destroyed, pending protocol commands, service workers, frames, extension content scripts, observers, event listeners, memory allocators, caches, and work queues.

That history can fail in several shapes:

1. Admission failure. The automation call exists, but the browser never starts the network request. A navigation timeout cannot be blamed on the origin when an owned server saw no request.
2. Amplification failure. One document mutation schedules more internal work than the previous mutation. Individual actions remain valid while their marginal CPU cost grows until the event loop stops serving useful work.
3. Settlement failure. A protocol acknowledgement says a close request was accepted, but the target-destroyed event never arrives. The command response and the browser state contradict each other.
4. Revision cost drift. The same logical workload completes, yet consumes materially more CPU. Success-rate-only canaries pass while fleet capacity, queue delay, and timeout risk deteriorate.

A fresh context helps isolate cookies, local storage, permissions, and some page state. It does not necessarily create a fresh browser process, automation connection, extension realm, or resource baseline. Conversely, restarting after every page maximizes isolation but increases launch cost and can hide bugs that occur only in the reuse pattern production actually depends on.

The right unit is therefore a worker generation: one browser process and control connection from successful admission until verified shutdown or forced retirement. Context and page generations sit inside it.

Issue a cumulative worker receipt

Attach a compact receipt to each important operation and roll it up by worker generation:

• worker ID and generation, automation version, browser product and revision, operating system, image digest, protocol, and launch-argument digest;
• cumulative pages, contexts, navigations, redirects, document generations, frame attaches and detaches, crashes, downloads, and policy stops;
• operation ordinal within the worker, operation type, target generation, dispatch time, and deadline source;
• navigation milestones: command accepted, request event observed, owned-server receipt when available, response received, commit, readiness, and extraction accepted;
• teardown milestones: close command response, target-destroyed event, page-set delta, context close, browser disconnect, and process exit;
• CPU seconds for the browser process tree, CPU per accepted record, peak CPU, memory, file descriptors, child-process count, event-loop lag, and pending protocol operations;
• retry location: same page, new page, new context, or new worker, plus whether evidence proves the earlier attempt reached the source;
• final worker verdict: healthy, degrading, pre_request_wedged, target_unsettled, renderer_unresponsive, forced_retirement, or unknown.

Do not broadly log private URLs, cookies, credentials, or page content. Origin classes, normalized route shapes, counts, timings, bounded hashes, and restricted artifact references are usually enough.

Two derived views matter. Plot operation duration and CPU cost by operation ordinal, not only by URL template. Then build a survival view showing how many worker generations remain healthy after each cumulative-work bucket. A spike at a particular ordinal is a diagnostic lead, not a magic fleet-wide retirement number.

Build a soak canary that keeps the history

A useful promotion test should resemble the longest production worker lifetime, not the smallest reproducible page:

1. Serve owned fixtures for a fast document, redirects, client-side navigation, frame replacement, service-worker-controlled navigation, DOM churn, and a deliberately slow response.
2. Run the exact production sequence of page reuse, context reuse, tracing, extensions, request interception, extraction, and teardown. A bare page.goto() loop tests a different system.
3. Continue beyond the intended production retirement horizon. If workers normally handle a bounded number of jobs, the canary must cross that bound with margin rather than stop after one clean pass.
4. Record both automation request events and fixture-server receipts. This separates request_not_started from request_started_but_unobserved and from source latency.
5. Measure CPU seconds per accepted fixture record and event-loop lag by ordinal. Alert on a sustained slope or step change, not only container CPU saturation.
6. Exercise document-level mutation repeatedly with every production extension and injected script enabled. Compare against an extension-free lane so page work and instrumentation work can be attributed separately.
7. Race teardown against controlled navigation commits. Require command settlement, target disappearance, and bounded escalation to context or worker shutdown.
8. Run pinned and candidate browser stacks in parallel, with several complete worker generations. One long run can find sequence dependence; repeated generations show whether the failure position is stable or probabilistic.
9. Include policy fixtures for login, denial, challenge, and 429. Require a stop verdict and prove that retirement does not trigger identity, route, or session rotation.

Keep a short cold-start suite too. It catches admission and packaging failures quickly. The soak suite answers a different question: whether useful work remains bounded as history accumulates.

Retirement should respond to contradictions

A fixed maximum navigation count is a reasonable safety cap, but it should not be the only policy. Retire early when the worker violates a semantic invariant:

• a navigation exceeds its admission budget with no request evidence;
• CPU per accepted record or event-loop lag grows beyond the canary envelope;
• a close acknowledgement is not followed by target disappearance;
• page, frame, process, or pending-operation counts fail to return to their post-job baseline;
• a renderer becomes unresponsive or a browser process crashes;
• the system cannot determine whether an operation reached the source.

When no request was emitted and an owned receipt confirms that fact, retrying once in a new worker can diagnose a local wedge without adding duplicate origin traffic. When request emission is unknown, treat the attempt as possibly sent and apply the ordinary site-safe retry budget. Retrying goto repeatedly on the same wedged page is not recovery evidence.

Quarantine artifacts and dependent fields from a worker whose history cannot be reconstructed. A later successful retry can supply a new result; it cannot retroactively prove that earlier page generations were sound.

The operator decision rule

When long crawls slow down, hang, or lose yield, split failures by the last completed milestone and operation ordinal. No request start points to browser admission or control-plane state. Increasing CPU per record points to cumulative page, extension, or engine work. A successful close response without target destruction is a teardown contradiction. A clean new worker succeeding on the same owned fixture is evidence against the source, but it is not permission to increase retries against third-party sites.

The sixty-fifth navigation is not special outside one fresh reproduction. What matters is that it existed beyond the horizon of a short canary. Preserve realistic history, measure the marginal cost of each accepted record, and retire workers when their evidence stops converging. That turns cumulative browser failure from mysterious source flakiness into a bounded, testable property of the fetch fleet.