Better Fetch

· Paul Crossland

The Browser Download Commit Point

Three fresh automation bugs show why download completion is a correlated commit protocol, not a file-exists check or callback.

A browser clicks Export, a file appears in the download directory, and a completion callback fires. That sounds like success. In production automation, all three observations can still describe the wrong outcome.

The file may be a growing partial. The callback may represent an interrupted transfer. A second watcher may emit the same completion again. A fast download may finish before the waiter establishes its baseline and then time out as if nothing happened. Worse, an old file with the expected name can survive a failed replacement and be accepted as the new export.

The practical thesis is: browser download completion is a correlated commit protocol; accept a download only when one operation reaches terminal success exactly once and its intended artifact passes identity and integrity checks.

Fresh evidence from three failure modes

These current changes expose different parts of the same boundary:

Primary sourceDateWhat it contributes
Browser Use issue 55152026-08-22Reports a filesystem poller treating Chromium .crdownload files larger than four bytes as complete, while a separate CDP progress path can dispatch a second completion.
Browser Use pull request 55192026-08-22Proposes filtering temporary suffixes, claiming a download GUID before callbacks, retaining bounded terminal tombstones, and testing duplicate, canceled, local, and remote paths. It remains open at publication time.
AviUtl Package Manager pull request 24472026-08-22Shows an Electron done handler that did not distinguish completed, cancelled, and interrupted; a failed same-name replacement could leave an older valid archive to be installed. The pull request is open.
Flowproof pull request 4922026-08-18Fixes the opposite race: a fast successful download could land before wait_for_download took its snapshot, become part of the baseline, and never be observed as new. It merged on August 21.

Older browser protocol and automation APIs provide the background distinction between download-start, progress, and terminal events. The new information surplus is operational: independent event, filesystem, and waiter races all point to one design requirement—a download must carry identity through a state machine and cross an explicit acceptance boundary.

This avoids two repeated angles from earlier Better Fetch posts. It is not about converting a correctly retrieved PDF or package, and it is not a general browser-lifecycle teardown checklist. The focus here is the commit point between an authorized browser action and downstream ingestion.

Why every single-signal detector fails

A production download crosses several planes:

  1. Intent: a particular action requested an export.
  2. Browser transfer: the browser assigned an identifier and reports progress.
  3. Filesystem publication: temporary bytes become a final pathname, if the browser is local.
  4. Artifact acceptance: the pipeline verifies that the object is the expected, usable result.
  5. Downstream publication: extraction or storage receives one accepted artifact.

A callback alone is ambiguous unless its terminal state is checked. A pathname alone is ambiguous because temporary and stale files exist. A size threshold only proves that some bytes were written. Repeated size stability reduces the chance of reading a growing file, but it still cannot prove that the transfer completed or that the file belongs to this action.

The races also run in both directions. Start watching too late and a fast file disappears into the baseline. Run multiple watchers without shared ownership and both can publish. Mark an identifier handled after invoking callbacks and re-entrant or concurrent code can win the gap. Reuse a non-empty directory and a failed replacement can make yesterday's valid file look like today's success.

Define an explicit download state machine

Use states with different meanings rather than one done boolean:

requested -> started -> receiving -> browser_completed -> artifact_accepted
                              |              |
                              +-> canceled   +-> artifact_rejected
                              +-> interrupted
                              +-> timed_out

Only artifact_accepted may wake extraction or publish a success record. canceled, interrupted, and timed_out are terminal failures even when a plausible pathname exists. A completed browser transfer can still be rejected because its digest, media type, container structure, size policy, or parser smoke test fails.

Give each request a download_operation_id before triggering the click. Correlate it with the browser's download GUID, source URL, suggested filename, target page, and action span. If the browser library exposes terminal states, preserve the original state instead of collapsing it into success or exception.

For exactly-once notification, claim the operation before calling consumers. Keep a short-lived terminal record so duplicate terminal events become no-ops. Bound that record by both time and count, and clear it when the browser session ends. This is not business-level exactly-once delivery across every service; it is a local invariant that prevents two detection paths from publishing one browser operation twice.

Make filesystem fallback narrow and subordinate

Protocol events should be authoritative when they are available and reliable. A filesystem watcher can be a useful fallback, but it should not scan a shared downloads folder and select any recent-looking file.

Use a fresh per-job directory or snapshot it before the triggering action. Exclude known temporary forms such as .crdownload, .part, and .tmp, while recognizing that suffix filtering is only a guard, not proof of completion. Match the strongest available identity: browser GUID mapping, expected final path, suggested filename, creation interval, and source action.

Never accept an existing same-name file merely because it parses or has a valid checksum for some version. Remove or quarantine pre-existing candidates before the run, write accepted artifacts to immutable names, and record a content digest. When the browser is remote, do not pretend a server-side path is locally readable; obtain the artifact through the remote service's supported transfer mechanism and preserve that handoff as another state transition.

Put artifact acceptance after transport completion

The acceptance policy should match the data product. Useful checks include:

  • browser terminal state is exactly completed;
  • final artifact is associated with the expected operation and authorized source;
  • file exists in the isolated run directory and is not a temporary candidate;
  • observed size is compatible with protocol totals when totals are available;
  • bytes match an allowed media type by content, not only extension;
  • digest is computed after the final handle is opened;
  • ZIP, PDF, CSV, JSON, or other format-specific smoke checks pass;
  • an expected schema, header set, page count, or minimum record count passes;
  • the artifact is published downstream through an idempotency key derived from operation identity and digest.

Do not turn a parser failure into an automatic re-click loop without limits. A repeated export can create multiple server-side jobs, duplicate billing, or inconsistent snapshots. Retry according to the site's permitted workflow, with a bounded attempt number and a new operation identifier.

Log enough to reconstruct the commit

A useful download record separates observation from verdict:

Field groupSuggested fields
Intentjob_id, download_operation_id, action_span_id, target page URL, action timestamp
Browserlibrary and browser versions, session ID, download GUID, source URL, suggested filename
Progressstate, received bytes, total bytes, first-progress time, terminal time, terminal reason
Filesystemisolated directory ID, first-seen path, final path, temporary suffix seen, rename time
Acceptancecontent length, detected media type, digest, validator name and version, validation result
Deduplicationclaim winner, duplicate event count, terminal-record age, downstream idempotency key
Outcomeaccepted, canceled, interrupted, timed_out, rejected, or unknown

Alert on contradictions, not just timeouts: terminal completed with no artifact, an artifact before started, received bytes above a known total, multiple final paths for one GUID, one path claimed by multiple operations, or a downstream publish before acceptance.

A regression matrix for download workers

Test the detector against controlled fixtures before trusting real exports:

  1. A slow download exposes a growing temporary file; assert no success occurs before the browser's completed state and final artifact checks.
  2. The filesystem path becomes final before the protocol completion; assert one notification, not two.
  3. Protocol completion arrives twice; assert the second event is recorded as a duplicate and causes no downstream side effect.
  4. A tiny download finishes immediately after the click; assert a watcher established before the action still claims it.
  5. A transfer is canceled and another is interrupted; assert both fail even if bytes or an old same-name final file exist.
  6. Two downloads use the same suggested filename; assert operation identity and immutable storage names keep them separate.
  7. Two downloads finish concurrently in one session; assert neither fallback path claims the other's file.
  8. A completed artifact has the wrong media type, truncated container, or invalid schema; assert transport success but artifact rejection.
  9. A remote browser reports a server-side path; assert local consumers do not open it and the supported artifact-transfer step is required.
  10. Restart the worker with terminal records present; assert retry and deduplication behavior is explicit rather than accidental.

The operational decision rule is simple: no browser event, filesystem observation, or parser result can independently commit a download. Require correlated terminal success, exclusive claim, and artifact acceptance before publishing data.

Use this pattern only for downloads your automation is authorized to request. Completion handling is reliability engineering, not a way to bypass authentication, rate limits, access controls, or a site's terms. A trustworthy fetch system should make both permission and artifact provenance visible at the same boundary where it declares success.