When Queue Time Becomes Document Damage

A PDF extraction request can return polished Markdown and still be the product of a timeout. The document may have spent most of its deadline waiting for a worker, reached expensive scanned pages with seconds left, fallen back to a lower-fidelity path, and then stored that degraded result under the source file's digest. Every retry can now “succeed” faster by adopting the same damaged artifact.

That failure is more dangerous than a clean timeout because it looks complete. Increasing the caller timeout or automatically resuming the request does not repair an output that crossed a quality boundary inside the document service.

The practical thesis is: large-document extraction should carry separate receipts for queue residence, processing budget, page coverage, quality fallback, cache admission, and client continuation; no successful response is publishable when a deadline-shaped artifact can be mistaken for a content-shaped one.

This is reliability guidance for public or properly authorized documents. Login, payment, publisher, encryption, rate-limit, and access-control boundaries remain stop or scheduling outcomes. Better timeout accounting must not become a reason to expand access or repeatedly submit costly work against source policy.

Fresh evidence across the document path

Current Firecrawl work exposes several boundaries that a single request duration hides:

Earlier Better Fetch posts already cover packaged-document provenance, download acceptance, clean-Markdown loss gates, and crawl materialization. The repeated article would say to hash the source file, record the converter version, or reject partial output. The new information surplus is the budget-to-quality chain: waiting, compute isolation, fallback, cache identity, and client continuation can convert one timing decision into a durable data-quality defect.

One timeout contains several clocks

An end-to-end request deadline is useful for protecting callers, but it is the wrong clock for every internal decision. At least four budgets matter:

1. Caller budget: how long the client will keep the request open.
2. Queue budget: how long admitted work may wait before capacity or priority policy must reconsider it.
3. Processing budget: how much claimed worker time the document class may consume.
4. Quality budget: which fallback, OCR, layout, or truncation changes are permitted before the representation becomes ineligible.

Starting the processing clock at submission lets queue congestion spend extraction quality. A worker can receive a nearly expired job and behave correctly according to its deadline while producing a fleet-wide quality incident. Conversely, giving every large document a longer global deadline can let stuck work occupy scarce workers and increase backlog age.

Keep an end-to-end service objective, but start a distinct processing lease when a worker claims the job. Estimate that lease from evidence such as page count, byte size, born-digital versus scanned classification, requested layout fidelity, OCR requirement, and historical cost by converter version. If the estimate does not fit available capacity, reject, schedule, or require an explicit bounded page range. Do not quietly purchase time by changing the output class.

Make fallback change artifact identity

A raw document digest identifies input bytes. It does not identify the derived representation. The same bytes can produce different text under different page ranges, OCR models, layout modes, deadlines, parser builds, or fallback decisions.

Use a derived-artifact key that includes at least:

• source digest and media type;
• converter, OCR, and layout-engine versions;
• requested and processed page range;
• document classification and quality profile;
• fallback rules and every fallback actually used;
• processing-budget policy version;
• output schema or Markdown-normalization version.

More importantly, gate cache admission by quality verdict. complete_declared_quality may enter a reusable result cache. complete_with_declared_fallback may enter a separate, explicitly acceptable class if the data product permits it. deadline_degraded, page_range_unknown, partial, and quality_unknown must not become canonical results merely because they contain text.

Content adoption needs the same rule. Reattaching a retry to existing work prevents duplicate compute, but identity alone is insufficient. The adopting caller must receive the existing job's quality profile, page coverage, deadline state, and final verdict. A new request cannot promote an old degraded artifact by waiting longer.

Resume work without replaying ambiguity

The fresh SDK change demonstrates a sound distinction: resume only after an explicit processing_continues signal, use the server's bounded delay, cap attempts and total waiting, and attach to the known in-flight computation. A plain timeout does not prove that work continues, that no result exists, or that submitting again is safe.

Issue a continuation receipt containing:

• client operation, server job, source-content, and derived-artifact IDs;
• original submission, queue-entry, claim, processing-start, and terminal timestamps;
• timeout source and the stage active when it fired;
• server state, remaining estimate, estimate version, and retry delay;
• resume ordinal, cumulative wait, adoption result, and whether new compute was created;
• page total, pages attempted, pages accepted, page ranges missing, and duplicate-page count;
• fallback counts by reason and affected page range;
• cache lookup, candidate verdict, admission decision, and cache key version;
• final verdict: complete_declared_quality, complete_with_declared_fallback, partial_resumable, deadline_degraded, rejected_limit, failed, or unknown.

Do not expose document text, signed URLs, credentials, or private storage paths in broad telemetry. Counts, timing buckets, policy versions, hashes, and restricted artifact references are usually enough.

Protect unrelated fetches from conversion work

Document conversion is often CPU and memory intensive. The open native-binding fix shows a separate failure mode: syntax that looks asynchronous does not guarantee execution leaves the event loop. One large office document can delay status endpoints, scrape responses, cancellation handling, and health checks for unrelated jobs.

Measure event-loop lag, worker CPU, memory, queue age, and accepted pages by document class. Isolate blocking conversion in a worker pool or process with bounded concurrency and memory. Backpressure admission before workers saturate. A longer extraction deadline is harmful if the control plane becomes too delayed to report, cancel, or reject the work accurately.

This isolation also improves diagnosis. If queue time rises while processing cost stays stable, investigate capacity and admission. If claimed processing time rises only for scanned documents, adjust the class model or OCR path. If unrelated API latency rises with conversion CPU, repair execution isolation before tuning source timeouts.

A large-document degradation test

Run this matrix on owned or redistribution-safe fixtures:

1. Queue a born-digital and a scanned document behind controlled backlog. Prove queue residence cannot consume the worker's quality budget.
2. Give both documents equal page counts. Require estimates and outcomes to reflect their different OCR costs rather than one fleet-wide milliseconds-per-page constant.
3. Force fallback after a known page. Require affected page ranges and quality changes to appear in the receipt, then prohibit canonical cache admission.
4. Retry the same source digest with a larger budget. Confirm it does not adopt a degraded artifact as complete and does not start duplicate compute when eligible work is still running.
5. Trigger the explicit processing-continues response. Verify bounded delay, attempt count, cumulative wait, and attachment to the same server job. A plain timeout must not enter that path.
6. Request an explicit page subset. Require the subset in artifact identity and label the result as a declared range, never a complete document.
7. Run a large native office conversion beside fast status and cancellation requests. Assert event-loop lag and control-plane latency stay inside their own budgets.
8. Return encrypted, denied, payment-bound, rate-limited, malformed, and unknown-page-count fixtures. Preserve each outcome instead of routing around it or publishing plausible partial text.

The operator decision rule

When a large document times out or returns suspiciously smooth but incomplete text, find the first clock or verdict that changed the representation. If queue wait consumed processing time, separate the budgets before increasing the cap. If fallback touched pages, quarantine the artifact and its cache entry. If a continuation signal proves work remains active, resume by job and content identity within a bounded client budget. If page coverage or quality lineage is unknown, do not publish. If conversion blocks unrelated control traffic, isolate compute rather than granting more wall time.

A long-running document request is not one slow fetch. It is admission, waiting, conversion, page-level quality, caching, and delivery joined across several systems. Make those stages reconcile, and a timeout remains an operational event instead of becoming a polished, reusable source of damaged data.