Three Ledgers Behind One Crawl Status

A distributed crawl can report running while a status poll downloads thousands of completed documents. It can report completed while one failed result page silently removes records. It can respect a configured URL limit in every worker and still exceed that limit when workers race.

Those outcomes look unrelated because they appear in different components. In practice they expose one design error: treating a crawl as a single status object when it is really three changing ledgers.

The practical thesis is: a crawl is publishable only when frontier admission stayed inside an atomic budget, control-plane observation remained bounded independently of result traversal, and terminal result materialization accounted for every page, cursor, failure, and truncation. A green job state proves none of those ledgers by itself.

This is reliability guidance for public or properly authorized collection. Robots rules, rate limits, login or payment boundaries, challenges, denials, and publisher restrictions remain stop or scheduling outcomes. Stronger crawl accounting must reduce accidental load and incomplete data, not justify broader access.

Fresh evidence at three boundaries

Current Firecrawl work makes the boundaries unusually visible:

Recent Better Fetch posts already cover clicking pagination controls, rate-aware scheduling, and browser-worker lifetime. The repeated angle would say to verify that Next changes the collection or to cap retries. The new information surplus here is the distributed job contract after discovery: admitted work, observed progress, and materialized output advance on different clocks and must reconcile before publication.

Ledger one records admitted work

A crawl limit is an admission invariant, not a dashboard preference. Suppose 20 workers each read visited_count = 99 under a limit of 100. If counting and claiming are separate operations, all 20 can decide capacity remains and admit different URLs. Every local decision is consistent; the global result is not.

The commit point must combine deduplication, budget comparison, and claim creation atomically for the chosen frontier identity. Record at least:

• crawl and frontier-generation IDs;
• discovered URL identity and normalized deduplication identity;
• parent URL or cursor and discovery rule;
• budget limit and count observed at the atomic decision;
• admitted, duplicate, limit_reached, or policy_stop;
• claim ID, decision timestamp, and store or script version.

Do not infer admitted work from queued messages alone. A producer may publish twice, a consumer may lease and return work, or a queue may retain canceled entries. The admission ledger answers how many distinct units were authorized to enter the crawl. Execution and completion counters answer different questions.

Ledger two observes the job without consuming it

A health probe should have bounded cost even while the job grows. Following every result cursor during each poll couples observability cost to crawl output and creates a moving target: while the client fetches page two, workers can append page three.

Separate two operations:

1. get_job_state returns one bounded control record: state, counters, heartbeat, budget use, failure summary, and a result-generation reference.
2. list_results traverses a declared result generation with explicit page, byte, record, and time budgets.

A convenience SDK may expose both through one method, but its defaults must preserve the distinction. While a job is non-terminal, polling should not drain result pages. At terminal state, automatic materialization is safe only if the result generation is frozen or the API supplies equivalent snapshot semantics.

Log poll latency, response bytes, HTTP-call count, cursor-follow count, state before and after the poll, and whether result traversal was requested explicitly. Alert when a status poll makes more than one result request, exceeds its control-plane byte budget, or returns records without naming their generation.

Ledger three proves what was materialized

completed means producers stopped according to the job state machine. It does not prove that the client received every result.

A materialization receipt should include:

• terminal job state and terminal timestamp;
• frozen result generation or snapshot token, when available;
• expected, succeeded, failed, skipped, and policy-stopped counts;
• page ordinal, incoming cursor, outgoing cursor, and cursor-consumed state;
• stable record identity and per-page identity digest;
• page request outcome, retry count, and error class;
• truncation cause and the exact unconsumed continuation;
• duplicate and missing-identity counts;
• final verdict: complete, complete_with_declared_failures, partial_resumable, partial_unstable, failed, or unknown.

Cursor semantics must be unambiguous. A returned cursor can mean “next page remains,” “this page was already consumed,” or “resume after a bounded output.” Reusing a consumed cursor duplicates data; clearing an unconsumed cursor turns truncation into false completion.

Offsets add another hazard. If rows are ordered only by timestamps and ties have no unique secondary key, inserts or equal timestamps can shift records across page boundaries. Use a server-issued snapshot cursor or a total order such as (finished_at, record_id). Without one, label resumed output partial_unstable and reconcile stable record IDs rather than claiming an exact continuation.

A reconciliation test for crawl platforms

Run this matrix on owned fixtures before changing an SDK, queue, cursor format, or worker count:

1. Race more distinct URL claims than the remaining budget. Assert the admission ledger never crosses the limit.
2. Poll a running crawl while it continuously emits results. Require one bounded control request per poll and no implicit result walk.
3. Freeze the job, then materialize multiple pages. Assert every cursor advances once and the final cursor is empty.
4. Stop at page or record limits. Require the first unconsumed cursor to survive with partial_resumable.
5. Make an intermediate page return an error or success: false. Prohibit a shorter success-shaped dataset.
6. Return a cursor that points to itself. Stop with a loop verdict rather than waiting forever.
7. Give several records equal timestamps and resume by offset while another record appears. Confirm the system detects duplicates or gaps unless a snapshot and unique tie-breaker exist.
8. Finish as failed or cancelled after producing some records. Preserve both the terminal failure and the bounded partial output; never convert either into empty success.
9. Return 429, login, challenge, denial, and robots-policy fixtures. Require scheduling or policy verdicts without identity or route rotation.

The operator decision rule

When crawl totals disagree, reconcile in order. First compare atomic admissions with the configured budget. Then prove that status polling did not traverse or mutate the result view. Finally walk one frozen result generation and account for every cursor and terminal failure.

If admission exceeded budget, stop expanding the frontier. If polls chased a moving cursor, repair the control API before tuning timeouts. If a page failed or continuation stability is unknown, quarantine the dataset as partial. A later successful rerun can produce a new result generation; it cannot make the earlier generation complete.

One crawl ID can name all three ledgers, but it cannot collapse them. Keep admission, observation, and materialization separate, then require them to reconcile. That is how a production crawler proves not merely that a job ended, but that it stayed bounded and delivered the dataset it claims.