One Green exec() Can Hide a Broken Crawl

A crawler marks a URL visited, adds it to a qualified set, gives both keys a TTL, and awaits one Redis pipeline. The promise resolves, so the worker reports success. Hours later, one URL was never queued, a completion counter is short, or a negative cache marker still says that fresh data does not exist.

The transport did not necessarily fail. One command inside the resolved batch did.

The practical thesis is: a crawl datastore client must interpret every batched command result and recover according to the state that command owns; transport-level success is neither batch success nor permission to replay the whole operation. The hard part is not detecting an error tuple. It is deciding whether to reject, retry, compensate, repair asynchronously, or safely treat the failure as a cache miss.

This is reliability guidance for public or properly authorized collection. Better queue and deduplication accounting must reinforce robots rules, rate limits, consent, denials, and access boundaries, not work around them.

Fresh evidence from crawler control paths

Two merged Firecrawl changes on September 6 expose opposite but compatible failure policies:

The ioredis pipelining documentation confirms that results are ordered by command and each entry has the form [err, result]. Redis's pipelining documentation explains the latency benefit, but fewer round trips do not create one application-level commit.

A recent Better Fetch article separated frontier admission, progress observation, and result materialization. Repeating that angle would merely ask operators to keep better ledgers. The new information surplus is one layer lower: a single datastore call can partially advance several ledgers, so recovery must be selected command by command from authority, idempotency, and repairability.

A resolved batch has two outcome planes

Treat pipeline execution as producing at least two independent outcomes:

1. Transport outcome: did the client send the batch and receive a reply sequence?
2. Command outcome: which commands succeeded, returned errors, or have an ambiguous result?

Partial progress matters. Suppose SADD visited URL succeeds and EXPIRE visited fails. Blindly retrying the batch is mechanically safe for SADD, but its second result will now be zero. If the caller interprets zero as “another worker already claimed this URL,” it can skip work that this same attempt claimed. The retry changed the evidence, not necessarily the business outcome.

The same pattern can reverse a completion result. A scrape may have succeeded and its webhook may already have been sent, while writing the completion marker fails. Routing that bookkeeping exception through the scrape failure handler can relabel completed work as failed. Recovery must preserve the prior fact and repair the missing marker separately.

Classify state before choosing recovery

Use a recovery matrix rather than one global “Redis is optional” or “retry three times” rule.

Three questions drive the choice:

• Authority: Is this store the source of truth, a claim, a cache, or a projection?
• Replay: Are all commands idempotent, and does the caller preserve evidence from earlier attempts?
• Repair: Is there a durable, bounded, monitored path that reconstructs missing state from an authoritative record?

A time-to-live is cleanup, not reconciliation. “The stale entry expires tomorrow” can be acceptable for a derived index after cancellation; it is not a recovery plan for a missing completion marker that prevents a crawl from terminating.

Batch size is part of correctness

Bulk command failures are often treated as capacity tuning. They can be correctness failures when collection size controls argument count.

Pull request 4557 traces one failure class to a Dragonfly command-argument limit and changes variadic set additions to fixed-size chunks. Bound both commands per pipeline and arguments per command, then record configured and observed maxima. One sitemap, feed, or highly connected page can create the pathological batch.

Chunking introduces another partial-progress boundary. If chunks one and two land and chunk three fails, a replay sees different insertion counts. Therefore counts from the final clean attempt cannot prove how many claims the operation acquired across all attempts. Maintain stable member identities and reconcile membership, not just aggregate return values.

Build a pipeline receipt

For each batched control-path write, retain a structured receipt with:

• operation, crawl, job, and attempt IDs;
• datastore role: authority, claim, cache, or derived_index;
• client and backend versions;
• command count, chunk count, and redacted command classes;
• transport outcome and latency;
• every failed command index, error class, and backend code;
• known successful command indexes and any ambiguous range;
• idempotency classification and whether an earlier attempt acquired a claim;
• chosen action: reject, retry, compensate, repair, rebuild, or cache_miss;
• repair-item identity, backlog age, and reconciliation result;
• final semantic verdict such as admitted, already_claimed, completed_repair_pending, or unknown.

Never log raw credentials, authorization values, or cache keys that embed them. Log a nonreversible operation identifier or an approved tenant identifier instead. The logging change in pull request 4554 is a reminder that better failure visibility can create a second incident if diagnostic context leaks secrets.

Alert on semantic damage, not merely Redis error rate. Useful signals include completion-repair backlog age, claimed-but-not-queued URLs, positive entries coexisting with negative markers, derived-index drift, command-error rate by pipeline, and batches approaching argument bounds.

Test the unhappy result array

A production test plan should inject outcomes at command granularity:

1. Resolve the pipeline promise with an error in the middle tuple. Assert that the operation does not report success.
2. Let a claim command land, fail its TTL or companion write, then retry. Assert that the URL is queued once rather than skipped as a duplicate.
3. Complete a scrape, fail the completion marker, and verify that the scrape remains successful while one idempotent repair item is created.
4. Fail the repair store and reconciler repeatedly. Require a bounded backlog alert and an unknown or repair-pending terminal state, not silent completion.
5. Land positive cache data while negative-marker deletion fails. Require an independent safe invalidation retry.
6. Make an evictable cache unavailable with each allowed error class, then inject an unexpected programming error. Only the former may become cache misses.
7. Exceed each bulk-command threshold on owned fixtures. Verify deterministic chunking and reconcile identities after a late-chunk failure.
8. Simulate a lost response after commands may have landed. Ensure the retry policy does not depend on assuming that no reply means no mutation.

End each test with a business invariant: every admitted URL is queued or explicitly released; every finished job is represented or durably pending repair; no cache marker contradicts authoritative data; and no policy or billing operation is silently replayed.

The operator decision rule

When a batched datastore call fails, do not ask only whether Redis is “critical.” Identify what each command could already have changed. Reject authoritative uncertainty, retry only idempotent work with prior-attempt evidence, compensate claims that would suppress future work, rebuild projections from durable state, and fail open only for explicitly disposable cache reads.

A green exec() means the reply envelope arrived. A healthy crawl requires a stronger statement: every command outcome was inspected, every partial mutation was classified, and the recovery path preserved the facts the crawler had already established.