Before the First URL, Compile the Crawl Boundary

A crawl request says to keep one section and exclude everything else. The API accepts the patterns, the job starts, and workers discover far more pages than intended. Nothing crashed: the client and API accepted regex syntax that the worker's engine could not compile, then the worker silently discarded the failed rule.

Fixing that by compiling every pattern during request validation creates a second risk. Short expressions can expand into large automata, and hundreds of synchronous compilations can stall the API before the crawl is even admitted.

The practical thesis is: path filters are executable crawl-boundary programs. A crawler should bound and compile them under the exact worker semantics before enqueueing work, identify the accepted policy as an artifact, and attach that identity to every frontier decision. A valid JSON request or successful client-side regex test proves neither scope nor safe compilation cost.

This is defensive reliability guidance for public or properly authorized collection. Include and exclude rules do not grant access. Robots policy, rate limits, login or payment boundaries, challenges, denials, and publisher restrictions remain separate stop or scheduling outcomes; filter failures must narrow or reject work, never trigger wider discovery.

Fresh evidence from one filter boundary

Two Firecrawl changes on September 4 expose both sides of the admission problem:

The timing figures and limits in pull request 4533 are project-specific measurements, not universal regex benchmarks. They still demonstrate a durable mechanism documented by the Rust regex crate guidance for untrusted patterns: linear-time matching does not make compilation free, and compiled size can be much larger than source length.

Recent Better Fetch posts already cover atomic frontier budgets, request-graph policy, verified pagination transitions, and policy-aware crawling. Repeating “fail closed” or “log crawl scope” would add little. The new information surplus is the filter compilation receipt: source text, regex dialect, match target, resource limits, compiled policy, and worker execution are separate facts. A boundary is trustworthy only when they reconcile before the first discovered URL enters the frontier.

Accepted syntax is not executed syntax

Path filters often cross several runtimes. A JavaScript SDK may perform schema checks, an API may parse JSON with another library, and a native worker may compile the expression with Rust regex, RE2, PCRE, or a browser engine. The same string does not imply the same language. Look-around, backreferences, Unicode classes, flags, anchors, and escaping vary.

Silently ignoring a failed filter is especially dangerous because failure changes scope without producing a fetch error. Dropping an exclusion admits pages that should have been denied. Dropping the last usable inclusion can also make a worker treat the include set as empty and admit the default scope. The job may finish successfully while spending more source load, credits, queue capacity, and extraction time on an invalid dataset.

Client-side preflight is useful for feedback but cannot be authoritative unless it executes the same compiler configuration. The admission service should return a structured rejection naming the field and failed rule. It should not “simplify” a pattern, swap engines, or continue with the remaining rules unless the caller explicitly requested and reviewed that semantics.

Compile cost belongs to admission

Moving compilation earlier closes the silent-scope gap but moves untrusted work into the control plane. Source length alone is a weak budget: nested or counted repetitions can create a large compiled representation from a small string. Pattern count multiplies that work. If validation runs synchronously on an event loop, one request can delay unrelated status, cancellation, and crawl-start requests.

Apply cheap limits before expensive work:

1. Bound request bytes, pattern count, and per-pattern length before invoking a compiler.
2. Configure compiled-size, nesting, and runtime-specific automaton or cache limits.
3. Define the match alphabet deliberately. Firecrawl's follow-up uses byte regexes with Unicode disabled because its match targets are serialized, percent-encoded URL paths or hrefs. That optimization is valid only if every execution path preserves that target contract.
4. Compile once per admitted policy and worker generation where the runtime permits it. Do not recompile the full rule set for every discovered-link batch.
5. Isolate or budget compilation so a pathological request cannot block the API's control functions.

Do not copy another project's numeric limits blindly. Select limits from real approved patterns, worst-case compile measurements on production hardware, API latency objectives, and worker memory budgets. The invariant is more important than the values: an over-budget policy is rejected before any crawl work exists.

Make the match target explicit

Even one compiler can produce the wrong boundary if different components feed it different text. A pattern may run against only the path, the path plus query, an absolute URL, a pre-normalized input, or a serialized URL whose non-ASCII characters are percent-encoded. Anchors and case behavior make those distinctions observable.

Create a policy manifest at admission containing:

• ordered include and exclude source digests, never only a combined opaque hash;
• regex engine, library version, mode, flags, and compiler-limit profile;
• match-target class such as serialized_path, path_and_query, or absolute_url;
• URL parser and canonicalization version, percent-encoding policy, and query treatment;
• include/exclude precedence and empty-set semantics;
• successful rule count, rejected rule count, compile duration, and compiled-size class where available;
• policy generation and a digest over all semantics above.

Workers should refuse a job if they cannot reproduce or consume that policy generation. If workers compile locally, compare engine build and compiler profile with admission and cache the result for the job. A rolling deployment must not let old and new workers apply different dialects under one crawl ID.

Then record one compact decision for each frontier candidate: candidate URL identity, normalized match target digest, policy generation, matched include rule, matched exclude rule, final admit or deny, reason, and worker build. This extends the frontier ledger rather than replacing it. Atomic URL and item budgets still decide how much work may enter after the filter says the URL is in scope.

A boundary-compiler canary

Run the exact API, native module, worker image, and URL serializer against controlled sites:

1. Submit supported anchored, alternation, class, and quantifier patterns. Require identical admission and worker decisions.
2. Submit syntax accepted by a common client engine but unsupported by the worker, plus malformed syntax. Require a pre-enqueue rejection and zero fixture requests.
3. Test short patterns with large or stacked repetitions, then count and length values just below and above each limit. Measure compile latency, event-loop lag, memory, and rejection reason.
4. Compare raw Unicode URLs, percent-encoded equivalents, mixed-case hosts, paths with reserved characters, queries, and fragments. Assert the documented match target rather than whichever string a component happens to expose.
5. Exercise no includes, one include, several includes, exclusions that overlap includes, and a rule set where every expression is invalid. No failure may collapse into an unrestricted default.
6. Restart and roll workers across engine versions while a crawl is queued. Require the policy generation to remain reproducible or stop the job before discovery continues.
7. Force the compiler service or native module to be unavailable. Admission should return policy_unavailable, not enqueue a crawl with empty filters.
8. Return robots disallow, 429, login, challenge, denial, and payment fixtures inside otherwise matching paths. Preserve those independent policy outcomes and prohibit filter changes from seeking broader access.

Useful fleet alerts include any job with a nonzero dropped-rule count, worker/admission policy-digest mismatch, compile latency above budget, candidate decision without a policy generation, or fetched URL lacking a prior frontier admission record.

The operator decision rule

When a crawl exceeds its intended section, stop expansion and locate the first disagreement. If the submitted expression was invalid for the worker dialect, reject and repair the policy rather than interpreting partial results. If admission accepted it but a worker used another compiler profile or match target, quarantine that crawl generation. If compilation exhausted the control-plane budget, tighten pre-compilation limits or isolate the work before raising timeouts. If all filter evidence agrees, continue with frontier, redirect, robots, and source-policy diagnosis; a matching path is not sufficient authorization.

A path filter is not metadata attached to a crawl. It is code that decides which requests can exist. Compile it as deliberately as production code, bind the result to frontier admission, and a bad expression becomes a bounded request error instead of a wider crawl—or a control plane that stops serving everyone else.