Allowed by Which Matcher, Under Whose Policy, at Which Hop?

A browser agent has an administrator-defined domain allowlist. A crawl API rejects blocked starting URLs. Both controls appear in configuration and both pass ordinary tests. Yet equivalent hostname forms can receive different decisions, one MCP call can alter the policy seen by the next call, and an admitted URL can redirect into a destination the original check never evaluated.

They are disagreements about the input, owner, lifetime, and evaluation point of one security decision.

The practical thesis is: a destination verdict should identify the canonical subject, exact matching semantics, immutable policy generation, and request-graph hop it evaluated; if any of those four inputs is unknown, allowed is not a reusable fact. This turns a domain list from mutable configuration into a versioned decision function that can be tested and audited.

This is defensive guidance for public or properly authorized fetching. Destination controls should reduce authority. Do not use parser differences, redirects, client overrides, or policy exceptions to reach a source outside the approved task. A denial, login, challenge, payment boundary, publisher restriction, robots decision, or rate limit remains a stop, review, or scheduling outcome.

Fresh evidence from four policy seams

Four open pull requests from September 8 and 9 expose different inputs to the same verdict:

These are proposed changes, not guarantees about released packages or hosted deployments. Test the exact versions and configuration paths you operate.

An earlier Better Fetch post covered policy closure over every request-producing edge. Repeating “check every redirect” would add little. The new information surplus starts one layer earlier: even when an edge reaches a policy hook, the hook can evaluate the wrong hostname semantics or a policy object whose authority changed between calls. Request-graph coverage and policy identity must reconcile.

Compile the subject before matching it

A URL can support several legitimate identities:

• exact origin: scheme, canonical host, and effective port;
• exact host, independent of port;
• subdomain relationship;
• registrable domain under a Public Suffix List generation;
• IP literal, which should normally remain an exact address rather than enter label arithmetic.

Those are different predicates. Calling all of them same_domain makes review difficult and invites a helper designed for cookie grouping to become an egress rule.

The Firecrawl change demonstrates why taking the last two labels is not registrable-domain parsing. Multipart public suffixes break that shortcut, while IP addresses are not DNS label trees at all. The Browser Use change shows a separate issue: the matcher and browser can disagree about equivalent DNS spelling or case unless normalization occurs consistently in list, glob, and set paths.

Define the predicate before choosing the parser. For a strict destination allowlist, exact origin or exact host is usually easier to reason about than registrable-domain equivalence. If a business rule intentionally includes subdomains or a registrable domain, name that expansion explicitly, pin the PSL implementation and data generation, and test tenant-style public suffixes. Never silently widen exact-host policy because a convenience helper returns a shared parent.

Keep the original URL for evidence, but match a structured parser result. Record parse failure instead of repairing ambiguous input differently in separate components.

Policy ownership is not a dictionary update

A server default, tenant policy, task grant, and per-call request do not have equal authority. A client request can usually narrow its usable destinations; it should not rewrite the administrator's ceiling or change what another client receives later.

The Browser Use report is a lifecycle failure: a function returned a live nested dictionary, and the call path assigned its override into that shared object. Copying before modification prevents persistence into later calls. Production systems still need an explicit composition rule. A safe default is:

• allowed destinations are the intersection of the administrator ceiling, tenant or task grant, and requested subset;
• prohibited destinations are the union of applicable denials;
• an omitted per-call field means “use the already compiled task policy,” not “reuse the previous call”;
• an empty intersection denies work rather than falling back to unrestricted access.

Wildcard intersections are easy to implement incorrectly. Prefer compiling each candidate through every authoritative layer and retaining every layer's verdict over trying to manufacture one clever merged glob list.

After compilation, freeze the effective policy. Give it a generation and digest over normalized rule semantics, parser and PSL versions, composition algorithm, principal, task, and enabled tools. Workers should receive that artifact or fail admission; they should not reread a mutable process-global dictionary halfway through a run.

A redirect is a new subject, not a policy continuation

The Firecrawl redirect change shows how coverage can vary by job type even inside one worker. A middleware checked the submitted URL, while a final-URL check lived in a crawl-specific branch. Single and batch scrape paths could therefore report success after a different final destination supplied the document.

A final-destination check is valuable for quarantine, billing, status semantics, and detecting gaps. It is not equivalent to a pre-request decision: by the time metadata contains the final URL, the redirected request may already have reached that destination and received bytes. Browser and HTTP redirect handlers should evaluate the next canonical destination before following whenever the stack permits it. The post-response check should then reconcile what actually happened with what was admitted.

Apply the same policy compiler to every request-producing job type and engine. Exceptions for monitors, uploaded-file parsing, licensed exchanges, internal callbacks, or other business paths need named rules, owners, tests, and expiry—not branches inferred from missing metadata. If one engine cannot expose redirect hops before transmission, label its enforcement post_response_only and use independent egress containment rather than claiming equivalent protection.

Issue a destination-decision receipt

For each attempted hop, retain:

• run, principal, client, task, job type, engine, request, parent-edge, and redirect-hop IDs;
• original URL reference, parsed scheme, canonical host, effective port, and parse verdict;
• subject predicate: exact origin, exact host, declared subdomain scope, registrable domain, or exact IP;
• URL parser, normalization profile, PSL implementation, and PSL data generation;
• administrator, tenant, task, and per-call policy generation IDs;
• each layer's matched rule and verdict, composition result, effective-policy digest, and object generation;
• decision time relative to request start, redirect receipt, and response consumption;
• final observed URL, final policy verdict, artifact eligibility, and containment-layer result;
• normalized outcome such as allowed_pre_request, denied_pre_request, policy_generation_mismatch, subject_unknown, denied_after_redirect, or post_response_only.

Avoid broad logging of credentials, complete private URLs, or query values. Restricted URL references, canonical destination classes, rule IDs, keyed digests, and timing are usually enough to diagnose policy drift.

A four-axis fixture matrix

Run controlled tests through the exact agent server, crawler API, browser or HTTP engine, and worker deployment:

1. Compare hostname case and equivalent root-label spelling across exact lists, wildcard lists, and any optimized set representation. Require one decision per declared predicate.
2. Test exact hosts, intended subdomains, unrelated registrable domains under multipart suffixes, IP literals, IPv6 forms, local test names, and parser failures. Never infer policy from string suffixes alone.
3. Make one synthetic client request a narrower domain subset, then start another request with no override. Require the second to receive the immutable administrator and task policy, with a different per-call generation.
4. Attempt a per-call expansion beyond the administrator ceiling. Require pre-network denial rather than a merged or replaced allowlist.
5. Exercise direct fetch, batch scrape, crawl root, discovered child, search or extract helper, monitor, browser navigation, subresource, and backend side request. Inventory which policy hook governs each path.
6. Redirect an owned allowed fixture to another allowed fixture and to a denied fixture. Require a pre-hop decision where supported, zero trusted artifacts after denial, and a final reconciliation check.
7. Change the policy or PSL generation while work is queued. Require the declared update behavior—finish under the pinned generation or stop and re-admit—rather than mixed decisions.
8. Make parsing or policy lookup unavailable. Require subject_unknown or policy_unavailable, no unrestricted fallback, and no retry that changes identity or route.

The operator decision rule

When a run reaches an unexpected destination, do not begin with the domain list printed in configuration. Reconstruct the exact decision. Which structured subject did the matcher evaluate? Which predicate and parser generation defined “same”? Which administrator, task, and client layers produced the effective policy? Was that object immutable for this call? Did the verdict apply to this redirect hop and job type before traffic left?

If any answer is missing, stop dependent work and quarantine the artifact. Repair subject parsing, policy composition, object lifetime, or path coverage at the first disagreement. A domain list is only source material. The production control is the reproducible receipt proving who allowed which canonical destination, under which authority, at which hop.