Blog
Notes from building Better Fetch.
· Paul Crossland
Allowed by Which Matcher, Under Whose Policy, at Which Hop?
Fresh crawler and browser-agent bugs show how hostname semantics, mutable config, and redirects can disagree with one domain policy.
· Paul Crossland
Open Socket, Wrong Session: Three Browser Control-Plane Gates
Fresh browser and MCP bugs show why route identity, protocol readiness, and feature ownership must be proven before browser commands run.
· Paul Crossland
One Green exec() Can Hide a Broken Crawl
Redis pipelines can resolve while commands fail. A state-aware recovery matrix prevents lost URLs, hung crawls, and unsafe retries.
· Paul Crossland
Who Settles the Body After the Caller Is Gone?
Fresh Undici changes expose body-ownership failures across redirects, retries, and deduplicated requests—and how to test them.
· Paul Crossland
Before the First URL, Compile the Crawl Boundary
Fresh crawler fixes show how regex dialect drift and unbounded compilation can widen a crawl or stall its control plane.
· Paul Crossland
The Certificate You Did Not Send Still Belongs in the Connection Key
Fresh Playwright and curl changes show why explicit no-certificate policy must survive origin routing, TLS interception, and connection reuse.
· Paul Crossland
From Header to Bytes to Blank Text: The OCR Receipt
Fresh crawler changes show why file extraction must reconcile headers, byte signatures, parser capability, empty output, and cache admission.
· Paul Crossland
Shared, Enabled, Admitted: The Cookie Receipt for HTTP Fetch Fleets
Fresh libcurl changes expose separate cookie states that shared HTTP fetch workers must reconcile before trusting session continuity.
· Paul Crossland
The First Rule Can Disappear Before the Browser Agent Runs
Fresh serializer bugs show how browser-agent instructions can vanish before a model request, and how to detect and contain that loss.
· Paul Crossland
The Last Chromium Flag Can Rewrite the Crawl
Fresh launch bugs show why browser workers must reconcile config intent, final process arguments, runtime effects, and startup outcomes.
· Paul Crossland
Punctuation, Placeholders, and Pressed Buttons
Fresh browser-automation bugs show why input intent, device state, page events, and application outcomes must reconcile.
· Paul Crossland
One Placeholder, Two Opposite Credential Failures
Fresh browser-agent bugs show why secret injection and redaction need separate, reconciling coverage proofs.
· Paul Crossland
When Queue Time Becomes Document Damage
Fresh document-pipeline changes show how queue wait, OCR cost, caches, and client resumes can turn one timeout into plausible incomplete text.
· Paul Crossland
Three Ledgers Behind One Crawl Status
Fresh crawl SDK changes show why frontier admission, job observation, and result materialization require separate evidence.
· Paul Crossland
Before the Agent Sees the Screenshot
Fresh browser-tool changes expose how capture timing, silent resizing, encoding, and model delivery can alter visual fetch evidence.
· Paul Crossland
The Sixty-Fifth Navigation Is Part of the Test
Fresh browser failures show why crawler canaries must measure cumulative work, pre-request stalls, CPU cost, and teardown contradictions.
· Paul Crossland
The Three Loss Gates Before Clean Markdown
Fresh crawler bugs show how clean-content pipelines lose intent, structure, and text before Markdown reaches an extractor.
· Paul Crossland
The Browser Download Commit Point
Three fresh automation bugs show why download completion is a correlated commit protocol, not a file-exists check or callback.
· Paul Crossland
The Button Said Next. The Crawl Changed the Wrong State.
Fresh pagination-classifier bugs show why crawlers must verify list transitions instead of trusting labels, roles, or clicks.
· Paul Crossland
A Stopped Agent Can Leave a Live Browser Behind
Fresh WebMCP and Playwright cancellation changes expose the gap between stopping a tool call and proving its browser is quiet.
· Paul Crossland
After the Tab Switch: Proving Which Page Supplied the Data
Fresh agent and MCP races show why browser extraction must verify the active target generation instead of trusting a switch acknowledgement.
· Paul Crossland
Eight-Hour Tokens and Type-Scoped Kill Switches
GitHub's type-scoped revocation and eight-hour OAuth tokens, plus Cloudflare's pre-registered MCP clients, make credential class operational for fetch fleets.
· Paul Crossland
The URL Passed. The Request Graph Did Not.
Fresh browser-agent guardrail gaps show why URL policy must cover every producer, scheme, and redirect hop before a request is sent.
· Paul Crossland
The Browser Network Archive Is a Derived Artifact
Fresh Playwright and WebKit changes show why HAR and trace consumers must label missing, reconstructed, and unfinalized network evidence.
· Paul Crossland
The Agent Saw It. The Accessibility Tree Did Not.
A fresh Playwright mismatch shows why agent snapshots need visibility, accessibility, and actionability labels before extraction.
· Paul Crossland
A Browser State File Is a Checkpoint, Not a Cookie Jar
Fresh IndexedDB and OPFS changes show why browser session exports need quiescence, coverage, and round-trip verification.
· Paul Crossland
An MCP Fetch Crosses Five Evidence Planes
Portable plugins and gateway detection expose five evidence planes between an installed MCP tool and the web data it returns.
· Paul Crossland
One Browser Run Can Take Three Network Routes
Proxy exclusions and dual-stack fallback can change a browser run before navigation. Audit effective routes per hop, not per job.
· Paul Crossland
One Browser Session, Two Principals: Audit the Agent Handoff
Human-to-agent handoffs can preserve browser state without preserving authority. Segment the journey before trusting extracted data.
· Paul Crossland
When an Iframe Vanishes, What Did the Crawl Actually Prove?
Fresh Puppeteer fixes show why crawlers should separate main-document success, optional frame churn, and incomplete frame evidence.
· Paul Crossland
Browser-Grade Fetching Now Has Fidelity Tiers
Agent-first browsers and WebMCP make evidence requirements—not a browser flag—the right way to route production fetches.
· Paul Crossland
Automation Control Planes Deserve Fetch Telemetry
Fresh Puppeteer and Chrome updates show why attach, launch, protocol, and browser-discovery evidence belong in crawler logs.
· Paul Crossland
HTTP/2 Stream Resets Are Fetch Evidence, Not Generic Retries
Fresh Undici and Node releases show why crawlers should log stream-level HTTP/2 failures before changing retry policy.
· Paul Crossland
Resolve Agent Identity Before Trusting Fetch Evidence
Fresh W3C DID and credential updates show why agentic fetch systems should log identity resolution, not just trust labels.
· Paul Crossland
Comment-Triggered Agents Need Fetch Run Guardrails
Fresh Copilot agent updates show why web-data agents need trigger provenance, reasoning budgets, and approval gates.
· Paul Crossland
Node HTTP Client Semantics Can Shift Fetch Evidence
Fresh Node releases show why HTTP parser, TLS session, and Undici changes deserve canaries before crawler rollout.
· Paul Crossland
WebTransport Turns API Discovery Into Flow Observability
WebTransport's Candidate Recommendation shows why crawlers need stream, datagram, and fallback evidence, not just XHR logs.
· Paul Crossland
Security Rollouts Need Fetch Evidence Quarantine
Fresh Chrome and WebKit releases show how to handle browser security updates without mislabeling data drift.
· Paul Crossland
Automation Lifecycle Is Fetch Evidence
Fresh Playwright and Puppeteer releases show why crawler incidents need cancellation, browser roll, and resource-lifetime telemetry.
· Paul Crossland
Stale Service Workers Distort Fetch Evidence
Fresh Safari fixes show why crawler incidents need service-worker, partitioned-cookie, and storage cleanup evidence.
· Paul Crossland
Release-Channel Skew Deserves a Fetch Canary
Fresh Chrome and WebKit releases show why crawler QA should sample browser channels before rollout drift becomes a data incident.
· Paul Crossland
Packaged Web Documents Need Extraction Provenance
Fresh W3C EPUB and WCAG updates show why document crawls need reading-system, accessibility, and sampling evidence.
· Paul Crossland
Auth State Is a Fetch Boundary, Not a Retry Problem
Fresh WebAuthn and Chrome updates show why browser fetch pipelines need explicit auth-state evidence before escalating failures.
· Paul Crossland
Agent Fetch Runs Need Reviewable Control Points
Fresh GitHub agent updates show why web-data agents need approvals, provenance, and replayable evidence around every automated action.
· Paul Crossland
Accessibility Sampling Catches Rendered Data Drift
Fresh W3C and WebKit updates show how accessibility evidence can expose crawler failures that DOM selectors miss.
· Paul Crossland
Resolver View Is Fetch State
Split-horizon DNS and resolver exceptions can change crawl results before HTTP begins. Log DNS provenance with every fetch.
· Paul Crossland
WAF Change Windows Are Crawl State
Fresh Cloudflare WAF releases show why crawlers should correlate policy-rule changes with fetch outcomes before retrying.
· Paul Crossland
Unicode Matching Is Extraction Observability
New W3C drafts show why crawlers should log normalization, language, and direction decisions before treating text drift as parser failure.
· Paul Crossland
DNSSEC Validation Belongs in Fetch Incidents
A fresh .AL DNSSEC outage shows why crawlers should log resolver, DNSSEC, EDE, cache, and fallback evidence before blaming parsers.
· Paul Crossland
Session Evidence for Agentic Fetch Incidents
Cloudflare's new agent and bot signals show why browser-grade fetch logs need journey-level context, not just request outcomes.
· Better Fetch
Scraping belongs in the AI layer
Agents need reliable web data as a native tool, not another scraping API they have to wire together.
· Paul Crossland
Browser Family Is Fetch State
Chrome 150 landed across desktop, Android, and iOS. Treat browser family and platform as crawl inputs, not labels.
· Paul Crossland
TLS Groups Belong in Fetch Incident Logs
Node and Undici updates show why crawler incidents need TLS, HTTP/2, proxy, and stream evidence before retry tuning.
· Paul Crossland
AI Traffic Policy Is Fetch Telemetry
New AI bot controls make crawler category, attribution, and monetization policy part of production fetch observability.
· Paul Crossland
Safari MCP Makes Browser State Testable
Safari's MCP server turns DOM, network, screenshots, and console output into agent-visible fetch evidence.
· Paul Crossland
Fetch Correctness Is Runtime Specific
Recent Undici and WebKit networking fixes show why production crawlers need runtime-level fetch conformance tests.
· Paul Crossland
Browser Extensions Are Fetch State
Puppeteer's new extension support is a reminder to log, test, and isolate extension-influenced browser fetches.
· Paul Crossland
Redirect Timing Is Fetch Observability
New Fetch, HTML, and WPT work on TAO redirect chains shows why crawlers should treat timing data as conditional evidence.
· Paul Crossland
Node Runtime Updates Are Fetch Infrastructure Changes
Recent Node releases show why production fetchers should canary HTTP clients, TLS roots, and connection-pool behavior.
· Paul Crossland
Browser Version Drift Is Fetch State
Chrome's latest channel movement is a reminder to pin, log, and canary browser builds in production fetch systems.
· Paul Crossland
Test Agent Browsing as a Fetch Surface
Chrome's agent tooling makes browser access more observable. Treat agent browsing as a new fetch surface, not magic.
· Paul Crossland
Pin Browser Automation Revisions
Recent Playwright and Puppeteer releases show why crawler fleets need explicit browser, config, and trace versioning.
· Paul Crossland
Treat Consent as Fetch State
Cookie banners change sessions, markup, and analytics payloads. Capture consent state before comparing browser-grade fetches.
· Paul Crossland
Log Accept-Language Before Scaling
Locale can change rendered content and API payloads. Record language, Vary, and geo state before comparing fetch results.
· Paul Crossland
Replay Fetch Metadata, Not Just URLs
Internal APIs often check Sec-Fetch-* context. Capture request intent before replaying browser traffic outside the page.
· Paul Crossland
Stop Spoofing User Agents
User-Agent strings are shrinking as a signal. Test Client Hints, TLS fingerprints, and session behavior together instead.
· Paul Crossland
Stop Waiting on Network Idle
Network idle is a weak readiness signal for modern apps. Wait for the data, selector, or API response that proves the page is useful.
· Paul Crossland
Extract JSON-LD Before You Render
Check structured data before spending a browser render. JSON-LD can turn pages into cleaner extraction contracts.
· Paul Crossland
Make Crawlers Conditional, Not Repetitive
Use ETags, If-None-Match, and 304s to avoid needless renders, reduce load, and make browser-grade crawlers easier to operate.
· Paul Crossland
Treat 429 as Flow Control, Not a Proxy Cue
A 429 is not automatically a bot wall. Log Retry-After, quota scope, and replay mode before rotating proxies or browsers.
· Paul Crossland
Headless Chrome Is Not a Fetch Strategy
Modern headless Chrome runs like real Chrome, but bot walls still score sessions. Treat browser mode, clearance, and replay as separate signals.
· Paul Crossland
Test Fetch Sessions Against Cookie Partitioning
Browser privacy changes make session replay less portable. Test cookies by top-level site, region, and browser context before scaling.
· Paul Crossland
Make Fetch Pipelines Policy-Aware
AI crawl controls are turning robots.txt into runtime policy. Treat 402, 403, robots, and crawler identity as first-class fetch signals.
· Paul Crossland
Discover APIs Before Scraping HTML
Modern pages often expose cleaner JSON behind the UI. Capture network calls first, then scrape rendered HTML only when you must.
· Better Fetch
Better Fetch vs. the alternatives — choosing a fetch API in 2026
An honest comparison of Better Fetch against Firecrawl, ScrapingBee, ScraperAPI, Bright Data, and the AI-agent fetch tools — and when you shouldn't pick us.