Blog
Notes from building Better Fetch.
· Paul Crossland
Stale Service Workers Distort Fetch Evidence
Fresh Safari fixes show why crawler incidents need service-worker, partitioned-cookie, and storage cleanup evidence.
· Paul Crossland
Release-Channel Skew Deserves a Fetch Canary
Fresh Chrome and WebKit releases show why crawler QA should sample browser channels before rollout drift becomes a data incident.
· Paul Crossland
Packaged Web Documents Need Extraction Provenance
Fresh W3C EPUB and WCAG updates show why document crawls need reading-system, accessibility, and sampling evidence.
· Paul Crossland
Auth State Is a Fetch Boundary, Not a Retry Problem
Fresh WebAuthn and Chrome updates show why browser fetch pipelines need explicit auth-state evidence before escalating failures.
· Paul Crossland
Agent Fetch Runs Need Reviewable Control Points
Fresh GitHub agent updates show why web-data agents need approvals, provenance, and replayable evidence around every automated action.
· Paul Crossland
Accessibility Sampling Catches Rendered Data Drift
Fresh W3C and WebKit updates show how accessibility evidence can expose crawler failures that DOM selectors miss.
· Paul Crossland
Resolver View Is Fetch State
Split-horizon DNS and resolver exceptions can change crawl results before HTTP begins. Log DNS provenance with every fetch.
· Paul Crossland
WAF Change Windows Are Crawl State
Fresh Cloudflare WAF releases show why crawlers should correlate policy-rule changes with fetch outcomes before retrying.
· Paul Crossland
Unicode Matching Is Extraction Observability
New W3C drafts show why crawlers should log normalization, language, and direction decisions before treating text drift as parser failure.
· Paul Crossland
DNSSEC Validation Belongs in Fetch Incidents
A fresh .AL DNSSEC outage shows why crawlers should log resolver, DNSSEC, EDE, cache, and fallback evidence before blaming parsers.
· Paul Crossland
Session Evidence for Agentic Fetch Incidents
Cloudflare's new agent and bot signals show why browser-grade fetch logs need journey-level context, not just request outcomes.
· Better Fetch
Scraping belongs in the AI layer
Agents need reliable web data as a native tool, not another scraping API they have to wire together.
· Paul Crossland
Browser Family Is Fetch State
Chrome 150 landed across desktop, Android, and iOS. Treat browser family and platform as crawl inputs, not labels.
· Paul Crossland
TLS Groups Belong in Fetch Incident Logs
Node and Undici updates show why crawler incidents need TLS, HTTP/2, proxy, and stream evidence before retry tuning.
· Paul Crossland
AI Traffic Policy Is Fetch Telemetry
New AI bot controls make crawler category, attribution, and monetization policy part of production fetch observability.
· Paul Crossland
Safari MCP Makes Browser State Testable
Safari's MCP server turns DOM, network, screenshots, and console output into agent-visible fetch evidence.
· Paul Crossland
Fetch Correctness Is Runtime Specific
Recent Undici and WebKit networking fixes show why production crawlers need runtime-level fetch conformance tests.
· Paul Crossland
Browser Extensions Are Fetch State
Puppeteer's new extension support is a reminder to log, test, and isolate extension-influenced browser fetches.
· Paul Crossland
Redirect Timing Is Fetch Observability
New Fetch, HTML, and WPT work on TAO redirect chains shows why crawlers should treat timing data as conditional evidence.
· Paul Crossland
Node Runtime Updates Are Fetch Infrastructure Changes
Recent Node releases show why production fetchers should canary HTTP clients, TLS roots, and connection-pool behavior.
· Paul Crossland
Browser Version Drift Is Fetch State
Chrome's latest channel movement is a reminder to pin, log, and canary browser builds in production fetch systems.
· Paul Crossland
Test Agent Browsing as a Fetch Surface
Chrome's agent tooling makes browser access more observable. Treat agent browsing as a new fetch surface, not magic.
· Paul Crossland
Pin Browser Automation Revisions
Recent Playwright and Puppeteer releases show why crawler fleets need explicit browser, config, and trace versioning.
· Paul Crossland
Treat Consent as Fetch State
Cookie banners change sessions, markup, and analytics payloads. Capture consent state before comparing browser-grade fetches.
· Paul Crossland
Log Accept-Language Before Scaling
Locale can change rendered content and API payloads. Record language, Vary, and geo state before comparing fetch results.
· Paul Crossland
Replay Fetch Metadata, Not Just URLs
Internal APIs often check Sec-Fetch-* context. Capture request intent before replaying browser traffic outside the page.
· Paul Crossland
Stop Spoofing User Agents
User-Agent strings are shrinking as a signal. Test Client Hints, TLS fingerprints, and session behavior together instead.
· Paul Crossland
Stop Waiting on Network Idle
Network idle is a weak readiness signal for modern apps. Wait for the data, selector, or API response that proves the page is useful.
· Paul Crossland
Extract JSON-LD Before You Render
Check structured data before spending a browser render. JSON-LD can turn pages into cleaner extraction contracts.
· Paul Crossland
Make Crawlers Conditional, Not Repetitive
Use ETags, If-None-Match, and 304s to avoid needless renders, reduce load, and make browser-grade crawlers easier to operate.
· Paul Crossland
Treat 429 as Flow Control, Not a Proxy Cue
A 429 is not automatically a bot wall. Log Retry-After, quota scope, and replay mode before rotating proxies or browsers.
· Paul Crossland
Headless Chrome Is Not a Fetch Strategy
Modern headless Chrome runs like real Chrome, but bot walls still score sessions. Treat browser mode, clearance, and replay as separate signals.
· Paul Crossland
Test Fetch Sessions Against Cookie Partitioning
Browser privacy changes make session replay less portable. Test cookies by top-level site, region, and browser context before scaling.
· Paul Crossland
Make Fetch Pipelines Policy-Aware
AI crawl controls are turning robots.txt into runtime policy. Treat 402, 403, robots, and crawler identity as first-class fetch signals.
· Paul Crossland
Discover APIs Before Scraping HTML
Modern pages often expose cleaner JSON behind the UI. Capture network calls first, then scrape rendered HTML only when you must.
· Better Fetch
Better Fetch vs. the alternatives — choosing a fetch API in 2026
An honest comparison of Better Fetch against Firecrawl, ScrapingBee, ScraperAPI, Bright Data, and the AI-agent fetch tools — and when you shouldn't pick us.