Blog
Notes from building Better Fetch.
· Paul Crossland
Make Fetch Pipelines Policy-Aware
AI crawl controls are turning robots.txt into runtime policy. Treat 402, 403, robots, and crawler identity as first-class fetch signals.
· Paul Crossland
Discover APIs Before Scraping HTML
Modern pages often expose cleaner JSON behind the UI. Capture network calls first, then scrape rendered HTML only when you must.
· Better Fetch
Better Fetch vs. the alternatives — choosing a fetch API in 2026
An honest comparison of Better Fetch against Firecrawl, ScrapingBee, ScraperAPI, Bright Data, and the AI-agent fetch tools — and when you shouldn't pick us.