Scraping belongs in the AI layer

An AI agent can write scraper code. That does not mean it should have to build a new retrieval stack every time someone asks a question about the live web.

The useful abstraction is not “open a remote browser.” It is “get the right web data, using the cheapest reliable strategy, and explain what happened.”

That belongs in the AI layer.

The gap between fetch and data

A plain HTTP request is ideal when it works. It is fast, cheap, and easy to inspect. But an agent regularly encounters pages that need JavaScript, browser cookies, a consistent session, an API hidden behind the rendered app, or a regional network route.

Without a retrieval layer, the model has to rediscover that escalation path in the middle of the task. It writes glue code, retries failures without enough context, and moves large page bodies through the conversation before it knows which fields matter.

Better Fetch puts that escalation behind one hosted MCP connection:

1. Start with direct HTTP.
2. Move to Chromium when the page needs a browser.
3. Reuse a named session when several calls should look like one visitor.
4. Discover the structured API behind the page before parsing HTML.
5. Use residential routing only when direct egress is blocked or the task is genuinely regional.
6. Return the content together with block, retry, transport, and routing metadata.

The agent gets a stable tool contract. The retrieval runtime absorbs the browser and network complexity.

MCP is the product surface

An API is still useful for applications that need direct control, but most people should not have to write an integration before their AI can retrieve a page.

With MCP, the retrieval tools arrive where the work already happens. Claude, ChatGPT, Codex, and other clients can authenticate once, inspect the available capabilities, and call them during a task.

The server can also provide operating instructions: prefer structured data, reuse sessions, avoid blind retries, and make credit-consuming escalation visible. Those rules matter as much as the individual tools.

More tools are not always better

A catalogue can contain dozens of focused scrapers without exposing dozens of top-level schemas on every conversation.

Better Fetch keeps the default MCP surface compact. The agent searches the catalogue when it needs a specialist capability, reviews the matching schema and estimated cost, then invokes that exact tool. General retrieval remains available through a small set of primitives.

That separation lets the catalogue grow without making tool choice less reliable.

Reliability includes honest failure

No browser or proxy stack can promise that every target will cooperate.

A useful retrieval layer classifies the result instead:

• Was the response blocked?
• What kind of block was detected?
• Did the service retry?
• Which transport produced the result?
• Was residential routing used?
• How much content was truncated?

The agent can then stop, change strategy, or report the gap. “Could not retrieve this target after one controlled escalation” is a better product outcome than an opaque loop that spends credits and still returns an interstitial.

Give your AI a better fetch

Better Fetch is available as a hosted MCP server at https://betterfetch.co/api/mcp. Add it to Claude or ChatGPT, sign in, and the web-data layer is ready when the model needs it.

The REST API remains available underneath. It is infrastructure, not the first thing a user should have to understand.