Better Fetch

Overview

Better Fetch retrieves difficult web pages, then lets you choose the output layer. Use /v1/fetchfor raw transport data and diagnostics; use /v1/scrape for clean text, Markdown, metadata, links, and images. Both share direct HTTP, stealth Chromium, browser geo-emulation, account-scoped sticky sessions, screenshots, network capture, or Cloudflare/DataDome cookie collection when issued. The base URL is https://api.betterfetch.co and endpoints are versioned under /v1.

Simple requests can complete over direct HTTP. When Chromium is needed, it runs in a real persistent browser profile—not incognito. Requests with a session reuse a warm, pooled context keyed to your account, session, and browser options. If a response looks blocked and country is set, the service can retry on a fresh browser identity and escalate to a headed browser. Egress changes only when proxy is auto or residential.

Social data APIs

Better Fetch also publishes platform-specific tools for public social profiles, posts, videos, comments, transcripts, search, ads, commerce, and creator intelligence. The social reference keeps live runner contracts separate from in-development coverage targets.

Browse the platform-by-platform social data reference

Authentication

All fetch and scrape requests require a bearer token. Create and revoke keys on the API keys page. Keep keys server-side — never in browser JavaScript, query strings, or shared logs.

Authorization: Bearer <your-api-key>
Content-Type: application/json

Keys require an active subscription and are metered against your plan's monthly credit quota. Ordinary accepted retrieval costs 1 credit. Pro and Scale residential attempts cost 25 credits each when they actually occur. Stored browser sessions are also plan-limited and can be cleared from the dashboard.

POST /v1/scrape

POST/v1/scrape

Retrieve one HTML page through the same engine as fetch, then run a deterministic, network-free document normalizer. Ordinary retrieval costs one credit; actual residential attempts use the higher published rate. The response keeps retrieval diagnostics separate, so a target block remains visible under retrieval.blocked without confusing it with an API failure.

curl -sS -X POST "https://api.betterfetch.co/v1/scrape" \
  -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/article",
    "strategy": "auto",
    "proxy": "none",
    "country": "au",
    "session": "research",
    "only_main_content": true,
    "max_chars": 60000,
    "include_raw_html": false
  }'

Document options

All fetch request fields also work here. These fields control only normalization and output size.

only_main_contentbooleandefault: true

Prefer substantial main/article content and remove common navigation, header, footer, form, and sidebar chrome.

max_charsintegerdefault: 500000

Maximum characters returned independently in text and Markdown. Range 1000-2000000; truncation flags tell you when the limit was reached.

max_linksintegerdefault: 500

Maximum deduplicated HTTP(S) links. Range 0-5000.

max_imagesintegerdefault: 200

Maximum deduplicated HTTP(S) images. Range 0-2000.

include_raw_htmlbooleandefault: false

Include the retrieved HTML alongside the normalized document. Leave false for model and RAG workflows.

Document response

urlstring

Final target URL after redirects.

titlestring

Resolved document/page title.

textstring

Clean readable document text.

markdownstring

Clean content with headings, lists, quotes, and code structure preserved.

text_truncatedboolean

Whether text exceeded max_chars.

markdown_truncatedboolean

Whether Markdown exceeded max_chars.

word_countinteger

Word count of the returned text.

credits_usedinteger

Credits charged for the retrieval work that actually occurred.

linksarray

Deduplicated absolute links with visible labels when available.

imagesarray

Deduplicated absolute image URLs with alt text when available.

metadataobject

Title, description, author, publication time, site, language, canonical URL, and JSON-LD types when present.

retrievalobject

Target status, block verdict, attempts, transport, proxy/cache use, timing, content type, and body size from the underlying retrieval.

htmlstring

Present only when include_raw_html is true.

JSON, plain-text, binary, and empty targets return 415 unsupported_content. Use /v1/fetch for those. Try the signed-in scrape playgroundwithout copying an API key.

POST /v1/fetch

POST/v1/fetch

Fetch a URL through the browser and return rendered page data: target status, final URL, title, rendered HTML, headers, timing, block classification, and optionally the raw body, parsed JSON, captured network calls, or a screenshot. Unknown request fields are rejected with 400.

curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \
  -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "wait_until": "domcontentloaded",
    "timeout_ms": 60000
  }'

Request fields

Only url is required. Everything else has a sensible default.

urlstringdefault: required

Public URL or domain to fetch. Bare domains default to HTTPS; explicit HTTP and HTTPS URLs are preserved.

wait_untilstringdefault: "load"

Browser navigation state when the browser path is selected: load, domcontentloaded, networkidle, or commit.

wait_selectorstring

CSS selector to wait for after navigation. Use this for pages that render content after the initial load.

scroll_selectorstring

CSS selector to scroll into view after navigation. Use it to trigger lazy-loaded feeds, comments, and media before wait_ms and network capture.

wait_msnumber

Extra fixed wait after navigation, in milliseconds. Capped by timeout_ms. Use only when there is no reliable selector.

timeout_msnumberdefault: 90000

Navigation and selector timeout in milliseconds. Maximum 240000. The default is deliberately generous so the first call after a cold browser start has room to finish.

cache_ttl_msnumberdefault: 0

Optional short-lived in-process response cache TTL for identical synchronous fetch payloads. Range 0-60000. Hits are best effort per API worker; the cache key ignores cache_ttl_ms itself.

strategy"auto" | "http" | "browser"default: "auto"

Execution strategy. auto uses direct HTTP for simple body/JSON fetches and Chromium when the payload asks for browser-only features or the fast path looks blocked. http returns the raw HTTP response without browser-only features. browser forces the rendered browser path.

return_response_textbooleandefault: false

Include the raw response body in body_text. JSON responses include it automatically. When set with an API-shaped URL (a path with no .html/.php-style extension), this also participates in auto strategy selection and can auto-select browser JSON mode — unless screenshot, full_page, or wait_selector is also set, which keeps the request on the rendered-page path. Site-root URLs with no path (https://example.com) are always treated as pages, never APIs.

include_htmlbooleandefault: true

Include rendered/raw HTML in html. Set false for JSON/body workflows to reduce response size and skip origin DOM serialization in browser JSON mode.

return_cf_clearancebooleandefault: false

Attempt to collect Cloudflare cf_clearance token data and return it when the browser receives that cookie. When false, cookies are not read or returned.

return_datadome_cookiebooleandefault: false

Attempt to collect DataDome datadome cookie data and return it when the rendered browser session receives that cookie. When false, DataDome cookie fields are not read or returned.

return_cookiesbooleandefault: false

Return storage-ready cookies visible to the final rendered page in cookies.

cookiesarraydefault: []

Preload cookies into the browser context before navigation. Each cookie needs name and value plus either url or domain; path defaults to / when domain is used.

capture_networkbooleandefault: false

Capture matching browser network calls and return them in network. Defaults to XHR/fetch only — useful for API discovery and debugging.

network_resource_typesstring[]default: ["xhr","fetch"]

Playwright resource types to capture, e.g. xhr, fetch, document, script, or websocket. Keep this narrow for most workloads.

network_include_bodiesbooleandefault: true

Include capped response bodies for captured network responses.

network_include_headersbooleandefault: false

Include request and response headers for captured entries. Off by default because headers can contain cookies, bearer tokens, or other secrets.

network_max_entriesnumberdefault: 100

Maximum matching network entries to return. Range 1–500.

network_max_body_bytesnumberdefault: 262144

Maximum bytes kept from each captured response body. Range 0–1048576.

network_capture_streamsbooleandefault: false

When capture_network is true, capture streamed fetch/XHR chunks, EventSource messages, and WebSocket messages in network_streams.

network_stream_max_eventsnumberdefault: 100

Maximum streamed network values to return. Range 1–500.

network_stream_max_value_bytesnumberdefault: 65536

Maximum bytes kept from each streamed network value. Range 0–262144.

screenshotbooleandefault: false

Include a PNG screenshot encoded as base64. Works together with return_response_text: requesting a screenshot keeps the request on the rendered-page path, so both come back. Only explicit json_mode: true disables screenshots. Screenshots can make responses much larger, so request them only when needed.

full_pagebooleandefault: false

Capture the full scrollable page when screenshot is true.

countrystring

Two-letter country code for browser geo-emulation, e.g. us, gb, de, au, ca. When geoip is true and locale/timezone are unset, Better Fetch applies representative browser defaults for that country. This does not change network egress IP.

proxy"none" | "auto" | "residential"default: "none"

Pro/Scale network routing. none uses Better Fetch datacenter egress. auto starts direct and escalates blocked retries through residential egress. residential routes every attempt through residential egress. When country is also set, proxied attempts use that country for the exit and browser identity. Each residential attempt that occurs costs 25 credits.

sessionstring

Account-scoped browser identity key. Reuse it to keep the same warm browser context, fingerprint, cookies, localStorage, and encrypted portable snapshot. Only letters and numbers form the canonical key today, so punctuation is ignored.

geoipbooleandefault: true when country is set

When country is set, apply country-derived browser timezone/locale defaults unless explicit locale/timezone values are supplied. This does not spoof WebRTC IP or change network egress IP.

localestringdefault: automatic

Browser locale, e.g. en-GB. Overrides the country-derived locale.

timezonestringdefault: automatic

Browser timezone, e.g. Europe/London. Overrides the country-derived timezone.

user_agentstringdefault: browser default

Custom user agent applied to the browser context. Usually leave unset — it forms part of a session's warm-context identity.

extra_headersobjectdefault: {}

Additional HTTP headers applied to the direct HTTP request or inside the browser context.

humanizebooleandefault: auto

Human-like mouse, keyboard, and scroll behavior for browser fetches. Defaults to true for rendered browser pages and false for direct/API calls. Set explicitly to override.

json_modebooleandefault: auto

Fetch via an in-page browser fetch() call instead of a top-level browser navigation. Sends Sec-Fetch-Mode: cors and a natural Referer from the URL's origin. Auto-detected from Accept: application/json or from return_response_text on an API-shaped URL — but never when screenshot, full_page, or wait_selector is set (those signal a rendered page), and never for site-root URLs with no path. An explicit true/false always wins; note screenshots are unavailable in json_mode.

Response fields

ok: true means Better Fetch completed the browser request — check status and blocked / block_reasonfor the target's verdict.

okboolean

true when Better Fetch completed the request — not whether the target accepted it. Check status and blocked for the target's verdict.

statusnumber | null

HTTP status from the target response (navigation or in-page fetch). On challenge-fronted sites this can be stale: after Better Fetch observes a challenge clear, the solved page can render while status keeps the original 403/503. Use blocked, not status alone, as the browser-fetch outcome signal.

final_urlstring

Final target URL after redirects.

titlestring

Page title after rendering, or the parsed title for direct HTTP HTML responses.

htmlstring

Rendered DOM HTML from the browser, or raw HTML from the direct HTTP transport. Empty when include_html is false.

body_textstring | null

Raw response body when requested, when the target response is JSON, or when strategy=http. Capped at 50 MB — see body_truncated.

body_bytesnumber

Total response body size in bytes when measured by a body-fetch strategy. When body_truncated is true, this is larger than body_text.length.

body_truncatedboolean

True when the response body exceeded the 50 MB transfer cap and was truncated. Reduce per_page or narrow the query if you need the complete body.

content_typestring

Normalized response Content-Type media type without parameters, for example text/html or application/json. Empty when unavailable.

content_kind"html" | "json" | "text" | "binary" | "empty" | "unknown"

Best-effort response body category for routing parser logic.

json_parse_okboolean

Whether body_text parsed as JSON.

jsonany | null

Parsed JSON payload when json_parse_ok is true; otherwise null.

headersobject

Response headers from the target navigation response (string values).

screenshot_b64string | null

Base64-encoded PNG when screenshot is true; otherwise null. Always null in json_mode — in-page fetch() never renders the target as a page, which is why screenshot: true suppresses json_mode auto-detection.

cf_clearancestring | null

Cloudflare cf_clearance token value when return_cf_clearance is true and the target issued it.

cf_clearance_cookieobject | null

Storage-ready cookie metadata (name, value, domain, path, expires, httpOnly, secure, sameSite) when requested and present.

cf_clearance_sessionstring | null

The session that produced the clearance result. Blocked retries may rotate to a fresh session before the final result.

datadome_cookiestring | null

DataDome datadome cookie value when return_datadome_cookie is true and the target issued it. Otherwise null or omitted.

datadome_cookie_detailobject | null

Storage-ready datadome cookie metadata when requested and present. Otherwise null or omitted.

datadome_sessionstring | null

The session that produced the DataDome cookie result. Blocked retries may rotate to a fresh session before the final result.

datadome_detectedboolean

Present when return_datadome_cookie is true. true when the rendered page or response showed DataDome signals, even if no datadome cookie was returned.

cookiesarray

Storage-ready cookies visible to the final rendered page when return_cookies is true. Otherwise omitted.

networkarray

Captured browser network entries when capture_network is true. Otherwise omitted.

network_streamsarray

Captured streamed fetch/XHR chunks, EventSource messages, and WebSocket messages when network_capture_streams is true. Otherwise omitted.

blockedboolean

true when the response looks like a bot wall or unsolved challenge — even when the target returns HTTP 200. The inverse also holds: a 403/503 whose rendered body is substantial real content with no live interstitial is a solved challenge and reports blocked: false. Trust blocked over status in both directions.

block_reason"none" | "http_401" | "http_403" | "http_429" | "http_503" | "cloudflare" | "datadome" | "captcha" | "block_title" | "challenge_interstitial"

Stable reason for the blocked verdict. none means the response is not classified as blocked.

headedboolean

Whether the attempt that produced this result ran a headed browser (set on escalated retries).

pooledboolean

true when served from a warm pooled context (session requests); false for sessionless ephemeral contexts.

transport"http" | "browser"

Execution transport that produced the result.

cache_status"bypass" | "miss" | "hit" | "coalesced"

Synchronous fetch cache status. bypass means no cache was requested or enabled; miss means the target was fetched; hit means a completed prior result was reused; coalesced means an identical in-flight fetch was shared.

attemptsnumber

Total attempts including the first. Greater than 1 means the service retried on a fresh session — after a block or a transient navigation timeout.

proxy_usedboolean

Whether the final attempt used managed residential egress.

proxy_attemptsinteger

Residential attempts that actually occurred. Cache hits and coalesced followers have no billable proxy attempts.

credits_usedinteger

Total charged credits. Ordinary retrieval is 1; each actual residential attempt is 25 credits.

remaining_creditsinteger

Remaining period credits after a residential post-charge, when available.

timing_msnumber

Time spent inside the fetch request (final attempt).

Examples

Fetch rendered HTML

The basic call: navigate with a real browser and return the rendered DOM.

curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \
  -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "wait_until": "domcontentloaded",
    "timeout_ms": 60000
  }' | jq '.status, .title, .final_url'

Fetch with country browser identity

Use country when browser locale/timezone should match a country, and session when several requests should share one browser profile.

curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \
  -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "country": "gb",
    "session": "examplegb",
    "wait_until": "domcontentloaded",
    "timeout_ms": 60000
  }'

Session names are scoped to your Better Fetch account; other accounts using the same name get isolated browser profiles and encrypted snapshots. Use alphanumeric names when you need distinct sessions; today example-gb, example_gb, and examplegb point at the same session.

Fetch a JSON API (quick start)

Minimal JSON call. With strategy auto, Better Fetch uses the direct HTTP fast path for this shape and returns parsed JSON plus body_bytes.

curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \
  -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://jsonplaceholder.typicode.com/todos/1",
    "include_html": false,
    "extra_headers": { "Accept": "application/json" }
  }' | jq '{ ok, status, transport, cache_status, content_type, content_kind, json_parse_ok, body_bytes, timing_ms }'

If jq shows null for every field, the call failed — inspect ok, error, and message first (see Tips). Keys must start with bf_ and come from your keys page.

Fetch a JSON API

For harder JSON APIs, pass the SPA Referer/Origin plus a stable session. Add country only when browser locale/timezone should match the target. auto starts with the fast HTTP path and falls back to the browser path if the response looks blocked; set strategy: browser to force the old in-page fetch behavior.

curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \
  -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://partner-api.example.com/projections?per_page=500",
    "country": "us",
    "session": "exampleus",
    "return_response_text": true,
    "include_html": false,
    "extra_headers": {
      "Accept": "application/json",
      "Referer": "https://app.example.com/",
      "Origin": "https://app.example.com"
    },
    "timeout_ms": 60000
  }' | jq '{ ok, status, transport, cache_status, content_type, content_kind, json_parse_ok, body_bytes, body_truncated, timing_ms, blocked, block_reason }'

strategy auto uses the HTTP fast path for JSON/API body requests, then escalates to the browser if that response looks blocked. Set strategy: browser when the target specifically requires browser CORS/fetch semantics, or strategy: http when you explicitly want raw HTTP only.

Wait for rendered content

For client-rendered pages, wait for a selector instead of adding a long fixed delay.

curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \
  -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://app.example.com/items/123",
    "wait_until": "domcontentloaded",
    "wait_selector": "#content",
    "timeout_ms": 90000
  }'

Capture network calls

Capture the XHR/fetch calls a page makes while rendering — the fastest way to discover a site's internal APIs.

curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \
  -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://app.example.com/items/123",
    "wait_until": "networkidle",
    "timeout_ms": 90000,
    "capture_network": true,
    "network_max_entries": 50
  }' | jq '.network[] | { method, url, status, json }'

Enable network_include_headers only when you need it — headers can contain credentials.

Capture streamed values

Capture values delivered through streaming fetch/XHR, EventSource, or WebSocket while the page is open.

curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \
  -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://app.example.com/live",
    "wait_until": "domcontentloaded",
    "wait_ms": 10000,
    "timeout_ms": 90000,
    "capture_network": true,
    "network_capture_streams": true,
    "network_resource_types": ["fetch", "xhr", "eventsource", "websocket"],
    "network_stream_max_events": 100,
    "network_stream_max_value_bytes": 65536
  }' | jq '.network_streams[] | { source, event_type, url, value_text, json }'

Stream capture is opt-in because it instruments page fetch/XHR/EventSource/WebSocket APIs and can produce large responses.

Capture a screenshot

Return a base64-encoded PNG of the rendered page.

curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \
  -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "screenshot": true,
    "full_page": true,
    "wait_until": "domcontentloaded"
  }' | jq -r '.screenshot_b64'

Collect a Cloudflare clearance token when issued

Attempt the page flow and return the cf_clearance cookie when the target issues it, with storage-ready metadata.

curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \
  -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.example.com/",
    "country": "us",
    "session": "clearanceus",
    "wait_until": "domcontentloaded",
    "timeout_ms": 90000,
    "return_cf_clearance": true
  }' | jq '{ status, blocked, block_reason, cf_clearance, cf_clearance_cookie }'

cf_clearance is null when the target doesn't issue the cookie or the challenge remains unsolved. Store cf_clearance_session too — retries may rotate sessions.

Collect a DataDome cookie when issued

Attempt the page flow and return the datadome cookie when the target issues it, with storage-ready metadata.

curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \
  -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.example.com/",
    "country": "us",
    "session": "datadomeus",
    "wait_until": "domcontentloaded",
    "timeout_ms": 90000,
    "return_datadome_cookie": true
  }' | jq '{ status, blocked, block_reason, datadome_detected, datadome_cookie, datadome_cookie_detail }'

datadome_cookie is null when the target doesn't issue the cookie or the challenge remains unsolved. Reusing the Better Fetch session is usually more reliable than replaying the raw cookie elsewhere.

Export and replay browser cookies

Return cookies from a rendered browser session, store them in your system, then send them back on a later request.

curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \
  -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "session": "examplelogin",
    "wait_until": "domcontentloaded",
    "return_cookies": true
  }' | jq '.cookies'

curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \
  -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/account",
    "session": "examplelogin",
    "cookies": [
      {
        "name": "session",
        "value": "abc123",
        "domain": ".example.com",
        "path": "/",
        "expires": 1790000000,
        "httpOnly": true,
        "secure": true,
        "sameSite": "Lax"
      }
    ]
  }'

Using the same session also reuses your account-scoped server-side browser state. Stored session limits are plan-based: Free 1, Starter 10, Pro 50, Scale 250; sessions expire after 7 idle days.

Submit an async job

When you don't want to hold a connection open — or want to fan out many URLs — submit a fetch as a background job. Returns 202 immediately; poll until status is done or failed.

curl -sS -X POST "https://api.betterfetch.co/v1/jobs" \
  -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "wait_until": "domcontentloaded",
    "timeout_ms": 60000
  }'

# → {"ok": true, "id": "a1b2c3d4-...", "status": "queued"}

curl -sS "https://api.betterfetch.co/v1/jobs/a1b2c3d4-..." \
  -H "Authorization: Bearer $BETTER_FETCH_API_KEY"

# → {"ok": true, "id": "...", "status": "done", "result": {...}, "error": null, ...}

Jobs run under a separate concurrency budget so they never block synchronous /v1/fetch. One base credit is charged when admitted; actual residential attempts are post-charged at the published rate.

Errors

Every non-2xx response uses a single JSON envelope. Switch on the stable error code, not the message.

{ "ok": false, "error": "unauthorized", "message": "invalid or missing bearer token", "status": 401 }
StatusCodeMeaning
400bad_requestInvalid JSON, unknown request field, missing or non-HTTP url, or an out-of-range parameter.
401unauthorizedMissing, incorrect, or revoked bearer token.
402payment_requiredValid key but no active subscription.
402residential_plan_requiredproxy auto/residential was requested from a Free or Starter account. Upgrade to Pro or Scale; the rejected request is not metered.
415unsupported_contentPOST /v1/scrape retrieved JSON, plain text, binary, or empty content instead of HTML. Use /v1/fetch for non-HTML targets.
429quota_exceededMonthly credit quota exhausted; resets at the next billing cycle. Ordinary accepted retrieval costs 1 credit; actual residential attempts use the published higher rate.
429session_limit_exceededStored browser session limit reached; clear a session from the dashboard or upgrade.
502fetch_failedBrowser launch, navigation, or target fetch failed. The message includes the underlying detail. In json_mode, Failed to fetch often means the API's CORS policy rejected the in-page call — set Referer to the site's app origin or try json_mode: false.
504timeoutRequest timed out at the API layer.

Target errors are different from API errors: if Better Fetch returns HTTP 200 and the JSON contains "status": 403 or "blocked": true, the API worked and the target denied the browser request; use block_reason for the category.

Guides

High-volume same-site scraping

When you fetch many URLs on one site in a single run — scanning dozens of markets on a bookmaker, paging through a listing, etc. — these habits avoid most blocks and connection timeouts.

  • Reuse one session per site for the whole run (and across runs) — e.g. session: "skybet". A warm session keeps a stable fingerprint and persisted cookies/localStorage that mark you as a returning visitor. Do not generate a fresh session name per URL or per run: each distinct name — and each distinct locale/timezone/user_agent value — is a separate stored session that counts against your plan limit and starts cold.
  • Use country for coherent browser identity defaults (gb for UK locale/timezone, au for Australian locale/timezone, etc.). It changes network egress only when proxy is auto or residential.
  • Leave user_agent, locale, and timezone unset unless the integration requires them, so country can apply coherent defaults. If you do set them, keep them byte-identical across every call for that session — changing them splits the warm pool into separate cold contexts.
  • Pace the run: keep concurrency to a few in-flight requests and add a small jittered delay (1–3s) between calls. A sub-second burst of dozens of requests from one IP looks robotic and trips bot detection that can then poison the whole session.
  • Prefer fast waits on browser pages: wait_until: "domcontentloaded" plus wait_selector over networkidle or long fixed waits. For JSON endpoints, set extra_headers {"Accept":"application/json"}; auto uses direct HTTP first, skips humanization, and returns parsed JSON faster.
  • For JSON polling, use cache_ttl_ms for short identical bursts and cache longer-lived responses in your application. Reuse one session, shrink query params (per_page, include=), and poll only as often as the data actually changes.
  • Handle responses defensively in your loop. If blocked is true, inspect block_reason, then pause and back off (5–10s) before continuing. Retry an individual 502 fetch_failed after a short backoff — sustained failures mean the target is pushing back. Keep timeout_ms moderate (45–60s).

Marketplace tools

Ready-made tools built on the Better Fetch engine — structured extraction, monitoring, screenshots, and more — that you can run without writing fetch code yourself. Browse them at https://betterfetch.co/tools.

  • Each tool page documents its input/output schema, examples, and a playground to try it against your account.
  • Call search_tools from Claude, ChatGPT, Codex, or another MCP client to find the best live tool, then pass its exact name and input schema to run_tool. The catalogue stays searchable without flooding the default MCP surface.
  • Or call them over REST: POST https://betterfetch.co/api/tools/{name}/run with Authorization: Bearer bf_... and a JSON body of {"input": {...}}.
  • Tool runs are metered against your plan like direct fetch calls; each tool lists a ~credits-per-run estimate.

POST /v1/jobs

POST/v1/jobs

Submit a fetch as a background job. Accepts the same body as POST /v1/fetch and returns 202 with a job id immediately. The browser work runs asynchronously under a separate concurrency budget, so it never blocks synchronous /v1/fetch traffic. Jobs are a short-lived convenience queue, not a durable crawl service: queued work and results can be lost during a process restart and results expire after roughly 24 hours.

curl -sS -X POST "https://api.betterfetch.co/v1/jobs" \
  -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "wait_until": "domcontentloaded",
    "timeout_ms": 60000
  }'

# → {"ok": true, "id": "a1b2c3d4-...", "status": "queued"}
GET/v1/jobs/{id}

Poll a job. Returns the current status (queued, running, done, failed) and the full result when complete. Jobs are scoped to the authenticated account.

curl -sS "https://api.betterfetch.co/v1/jobs/a1b2c3d4-..." \
  -H "Authorization: Bearer $BETTER_FETCH_API_KEY"

result contains the same FetchSuccess shape as /v1/fetch. One base credit is charged when the job is admitted; actual residential attempts are post-charged.

okboolean

true when the job was found and belongs to the authenticated account.

idstring

Job identifier (UUID).

statusstring

Current job state: queued, running, done, or failed.

resultobject | null

Full FetchSuccess payload (same shape as POST /v1/fetch) when status is done; otherwise null.

errorstring | null

Error message when status is failed; otherwise null.

created_atstring

ISO timestamp when the job was admitted.

started_atstring | null

ISO timestamp when the browser work began.

completed_atstring | null

ISO timestamp when the job reached done or failed.

expires_atstring

ISO timestamp after which the job result is cleaned up (24 hours after creation by default).

GET /v1/sessions

GET/v1/sessions

List active account-scoped browser sessions without exposing cookie values. Clear one with DELETE /v1/sessions/<id>.

Session names are account-scoped but canonicalized for backend routing: only letters and numbers form the durable key today. For example, shop-us, shop_us, and shopus target the same stored browser session.

curl -sS "https://api.betterfetch.co/v1/sessions" \
  -H "Authorization: Bearer $BETTER_FETCH_API_KEY"

curl -sS -X DELETE "https://api.betterfetch.co/v1/sessions/<id>" \
  -H "Authorization: Bearer $BETTER_FETCH_API_KEY"

GET /v1/health

GET/v1/health

Liveness check. No authentication required.

curl -sS "https://api.betterfetch.co/v1/health"
curl -sS "https://api.betterfetch.co/v1/health?geo=1&country=us"

{
  "ok": true,
  "version": "0.4.0",
  "browser": {
    "version": "146.0.7680.177.5",
    "platform": "linux-x64",
    "installed": true
  },
  "managed_proxy": {
    "enabled": true,
    "message": "Use proxy=auto for direct-first escalation or proxy=residential for every attempt."
  },
  "geo_emulation": {
    "ok": true,
    "status": "geo_emulation",
    "country": "us",
    "timezone": "America/New_York",
    "locale": "en-US",
    "egress_ip_changed": false,
    "message": "country sets browser timezone/locale defaults only"
  }
}

MCP tools

The remote MCP server lives at https://betterfetch.co/api/mcp. Claude-style OAuth connectors can sign in without handling a key; manual MCP clients can send Authorization: Bearer bf_.... Browser fetch tool calls are metered like REST fetch calls.

ToolContract
search_toolsFind a ready-made scraper or extractor in the live catalogue and return its schema, example inputs, and estimated credit cost.
run_toolRun one exact catalogue tool selected with search_tools, without loading every specialist tool into the default MCP surface.
fetch_urlFetch raw HTML or return clean readable text/Markdown. Better Fetch uses direct HTTP first, Chromium when rendering is needed, and optional residential escalation, with block, cache, transport, attempt, and routing metadata.
scrape_pageRetrieve one HTML page as a deterministic model-ready document: clean text, Markdown, metadata, canonical links, images, and retrieval diagnostics.
scrape_jsonFetch a JSON endpoint with browser-compatible headers and session reuse when needed, returning parsed JSON, fallback body_text, and response metadata.
screenshot_urlRender a page and capture a viewport or full-page PNG screenshot.
discover_apisLoad a page and capture the XHR/fetch calls it makes, with optional response previews and streamed values, to find internal APIs behind the page.
get_clearanceAttempt a Cloudflare challenge flow and return cf_clearance cookie metadata when the browser receives it, plus the reusable Better Fetch session.
get_datadome_cookieRender a DataDome-protected page and return datadome cookie metadata when the browser receives it, plus the reusable Better Fetch session.
get_usageCheck the connected Better Fetch account: plan, credits used this billing period, remaining quota, stored browser sessions, and reset time.
list_sessionsList active account-scoped browser sessions without exposing cookies, localStorage, or snapshot object paths.
clear_sessionClear a stored browser session through the backend, delete its portable snapshot, and make future requests with that session name start from a fresh profile key.

Tips

  • Every response includes ok. On failure you get { ok: false, error, message, status } — not body_bytes or timing_ms. When debugging with jq, always select ok and error first.
  • Use content_kind and content_type to route parser logic before inspecting body_text, html, or json.
  • Set include_html: false for JSON/body workflows when you only need body_text or json, especially with strategy: browser JSON mode.
  • Use cache_ttl_ms for short scraper bursts that repeat the exact same synchronous fetch payload. It is explicit, off by default, and best effort per API worker; cache_status reports miss, hit, coalesced, or bypass.
  • For simple page/body fetches, leave strategy as auto; Better Fetch uses direct HTTP and only moves to Chromium when the request asks for browser-only features or the fast path looks blocked.
  • Use strategy: browser with wait_selector, cookies, screenshots, or network capture when you specifically need rendered DOM behavior.
  • For JSON APIs, set extra_headers {"Accept":"application/json"}. strategy auto uses direct HTTP first, then falls back to the browser path if the response looks blocked.
  • Set strategy: browser when the target specifically requires in-page fetch/CORS semantics from a SPA origin; set strategy: http when you explicitly want raw HTTP and no browser-only features.
  • When using json_mode, include a Referer header pointing to the site's app/SPA origin (e.g. "https://app.example.com/"). Better Fetch navigates to that origin first so CORS and Referer match what the API expects. Without a Referer, the URL's own origin is used.
  • Large body responses (> 50 MB) are truncated in body-fetch strategies. Check body_truncated and body_bytes — reduce per_page or narrow the query if you need the complete body.
  • country sets representative browser locale/timezone defaults when geoip is true and explicit values are omitted. It changes egress IP only when proxy is auto or residential.
  • Check GET /v1/health?geo=1&country=us to see the country defaults Better Fetch will apply.
  • Pass a session for hard targets: it enables an account-scoped warm pooled context, encrypted portable cookie/localStorage snapshots, and a stable fingerprint.
  • Session names are canonicalized to letters and numbers for backend routing today; punctuation is ignored, so use clearly distinct alphanumeric names when you need separate sessions.
  • Reuse the same session to keep browser cookies/localStorage across machines; use return_cookies plus cookies when you want caller-managed cookie replay.
  • Use return_cf_clearance or return_datadome_cookie when you need a specific protection cookie; for DataDome, the Better Fetch session is usually the stronger reuse primitive because cookies can be bound to IP, fingerprint, and browser state.
  • Stored session limits are plan-based: Free 1, Starter 10, Pro 50, Scale 250; named sessions expire after 7 idle days.
  • Pro and Scale can use proxy: auto for cost-aware escalation: the first attempt stays on datacenter egress and blocked retries move to residential routing. Use proxy: residential only when every attempt must use a regional residential exit. Each residential attempt that actually occurs costs 25 credits.
  • Check blocked and block_reason to detect bot walls even when the target returns HTTP 200. The attempts field is greater than 1 when the service retried — retries cover both blocks and transient navigation timeouts (net::ERR_TIMED_OUT).
  • A browser fetch can return status 403 with blocked: false only when Better Fetch observed the challenge clear and the real rendered page replace it. A large generic 403 shell remains blocked. Treat blocked: false plus rendered content as success.
  • Leave user_agent, locale, and timezone unset unless required — they form part of a session's warm-context identity, so changing them splits the warm pool.
  • First requests after a deploy can be slower while Chromium starts; reused session requests are served from a warm pool.
  • Do not send raw proxy credentials. Select none, auto, or residential with the proxy field; Better Fetch owns provider credentials and reports proxy_used in the result.
  • Use POST /v1/jobs for long-running fetches or batch fan-out: it returns a job id immediately and runs the browser work asynchronously. Poll GET /v1/jobs/{id} for the result.
  • Jobs run under a separate concurrency budget from synchronous /v1/fetch, so they never block real-time requests.

This frontend page is the canonical API reference. The same content is also available as markdown for agents. Legacy API-served schema/reference endpoints may exist for compatibility, but they are not the source of truth.