Overview
Better Fetch retrieves difficult web pages, then lets you choose the output layer. Use /v1/fetchfor raw transport data and diagnostics; use /v1/scrape for clean text, Markdown, metadata, links, and images. Both share direct HTTP, stealth Chromium, browser geo-emulation, account-scoped sticky sessions, screenshots, network capture, or Cloudflare/DataDome cookie collection when issued. The base URL is https://api.betterfetch.co and endpoints are versioned under /v1.
Simple requests can complete over direct HTTP. When Chromium is needed, it runs in a real persistent browser profile—not incognito. Requests with a session reuse a warm, pooled context keyed to your account, session, and browser options. If a response looks blocked and country is set, the service can retry on a fresh browser identity and escalate to a headed browser. Egress changes only when proxy is auto or residential.
Authentication
All fetch and scrape requests require a bearer token. Create and revoke keys on the API keys page. Keep keys server-side — never in browser JavaScript, query strings, or shared logs.
Authorization: Bearer <your-api-key> Content-Type: application/json
Keys require an active subscription and are metered against your plan's monthly credit quota. Ordinary accepted retrieval costs 1 credit. Pro and Scale residential attempts cost 25 credits each when they actually occur. Stored browser sessions are also plan-limited and can be cleared from the dashboard.
POST /v1/scrape
/v1/scrapeRetrieve one HTML page through the same engine as fetch, then run a deterministic, network-free document normalizer. Ordinary retrieval costs one credit; actual residential attempts use the higher published rate. The response keeps retrieval diagnostics separate, so a target block remains visible under retrieval.blocked without confusing it with an API failure.
curl -sS -X POST "https://api.betterfetch.co/v1/scrape" \
-H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/article",
"strategy": "auto",
"proxy": "none",
"country": "au",
"session": "research",
"only_main_content": true,
"max_chars": 60000,
"include_raw_html": false
}'Document options
All fetch request fields also work here. These fields control only normalization and output size.
only_main_contentbooleandefault: truePrefer substantial main/article content and remove common navigation, header, footer, form, and sidebar chrome.
max_charsintegerdefault: 500000Maximum characters returned independently in text and Markdown. Range 1000-2000000; truncation flags tell you when the limit was reached.
max_linksintegerdefault: 500Maximum deduplicated HTTP(S) links. Range 0-5000.
max_imagesintegerdefault: 200Maximum deduplicated HTTP(S) images. Range 0-2000.
include_raw_htmlbooleandefault: falseInclude the retrieved HTML alongside the normalized document. Leave false for model and RAG workflows.
Document response
urlstringFinal target URL after redirects.
titlestringResolved document/page title.
textstringClean readable document text.
markdownstringClean content with headings, lists, quotes, and code structure preserved.
text_truncatedbooleanWhether text exceeded max_chars.
markdown_truncatedbooleanWhether Markdown exceeded max_chars.
word_countintegerWord count of the returned text.
credits_usedintegerCredits charged for the retrieval work that actually occurred.
linksarrayDeduplicated absolute links with visible labels when available.
imagesarrayDeduplicated absolute image URLs with alt text when available.
metadataobjectTitle, description, author, publication time, site, language, canonical URL, and JSON-LD types when present.
retrievalobjectTarget status, block verdict, attempts, transport, proxy/cache use, timing, content type, and body size from the underlying retrieval.
htmlstringPresent only when include_raw_html is true.
JSON, plain-text, binary, and empty targets return 415 unsupported_content. Use /v1/fetch for those. Try the signed-in scrape playgroundwithout copying an API key.
POST /v1/fetch
/v1/fetchFetch a URL through the browser and return rendered page data: target status, final URL, title, rendered HTML, headers, timing, block classification, and optionally the raw body, parsed JSON, captured network calls, or a screenshot. Unknown request fields are rejected with 400.
curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \
-H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"wait_until": "domcontentloaded",
"timeout_ms": 60000
}'Request fields
Only url is required. Everything else has a sensible default.
urlstringdefault: requiredPublic URL or domain to fetch. Bare domains default to HTTPS; explicit HTTP and HTTPS URLs are preserved.
wait_untilstringdefault: "load"Browser navigation state when the browser path is selected: load, domcontentloaded, networkidle, or commit.
wait_selectorstringCSS selector to wait for after navigation. Use this for pages that render content after the initial load.
scroll_selectorstringCSS selector to scroll into view after navigation. Use it to trigger lazy-loaded feeds, comments, and media before wait_ms and network capture.
wait_msnumberExtra fixed wait after navigation, in milliseconds. Capped by timeout_ms. Use only when there is no reliable selector.
timeout_msnumberdefault: 90000Navigation and selector timeout in milliseconds. Maximum 240000. The default is deliberately generous so the first call after a cold browser start has room to finish.
cache_ttl_msnumberdefault: 0Optional short-lived in-process response cache TTL for identical synchronous fetch payloads. Range 0-60000. Hits are best effort per API worker; the cache key ignores cache_ttl_ms itself.
strategy"auto" | "http" | "browser"default: "auto"Execution strategy. auto uses direct HTTP for simple body/JSON fetches and Chromium when the payload asks for browser-only features or the fast path looks blocked. http returns the raw HTTP response without browser-only features. browser forces the rendered browser path.
return_response_textbooleandefault: falseInclude the raw response body in body_text. JSON responses include it automatically. When set with an API-shaped URL (a path with no .html/.php-style extension), this also participates in auto strategy selection and can auto-select browser JSON mode — unless screenshot, full_page, or wait_selector is also set, which keeps the request on the rendered-page path. Site-root URLs with no path (https://example.com) are always treated as pages, never APIs.
include_htmlbooleandefault: trueInclude rendered/raw HTML in html. Set false for JSON/body workflows to reduce response size and skip origin DOM serialization in browser JSON mode.
return_cf_clearancebooleandefault: falseAttempt to collect Cloudflare cf_clearance token data and return it when the browser receives that cookie. When false, cookies are not read or returned.
return_datadome_cookiebooleandefault: falseAttempt to collect DataDome datadome cookie data and return it when the rendered browser session receives that cookie. When false, DataDome cookie fields are not read or returned.
return_cookiesbooleandefault: falseReturn storage-ready cookies visible to the final rendered page in cookies.
cookiesarraydefault: []Preload cookies into the browser context before navigation. Each cookie needs name and value plus either url or domain; path defaults to / when domain is used.
capture_networkbooleandefault: falseCapture matching browser network calls and return them in network. Defaults to XHR/fetch only — useful for API discovery and debugging.
network_resource_typesstring[]default: ["xhr","fetch"]Playwright resource types to capture, e.g. xhr, fetch, document, script, or websocket. Keep this narrow for most workloads.
network_include_bodiesbooleandefault: trueInclude capped response bodies for captured network responses.
network_include_headersbooleandefault: falseInclude request and response headers for captured entries. Off by default because headers can contain cookies, bearer tokens, or other secrets.
network_max_entriesnumberdefault: 100Maximum matching network entries to return. Range 1–500.
network_max_body_bytesnumberdefault: 262144Maximum bytes kept from each captured response body. Range 0–1048576.
network_capture_streamsbooleandefault: falseWhen capture_network is true, capture streamed fetch/XHR chunks, EventSource messages, and WebSocket messages in network_streams.
network_stream_max_eventsnumberdefault: 100Maximum streamed network values to return. Range 1–500.
network_stream_max_value_bytesnumberdefault: 65536Maximum bytes kept from each streamed network value. Range 0–262144.
screenshotbooleandefault: falseInclude a PNG screenshot encoded as base64. Works together with return_response_text: requesting a screenshot keeps the request on the rendered-page path, so both come back. Only explicit json_mode: true disables screenshots. Screenshots can make responses much larger, so request them only when needed.
full_pagebooleandefault: falseCapture the full scrollable page when screenshot is true.
countrystringTwo-letter country code for browser geo-emulation, e.g. us, gb, de, au, ca. When geoip is true and locale/timezone are unset, Better Fetch applies representative browser defaults for that country. This does not change network egress IP.
proxy"none" | "auto" | "residential"default: "none"Pro/Scale network routing. none uses Better Fetch datacenter egress. auto starts direct and escalates blocked retries through residential egress. residential routes every attempt through residential egress. When country is also set, proxied attempts use that country for the exit and browser identity. Each residential attempt that occurs costs 25 credits.
sessionstringAccount-scoped browser identity key. Reuse it to keep the same warm browser context, fingerprint, cookies, localStorage, and encrypted portable snapshot. Only letters and numbers form the canonical key today, so punctuation is ignored.
geoipbooleandefault: true when country is setWhen country is set, apply country-derived browser timezone/locale defaults unless explicit locale/timezone values are supplied. This does not spoof WebRTC IP or change network egress IP.
localestringdefault: automaticBrowser locale, e.g. en-GB. Overrides the country-derived locale.
timezonestringdefault: automaticBrowser timezone, e.g. Europe/London. Overrides the country-derived timezone.
user_agentstringdefault: browser defaultCustom user agent applied to the browser context. Usually leave unset — it forms part of a session's warm-context identity.
extra_headersobjectdefault: {}Additional HTTP headers applied to the direct HTTP request or inside the browser context.
humanizebooleandefault: autoHuman-like mouse, keyboard, and scroll behavior for browser fetches. Defaults to true for rendered browser pages and false for direct/API calls. Set explicitly to override.
json_modebooleandefault: autoFetch via an in-page browser fetch() call instead of a top-level browser navigation. Sends Sec-Fetch-Mode: cors and a natural Referer from the URL's origin. Auto-detected from Accept: application/json or from return_response_text on an API-shaped URL — but never when screenshot, full_page, or wait_selector is set (those signal a rendered page), and never for site-root URLs with no path. An explicit true/false always wins; note screenshots are unavailable in json_mode.
Response fields
ok: true means Better Fetch completed the browser request — check status and blocked / block_reasonfor the target's verdict.
okbooleantrue when Better Fetch completed the request — not whether the target accepted it. Check status and blocked for the target's verdict.
statusnumber | nullHTTP status from the target response (navigation or in-page fetch). On challenge-fronted sites this can be stale: after Better Fetch observes a challenge clear, the solved page can render while status keeps the original 403/503. Use blocked, not status alone, as the browser-fetch outcome signal.
final_urlstringFinal target URL after redirects.
titlestringPage title after rendering, or the parsed title for direct HTTP HTML responses.
htmlstringRendered DOM HTML from the browser, or raw HTML from the direct HTTP transport. Empty when include_html is false.
body_textstring | nullRaw response body when requested, when the target response is JSON, or when strategy=http. Capped at 50 MB — see body_truncated.
body_bytesnumberTotal response body size in bytes when measured by a body-fetch strategy. When body_truncated is true, this is larger than body_text.length.
body_truncatedbooleanTrue when the response body exceeded the 50 MB transfer cap and was truncated. Reduce per_page or narrow the query if you need the complete body.
content_typestringNormalized response Content-Type media type without parameters, for example text/html or application/json. Empty when unavailable.
content_kind"html" | "json" | "text" | "binary" | "empty" | "unknown"Best-effort response body category for routing parser logic.
json_parse_okbooleanWhether body_text parsed as JSON.
jsonany | nullParsed JSON payload when json_parse_ok is true; otherwise null.
headersobjectResponse headers from the target navigation response (string values).
screenshot_b64string | nullBase64-encoded PNG when screenshot is true; otherwise null. Always null in json_mode — in-page fetch() never renders the target as a page, which is why screenshot: true suppresses json_mode auto-detection.
cf_clearancestring | nullCloudflare cf_clearance token value when return_cf_clearance is true and the target issued it.
cf_clearance_cookieobject | nullStorage-ready cookie metadata (name, value, domain, path, expires, httpOnly, secure, sameSite) when requested and present.
cf_clearance_sessionstring | nullThe session that produced the clearance result. Blocked retries may rotate to a fresh session before the final result.
datadome_cookiestring | nullDataDome datadome cookie value when return_datadome_cookie is true and the target issued it. Otherwise null or omitted.
datadome_cookie_detailobject | nullStorage-ready datadome cookie metadata when requested and present. Otherwise null or omitted.
datadome_sessionstring | nullThe session that produced the DataDome cookie result. Blocked retries may rotate to a fresh session before the final result.
datadome_detectedbooleanPresent when return_datadome_cookie is true. true when the rendered page or response showed DataDome signals, even if no datadome cookie was returned.
cookiesarrayStorage-ready cookies visible to the final rendered page when return_cookies is true. Otherwise omitted.
networkarrayCaptured browser network entries when capture_network is true. Otherwise omitted.
network_streamsarrayCaptured streamed fetch/XHR chunks, EventSource messages, and WebSocket messages when network_capture_streams is true. Otherwise omitted.
blockedbooleantrue when the response looks like a bot wall or unsolved challenge — even when the target returns HTTP 200. The inverse also holds: a 403/503 whose rendered body is substantial real content with no live interstitial is a solved challenge and reports blocked: false. Trust blocked over status in both directions.
block_reason"none" | "http_401" | "http_403" | "http_429" | "http_503" | "cloudflare" | "datadome" | "captcha" | "block_title" | "challenge_interstitial"Stable reason for the blocked verdict. none means the response is not classified as blocked.
headedbooleanWhether the attempt that produced this result ran a headed browser (set on escalated retries).
pooledbooleantrue when served from a warm pooled context (session requests); false for sessionless ephemeral contexts.
transport"http" | "browser"Execution transport that produced the result.
cache_status"bypass" | "miss" | "hit" | "coalesced"Synchronous fetch cache status. bypass means no cache was requested or enabled; miss means the target was fetched; hit means a completed prior result was reused; coalesced means an identical in-flight fetch was shared.
attemptsnumberTotal attempts including the first. Greater than 1 means the service retried on a fresh session — after a block or a transient navigation timeout.
proxy_usedbooleanWhether the final attempt used managed residential egress.
proxy_attemptsintegerResidential attempts that actually occurred. Cache hits and coalesced followers have no billable proxy attempts.
credits_usedintegerTotal charged credits. Ordinary retrieval is 1; each actual residential attempt is 25 credits.
remaining_creditsintegerRemaining period credits after a residential post-charge, when available.
timing_msnumberTime spent inside the fetch request (final attempt).
Examples
Fetch rendered HTML
The basic call: navigate with a real browser and return the rendered DOM.
curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \
-H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"wait_until": "domcontentloaded",
"timeout_ms": 60000
}' | jq '.status, .title, .final_url'Fetch with country browser identity
Use country when browser locale/timezone should match a country, and session when several requests should share one browser profile.
curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \
-H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"country": "gb",
"session": "examplegb",
"wait_until": "domcontentloaded",
"timeout_ms": 60000
}'Session names are scoped to your Better Fetch account; other accounts using the same name get isolated browser profiles and encrypted snapshots. Use alphanumeric names when you need distinct sessions; today example-gb, example_gb, and examplegb point at the same session.
Fetch a JSON API (quick start)
Minimal JSON call. With strategy auto, Better Fetch uses the direct HTTP fast path for this shape and returns parsed JSON plus body_bytes.
curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \
-H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://jsonplaceholder.typicode.com/todos/1",
"include_html": false,
"extra_headers": { "Accept": "application/json" }
}' | jq '{ ok, status, transport, cache_status, content_type, content_kind, json_parse_ok, body_bytes, timing_ms }'If jq shows null for every field, the call failed — inspect ok, error, and message first (see Tips). Keys must start with bf_ and come from your keys page.
Fetch a JSON API
For harder JSON APIs, pass the SPA Referer/Origin plus a stable session. Add country only when browser locale/timezone should match the target. auto starts with the fast HTTP path and falls back to the browser path if the response looks blocked; set strategy: browser to force the old in-page fetch behavior.
curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \
-H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://partner-api.example.com/projections?per_page=500",
"country": "us",
"session": "exampleus",
"return_response_text": true,
"include_html": false,
"extra_headers": {
"Accept": "application/json",
"Referer": "https://app.example.com/",
"Origin": "https://app.example.com"
},
"timeout_ms": 60000
}' | jq '{ ok, status, transport, cache_status, content_type, content_kind, json_parse_ok, body_bytes, body_truncated, timing_ms, blocked, block_reason }'strategy auto uses the HTTP fast path for JSON/API body requests, then escalates to the browser if that response looks blocked. Set strategy: browser when the target specifically requires browser CORS/fetch semantics, or strategy: http when you explicitly want raw HTTP only.
Wait for rendered content
For client-rendered pages, wait for a selector instead of adding a long fixed delay.
curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \
-H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://app.example.com/items/123",
"wait_until": "domcontentloaded",
"wait_selector": "#content",
"timeout_ms": 90000
}'Capture network calls
Capture the XHR/fetch calls a page makes while rendering — the fastest way to discover a site's internal APIs.
curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \
-H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://app.example.com/items/123",
"wait_until": "networkidle",
"timeout_ms": 90000,
"capture_network": true,
"network_max_entries": 50
}' | jq '.network[] | { method, url, status, json }'Enable network_include_headers only when you need it — headers can contain credentials.
Capture streamed values
Capture values delivered through streaming fetch/XHR, EventSource, or WebSocket while the page is open.
curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \
-H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://app.example.com/live",
"wait_until": "domcontentloaded",
"wait_ms": 10000,
"timeout_ms": 90000,
"capture_network": true,
"network_capture_streams": true,
"network_resource_types": ["fetch", "xhr", "eventsource", "websocket"],
"network_stream_max_events": 100,
"network_stream_max_value_bytes": 65536
}' | jq '.network_streams[] | { source, event_type, url, value_text, json }'Stream capture is opt-in because it instruments page fetch/XHR/EventSource/WebSocket APIs and can produce large responses.
Capture a screenshot
Return a base64-encoded PNG of the rendered page.
curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \
-H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"screenshot": true,
"full_page": true,
"wait_until": "domcontentloaded"
}' | jq -r '.screenshot_b64'Collect a Cloudflare clearance token when issued
Attempt the page flow and return the cf_clearance cookie when the target issues it, with storage-ready metadata.
curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \
-H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://www.example.com/",
"country": "us",
"session": "clearanceus",
"wait_until": "domcontentloaded",
"timeout_ms": 90000,
"return_cf_clearance": true
}' | jq '{ status, blocked, block_reason, cf_clearance, cf_clearance_cookie }'cf_clearance is null when the target doesn't issue the cookie or the challenge remains unsolved. Store cf_clearance_session too — retries may rotate sessions.
Collect a DataDome cookie when issued
Attempt the page flow and return the datadome cookie when the target issues it, with storage-ready metadata.
curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \
-H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://www.example.com/",
"country": "us",
"session": "datadomeus",
"wait_until": "domcontentloaded",
"timeout_ms": 90000,
"return_datadome_cookie": true
}' | jq '{ status, blocked, block_reason, datadome_detected, datadome_cookie, datadome_cookie_detail }'datadome_cookie is null when the target doesn't issue the cookie or the challenge remains unsolved. Reusing the Better Fetch session is usually more reliable than replaying the raw cookie elsewhere.
Submit an async job
When you don't want to hold a connection open — or want to fan out many URLs — submit a fetch as a background job. Returns 202 immediately; poll until status is done or failed.
curl -sS -X POST "https://api.betterfetch.co/v1/jobs" \
-H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"wait_until": "domcontentloaded",
"timeout_ms": 60000
}'
# → {"ok": true, "id": "a1b2c3d4-...", "status": "queued"}
curl -sS "https://api.betterfetch.co/v1/jobs/a1b2c3d4-..." \
-H "Authorization: Bearer $BETTER_FETCH_API_KEY"
# → {"ok": true, "id": "...", "status": "done", "result": {...}, "error": null, ...}Jobs run under a separate concurrency budget so they never block synchronous /v1/fetch. One base credit is charged when admitted; actual residential attempts are post-charged at the published rate.
Errors
Every non-2xx response uses a single JSON envelope. Switch on the stable error code, not the message.
{ "ok": false, "error": "unauthorized", "message": "invalid or missing bearer token", "status": 401 }| Status | Code | Meaning |
|---|---|---|
| 400 | bad_request | Invalid JSON, unknown request field, missing or non-HTTP url, or an out-of-range parameter. |
| 401 | unauthorized | Missing, incorrect, or revoked bearer token. |
| 402 | payment_required | Valid key but no active subscription. |
| 402 | residential_plan_required | proxy auto/residential was requested from a Free or Starter account. Upgrade to Pro or Scale; the rejected request is not metered. |
| 415 | unsupported_content | POST /v1/scrape retrieved JSON, plain text, binary, or empty content instead of HTML. Use /v1/fetch for non-HTML targets. |
| 429 | quota_exceeded | Monthly credit quota exhausted; resets at the next billing cycle. Ordinary accepted retrieval costs 1 credit; actual residential attempts use the published higher rate. |
| 429 | session_limit_exceeded | Stored browser session limit reached; clear a session from the dashboard or upgrade. |
| 502 | fetch_failed | Browser launch, navigation, or target fetch failed. The message includes the underlying detail. In json_mode, Failed to fetch often means the API's CORS policy rejected the in-page call — set Referer to the site's app origin or try json_mode: false. |
| 504 | timeout | Request timed out at the API layer. |
Target errors are different from API errors: if Better Fetch returns HTTP 200 and the JSON contains "status": 403 or "blocked": true, the API worked and the target denied the browser request; use block_reason for the category.
Guides
High-volume same-site scraping
When you fetch many URLs on one site in a single run — scanning dozens of markets on a bookmaker, paging through a listing, etc. — these habits avoid most blocks and connection timeouts.
- Reuse one session per site for the whole run (and across runs) — e.g. session: "skybet". A warm session keeps a stable fingerprint and persisted cookies/localStorage that mark you as a returning visitor. Do not generate a fresh session name per URL or per run: each distinct name — and each distinct locale/timezone/user_agent value — is a separate stored session that counts against your plan limit and starts cold.
- Use country for coherent browser identity defaults (gb for UK locale/timezone, au for Australian locale/timezone, etc.). It changes network egress only when proxy is auto or residential.
- Leave user_agent, locale, and timezone unset unless the integration requires them, so country can apply coherent defaults. If you do set them, keep them byte-identical across every call for that session — changing them splits the warm pool into separate cold contexts.
- Pace the run: keep concurrency to a few in-flight requests and add a small jittered delay (1–3s) between calls. A sub-second burst of dozens of requests from one IP looks robotic and trips bot detection that can then poison the whole session.
- Prefer fast waits on browser pages: wait_until: "domcontentloaded" plus wait_selector over networkidle or long fixed waits. For JSON endpoints, set extra_headers {"Accept":"application/json"}; auto uses direct HTTP first, skips humanization, and returns parsed JSON faster.
- For JSON polling, use cache_ttl_ms for short identical bursts and cache longer-lived responses in your application. Reuse one session, shrink query params (per_page, include=), and poll only as often as the data actually changes.
- Handle responses defensively in your loop. If blocked is true, inspect block_reason, then pause and back off (5–10s) before continuing. Retry an individual 502 fetch_failed after a short backoff — sustained failures mean the target is pushing back. Keep timeout_ms moderate (45–60s).
Marketplace tools
Ready-made tools built on the Better Fetch engine — structured extraction, monitoring, screenshots, and more — that you can run without writing fetch code yourself. Browse them at https://betterfetch.co/tools.
- Each tool page documents its input/output schema, examples, and a playground to try it against your account.
- Call search_tools from Claude, ChatGPT, Codex, or another MCP client to find the best live tool, then pass its exact name and input schema to run_tool. The catalogue stays searchable without flooding the default MCP surface.
- Or call them over REST: POST https://betterfetch.co/api/tools/{name}/run with Authorization: Bearer bf_... and a JSON body of {"input": {...}}.
- Tool runs are metered against your plan like direct fetch calls; each tool lists a ~credits-per-run estimate.
POST /v1/jobs
/v1/jobsSubmit a fetch as a background job. Accepts the same body as POST /v1/fetch and returns 202 with a job id immediately. The browser work runs asynchronously under a separate concurrency budget, so it never blocks synchronous /v1/fetch traffic. Jobs are a short-lived convenience queue, not a durable crawl service: queued work and results can be lost during a process restart and results expire after roughly 24 hours.
curl -sS -X POST "https://api.betterfetch.co/v1/jobs" \
-H "Authorization: Bearer $BETTER_FETCH_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"wait_until": "domcontentloaded",
"timeout_ms": 60000
}'
# → {"ok": true, "id": "a1b2c3d4-...", "status": "queued"}/v1/jobs/{id}Poll a job. Returns the current status (queued, running, done, failed) and the full result when complete. Jobs are scoped to the authenticated account.
curl -sS "https://api.betterfetch.co/v1/jobs/a1b2c3d4-..." \ -H "Authorization: Bearer $BETTER_FETCH_API_KEY"
result contains the same FetchSuccess shape as /v1/fetch. One base credit is charged when the job is admitted; actual residential attempts are post-charged.
okbooleantrue when the job was found and belongs to the authenticated account.
idstringJob identifier (UUID).
statusstringCurrent job state: queued, running, done, or failed.
resultobject | nullFull FetchSuccess payload (same shape as POST /v1/fetch) when status is done; otherwise null.
errorstring | nullError message when status is failed; otherwise null.
created_atstringISO timestamp when the job was admitted.
started_atstring | nullISO timestamp when the browser work began.
completed_atstring | nullISO timestamp when the job reached done or failed.
expires_atstringISO timestamp after which the job result is cleaned up (24 hours after creation by default).
GET /v1/sessions
/v1/sessionsList active account-scoped browser sessions without exposing cookie values. Clear one with DELETE /v1/sessions/<id>.
Session names are account-scoped but canonicalized for backend routing: only letters and numbers form the durable key today. For example, shop-us, shop_us, and shopus target the same stored browser session.
curl -sS "https://api.betterfetch.co/v1/sessions" \ -H "Authorization: Bearer $BETTER_FETCH_API_KEY" curl -sS -X DELETE "https://api.betterfetch.co/v1/sessions/<id>" \ -H "Authorization: Bearer $BETTER_FETCH_API_KEY"
GET /v1/health
/v1/healthLiveness check. No authentication required.
curl -sS "https://api.betterfetch.co/v1/health"
curl -sS "https://api.betterfetch.co/v1/health?geo=1&country=us"
{
"ok": true,
"version": "0.4.0",
"browser": {
"version": "146.0.7680.177.5",
"platform": "linux-x64",
"installed": true
},
"managed_proxy": {
"enabled": true,
"message": "Use proxy=auto for direct-first escalation or proxy=residential for every attempt."
},
"geo_emulation": {
"ok": true,
"status": "geo_emulation",
"country": "us",
"timezone": "America/New_York",
"locale": "en-US",
"egress_ip_changed": false,
"message": "country sets browser timezone/locale defaults only"
}
}MCP tools
The remote MCP server lives at https://betterfetch.co/api/mcp. Claude-style OAuth connectors can sign in without handling a key; manual MCP clients can send Authorization: Bearer bf_.... Browser fetch tool calls are metered like REST fetch calls.
| Tool | Contract |
|---|---|
| search_tools | Find a ready-made scraper or extractor in the live catalogue and return its schema, example inputs, and estimated credit cost. |
| run_tool | Run one exact catalogue tool selected with search_tools, without loading every specialist tool into the default MCP surface. |
| fetch_url | Fetch raw HTML or return clean readable text/Markdown. Better Fetch uses direct HTTP first, Chromium when rendering is needed, and optional residential escalation, with block, cache, transport, attempt, and routing metadata. |
| scrape_page | Retrieve one HTML page as a deterministic model-ready document: clean text, Markdown, metadata, canonical links, images, and retrieval diagnostics. |
| scrape_json | Fetch a JSON endpoint with browser-compatible headers and session reuse when needed, returning parsed JSON, fallback body_text, and response metadata. |
| screenshot_url | Render a page and capture a viewport or full-page PNG screenshot. |
| discover_apis | Load a page and capture the XHR/fetch calls it makes, with optional response previews and streamed values, to find internal APIs behind the page. |
| get_clearance | Attempt a Cloudflare challenge flow and return cf_clearance cookie metadata when the browser receives it, plus the reusable Better Fetch session. |
| get_datadome_cookie | Render a DataDome-protected page and return datadome cookie metadata when the browser receives it, plus the reusable Better Fetch session. |
| get_usage | Check the connected Better Fetch account: plan, credits used this billing period, remaining quota, stored browser sessions, and reset time. |
| list_sessions | List active account-scoped browser sessions without exposing cookies, localStorage, or snapshot object paths. |
| clear_session | Clear a stored browser session through the backend, delete its portable snapshot, and make future requests with that session name start from a fresh profile key. |
Tips
- Every response includes ok. On failure you get { ok: false, error, message, status } — not body_bytes or timing_ms. When debugging with jq, always select ok and error first.
- Use content_kind and content_type to route parser logic before inspecting body_text, html, or json.
- Set include_html: false for JSON/body workflows when you only need body_text or json, especially with strategy: browser JSON mode.
- Use cache_ttl_ms for short scraper bursts that repeat the exact same synchronous fetch payload. It is explicit, off by default, and best effort per API worker; cache_status reports miss, hit, coalesced, or bypass.
- For simple page/body fetches, leave strategy as auto; Better Fetch uses direct HTTP and only moves to Chromium when the request asks for browser-only features or the fast path looks blocked.
- Use strategy: browser with wait_selector, cookies, screenshots, or network capture when you specifically need rendered DOM behavior.
- For JSON APIs, set extra_headers {"Accept":"application/json"}. strategy auto uses direct HTTP first, then falls back to the browser path if the response looks blocked.
- Set strategy: browser when the target specifically requires in-page fetch/CORS semantics from a SPA origin; set strategy: http when you explicitly want raw HTTP and no browser-only features.
- When using json_mode, include a Referer header pointing to the site's app/SPA origin (e.g. "https://app.example.com/"). Better Fetch navigates to that origin first so CORS and Referer match what the API expects. Without a Referer, the URL's own origin is used.
- Large body responses (> 50 MB) are truncated in body-fetch strategies. Check body_truncated and body_bytes — reduce per_page or narrow the query if you need the complete body.
- country sets representative browser locale/timezone defaults when geoip is true and explicit values are omitted. It changes egress IP only when proxy is auto or residential.
- Check GET /v1/health?geo=1&country=us to see the country defaults Better Fetch will apply.
- Pass a session for hard targets: it enables an account-scoped warm pooled context, encrypted portable cookie/localStorage snapshots, and a stable fingerprint.
- Session names are canonicalized to letters and numbers for backend routing today; punctuation is ignored, so use clearly distinct alphanumeric names when you need separate sessions.
- Reuse the same session to keep browser cookies/localStorage across machines; use return_cookies plus cookies when you want caller-managed cookie replay.
- Use return_cf_clearance or return_datadome_cookie when you need a specific protection cookie; for DataDome, the Better Fetch session is usually the stronger reuse primitive because cookies can be bound to IP, fingerprint, and browser state.
- Stored session limits are plan-based: Free 1, Starter 10, Pro 50, Scale 250; named sessions expire after 7 idle days.
- Pro and Scale can use proxy: auto for cost-aware escalation: the first attempt stays on datacenter egress and blocked retries move to residential routing. Use proxy: residential only when every attempt must use a regional residential exit. Each residential attempt that actually occurs costs 25 credits.
- Check blocked and block_reason to detect bot walls even when the target returns HTTP 200. The attempts field is greater than 1 when the service retried — retries cover both blocks and transient navigation timeouts (net::ERR_TIMED_OUT).
- A browser fetch can return status 403 with blocked: false only when Better Fetch observed the challenge clear and the real rendered page replace it. A large generic 403 shell remains blocked. Treat blocked: false plus rendered content as success.
- Leave user_agent, locale, and timezone unset unless required — they form part of a session's warm-context identity, so changing them splits the warm pool.
- First requests after a deploy can be slower while Chromium starts; reused session requests are served from a warm pool.
- Do not send raw proxy credentials. Select none, auto, or residential with the proxy field; Better Fetch owns provider credentials and reports proxy_used in the result.
- Use POST /v1/jobs for long-running fetches or batch fan-out: it returns a job id immediately and runs the browser work asynchronously. Poll GET /v1/jobs/{id} for the result.
- Jobs run under a separate concurrency budget from synchronous /v1/fetch, so they never block real-time requests.
This frontend page is the canonical API reference. The same content is also available as markdown for agents. Legacy API-served schema/reference endpoints may exist for compatibility, but they are not the source of truth.
Social data APIs
Better Fetch also publishes platform-specific tools for public social profiles, posts, videos, comments, transcripts, search, ads, commerce, and creator intelligence. The social reference keeps live runner contracts separate from in-development coverage targets.
Browse the platform-by-platform social data reference