# Better Fetch — API reference > Retrieve raw web responses through direct HTTP or stealth Chromium, or normalize HTML into clean text and Markdown. Base URL https://api.betterfetch.co, endpoints under /v1. This file is generated from the same canonical frontend docs source as https://betterfetch.co/docs. ## Authentication All fetch and scrape requests require a bearer token: `Authorization: Bearer `. Create and revoke keys at https://betterfetch.co/keys. Ordinary accepted retrieval costs 1 credit. Pro and Scale unlock residential routing; each residential attempt that actually occurs costs 25 credits. Responses report `credits_used` and `proxy_attempts`. Stored browser sessions are scoped to the authenticated account, have plan limits, and sync encrypted cookie/localStorage snapshots for multi-machine reuse. ## POST /v1/fetch Fetch a URL through the browser and return rendered page data: target status, final URL, title, rendered HTML, headers, timing, block classification, and optionally the raw body, parsed JSON, captured network calls, or a screenshot. Only `url` is required; unknown request fields are rejected with 400. ```bash curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \ -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com", "wait_until": "domcontentloaded", "timeout_ms": 60000}' ``` ## Request fields - `url` (string, default required): Public URL or domain to fetch. Bare domains default to HTTPS; explicit HTTP and HTTPS URLs are preserved. - `wait_until` (string, default "load"): Browser navigation state when the browser path is selected: load, domcontentloaded, networkidle, or commit. - `wait_selector` (string): CSS selector to wait for after navigation. Use this for pages that render content after the initial load. - `scroll_selector` (string): CSS selector to scroll into view after navigation. Use it to trigger lazy-loaded feeds, comments, and media before wait_ms and network capture. - `wait_ms` (number): Extra fixed wait after navigation, in milliseconds. Capped by timeout_ms. Use only when there is no reliable selector. - `timeout_ms` (number, default 90000): Navigation and selector timeout in milliseconds. Maximum 240000. The default is deliberately generous so the first call after a cold browser start has room to finish. - `cache_ttl_ms` (number, default 0): Optional short-lived in-process response cache TTL for identical synchronous fetch payloads. Range 0-60000. Hits are best effort per API worker; the cache key ignores cache_ttl_ms itself. - `strategy` ("auto" | "http" | "browser", default "auto"): Execution strategy. auto uses direct HTTP for simple body/JSON fetches and Chromium when the payload asks for browser-only features or the fast path looks blocked. http returns the raw HTTP response without browser-only features. browser forces the rendered browser path. - `return_response_text` (boolean, default false): Include the raw response body in body_text. JSON responses include it automatically. When set with an API-shaped URL (a path with no .html/.php-style extension), this also participates in auto strategy selection and can auto-select browser JSON mode — unless screenshot, full_page, or wait_selector is also set, which keeps the request on the rendered-page path. Site-root URLs with no path (https://example.com) are always treated as pages, never APIs. - `include_html` (boolean, default true): Include rendered/raw HTML in html. Set false for JSON/body workflows to reduce response size and skip origin DOM serialization in browser JSON mode. - `return_cf_clearance` (boolean, default false): Attempt to collect Cloudflare cf_clearance token data and return it when the browser receives that cookie. When false, cookies are not read or returned. - `return_datadome_cookie` (boolean, default false): Attempt to collect DataDome datadome cookie data and return it when the rendered browser session receives that cookie. When false, DataDome cookie fields are not read or returned. - `return_cookies` (boolean, default false): Return storage-ready cookies visible to the final rendered page in cookies. - `cookies` (array, default []): Preload cookies into the browser context before navigation. Each cookie needs name and value plus either url or domain; path defaults to / when domain is used. - `capture_network` (boolean, default false): Capture matching browser network calls and return them in network. Defaults to XHR/fetch only — useful for API discovery and debugging. - `network_resource_types` (string[], default ["xhr","fetch"]): Playwright resource types to capture, e.g. xhr, fetch, document, script, or websocket. Keep this narrow for most workloads. - `network_include_bodies` (boolean, default true): Include capped response bodies for captured network responses. - `network_include_headers` (boolean, default false): Include request and response headers for captured entries. Off by default because headers can contain cookies, bearer tokens, or other secrets. - `network_max_entries` (number, default 100): Maximum matching network entries to return. Range 1–500. - `network_max_body_bytes` (number, default 262144): Maximum bytes kept from each captured response body. Range 0–1048576. - `network_capture_streams` (boolean, default false): When capture_network is true, capture streamed fetch/XHR chunks, EventSource messages, and WebSocket messages in network_streams. - `network_stream_max_events` (number, default 100): Maximum streamed network values to return. Range 1–500. - `network_stream_max_value_bytes` (number, default 65536): Maximum bytes kept from each streamed network value. Range 0–262144. - `screenshot` (boolean, default false): Include a PNG screenshot encoded as base64. Works together with return_response_text: requesting a screenshot keeps the request on the rendered-page path, so both come back. Only explicit json_mode: true disables screenshots. Screenshots can make responses much larger, so request them only when needed. - `full_page` (boolean, default false): Capture the full scrollable page when screenshot is true. - `country` (string): Two-letter country code for browser geo-emulation, e.g. us, gb, de, au, ca. When geoip is true and locale/timezone are unset, Better Fetch applies representative browser defaults for that country. This does not change network egress IP. - `proxy` ("none" | "auto" | "residential", default "none"): Pro/Scale network routing. none uses Better Fetch datacenter egress. auto starts direct and escalates blocked retries through residential egress. residential routes every attempt through residential egress. When country is also set, proxied attempts use that country for the exit and browser identity. Each residential attempt that occurs costs 25 credits. - `session` (string): Account-scoped browser identity key. Reuse it to keep the same warm browser context, fingerprint, cookies, localStorage, and encrypted portable snapshot. Only letters and numbers form the canonical key today, so punctuation is ignored. - `geoip` (boolean, default true when country is set): When country is set, apply country-derived browser timezone/locale defaults unless explicit locale/timezone values are supplied. This does not spoof WebRTC IP or change network egress IP. - `locale` (string, default automatic): Browser locale, e.g. en-GB. Overrides the country-derived locale. - `timezone` (string, default automatic): Browser timezone, e.g. Europe/London. Overrides the country-derived timezone. - `user_agent` (string, default browser default): Custom user agent applied to the browser context. Usually leave unset — it forms part of a session's warm-context identity. - `extra_headers` (object, default {}): Additional HTTP headers applied to the direct HTTP request or inside the browser context. - `humanize` (boolean, default auto): Human-like mouse, keyboard, and scroll behavior for browser fetches. Defaults to true for rendered browser pages and false for direct/API calls. Set explicitly to override. - `json_mode` (boolean, default auto): Fetch via an in-page browser fetch() call instead of a top-level browser navigation. Sends Sec-Fetch-Mode: cors and a natural Referer from the URL's origin. Auto-detected from Accept: application/json or from return_response_text on an API-shaped URL — but never when screenshot, full_page, or wait_selector is set (those signal a rendered page), and never for site-root URLs with no path. An explicit true/false always wins; note screenshots are unavailable in json_mode. ## Response fields - `ok` (boolean): true when Better Fetch completed the request — not whether the target accepted it. Check status and blocked for the target's verdict. - `status` (number | null): HTTP status from the target response (navigation or in-page fetch). On challenge-fronted sites this can be stale: after Better Fetch observes a challenge clear, the solved page can render while status keeps the original 403/503. Use blocked, not status alone, as the browser-fetch outcome signal. - `final_url` (string): Final target URL after redirects. - `title` (string): Page title after rendering, or the parsed title for direct HTTP HTML responses. - `html` (string): Rendered DOM HTML from the browser, or raw HTML from the direct HTTP transport. Empty when include_html is false. - `body_text` (string | null): Raw response body when requested, when the target response is JSON, or when strategy=http. Capped at 50 MB — see body_truncated. - `body_bytes` (number): Total response body size in bytes when measured by a body-fetch strategy. When body_truncated is true, this is larger than body_text.length. - `body_truncated` (boolean): True when the response body exceeded the 50 MB transfer cap and was truncated. Reduce per_page or narrow the query if you need the complete body. - `content_type` (string): Normalized response Content-Type media type without parameters, for example text/html or application/json. Empty when unavailable. - `content_kind` ("html" | "json" | "text" | "binary" | "empty" | "unknown"): Best-effort response body category for routing parser logic. - `json_parse_ok` (boolean): Whether body_text parsed as JSON. - `json` (any | null): Parsed JSON payload when json_parse_ok is true; otherwise null. - `headers` (object): Response headers from the target navigation response (string values). - `screenshot_b64` (string | null): Base64-encoded PNG when screenshot is true; otherwise null. Always null in json_mode — in-page fetch() never renders the target as a page, which is why screenshot: true suppresses json_mode auto-detection. - `cf_clearance` (string | null): Cloudflare cf_clearance token value when return_cf_clearance is true and the target issued it. - `cf_clearance_cookie` (object | null): Storage-ready cookie metadata (name, value, domain, path, expires, httpOnly, secure, sameSite) when requested and present. - `cf_clearance_session` (string | null): The session that produced the clearance result. Blocked retries may rotate to a fresh session before the final result. - `datadome_cookie` (string | null): DataDome datadome cookie value when return_datadome_cookie is true and the target issued it. Otherwise null or omitted. - `datadome_cookie_detail` (object | null): Storage-ready datadome cookie metadata when requested and present. Otherwise null or omitted. - `datadome_session` (string | null): The session that produced the DataDome cookie result. Blocked retries may rotate to a fresh session before the final result. - `datadome_detected` (boolean): Present when return_datadome_cookie is true. true when the rendered page or response showed DataDome signals, even if no datadome cookie was returned. - `cookies` (array): Storage-ready cookies visible to the final rendered page when return_cookies is true. Otherwise omitted. - `network` (array): Captured browser network entries when capture_network is true. Otherwise omitted. - `network_streams` (array): Captured streamed fetch/XHR chunks, EventSource messages, and WebSocket messages when network_capture_streams is true. Otherwise omitted. - `blocked` (boolean): true when the response looks like a bot wall or unsolved challenge — even when the target returns HTTP 200. The inverse also holds: a 403/503 whose rendered body is substantial real content with no live interstitial is a solved challenge and reports blocked: false. Trust blocked over status in both directions. - `block_reason` ("none" | "http_401" | "http_403" | "http_429" | "http_503" | "cloudflare" | "datadome" | "captcha" | "block_title" | "challenge_interstitial"): Stable reason for the blocked verdict. none means the response is not classified as blocked. - `headed` (boolean): Whether the attempt that produced this result ran a headed browser (set on escalated retries). - `pooled` (boolean): true when served from a warm pooled context (session requests); false for sessionless ephemeral contexts. - `transport` ("http" | "browser"): Execution transport that produced the result. - `cache_status` ("bypass" | "miss" | "hit" | "coalesced"): Synchronous fetch cache status. bypass means no cache was requested or enabled; miss means the target was fetched; hit means a completed prior result was reused; coalesced means an identical in-flight fetch was shared. - `attempts` (number): Total attempts including the first. Greater than 1 means the service retried on a fresh session — after a block or a transient navigation timeout. - `proxy_used` (boolean): Whether the final attempt used managed residential egress. - `proxy_attempts` (integer): Residential attempts that actually occurred. Cache hits and coalesced followers have no billable proxy attempts. - `credits_used` (integer): Total charged credits. Ordinary retrieval is 1; each actual residential attempt is 25 credits. - `remaining_credits` (integer): Remaining period credits after a residential post-charge, when available. - `timing_ms` (number): Time spent inside the fetch request (final attempt). ## POST /v1/scrape Retrieve one HTML page through the same engine, then normalize it deterministically into clean text, Markdown, metadata, links, and images without additional network requests. Normalization adds no charge beyond retrieval. Target diagnostics and credit evidence live under `retrieval`; a target block can therefore return API-level 200 with `retrieval.blocked=true`. ```bash curl -sS -X POST "https://api.betterfetch.co/v1/scrape" \ -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url":"https://example.com/article","strategy":"auto","proxy":"auto","only_main_content":true,"max_chars":60000}' ``` All fetch request fields are supported. Document-only fields and response fields follow. ## Scrape document options - `only_main_content` (boolean, default true): Prefer substantial main/article content and remove common navigation, header, footer, form, and sidebar chrome. - `max_chars` (integer, default 500000): Maximum characters returned independently in text and Markdown. Range 1000-2000000; truncation flags tell you when the limit was reached. - `max_links` (integer, default 500): Maximum deduplicated HTTP(S) links. Range 0-5000. - `max_images` (integer, default 200): Maximum deduplicated HTTP(S) images. Range 0-2000. - `include_raw_html` (boolean, default false): Include the retrieved HTML alongside the normalized document. Leave false for model and RAG workflows. ## Scrape response fields - `url` (string): Final target URL after redirects. - `title` (string): Resolved document/page title. - `text` (string): Clean readable document text. - `markdown` (string): Clean content with headings, lists, quotes, and code structure preserved. - `text_truncated` (boolean): Whether text exceeded max_chars. - `markdown_truncated` (boolean): Whether Markdown exceeded max_chars. - `word_count` (integer): Word count of the returned text. - `credits_used` (integer): Credits charged for the retrieval work that actually occurred. - `links` (array): Deduplicated absolute links with visible labels when available. - `images` (array): Deduplicated absolute image URLs with alt text when available. - `metadata` (object): Title, description, author, publication time, site, language, canonical URL, and JSON-LD types when present. - `retrieval` (object): Target status, block verdict, attempts, transport, proxy/cache use, timing, content type, and body size from the underlying retrieval. - `html` (string): Present only when include_raw_html is true. JSON, plain-text, binary, and empty targets return `415 unsupported_content`; use `/v1/fetch` for those. ## Examples ### Fetch rendered HTML The basic call: navigate with a real browser and return the rendered DOM. ```bash curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \ -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com", "wait_until": "domcontentloaded", "timeout_ms": 60000 }' | jq '.status, .title, .final_url' ``` ### Fetch with country browser identity Use country when browser locale/timezone should match a country, and session when several requests should share one browser profile. ```bash curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \ -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com", "country": "gb", "session": "examplegb", "wait_until": "domcontentloaded", "timeout_ms": 60000 }' ``` Session names are scoped to your Better Fetch account; other accounts using the same name get isolated browser profiles and encrypted snapshots. Use alphanumeric names when you need distinct sessions; today example-gb, example_gb, and examplegb point at the same session. ### Fetch a JSON API (quick start) Minimal JSON call. With strategy auto, Better Fetch uses the direct HTTP fast path for this shape and returns parsed JSON plus body_bytes. ```bash curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \ -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://jsonplaceholder.typicode.com/todos/1", "include_html": false, "extra_headers": { "Accept": "application/json" } }' | jq '{ ok, status, transport, cache_status, content_type, content_kind, json_parse_ok, body_bytes, timing_ms }' ``` If jq shows null for every field, the call failed — inspect ok, error, and message first (see Tips). Keys must start with bf_ and come from your keys page. ### Fetch a JSON API For harder JSON APIs, pass the SPA Referer/Origin plus a stable session. Add country only when browser locale/timezone should match the target. auto starts with the fast HTTP path and falls back to the browser path if the response looks blocked; set strategy: browser to force the old in-page fetch behavior. ```bash curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \ -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://partner-api.example.com/projections?per_page=500", "country": "us", "session": "exampleus", "return_response_text": true, "include_html": false, "extra_headers": { "Accept": "application/json", "Referer": "https://app.example.com/", "Origin": "https://app.example.com" }, "timeout_ms": 60000 }' | jq '{ ok, status, transport, cache_status, content_type, content_kind, json_parse_ok, body_bytes, body_truncated, timing_ms, blocked, block_reason }' ``` strategy auto uses the HTTP fast path for JSON/API body requests, then escalates to the browser if that response looks blocked. Set strategy: browser when the target specifically requires browser CORS/fetch semantics, or strategy: http when you explicitly want raw HTTP only. ### Wait for rendered content For client-rendered pages, wait for a selector instead of adding a long fixed delay. ```bash curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \ -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://app.example.com/items/123", "wait_until": "domcontentloaded", "wait_selector": "#content", "timeout_ms": 90000 }' ``` ### Capture network calls Capture the XHR/fetch calls a page makes while rendering — the fastest way to discover a site's internal APIs. ```bash curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \ -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://app.example.com/items/123", "wait_until": "networkidle", "timeout_ms": 90000, "capture_network": true, "network_max_entries": 50 }' | jq '.network[] | { method, url, status, json }' ``` Enable network_include_headers only when you need it — headers can contain credentials. ### Capture streamed values Capture values delivered through streaming fetch/XHR, EventSource, or WebSocket while the page is open. ```bash curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \ -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://app.example.com/live", "wait_until": "domcontentloaded", "wait_ms": 10000, "timeout_ms": 90000, "capture_network": true, "network_capture_streams": true, "network_resource_types": ["fetch", "xhr", "eventsource", "websocket"], "network_stream_max_events": 100, "network_stream_max_value_bytes": 65536 }' | jq '.network_streams[] | { source, event_type, url, value_text, json }' ``` Stream capture is opt-in because it instruments page fetch/XHR/EventSource/WebSocket APIs and can produce large responses. ### Capture a screenshot Return a base64-encoded PNG of the rendered page. ```bash curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \ -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com", "screenshot": true, "full_page": true, "wait_until": "domcontentloaded" }' | jq -r '.screenshot_b64' ``` ### Collect a Cloudflare clearance token when issued Attempt the page flow and return the cf_clearance cookie when the target issues it, with storage-ready metadata. ```bash curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \ -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://www.example.com/", "country": "us", "session": "clearanceus", "wait_until": "domcontentloaded", "timeout_ms": 90000, "return_cf_clearance": true }' | jq '{ status, blocked, block_reason, cf_clearance, cf_clearance_cookie }' ``` cf_clearance is null when the target doesn't issue the cookie or the challenge remains unsolved. Store cf_clearance_session too — retries may rotate sessions. ### Collect a DataDome cookie when issued Attempt the page flow and return the datadome cookie when the target issues it, with storage-ready metadata. ```bash curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \ -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://www.example.com/", "country": "us", "session": "datadomeus", "wait_until": "domcontentloaded", "timeout_ms": 90000, "return_datadome_cookie": true }' | jq '{ status, blocked, block_reason, datadome_detected, datadome_cookie, datadome_cookie_detail }' ``` datadome_cookie is null when the target doesn't issue the cookie or the challenge remains unsolved. Reusing the Better Fetch session is usually more reliable than replaying the raw cookie elsewhere. ### Export and replay browser cookies Return cookies from a rendered browser session, store them in your system, then send them back on a later request. ```bash curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \ -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com", "session": "examplelogin", "wait_until": "domcontentloaded", "return_cookies": true }' | jq '.cookies' curl -sS -X POST "https://api.betterfetch.co/v1/fetch" \ -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com/account", "session": "examplelogin", "cookies": [ { "name": "session", "value": "abc123", "domain": ".example.com", "path": "/", "expires": 1790000000, "httpOnly": true, "secure": true, "sameSite": "Lax" } ] }' ``` Using the same session also reuses your account-scoped server-side browser state. Stored session limits are plan-based: Free 1, Starter 10, Pro 50, Scale 250; sessions expire after 7 idle days. ### Submit an async job When you don't want to hold a connection open — or want to fan out many URLs — submit a fetch as a background job. Returns 202 immediately; poll until status is done or failed. ```bash curl -sS -X POST "https://api.betterfetch.co/v1/jobs" \ -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com", "wait_until": "domcontentloaded", "timeout_ms": 60000 }' # → {"ok": true, "id": "a1b2c3d4-...", "status": "queued"} curl -sS "https://api.betterfetch.co/v1/jobs/a1b2c3d4-..." \ -H "Authorization: Bearer $BETTER_FETCH_API_KEY" # → {"ok": true, "id": "...", "status": "done", "result": {...}, "error": null, ...} ``` Jobs run under a separate concurrency budget so they never block synchronous /v1/fetch. One base credit is charged when admitted; actual residential attempts are post-charged at the published rate. ## Errors Every non-2xx response uses one JSON envelope: `{ "ok": false, "error": "", "message": "...", "status": }`. Switch on the stable `error` code. - `400` `bad_request`: Invalid JSON, unknown request field, missing or non-HTTP url, or an out-of-range parameter. - `401` `unauthorized`: Missing, incorrect, or revoked bearer token. - `402` `payment_required`: Valid key but no active subscription. - `402` `residential_plan_required`: proxy auto/residential was requested from a Free or Starter account. Upgrade to Pro or Scale; the rejected request is not metered. - `415` `unsupported_content`: POST /v1/scrape retrieved JSON, plain text, binary, or empty content instead of HTML. Use /v1/fetch for non-HTML targets. - `429` `quota_exceeded`: Monthly credit quota exhausted; resets at the next billing cycle. Ordinary accepted retrieval costs 1 credit; actual residential attempts use the published higher rate. - `429` `session_limit_exceeded`: Stored browser session limit reached; clear a session from the dashboard or upgrade. - `502` `fetch_failed`: Browser launch, navigation, or target fetch failed. The message includes the underlying detail. In json_mode, Failed to fetch often means the API's CORS policy rejected the in-page call — set Referer to the site's app origin or try json_mode: false. - `504` `timeout`: Request timed out at the API layer. A 200 with `"blocked": true` (or a target `"status": 403`) means Better Fetch worked but the target denied the browser request — different from an API error. Use `block_reason` for the category. ## Guides ### High-volume same-site scraping When you fetch many URLs on one site in a single run — scanning dozens of markets on a bookmaker, paging through a listing, etc. — these habits avoid most blocks and connection timeouts. - Reuse one session per site for the whole run (and across runs) — e.g. session: "skybet". A warm session keeps a stable fingerprint and persisted cookies/localStorage that mark you as a returning visitor. Do not generate a fresh session name per URL or per run: each distinct name — and each distinct locale/timezone/user_agent value — is a separate stored session that counts against your plan limit and starts cold. - Use country for coherent browser identity defaults (gb for UK locale/timezone, au for Australian locale/timezone, etc.). It changes network egress only when proxy is auto or residential. - Leave user_agent, locale, and timezone unset unless the integration requires them, so country can apply coherent defaults. If you do set them, keep them byte-identical across every call for that session — changing them splits the warm pool into separate cold contexts. - Pace the run: keep concurrency to a few in-flight requests and add a small jittered delay (1–3s) between calls. A sub-second burst of dozens of requests from one IP looks robotic and trips bot detection that can then poison the whole session. - Prefer fast waits on browser pages: wait_until: "domcontentloaded" plus wait_selector over networkidle or long fixed waits. For JSON endpoints, set extra_headers {"Accept":"application/json"}; auto uses direct HTTP first, skips humanization, and returns parsed JSON faster. - For JSON polling, use cache_ttl_ms for short identical bursts and cache longer-lived responses in your application. Reuse one session, shrink query params (per_page, include=), and poll only as often as the data actually changes. - Handle responses defensively in your loop. If blocked is true, inspect block_reason, then pause and back off (5–10s) before continuing. Retry an individual 502 fetch_failed after a short backoff — sustained failures mean the target is pushing back. Keep timeout_ms moderate (45–60s). ### Marketplace tools Ready-made tools built on the Better Fetch engine — structured extraction, monitoring, screenshots, and more — that you can run without writing fetch code yourself. Browse them at https://betterfetch.co/tools. - Each tool page documents its input/output schema, examples, and a playground to try it against your account. - Call search_tools from Claude, ChatGPT, Codex, or another MCP client to find the best live tool, then pass its exact name and input schema to run_tool. The catalogue stays searchable without flooding the default MCP surface. - Or call them over REST: POST https://betterfetch.co/api/tools/{name}/run with Authorization: Bearer bf_... and a JSON body of {"input": {...}}. - Tool runs are metered against your plan like direct fetch calls; each tool lists a ~credits-per-run estimate. ## POST /v1/jobs Submit a fetch as a background job. Accepts the same body as POST /v1/fetch and returns 202 with a job id immediately. The browser work runs asynchronously under a separate concurrency budget so it never blocks synchronous /v1/fetch. One base credit is charged when admitted; residential attempts are post-charged when they occur. Jobs are a short-lived convenience queue, not a durable crawl service: queued work and results can be lost during a process restart, and results expire after roughly 24 hours. ```bash curl -sS -X POST "https://api.betterfetch.co/v1/jobs" \ -H "Authorization: Bearer $BETTER_FETCH_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com", "wait_until": "domcontentloaded", "timeout_ms": 60000}' # → {"ok": true, "id": "a1b2c3d4-...", "status": "queued"} ``` Poll the job with GET /v1/jobs/{id}. Returns the current status (queued, running, done, failed) and the full FetchSuccess result when complete. Jobs are scoped to the authenticated account. ```bash curl -sS "https://api.betterfetch.co/v1/jobs/a1b2c3d4-..." \ -H "Authorization: Bearer $BETTER_FETCH_API_KEY" ``` ## Job response fields - `ok` (boolean): true when the job was found and belongs to the authenticated account. - `id` (string): Job identifier (UUID). - `status` (string): Current job state: queued, running, done, or failed. - `result` (object | null): Full FetchSuccess payload (same shape as POST /v1/fetch) when status is done; otherwise null. - `error` (string | null): Error message when status is failed; otherwise null. - `created_at` (string): ISO timestamp when the job was admitted. - `started_at` (string | null): ISO timestamp when the browser work began. - `completed_at` (string | null): ISO timestamp when the job reached done or failed. - `expires_at` (string): ISO timestamp after which the job result is cleaned up (24 hours after creation by default). ## GET /v1/sessions List active account-scoped browser sessions without exposing cookie values. Clear one with `DELETE /v1/sessions/`. Session names are account-scoped but canonicalized for backend routing: only letters and numbers form the durable key today, so `shop-us`, `shop_us`, and `shopus` target the same stored browser session. ## GET /v1/health Liveness check, no auth: `curl -sS https://api.betterfetch.co/v1/health`. The response includes the service `version`, managed residential routing availability under `managed_proxy`, and pinned browser metadata under `browser`. Add `?geo=1&country=us` to include country browser-identity defaults. Select routing per fetch with `proxy: "none" | "auto" | "residential"`. ## Tips - Every response includes ok. On failure you get { ok: false, error, message, status } — not body_bytes or timing_ms. When debugging with jq, always select ok and error first. - Use content_kind and content_type to route parser logic before inspecting body_text, html, or json. - Set include_html: false for JSON/body workflows when you only need body_text or json, especially with strategy: browser JSON mode. - Use cache_ttl_ms for short scraper bursts that repeat the exact same synchronous fetch payload. It is explicit, off by default, and best effort per API worker; cache_status reports miss, hit, coalesced, or bypass. - For simple page/body fetches, leave strategy as auto; Better Fetch uses direct HTTP and only moves to Chromium when the request asks for browser-only features or the fast path looks blocked. - Use strategy: browser with wait_selector, cookies, screenshots, or network capture when you specifically need rendered DOM behavior. - For JSON APIs, set extra_headers {"Accept":"application/json"}. strategy auto uses direct HTTP first, then falls back to the browser path if the response looks blocked. - Set strategy: browser when the target specifically requires in-page fetch/CORS semantics from a SPA origin; set strategy: http when you explicitly want raw HTTP and no browser-only features. - When using json_mode, include a Referer header pointing to the site's app/SPA origin (e.g. "https://app.example.com/"). Better Fetch navigates to that origin first so CORS and Referer match what the API expects. Without a Referer, the URL's own origin is used. - Large body responses (> 50 MB) are truncated in body-fetch strategies. Check body_truncated and body_bytes — reduce per_page or narrow the query if you need the complete body. - country sets representative browser locale/timezone defaults when geoip is true and explicit values are omitted. It changes egress IP only when proxy is auto or residential. - Check GET /v1/health?geo=1&country=us to see the country defaults Better Fetch will apply. - Pass a session for hard targets: it enables an account-scoped warm pooled context, encrypted portable cookie/localStorage snapshots, and a stable fingerprint. - Session names are canonicalized to letters and numbers for backend routing today; punctuation is ignored, so use clearly distinct alphanumeric names when you need separate sessions. - Reuse the same session to keep browser cookies/localStorage across machines; use return_cookies plus cookies when you want caller-managed cookie replay. - Use return_cf_clearance or return_datadome_cookie when you need a specific protection cookie; for DataDome, the Better Fetch session is usually the stronger reuse primitive because cookies can be bound to IP, fingerprint, and browser state. - Stored session limits are plan-based: Free 1, Starter 10, Pro 50, Scale 250; named sessions expire after 7 idle days. - Pro and Scale can use proxy: auto for cost-aware escalation: the first attempt stays on datacenter egress and blocked retries move to residential routing. Use proxy: residential only when every attempt must use a regional residential exit. Each residential attempt that actually occurs costs 25 credits. - Check blocked and block_reason to detect bot walls even when the target returns HTTP 200. The attempts field is greater than 1 when the service retried — retries cover both blocks and transient navigation timeouts (net::ERR_TIMED_OUT). - A browser fetch can return status 403 with blocked: false only when Better Fetch observed the challenge clear and the real rendered page replace it. A large generic 403 shell remains blocked. Treat blocked: false plus rendered content as success. - Leave user_agent, locale, and timezone unset unless required — they form part of a session's warm-context identity, so changing them splits the warm pool. - First requests after a deploy can be slower while Chromium starts; reused session requests are served from a warm pool. - Do not send raw proxy credentials. Select none, auto, or residential with the proxy field; Better Fetch owns provider credentials and reports proxy_used in the result. - Use POST /v1/jobs for long-running fetches or batch fan-out: it returns a job id immediately and runs the browser work asynchronously. Poll GET /v1/jobs/{id} for the result. - Jobs run under a separate concurrency budget from synchronous /v1/fetch, so they never block real-time requests. ## MCP connector AI agents call Better Fetch through the hosted Streamable HTTP MCP server at https://betterfetch.co/api/mcp. Claude and ChatGPT desktop can add the URL and sign in with OAuth. Codex uses `codex mcp add better-fetch --url https://betterfetch.co/api/mcp` followed by `codex mcp login better-fetch`. API-key bearer auth remains available for unattended clients. The Claude Code plugin adds retrieval skills and a scraper subagent: `/plugin marketplace add better-fetch/claude-plugins`. Tools: - `search_tools`: Find a ready-made scraper or extractor in the live catalogue and return its schema, example inputs, and estimated credit cost. - `run_tool`: Run one exact catalogue tool selected with search_tools, without loading every specialist tool into the default MCP surface. - `fetch_url`: Fetch raw HTML or return clean readable text/Markdown. Better Fetch uses direct HTTP first, Chromium when rendering is needed, and optional residential escalation, with block, cache, transport, attempt, and routing metadata. - `scrape_page`: Retrieve one HTML page as a deterministic model-ready document: clean text, Markdown, metadata, canonical links, images, and retrieval diagnostics. - `scrape_json`: Fetch a JSON endpoint with browser-compatible headers and session reuse when needed, returning parsed JSON, fallback body_text, and response metadata. - `screenshot_url`: Render a page and capture a viewport or full-page PNG screenshot. - `discover_apis`: Load a page and capture the XHR/fetch calls it makes, with optional response previews and streamed values, to find internal APIs behind the page. - `get_clearance`: Attempt a Cloudflare challenge flow and return cf_clearance cookie metadata when the browser receives it, plus the reusable Better Fetch session. - `get_datadome_cookie`: Render a DataDome-protected page and return datadome cookie metadata when the browser receives it, plus the reusable Better Fetch session. - `get_usage`: Check the connected Better Fetch account: plan, credits used this billing period, remaining quota, stored browser sessions, and reset time. - `list_sessions`: List active account-scoped browser sessions without exposing cookies, localStorage, or snapshot object paths. - `clear_session`: Clear a stored browser session through the backend, delete its portable snapshot, and make future requests with that session name start from a fresh profile key.