Web Scraper

Scrape one rendered web page into structured metadata, readable text, links, images, JSON-LD types, and optional simple selector matches.

Developer Toolsv0.1.0~1 credit/runApify StoreSource on GitHub

Overview

Web Scraper is the general-purpose single-page extraction tool in the Better Fetch marketplace. Provide any public URL and the tool renders the page through the browser engine, normalizes page metadata, extracts readable text, captures link and image inventories, reports structured-data types, and can pull text from a small list of simple selectors.

Last validated: Jul 3, 2026

Playground

Input

urlstring (uri)required

The public page URL to scrape.

max_linksintegerdefault: 25

Maximum links to return.

selectorsstring[]

Optional simple selectors to extract text from, e.g. h1, main, .price, #content, article.

max_imagesintegerdefault: 10

Maximum images to return.

max_text_charsintegerdefault: 12000

Maximum readable text characters to return.

Output

urlstringrequired

Requested URL

textstringrequired

Readable page text

linksobject[]

Links discovered in page order

titlestringrequired

Page title

imagesobject[]

Images discovered in page order

final_urlstringrequired

Final URL after redirects

html_bytesinteger

HTML byte length

word_countintegerrequired

Words in extracted text

descriptionstring

Meta description

canonical_urlstring

Canonical page URL

json_ld_typesstring[]

Structured data @type values

selector_resultsobject[]

Optional selector text matches

Examples

example-page

{
  "url": "https://example.com",
  "max_links": 10,
  "selectors": [
    "h1",
    "p"
  ],
  "max_images": 5,
  "max_text_chars": 2000
}

Use cases

Quick page-to-JSON extraction

Turn a rendered page into title, description, canonical URL, readable text, links, images, and structured-data hints without writing a custom scraper.

Agent research

Let an agent inspect a product page, landing page, directory entry, documentation page, or public profile and reason over clean structured fields.

Selector checks

Probe simple selectors such as h1, main, article, .price, or #content to see whether a page exposes the text you need before building a deeper tool.

FAQ

Is Web Scraper a full recursive crawler?

No. Version 0.1 focuses on a single rendered page. Use Website Content Crawler for small same-origin crawls, or add a future Web Scraper slice for recursive request queues.

Can it run arbitrary JavaScript page functions?

No. It returns a fixed, safe structured extraction. Arbitrary page functions and custom execution should be added as a separate validated capability.

What selectors does this version support?

The selector helper supports simple tag, class, id, and tag.class patterns. It is intended for lightweight text probes, not complete CSS selector coverage.

Use it anywhere

MCP (Claude, Cursor, any client)

# Add the Better Fetch MCP connector (or paste the URL into
# Claude → Settings → Connectors → Add custom connector):
claude mcp add --transport http better-fetch https://betterfetch.co/api/mcp \
  --header "Authorization: Bearer bf_your_key_here"

# Then ask for the tool by name: web_scraper

REST

curl -sS -X POST "https://betterfetch.co/api/tools/web_scraper/run" \
  -H "Authorization: Bearer bf_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{"input": {"url":"https://example.com","max_links":10,"selectors":["h1","p"],"max_images":5,"max_text_chars":2000}}'

Run locally

git clone https://github.com/better-fetch/web-scraper && cd web-scraper && npm i
BETTER_FETCH_API_KEY=bf_your_key_here npx bf-tool run --input '{"url":"https://example.com","max_links":10,"selectors":["h1","p"],"max_images":5,"max_text_chars":2000}'