Better Fetch

Robots.txt Analyzer

Parse robots.txt directives, user-agent groups, crawl rules, crawl delays, and advertised sitemap locations.

Developer Toolsv0.1.0~1 credit/runSource on GitHub

Workspace

Configure

Inputs stay here while results become exportable artifacts.

Start from an example

Website URL or direct robots.txt URL.

Sign in and run
~1 credit/run10 runs/minute

Results

Preview, inspect, copy, or export the complete output.

Your result will become an artifact

Run the tool to inspect structured rows, raw JSON, execution details, and downloadable JSON, CSV, and Markdown files.

countdirectivessource_url

Overview

Parse robots.txt directives, user-agent groups, crawl rules, crawl delays, and advertised sitemap locations. This first-party Better Fetch tool turns a focused request into predictable structured output that can be inspected in the web workspace, exported as JSON or CSV, called through REST, or composed by an AI agent over MCP. It is designed for repeatable production research rather than fragile copy-and-paste scraping.

Last validated: Jul 13, 2026

Input

urlstring (uri)required

Website URL or direct robots.txt URL.

Output

countintegerrequired

Number of records returned

directivesobject[]required

Normalized directives records

source_urlstringrequired

Fetched source URL

Examples

first-run

{
  "url": "https://www.google.com"
}

Use cases

Agent research

Give AI agents bounded, normalized robots.txt analyzer results without writing source-specific retrieval and parsing code.

Repeatable monitoring

Save the configuration, rerun it on a consistent source, and export comparable structured artifacts for operational review.

FAQ

What does Robots.txt Analyzer return?

It returns normalized structured fields documented in the output schema, plus the fetched source URL and a record count for receipts and exports.

Does this tool store the retrieved output?

No. Better Fetch records privacy-safe run metadata for reliability and retention analysis, but the output payload itself is not stored by the workspace.

Use it anywhere

MCP (Claude, ChatGPT, Codex)

# After connecting https://betterfetch.co/api/mcp, ask your AI to:
1. Call search_tools for "Robots.txt Analyzer"
2. Call run_tool with:
   name: "robots_txt_analyzer"
   input: {"url":"https://www.google.com"}

REST

curl -sS -X POST "https://betterfetch.co/api/tools/robots_txt_analyzer/run" \
  -H "Authorization: Bearer bf_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{"input": {"url":"https://www.google.com"}}'

Run locally

git clone https://github.com/better-fetch/tools/tree/main/tools/robots-txt-analyzer && cd robots-txt-analyzer && npm i
BETTER_FETCH_API_KEY=bf_your_key_here npx bf-tool run --input '{"url":"https://www.google.com"}'