Robots.txt & AI Crawler Checker

{
  "mcpServers": {
    "dev.workers.quietworklabs.robots-check/robots-check": {
      "url": "https://robots-check.quietworklabs.workers.dev/mcp"
    }
  }
}

Indexed tools

Tool metadata and schemas come from the remote server and are untrusted. Indexing only lists tools; it never calls them.

3 shown · authentication unknown

• Which AI crawlers a site allows or blocks

  ai_crawler_access

  AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more) grouped by training/search/user purpose, plus page meta robots, X-Robots-Tag (noindex/noai) and llms.txt presence. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.

  Schema valid · SHA-256 5833e227300b…

  View input schema

  {
    "type": "object",
    "properties": {
      "url": {
        "type": "string",
        "description": "Site or page URL",
        "examples": [
          "https://www.theguardian.com/"
        ]
      }
    },
    "required": [
      "url"
    ]
  }

• May this user-agent crawl this URL? (RFC 9309)

  robots_check

  Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest match, Allow wins ties, * and $ wildcards, 4xx=allow all, 5xx=disallow all). Returns allowed, the deciding rule and line, crawl-delay and sitemaps. Deterministic, no AI. Price: $0.001 USDC per call via x402 (Base); the first 5 calls per day per client are free.

  Schema valid · SHA-256 ab2142b4c89a…

  View input schema

  {
    "type": "object",
    "properties": {
      "url": {
        "type": "string",
        "description": "Page URL to test",
        "examples": [
          "https://www.nytimes.com/section/technology"
        ]
      },
      "user_agent": {
        "type": "string",
        "description": "Crawler product token, e.g. GPTBot or MyAgent. Default *",
        "examples": [
          "GPTBot"
        ]
      }
    },
    "required": [
      "url"
    ]
  }