RobotsGate

{
  "mcpServers": {
    "dev.workers.mike-tusa.robotsgate/robotsgate": {
      "url": "https://robotsgate.mike-tusa.workers.dev/mcp"
    }
  }
}

Indexed tools

Tool metadata and schemas come from the remote server and are untrusted. Indexing only lists tools; it never calls them.

4 shown · authentication unknown

• Check which AI crawlers a live site allows

  check_site

  Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5 minutes) and report, for one path, which registry AI crawlers are allowed or blocked, plus validation errors and warnings. Read-only: one GET of /robots.txt with a RobotsGate user-agent. Same engine as GET /api/check.

  Schema valid · SHA-256 8fbd55130bae…

  View input schema

  {
    "type": "object",
    "properties": {
      "url": {
        "type": "string",
        "minLength": 1,
        "maxLength": 2048,
        "description": "Site URL or bare hostname, e.g. example.com. Only the scheme and host are used."
      },
      "path": {
        "type": "string",
        "maxLength": 2048,
        "description": "URL path to test, starting with \"/\" (default \"/\")."
      }
    },
    "required": [
      "url"
    ],
    "additionalProperties": false
  }

• Generate a robots.txt for AI crawlers

  generate_robots

  Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default policy for other bots, paths, sitemaps and custom rules. Returns the file text, a per-crawler summary and notes. Read-only: it only returns text and changes nothing. Same engine as POST /api/generate.

  Schema valid · SHA-256 033fd27404d7…

  View input schema

  {
    "type": "object",
    "properties": {
      "categories": {
        "type": "object",
        "description": "Action per crawler category. Omitted categories get no group of their own.",
        "properties": {
          "training": {
            "type": "string",
            "enum": [
              "allow",
              "block",
              "omit"
            ]
          },
          "dataset": {
            "type": "string",
            "enum": [
              "allow",
              "block",
              "omit"
            ]
          },
          "search": {
            "type": "string",
            "enum": [
              "allow",
              "block",
              "omit"
            ]
          },
          "user_fetch": {
            "type": "string",
            "enum": [
              "allow",
              "block",
              "omit"
            ]
          },
          "other": {
            "type": "string",
            "enum": [
              "allow",
              "block",
              "omit"
            ]
          }
        },
        "additionalProperties": false
      },
      "agents": {
        "type": "object",
        "description": "Per-token overrides, e.g. {\"GPTBot\": \"block\"}. Tokens: letters, digits, \"_\", \".\" and \"-\", up to 64 characters (not \"*\"). At most 100.",
        "maxProperties": 100,
        "propertyNames": {
          "pattern": "^[A-Za-z0-9_.-]{1,64}$"
        },
        "additionalProperties": {
          "type": "string",
          "enum": [
            "allow",
            "block",
            "omit"
          ]
        }
      },
      "default": {
        "type": "string",
        "enum": [
          "allow",
          "block"
        ],
        "description": "Policy for \"User-agent: *\" (default allow)."
      },
      "disallow_paths": {
        "type": "array",
        "maxItems": 100,
        "items": {
          "type": "string",
          "minLength": 1,
          "maxLength": 512
        },
        "description": "Paths to disallow for \"*\" and for allowed AI groups. Each path starts with \"/\" or \"*\" and has no whitespace or \"#\". At most 100."
      },
      "allow_paths": {
        "type": "array",
        "maxItems": 100,
        "items": {
          "type": "string",
          "minLength": 1,
          "maxLength": 512
        },
        "description": "Paths to allow explicitly. Each path starts with \"/\" or \"*\" and has no whitespace or \"#\". At most 100."
      },
      "sitemaps": {
        "type": "array",
        "maxItems": 20,
        "items": {
          "type": "string",
          "maxLength": 2048
        },
        "description": "Absolute http(s) sitemap URLs. At most 20."
      },
      "sitemap": {
        "type": "string",
        "maxLength": 2048,
        "description": "A single sitemap URL (alternative to sitemaps)."
      },
      "custom_rules": {
        "type": "string",
        "maxLength": 20000,
        "description": "Extra robots.txt lines appended after validation (max 20000 bytes)."
      },
      "date": {
        "type": "string",
        "maxLength": 10,
        "pattern": "^\\d{4}-\\d{2}-\\d{2}$",
        "description": "Date for the header comment (YYYY-MM-DD). Default: today (UTC)."
      }
    },
    "additionalProperties": false
  }