Free tool · No signup · Runs in your browser
Robots.txt AI Crawler Access Checker
AI assistants can only cite your site if their crawlers can reach it. Paste your robots.txt below to see exactly which AI crawlers are allowed or blocked — including the difference between training bots, search-index bots, and user-triggered fetchers.
How it works
Open yourdomain.com/robots.txt in a browser tab and copy its contents.
Paste it into the checker and optionally set a path to test (defaults to the whole site).
The checker parses your rules the way crawlers do — most specific user-agent group wins, longest path rule wins.
Review the verdict for each AI crawler, grouped by what the bot is used for: model training, AI search indexing, or live page fetches when a user asks.
Frequently asked questions
Which AI crawlers does this tool check?
It checks the major published AI user agents, including OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User; Anthropic's ClaudeBot, Claude-SearchBot and Claude-User; PerplexityBot and Perplexity-User; Google-Extended; Applebot-Extended; Meta-ExternalAgent; Amazonbot; Bytespider; CCBot; and others.
Should I block AI crawlers or allow them?
It depends on your goal. If you want your brand mentioned and cited in AI answers, blocking search-oriented bots like OAI-SearchBot or PerplexityBot makes you invisible there. Many sites allow search and user-fetch bots while blocking pure training bots like GPTBot or CCBot — that is a policy choice, and this tool shows you what your current file actually does.
What is the difference between GPTBot and ChatGPT-User?
GPTBot crawls pages to improve OpenAI's models (training). ChatGPT-User fetches a page live when a ChatGPT user asks about it, and OAI-SearchBot builds the index behind ChatGPT search. Each obeys its own robots.txt rules, so you can allow one and block another.
Is my robots.txt uploaded to a server?
No. Parsing happens entirely in your browser — nothing is transmitted or stored.
More free tools
llms.txt Generator
Generate a spec-compliant llms.txt file for your website in seconds. Help ChatGPT, Claude, Perplexity, and other AI engines understand your content. Free, no signup.
Schema Generator
Generate valid JSON-LD structured data for FAQ, Article, and Organization schema in seconds. Copy-paste ready for Google and AI engines. Free, no signup.
Meta Tag Generator
Generate SEO meta tags, Open Graph, and Twitter Card markup with a live Google and social preview. Check title and description lengths. Free, no signup.