AI crawler access checker
robots.txt is the first thing an AI crawler reads. One stray Disallow line and ChatGPT, Claude or Perplexity never see your listings. Paste your file or enter your site address, and this tool reads the rules the way the crawlers do and shows which of the ten named AI agents may fetch each path.
The ten crawlers
- GPTBotOpenAI's crawler for training data.
- OAI-SearchBotOpenAI's crawler for ChatGPT search results.
- ChatGPT-UserChatGPT fetching a page on a user's behalf during a chat.
- ClaudeBotAnthropic's crawler for training data.
- Claude-SearchBotAnthropic's crawler for Claude's search results.
- Claude-UserClaude fetching a page on a user's behalf during a chat.
- PerplexityBotPerplexity's crawler for its search index.
- Perplexity-UserPerplexity fetching a page on a user's behalf.
- Google-ExtendedGoogle's control for Gemini training. It doesn't affect Google Search.
- Applebot-ExtendedApple's control for Apple Intelligence training. It doesn't affect Siri or Spotlight.
Questions
Which crawlers does it test?
The ten AI user agents listed above: three from OpenAI, three from Anthropic, two from Perplexity, and the training controls from Google and Apple. The same ten are used by the AI-Ready Check.
What's the difference between blocking GPTBot and blocking OAI-SearchBot?
GPTBot collects pages for training future models. OAI-SearchBot collects pages so ChatGPT can show your site as a source when someone asks a question. Many sites block the first and allow the second. If your goal is to be cited, the search and user agents are the ones that matter.
The tool says allowed, but AI still doesn't mention us.
Access is the first step, not the last. The crawler also has to find your listings (a sitemap helps), read them without JavaScript, and understand what they are (structured data helps). The AI-Ready Check tests those too.
Does the site-address mode fetch my whole site?
No. It fetches robots.txt once, from our server, with our own user agent, and reads nothing else. The paste mode fetches nothing at all.
Want the full picture? The AI-Ready Check reads your site the way ChatGPT, Claude and Perplexity do and reports back in about a minute.
Run the AI-Ready Check