Check which search engines and AI crawlers your robots.txt allows or blocks. Paste your robots.txt and see, bot by bot, whether Googlebot, Bingbot, ChatGPT, Perplexity, Claude and the main AI training crawlers can reach your pages, so you can stay visible in AI answers while choosing what you allow for training.
| Crawler | Company | What it is for | Status |
|---|
How to use the AI Crawler Checker
- Open
yoursite.com/robots.txtin a new browser tab. - Copy the whole file and paste it into the box, replacing the example.
- Enter a page path to test, such as
/for the home page or/blog/my-post/for one article. - Read the table. Each crawler shows Allowed or Blocked, and the rule group that decided it.
- Fix any search or answer bot marked Blocked in red, then upload the new robots.txt and test again.
Search bots and training bots are not the same
AI companies run separate crawlers for different jobs. Search and answer bots, such as OAI-SearchBot, PerplexityBot and Claude-SearchBot, find pages so they can be shown and cited in answers. Training bots, such as GPTBot, ClaudeBot, Google-Extended and CCBot, collect content to train models. Blocking a training bot does not remove you from that company’s search results, but blocking a search bot does. Many businesses allow every search bot and block the training bots they are not comfortable with. This tool colours the results to match: red means you are losing search visibility, while a blocked training bot shows green because it is a deliberate choice.
How the tool reads your rules
- It follows the same logic as Google: a crawler uses the most specific
User-agentgroup that names it, or the*group if none does. - When rules conflict, the longest matching path wins. If an Allow and a Disallow are the same length, Allow wins.
- Wildcards (
*) and end-of-URL markers ($) are supported. - An empty
Disallow:line means nothing is blocked.
Pros
- Covers 16 search and AI crawlers in one table
- Separates search visibility from training opt-out
- Tests any page path, not only the home page
- Runs in your browser; nothing is uploaded
Cons
- Cannot fetch robots.txt for you; you paste it in
- Does not see firewall or CDN blocks, which can stop bots even when robots.txt allows them
- Some crawlers ignore robots.txt entirely
- Bot names change over time, so check each company’s documentation
Frequently asked questions
Will blocking GPTBot remove me from ChatGPT?
No. ChatGPT search uses OAI-SearchBot. GPTBot is the training crawler, so you can block it and still appear in ChatGPT search results.
Does Google-Extended control AI Overviews?
No. AI Overviews and AI Mode use Googlebot. Google-Extended only controls whether your content is used to train Gemini models.
My robots.txt allows everything but AI tools still do not cite me. Why?
Check your firewall, CDN and security plugin for bot blocking, make sure key content is in the page HTML rather than loaded by JavaScript, and build mentions of your brand on other sites.
Want a full check of your site?
Our free SEO audit looks at every page, your speed, crawling and AI search visibility, then sends you a clear fix list.
Get a free SEO audit