AI Crawler Checker is a free tool that shows which search engines and AI crawlers your robots.txt allows or blocks. Paste your robots.txt and see, bot by bot, whether Googlebot, Bingbot, ChatGPT, Perplexity, Claude and the main AI training crawlers can reach your pages, so you can stay visible in AI answers while choosing what you allow for training.
AI Crawler Checker: common questions about AI crawlers
Will blocking GPTBot remove my site from ChatGPT answers? Not always. GPTBot is the training crawler, while OAI-SearchBot is the search crawler, so the two settings affect different things.
Does llms.txt control crawling? No. It is a summary file that helps AI tools understand the site, while robots.txt is the rule file that bots follow.
What to do after running the AI Crawler Checker
Decide the setting for each bot based on your goal: search visibility, training, or both. Write the decision down, so the robots.txt file and the llms.txt file stay consistent.
Check the file again after each theme or plugin update, because some updates reset robots rules without warning.
| Crawler | Company | What it is for | Status |
|---|
How to use the AI Crawler Checker
- Open
yoursite.com/robots.txtin a new browser tab. - Copy the whole file and paste it into the box, replacing the example.
- Enter a page path to test, such as
/for the home page or/blog/my-post/for one article. - Read the table. Each crawler shows Allowed or Blocked, and the rule group that decided it.
- Fix any search or answer bot marked Blocked in red, then upload the new robots.txt and test again.
Search bots and training bots are not the same
AI companies run separate crawlers for different jobs. Search and answer bots, such as OAI-SearchBot, PerplexityBot and Claude-SearchBot, find pages so they can be shown and cited in answers. Training bots, such as GPTBot, ClaudeBot, Google-Extended and CCBot, collect content to train models. Blocking a training bot does not remove you from that company’s search results, but blocking a search bot does. Many businesses allow every search bot and block the training bots they are not comfortable with. The AI Crawler Checker colours the results to match: red means you are losing search visibility, while a blocked training bot shows green because it is a deliberate choice.
How the AI Crawler Checker reads your rules
- It follows the same logic as Google: a crawler uses the most specific
User-agentgroup that names it, or the*group if none does. - When rules conflict, the longest matching path wins. If an Allow and a Disallow are the same length, Allow wins.
- Wildcards (
*) and end-of-URL markers ($) are supported. - An empty
Disallow:line means nothing is blocked.
Pros
- Covers 16 search and AI crawlers in one table
- Separates search visibility from training opt-out
- Tests any page path, not only the home page
- The AI Crawler Checker runs in your browser; nothing is uploaded
Cons
- Cannot fetch robots.txt for you; you paste it in
- Does not see firewall or CDN blocks, which can stop bots even when robots.txt allows them
- Some crawlers ignore robots.txt entirely
- Bot names change over time, so check each company’s documentation
How to read the AI Crawler Checker results
Green means the bot can read the pages covered by your file. Red means the rule blocks it.
Check Googlebot and Bingbot first, because blocking them affects search visibility.
Common decisions after using the AI Crawler Checker
Allow search crawlers, and decide on AI training crawlers based on your content policy. Use the AI Crawler Checker again after every change to confirm the result.
Check that your sitemap line is present, so crawlers can find the important pages.
Frequently asked questions
What does the AI Crawler Checker do?
The AI Crawler Checker reads the robots.txt you paste in and shows, bot by bot, whether Googlebot, Bingbot and the main AI crawlers are allowed or blocked for any page path you enter.
Will blocking GPTBot remove me from ChatGPT?
No. ChatGPT search uses OAI-SearchBot. GPTBot is the training crawler, so you can block it and still appear in ChatGPT search results.
Does Google-Extended control AI Overviews?
No. AI Overviews and AI Mode use Googlebot. Google-Extended only controls whether your content is used to train Gemini models.
My robots.txt allows everything but AI tools still do not cite me. Why?
Check your firewall, CDN and security plugin for bot blocking, make sure key content is in the page HTML rather than loaded by JavaScript, and build mentions of your brand on other sites.
Want a full check beyond the AI Crawler Checker?
Our free SEO audit looks at every page, your speed, crawling and AI search visibility, then sends you a clear fix list.
Get a free SEO auditWhich AI crawlers the AI Crawler Checker looks for
Several bots read websites for AI products. GPTBot is OpenAI training crawler, OAI-SearchBot is used for ChatGPT search results, ChatGPT-User is used when a user asks ChatGPT to open a page, and ClaudeBot and PerplexityBot are used by other AI services.
Google-Extended controls whether content is used for Gemini training, and it is separate from normal Google Search indexing.
How robots.txt rules work
A robots.txt rule names a bot in the User-agent line, then allows or blocks paths for that bot. A bot that is not named follows the rules for the wildcard user agent.
Rules only ask a bot not to crawl. Reputable crawlers follow them, but the file is not a security measure, so private content should be protected by a login.
What blocking training means for visibility
Blocking a training crawler does not always remove a site from AI answers, because some AI products use separate search bots. Blocking the search bot, however, can reduce the chance that a page is shown or cited in that product.
The right setting depends on the business goal. Some owners want visibility in AI search, others want to keep content out of training, and the AI Crawler Checker shows what each setting does.
Checking llms.txt and the sitemap
An llms.txt file gives AI tools a short, clear summary of the site, its services and its main pages. It does not replace the sitemap, which tells search engines which URLs exist.
The AI Crawler Checker reports whether your robots.txt lists a sitemap. To confirm that llms.txt and the sitemap load, open both in your browser and make sure the links inside them still work, so broken or old URLs can be corrected.