03 / CRAWLERS
Check AI crawler access and llms.txt
Crawl rules: robots.txt, canonical tags and noindex rules that decide whether the page can be found at all. AI search tools can only quote a page their crawlers are allowed to read. Your robots.txt decides that, one user agent at a time.
AI crawler access: whether GPTBot, ClaudeBot, PerplexityBot and Google-Extended are allowed in, and whether the site publishes an llms.txt file. Ask the checker to read your robots.txt and it will tell you which of these crawlers are allowed or blocked. Allowing a crawler does not mean an AI answer will mention you. To see where Google AI Overviews already cite your pages, use the AI visibility tracker. Our own robots.txt names GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended individually and allows each of them, and the site publishes an llms.txt file.
CRAWL RULESCan the page be read
robots.txt, canonical and noindex
GPTBot, ClaudeBot, PerplexityBot, Google-Extended
llms.txt published or not