Crawler access tool

Robots.txt Checker

See which rule blocks which crawler, line by line.

This robots.txt checker tests your paths against Googlebot and AI crawlers.

Tests Googlebot, Bingbot and 7 AI crawlersQuotes the rule line behind every resultA corrected file when something is wrong

START WITH A REQUEST

One file. Three ways to test it.

Each card is a real question site owners ask. Add your domain and the robots.txt checker opens it in SpringBrand.

WHAT YOU GET

What does your robots.txt actually block?

A one-character mistake in robots.txt can hide a whole site from search, and AI crawlers add new user agents every year. The robots.txt checker applies the same matching rules crawlers use and shows its work.

01 / PATH TESTS

Test any URL against nine crawlers

List up to 10 paths, or leave them out to test the homepage. Each path is checked for Googlebot, Bingbot, OAI-SearchBot, GPTBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended and CCBot.

Every result quotes the user-agent group and the rule line that decided it, or says no rule matched.

TEST

/checkout/

Googlebot · blocked · Disallow: /checkout/

Bingbot · blocked

OAI-SearchBot · allowed · no rule

02 / AI CRAWLER RULES

AI search bots and AI training bots, kept apart

Blocking a training bot like GPTBot is a policy choice. Blocking a search bot like OAI-SearchBot removes you from cited answers. The report shows each bot's type so you block the right one.

Google-Extended does not control AI Overviews; Googlebot does. The report says so when it matters.

AI CRAWLERS

Two kinds

Search: OAI-SearchBot, Claude-SearchBot, PerplexityBot

Training: GPTBot, ClaudeBot, Google-Extended, CCBot

03 / RFC 9309 MATCHING

Matched the way crawlers match

A crawler obeys only its most specific user-agent group, the longest matching path wins and Allow wins a tie. Wildcards and the $ end anchor are handled.

That is why a bot with its own group can ignore every rule you wrote under User-agent: *.

ORDER

How a result is decided

Pick the most specific group

Find the longest matching rule

Allow wins a tie

04 / CORRECTED FILE

A fixed file, only when something is wrong

If the check finds a medium or high issue, such as Disallow: / left from staging or blocked CSS and JS, you get a corrected robots.txt with each change listed.

Admin, cart, search and staging blocks stay in place. If nothing needs fixing, the report says no change is needed.

SEVERITY

What counts as high

Disallow: / for Googlebot

A ranking page blocked

Every AI search bot blocked

Numbers from Google's robots.txt rules

Limits the robots.txt checker applies when it reads your file, from Google's own documentation.

of robots.txt that Google reads
500 KiB
Google pauses crawling after a robots.txt 5xx
12 hours
errors (except 429) count as no robots.txt
4xx

From Google Search Central: how Google interprets the robots.txt specification, checked on 2026-11-19.

What Google and OpenAI say about robots.txt

The rules behind a robots.txt checker come from the crawler owners' own documentation. These are the points site owners most often get wrong.

Checked on

Does robots.txt keep a page out of Google?

No. It controls crawling. A blocked URL can still be indexed and shown if other sites link to it.

“This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google.”

— Google Search Central

Source: Google Search Central: introduction to robots.txt

What happens when Allow and Disallow conflict?

Google picks the most specific rule by path length; when rules still conflict, it uses the least restrictive one.

“In case of conflicting rules, including those with wildcards, Google uses the least restrictive rule.”

— Google Search Central

Source: Google Search Central: how Google interprets robots.txt

Are OpenAI's search and training crawlers separate?

Yes. OAI-SearchBot surfaces sites in ChatGPT search; GPTBot crawls content that may be used for training. You can allow one and block the other.

“OAI-SearchBot is used to surface websites in search results in ChatGPT's search features.”

— OpenAI documentation

Source: OpenAI: overview of OpenAI crawlers

How to test a robots.txt file

A run of the robots.txt checker needs your domain; the paths are optional. Here is how to go from a check to a fixed file.

1

Enter your domain and paths

Paste your homepage URL. Add the paths you care about, one per line, such as /blog/ or /checkout/; with none, the homepage is tested.

2

Read the results and the deciding rules

Check the path table and the AI crawler table. Every blocked result names the group and line, so you know exactly what to edit.

3

Replace the file and re-check

If the robots.txt checker returns a corrected file, review each listed change, upload it to your site root and run the check again.

A FEW THINGS TO KNOW

Before you edit your robots.txt

Indexing versus crawling, AI crawlers, rule order and what the robots.txt checker leaves out.

What is a robots.txt checker?

A robots.txt checker reads the robots.txt file at your site root and tells you which crawlers may fetch which URLs. SpringBrand's robots.txt checker tests your paths against Googlebot, Bingbot and the main AI crawlers, quotes the rule that decides each result and flags lines that block more than you meant.

Does blocking a page in robots.txt keep it out of Google?

No. Disallow stops crawling, not indexing. Google says a blocked URL can still appear in search results if other pages link to it. To keep a page out, let Google crawl it and add a noindex tag; a blocked page hides its noindex from Google.

Should I block GPTBot and other AI crawlers?

It depends on which kind. Training bots such as GPTBot, ClaudeBot, Google-Extended and CCBot collect content for model training; blocking them is a business choice. Search bots such as OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages for cited answers; block them and you drop out of those answers.

Which rule wins when Allow and Disallow both match?

The longest matching path wins. If an Allow and a Disallow rule are the same length, Google uses the least restrictive one, so Allow wins. A crawler with its own User-agent group ignores the * group entirely, which is a common source of surprises.

Does it check meta robots tags or Search Console?

No. This tool reads robots.txt only. Meta robots tags, X-Robots-Tag headers and indexing reports in Search Console are listed as not checked. For a page-level audit, use the SEO checker.

Can it check a robots.txt I haven't published yet?

The robots.txt checker reads the live file at your domain. If you are drafting a new one, paste the text into the chat in SpringBrand and the agent will test it the same way.

START WITH YOUR DOMAIN

Check your robots.txt

New accounts get 100 credits a month with no card. Run the robots.txt checker on your domain, read the deciding rules and paste in the corrected file if you need one.