1. Home
  2. Developer
  3. robots.txt Checker

robots.txt Checker

Parse robots.txt rules and test whether bots may crawl a path.

Server lookup · only the address you type is sent, nothing is stored
The domain you type is sent to a server to fetch its robots.txt and is not stored. The robots.txt text box and the paths you test are never sent anywhere; they are analyzed in this browser only.
Example: example.com | Any address works; we fetch /robots.txt of that host. www and non-www hosts have separate robots.txt files.
Edit the fetched text or paste your own to try rules. This box is never sent to a server.
User-agent groups
2
Rules (Allow/Disallow)
5
Sitemap lines
1
Size
0.2 KiB

Check results

No syntax errors or common mistakes found.

Parsed rules

BotsRulesNotes
*Disallow: /admin/Disallow: /cartAllow: /admin/helpDisallow: /*?sort=-
GPTBotDisallow: /-

Sitemaps

#URL
1https://example.com/sitemap.xml

Test whether paths are allowed

A full string like Googlebot/2.1 works; only the leading product name is used. Naver uses Yeti and Daum uses Daumoa.
Paths starting with /, or full URLs. The query string (?a=1) is included in the decision.

No group names googlebot, so User-agent: * applies.

PathResultRule applied
/AllowedNo matching rule (allowed by default)
/admin/loginBlockedDisallow: /admin/ (line 2)
/admin/helpAllowedAllow: /admin/help (line 4)
/products?sort=priceBlockedDisallow: /*?sort= (line 5)
/cartBlockedDisallow: /cart (line 3)

A testing aid that follows Google's reading of robots.txt (RFC 9309). Crawlers differ in details, and robots.txt is a crawl request, not access control.

How to use

  1. Enter example.com in the domain box and press Fetch robots.txt. Any address works; the tool fetches /robots.txt of that host.
  2. The fetched text fills the box below and is parsed immediately. You can edit it or paste your own, so you can try new rules before deploying them.
  3. Under Check results, see syntax errors and common mistakes with line numbers.
  4. The Parsed rules table shows how Allow and Disallow were read for each User-agent group.
  5. In Test whether paths are allowed, enter a bot name and paths (one per line, up to 50) to see whether each is allowed or blocked and which rule decided it.

The page opens with a sample robots.txt so you can try it right away. To write a new file, use the robots.txt generator.

How the rules are applied

robots.txt is read according to RFC 9309 and Google’s documentation:

  • Choosing the group: if a User-agent group matches the bot name, only that group applies; the User-agent: * group is used only when no group names the bot. Rules in * are not merged into a bot’s own group. Consecutive User-agent lines form one group.
  • Longest rule wins: when several rules match, the one with the longest path pattern wins. On a tie, Allow wins.
  • Patterns: * matches any characters and a trailing $ marks the end of the path. Paths are compared case-sensitively and include the query string (?a=1).
  • An empty Disallow blocks nothing. /robots.txt itself is always treated as allowed.
  • A bot name can be a full string such as Googlebot/2.1; only the leading product name is used. Googlebot-Image and similar fall back to the Googlebot group when they have none of their own. Naver uses Yeti and Daum uses Daumoa.

Results for sample rules (Disallow: /admin/, Allow: /admin/help, Disallow: /*?sort= and Disallow: /*.pdf$ in the * group):

Path Result Why
/admin/login Blocked Disallow: /admin/
/admin/help Allowed Allow: /admin/help is longer and wins
/list?sort=asc Blocked Disallow: /*?sort=
/list?page=2 Allowed No matching rule
/a/b.pdf Blocked Disallow: /*.pdf$
/a/b.pdf?x=1 Allowed Because of $, it does not apply when the path does not end in .pdf

Common mistakes it checks

Check Explanation
HTML came back A soft 404: the site answers a missing robots.txt with 200
Missing colon, typos Spellings such as Useragent or Dissallow, or lines like Disallow /x
Rule before User-agent Applies to no bot
Path not starting with /, full URL Disallow: admin, Disallow: https://...
$ in the middle Taken literally unless at the end
Noindex directive Not supported by Google since 2019
Block everything User-agent: * with only Disallow: /
Blocking CSS or JS Can interfere with rendering
Crawl-delay Ignored by Google, honored by a few such as Bing and Yandex
Over 500 KiB Google ignores the rest
Relative Sitemap URL Must be a full URL

If the fetched response is a 4xx such as 404, crawlers assume there are no restrictions; on a 5xx or 429, Google temporarily pauses crawling the whole site. The tool tells you about these statuses too.

Limits of robots.txt

robots.txt is a crawl request, not an access control. Some bots ignore it, and a blocked path can still show up in search results as a bare URL if other sites link to it. To keep a page out of the index use a noindex meta tag or an X-Robots-Tag header, and do not block that page in robots.txt or crawlers will never see the noindex. Protect private pages with a login or server configuration. To inspect a page’s meta tags use the meta tag checker.

What is sent to the server and the limits

  • The only value sent to the server is the domain (address), and it is not stored. The text box with the fetched content and the paths you test are never sent; they are analyzed in this browser.
  • A response must arrive within 8 seconds and at most 1 MiB of the body is read. Up to 5 redirects are followed, and visiting the same host 3 times stops the lookup.
  • Private addresses, internal networks and bare IP addresses are not checked, and only default ports (80, 443) are allowed.
  • Rate limits apply: 6 requests per minute per IP and 30 per minute per target host.
  • This is not a port scanner or a way to load someone else’s site. It fetches one file from one domain, once.
  • The test follows Google’s reading and is for reference; crawlers differ in details.

FAQ

Do www and non-www have different robots.txt files?

Yes. robots.txt applies per host (and scheme and port). example.com and www.example.com can serve different files, so check both. Usually one redirects to the other, and in that case the tool tells you it fetched the redirected address.

Why is Googlebot allowed while other bots are blocked?

If a Googlebot group exists, Google reads only that group and ignores *. A bot with no group of its own follows *. Change the bot name in the test box to see which group applies.

robots.txt only blocks crawling; it does not remove pages from the index. Links from other sites can still expose the bare URL. Use noindex to remove a page, and do not block that page in robots.txt.

Do AI crawlers follow this file?

Major AI companies say that bots such as GPTBot and ClaudeBot follow rules for their own User-agent names, but not every bot does. Enter GPTBot as the bot name to test whether the current rules block it. For sitemap checks use the sitemap checker.

It says there is no robots.txt. Do I need one?

It is not required. Without one, crawlers assume every path may be crawled. Create one if you want to block paths or announce your sitemap location. To see whether your site redirects, use the redirect checker.

The domain you type is sent to a server for the lookup and is never stored.

Can't find the tool you need?

Tell us what you want to do. We build the most requested tools first.