robots.txt checker: how to check yours by hand, AI crawlers included

Open the file

robots.txt lives at the top level of a site, for example https://www.example.com/robots.txt. Google says the rules apply only to the host, protocol, and port where the file sits. A subdomain such as shop.example.com has its own file, so open that one too if your booking or shop runs there.

If the file returns "not found" (404), Google treats it as if there were no robots.txt and assumes no crawl restrictions.

Read five common patterns

"User-agent" says which crawler a group of rules is for. "Disallow" says where it should not go. The readings below follow Google's specification and the crawler companies' own documentation. Other crawlers may read the file differently.

• "User-agent: *" then "Disallow: /": asks every crawler to skip the whole site. Unless Googlebot has its own group, this applies to Google Search too. On a live site, find out why it is there
• "User-agent: OAI-SearchBot" then "Disallow: /": blocks that crawler by name. OpenAI says such sites will not be shown in ChatGPT search answers, though they can still appear as navigational links
• "User-agent: *" then "Disallow: /wp-admin/": blocks only the WordPress admin area. Your public pages are not blocked
• "Disallow:" with nothing after it: Google ignores a rule without a path, so it blocks nothing
• Both "User-agent: *" and a named group: Google's crawlers follow only the most specific group that matches them and ignore the rest

Search crawlers and training crawlers are different

To check whether you can appear in AI search answers, look for the search crawlers. These names come from each company's documentation.

• Googlebot: Google Search. Google's AI features in Search are controlled through the same robots.txt rules for Googlebot
• OAI-SearchBot: ChatGPT search
• Claude-SearchBot: improves search results for Claude users
• PerplexityBot: Perplexity search results

GPTBot and ClaudeBot collect content that may be used for training. You can decide on those separately from search. ChatGPT-User and Perplexity-User act on a person's request. OpenAI says robots.txt rules may not apply to it, and Perplexity says its fetcher generally ignores them.

Check in Search Console

If you use Search Console, the robots.txt report in Settings shows which robots.txt files Google found for up to 20 hosts, when it last fetched them, and any errors. After a fix, you can request a recrawl of the file from that report. The report is available for Domain properties and for URL-prefix properties without a path.

What robots.txt cannot tell you

Knowing that you do not block a crawler still leaves these open.

• It is not a guarantee. Google says respectable crawlers follow the rules, but others might not
• A CDN, firewall, or server setting can block crawlers without anything in robots.txt. Perplexity notes that a firewall may need to allow its crawler
• To keep a page out of search, Google points to noindex. A page blocked in robots.txt cannot be read, so its noindex is not seen either
• Not blocking a crawler does not mean you will appear in AI answers

Changes also take time. Google generally caches robots.txt for up to 24 hours. OpenAI says its search systems can take about 24 hours to adjust.

If you find a block

Do not delete it right away. Ask whoever manages the site why it is there. It may be left over from development, or it may be a company decision. A "Disallow: /" under "User-agent: *" can also stop Google Search, so ask about that one today. That is our advice.

We are preparing a service that runs checks like this and sends one to three fixes each morning. It cannot see CDN or server settings, and it cannot promise a place in AI answers. It is not available in your country yet. Leave your email below to hear when it opens.