llms.txt vs robots.txt: what each file does, and which you need
Updated
Short answer
robots.txt controls which URLs crawlers may fetch, and search engines have used it for years. llms.txt is a newer proposal: a Markdown summary of your site for AI tools. Google says Google Search does not use llms.txt, so it neither helps nor harms your visibility there. Check robots.txt first. Treating llms.txt as optional is our view.
What robots.txt does
robots.txt is a plain text file at the top level of your site, for example https://www.example.com/robots.txt. Google says it tells crawlers which URLs they can access, and that it is mainly used to manage crawler traffic.
It is not a way to keep a page out of Google. Google points to noindex or password protection for that, because a blocked URL can still appear if other sites link to it. Google also notes that respectable crawlers follow the rules, but others might not.
What llms.txt is
llms.txt is a proposal published on llmstxt.org in September 2024. It describes a Markdown file that gives AI tools a short, curated guide to a site. The only required part is a top heading with the site name. Optional parts include a one-line summary and lists of links.
The proposal is open for community input, so read it as a suggestion under discussion rather than a settled standard. That is our reading.
How the two differ
The llms.txt proposal itself says the two files serve different purposes. robots.txt states what automated access is acceptable. llms.txt is meant to be read on demand when an AI tool needs information to help a user.
- robots.txt: rules about access. Read by search engine and AI crawlers. Mistakes can stop search engines from crawling your site
- llms.txt: a summary of content. Not used by Google Search. Mistakes mostly mean an out-of-date summary
- Sitemap: a list of pages you want found. The proposal says a sitemap cannot do the job of llms.txt
What Google says about llms.txt
Google's guide to generative AI features says you do not need new machine-readable files, AI text files, or Markdown to appear in Google Search, because Google Search does not use them. It adds that creating an llms.txt file for other services is fine, and that doing so will neither harm nor help your visibility or rankings in Google Search.
To be eligible for AI features in Google Search, Google says a page must be indexed and eligible to show with a snippet, and the site must not be excluded in the Search generative AI setting in Search Console.
What to check today
The order is our recommendation. One item today is enough.
- Open yoursite.com/robots.txt and look for "Disallow: /" under "User-agent: *"
- Search for your business name and confirm your pages appear
- If you still want an llms.txt, keep it in line with your pages and decide who updates it
Our own site has no llms.txt as of November 8, 2026. We will add one if a service we rely on asks for it in its official documentation.
Where we fit
We are preparing a service that sends a few concrete fixes each morning, including checks like these. It cannot promise a place in AI answers. It is not available in your country yet. Leave your email below to hear when it opens.