robots.txt
A file that tells crawlers which parts of a site they may or may not access.
Key takeaways
- robots.txt is a file that tells crawlers which parts of a site they may or may not access.
- Allow the crawlers you want, including AI search crawlers such as OAI-SearchBot and PerplexityBot.
- Avoid: using robots.txt to keep a page out of the index; use noindex instead.
What is robots.txt?
robots.txt controls crawling, not indexing. To prevent indexing use a noindex meta tag. A common mistake is blocking CSS or JS, which prevents Google from rendering the page properly.
Google's introduction to robots.txt explains what the file can and can't control.
Why robots.txt matters for SEO in 2026
robots.txt controls which crawlers can fetch which parts of your site. One wrong line can hide a whole site from Google or AI assistants, so it deserves a careful check whenever it changes.
For a wider view of where this fits, see our full SEO glossary or explore the SEO and GEO service we run for clients.
How to apply robots.txt on your site
Knowing the concept is only half of it. Work through these steps in order to put robots.txt into practice on your own pages:
- Allow the crawlers you want, including AI search crawlers such as OAI-SearchBot and PerplexityBot.
- Disallow only areas with no search value, such as admin and internal search.
- Point to your sitemap with a Sitemap line.
Common mistakes to avoid
Most sites that struggle with robots.txt fall into the same traps. Check your pages against each of these before moving on:
- Using robots.txt to keep a page out of the index; use noindex instead.
- Blocking CSS or JavaScript files that pages need to render.
- Copying a staging robots.txt that blocks everything into production.
Questions about robots.txt
A file that tells crawlers which parts of a site they may or may not access.
robots.txt controls which crawlers can fetch which parts of your site. One wrong line can hide a whole site from Google or AI assistants, so it deserves a careful check whenever it changes.
Start here: allow the crawlers you want, including AI search crawlers such as OAI-SearchBot and PerplexityBot. Then disallow only areas with no search value, such as admin and internal search.
The most common is using robots.txt to keep a page out of the index; use noindex instead. Another is blocking CSS or JavaScript files that pages need to render.