Term
robots.txt
Updated · 2 min read · By The ParSEO team
The short answer
A text file at a site's root that tells crawlers which parts of the site they may or may not crawl.
robots.txt sits at /robots.txt and gives crawlers rules, grouped by user agent: which paths they may crawl and which they may not. It can also point to your sitemap.
Two things it doesn't do: it doesn't hide a page from search results by itself (a blocked page can still be indexed from links, without its content), and it doesn't enforce anything; well-behaved crawlers follow it. To keep a page out of Google, use noindex instead.
For AI visibility it matters a lot: Google, OpenAI, Anthropic and Perplexity all document the user agents their crawlers use, and you decide which may read your site. A leftover Disallow: / from a staging site is one of the most common reasons a site disappears.
In practice
In practice, open yourdomain.com/robots.txt and read it once. Make sure it doesn't block pages you want found, that it allows the AI crawlers you want to be read by, and that it lists your sitemap. Keep private areas (an admin area, internal search results) blocked, and remember that robots.txt is public: anyone can read it.
Sources
Read next
See what Google and AI say about your business.
Add your website and get your Day 0 in minutes: where you rank, what AI answers say, and what to fix first.
Start your Day 0