Free tool
AI agent readiness checker: can ChatGPT and Google read your site?
Type your website. ParSEO reads its robots.txt the way each crawler does, visits as ChatGPT's robot, and shows who is allowed in and whether the page is easy for an AI agent to use.
Runs on ParSEO, with a small daily limit. No sign-up.
ParSEO reads only your public pages and files, the same ones every crawler reads. It changes nothing.
What this checker shows
Every crawler that reads your site looks for robots.txt first and follows the group that names it, or the * group if none does. The checker reads your robots.txt by the same rules (RFC 9309) and shows, for each crawler, whether it may read your home page. It also tells you whether you have an llms.txt file and which sitemaps robots.txt lists.
Search crawlers, AI search crawlers and training crawlers
They do different jobs, so block them separately:
- Search crawlers (Googlebot, Bingbot) decide whether you can appear in Google and Bing at all. Bing also feeds Microsoft Copilot.
- AI search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, and ChatGPT-User or Claude-User when a person asks the assistant to open a page) decide whether an AI assistant can read and cite your site in an answer.
- Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) collect pages to train future models. Blocking them doesn't remove you from search or AI answers.
How to fix a block you didn't mean
Many sites copied a "block all AI" robots.txt in 2023 and 2024. Those often block AI search too. To stay out of training but be found in AI answers, give each AI search crawler its own group that allows it, and block only the training crawlers:
User-agent: OAI-SearchBotthenAllow: /, and the same for PerplexityBot and Claude-SearchBot.User-agent: GPTBotthenDisallow: /if you don't want OpenAI to train on your pages; the same for ClaudeBot, Google-Extended and CCBot.
When the checker finds a block, it writes these lines for you to copy. In WordPress, an SEO plugin such as Yoast or Rank Math can edit robots.txt for you. On Shopify, edit the robots.txt.liquid template. Changes count the next time each crawler reads the file, usually within a day.
Where robots.txt stops
robots.txt is a request, not a lock: respectful crawlers follow it, and the companies above say theirs do. It also doesn't stop a page that is linked elsewhere from appearing in Google; for that you need a noindex tag. The website health check checks both.
Questions people ask
Does blocking GPTBot remove me from ChatGPT answers?
No. GPTBot collects pages for training. ChatGPT search reads pages with OAI-SearchBot, and ChatGPT-User when a person asks it to open a page, so those decide whether ChatGPT can cite you.
What does Google-Extended do?
It is a robots.txt token, not a separate crawler. Blocking it tells Google not to use your pages to train Gemini models. It doesn't change Google Search or AI Overviews, which use Googlebot.
I have no robots.txt. Is that a problem?
No. Without robots.txt every crawler may read the whole site. Many sites add one anyway to list their sitemap.
How often can I run the checker?
Twenty live checks a day from one connection, free. A site checked in the last hour shows its saved answer.
Sources
Updated .
More free tools
Read next
Want this checked every day, and fixed?
ParSEO runs these checks and many more on every page, asks AI assistants what they say about you, and fixes what holds you back with your OK.
Start your Day 0