The short answer
- Your robots.txt decides which AI bots may read your site. Blocking the wrong one can quietly keep you out of ChatGPT, Claude or Perplexity answers.
- Search bots put you in answers: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot. Allow them.
- Training bots (GPTBot, ClaudeBot) and usage tokens (Google-Extended, Applebot-Extended) control model training. Blocking them is your choice and doesn't remove you from search.
- A bot follows only the most specific group that names it, so a group for one bot must repeat every rule you want it to obey.
What is robots.txt?
robots.txt is a text file at yoursite.com/robots.txt that tells bots which pages they may fetch. It's an internet standard (RFC 9309), and every major search and AI crawler reads it before visiting.
It's a polite request, not a lock. It doesn't hide pages from people, and it doesn't remove a page that's already in search results (that's what noindex is for).
Which AI bots should I know about?
| User agent | Company | What it does | If you block it |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Finds pages for ChatGPT search | You won't be shown in ChatGPT search answers |
| ChatGPT-User | OpenAI | Opens a page when a ChatGPT user asks | OpenAI says robots.txt rules may not apply, because a person asked |
| GPTBot | OpenAI | Collects pages that may be used to train models | Opts you out of training; ChatGPT search is unaffected |
| Claude-SearchBot | Anthropic | Indexes pages for Claude's search results | May reduce your visibility in Claude's search results |
| Claude-User | Anthropic | Fetches a page when a Claude user asks | May reduce your visibility for user-directed web search |
| ClaudeBot | Anthropic | Collects training data | Opts you out of training |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity search | You won't appear in Perplexity's search results |
| Perplexity-User | Perplexity | Visits a page to answer a user's question | Perplexity says it generally ignores robots.txt, because a person asked |
| Googlebot | Google Search, including AI Overviews and AI Mode | You disappear from Google Search | |
| Google-Extended | Not a crawler: controls use for training and grounding in some of Google's other AI systems | Doesn't affect Google Search or AI Overviews | |
| Bingbot | Microsoft | Bing search, which Copilot's web answers draw on | You drop out of Bing and Copilot's web answers |
| Applebot | Apple | Search in Spotlight, Siri and Safari | You leave Apple's search features |
| Applebot-Extended | Apple | Not a crawler: controls training of Apple's foundation models | Doesn't affect Apple's search |
| DuckAssistBot | DuckDuckGo | Fetches pages for DuckAssist answers; not used for training | DuckDuckGo stops visiting within about 72 hours |
What should a small business allow?
If you want to be recommended, allow every search and answer bot. This file does that, keeps a private area closed and points bots to your sitemap:
robots.txt that welcomes search and answer engines
# Search and answer engines: allowed everywhere except private areas
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
User-agent: DuckAssistBot
Allow: /
Disallow: /admin/
# Everyone else
User-agent: *
Allow: /
Disallow: /admin/
Sitemap: https://www.example.com/sitemap.xmlHow do I block AI training but stay in AI search?
Add a group for the training bots and tokens. Your search bots keep their own group, so you stay in ChatGPT, Claude and Perplexity answers and in Google Search.
Add this group to opt out of model training
# Opt out of AI model training (search is unaffected)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /This is a business choice, not a technical requirement. OpenAI, Anthropic, Google and Apple document training and search as separate controls, so you can opt out of one and keep the other.
What are the most common mistakes?
- A leftover `Disallow: /`. Staging sites often block everything, and the rule survives the launch. One line hides the whole site from every bot.
- A bot group that forgets the shared rules. Because the most specific group wins, a group for Googlebot with no
Disallow: /admin/lets Googlebot into /admin/. - Blocking Googlebot to “stop AI”. It removes you from Google Search altogether. Use Google-Extended for AI training instead.
- Typos in bot names. Matching ignores upper and lower case, but not spelling:
PerplexityBotworks,Perplexity Botdoesn't. - The file in the wrong place. It must sit at the root of each host.
shop.example.comneeds its own robots.txt. - Firewalls that block bots anyway. Some security plugins and CDNs block AI crawlers before robots.txt is ever read. Check your host's or CDN's bot settings too.
How do I check my robots.txt?
- Open
yoursite.com/robots.txtin a browser and read it top to bottom. - Run the free AI visibility checker: it tests the file against each assistant's bot and says, in plain words, who can read your site.
- If you use Google Search Console, its robots.txt report shows the version Google last fetched.
Google says it generally caches robots.txt for up to 24 hours, and DuckDuckGo says changes take about 72 hours to reach DuckAssistBot, so give a fix a few days before judging it.
Questions people also ask
Type to search these questions, or ask anything about getting found on Google and AI.
Will blocking GPTBot remove me from ChatGPT?
No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, which you can keep allowed.
Do I need a separate rule for every AI bot?
No. If you don't name a bot, it follows your catch-all group (User-agent: *). Name bots only when you want them treated differently, and repeat any shared rules in that group.
Can robots.txt stop people asking ChatGPT about my site?
Not fully. OpenAI and Perplexity say their user-triggered fetchers (ChatGPT-User, Perplexity-User) may not follow robots.txt, because a person asked for the page. Anything private belongs behind a login, not just a robots.txt rule.
Sources
- RFC 9309: Robots Exclusion Protocol(opens in a new tab) · IETF
- How Google interprets the robots.txt specification(opens in a new tab) · Google Search Central
- Overview of OpenAI crawlers(opens in a new tab) · OpenAI
- Does Anthropic crawl data from the web, and how can site owners block the crawler?(opens in a new tab) · Anthropic
- Perplexity crawlers(opens in a new tab) · Perplexity
- AI features and your website(opens in a new tab) · Google Search Central
- About Applebot(opens in a new tab) · Apple
- DuckAssistBot(opens in a new tab) · DuckDuckGo
- Transparency note for Microsoft Copilot(opens in a new tab) · Microsoft
Checked on . AI products change quickly; the operators' own pages are the final word.