Skip to content

Glossary

AI crawler

A program run by an AI company that fetches web pages: for search answers, for a user's live question, or for training models.

Also called: AI bot

The companies run separate bots for separate jobs. OpenAI, for example, uses OAI-SearchBot for ChatGPT search, ChatGPT-User when a person asks ChatGPT to visit a page, and GPTBot for training data.

That split matters: you can block training bots and still allow the search bots that recommend you. Each bot is addressed by its user agent in robots.txt.

Go deeperrobots.txt rules for AI crawlers: what to allow and what to blockRead the guide

Sources

All terms

Be the business AI recommends.

Pick a plan, connect your site and let CiteMagnet fix what's in the way. Cancel any time.