AI crawlers
AI crawlers are automated programs operated by AI companies to fetch web documents, and each company runs several of them for different purposes.
OpenAI, for example, separates the crawler used for search answers from the one used for model training, and recommends allowing OAI-SearchBot to appear in search answers. Each can be allowed or blocked independently in robots.txt.
Related terms
Guides covering this term
- How ChatGPT, Claude, Perplexity, and Gemini Decide to Cite Your Site
- The Complete llms.txt Guide: Syntax and Examples
All terms
Back to glossary