What is an AI crawler?
An AI crawler is an automated agent that fetches web content to feed an AI system — training data pipelines, retrieval-augmented answer engines, and shopping/task agents — rather than to build a classic search index.
The distinction matters because AI crawlers vary widely in behavior: some render JavaScript, many do not; some respect robots.txt and dedicated user-agents for opt-out, others are more aggressive; and a growing set look for agent-oriented files (llms.txt, llms-full.txt) as a fast path to clean content. For a brand, the practical implications are: serve important content in server-rendered HTML rather than client-only rendering so non-rendering crawlers can read it; keep schema markup complete so facts are machine-parseable; use robots.txt and per-crawler user-agent rules deliberately to allow the crawlers you want and manage the ones you don't; and maintain an agent-readable layer that advertises your most citable pages. Blocking the wrong crawler can quietly remove a brand from the AI answers where its customers now research.
Related terms covered here
- PerplexityBot
- PerplexityBot is Perplexity's crawler that maintains the retrieval index feeding its answer engine; a companion user-agent, Perplexity-User, fetches a page live when a user follows a citation. Both feed the answer surface, so a brand that wants to be cited on Perplexity should allow both in robots.txt.
- GPTBot
- GPTBot is OpenAI's web crawler used to gather content for training and improving its foundation models. It is distinct from OpenAI's answer/retrieval crawlers (OAI-SearchBot and ChatGPT-User), which fetch pages to build cited answers rather than to train — a distinction that matters enormously for AI-search access control.
- Google-Extended
- Google-Extended is a robots.txt token that lets a site opt out of having its content used to train Google's generative-AI models (Gemini and Vertex AI) — without affecting how Googlebot crawls the site for Google Search or AI Overviews.