Last verified: August 2026. Crawler tokens and vendor policy change quickly — confirm against the official links below before applying any rule.
Short Answer: What are the major AI crawler user-agents in 2026?
AI crawlers fall into three functional types: training bots (GPTBot, ClaudeBot, Meta-ExternalAgent), search/citation bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer), and user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher). Google-Extended and Applebot-Extended are different again — robots.txt control tokens with no crawler of their own. Blocking a training bot is a licensing decision. Blocking a search bot can remove you from AI answers entirely.
Quick Summary
- AI crawlers split into three categories: training, search/citation, and user-triggered.
- Blocking a training bot stops future ingestion. It does not remove citations already generated by a search bot.
- Meta runs the same three-bot pattern as OpenAI and Anthropic — but its citation bot, Meta-WebIndexer, is easy to miss because it is rarely covered.
- Google-Extended and Applebot-Extended are policy tokens, not crawlers — they never appear in server logs.
- A user-agent string can be spoofed. Verify by IP range or reverse DNS before trusting a claimed identity.
Most robots.txt confusion comes from treating every AI bot the same way. It is not one decision. It is at least three, and they have different consequences.
This list only cites official vendor sources, checked directly against the pages themselves rather than secondary write-ups. For the deeper mechanics behind OpenAI’s three bots specifically, see GPTBot vs. OAI-SearchBot vs. ChatGPT-User.
What types of AI crawlers exist?
Three functional categories, and the distinction matters before you touch robots.txt at all.
| Type | Purpose | Consequence of blocking |
|---|---|---|
| Training crawler | Collects content for future model development | Stops future training ingestion; existing citations may persist |
| Search / citation crawler | Builds a retrieval index used to generate current AI answers | Can remove your pages from AI-generated answers |
| User-triggered fetcher | Fetches a specific page live when a person asks a question | Blocks on-demand fetches for individual user requests |
The distinction that trips people up: blocking a training crawler is a licensing opt-out. Blocking a search crawler is a visibility decision. They are not the same choice, and treating them the same is how a site loses AI citations by accident while trying to protect its content from training.
The complete AI crawler user-agent table
Every token below is checked against the vendor’s own current documentation, not a secondary aggregator. Where a vendor page was vague or wrong in earlier reporting, the correct source is linked directly.
| Operator | Token | Type | Official source |
|---|---|---|---|
| OpenAI | GPTBot | Training | OpenAI Bots |
| OpenAI | OAI-SearchBot | Search | OpenAI Bots |
| OpenAI | ChatGPT-User | User-triggered | OpenAI Bots |
| Anthropic | ClaudeBot | Training | Anthropic crawler policy |
| Anthropic | Claude-SearchBot | Search | Anthropic crawler policy |
| Anthropic | Claude-User | User-triggered | Anthropic crawler policy |
| Perplexity | PerplexityBot | Search | Perplexity Crawlers |
| Perplexity | Perplexity-User | User-triggered | Perplexity Crawlers |
| Meta | Meta-ExternalAgent | Training | Meta Web Crawlers |
| Meta | Meta-WebIndexer | Search | Meta Web Crawlers |
| Meta | Meta-ExternalFetcher | User-triggered | Meta Web Crawlers |
| Google-Extended | Training opt-out token | Google crawler docs | |
| Apple | Applebot-Extended | Training opt-out token | Apple Applebot support |
| Amazon | Amazonbot | Search | Amazon developer docs |
| Common Crawl | CCBot | Training | Common Crawl |
| ByteDance | Bytespider | Training | No official public documentation found |
That last row is deliberate. Several AI-crawler roundups link a specific “ByteDance docs” page for Bytespider. We looked. There is no official ByteDance page documenting it as of this verification date — only third-party observation of its behaviour in server logs. Treat any source claiming otherwise with suspicion.
The bot most crawler lists miss: Meta-WebIndexer
Most AI-crawler round-ups list only Meta-ExternalAgent and stop there, treating Meta as having a single, training-only bot. Meta’s own documentation says otherwise: Meta runs the identical three-bot pattern as OpenAI and Anthropic.
| Meta bot | Type | What it actually does |
|---|---|---|
| Meta-ExternalAgent | Training | Crawls the web for foundation-model training and product indexing |
| Meta-WebIndexer | Search / citation | Improves Meta AI search results — allowing it is what helps Meta AI cite and link your content |
| Meta-ExternalFetcher | User-triggered | Fetches a specific page on a user’s request; may bypass robots.txt for these fetches |
If your goal is to be cited by Meta AI specifically, Meta-WebIndexer is the bot that matters, not Meta-ExternalAgent. Blocking the training bot while ignoring the indexer is the exact same mistake as blocking GPTBot and assuming you have opted out of ChatGPT Search — it does nothing for the thing you actually wanted to control.
What is GPTBot, and what should you do about it?
OpenAI’s training crawler. It collects public content for future model development and does not power ChatGPT Search — that is OAI-SearchBot’s job. Blocking GPTBot stops future training ingestion. It does not remove content already used to train an existing model, and it does not touch ChatGPT Search eligibility.
User-agent: GPTBot Disallow: /
Verify claimed GPTBot traffic against OpenAI’s published GPTBot IP ranges before trusting the user-agent string alone.
What is OAI-SearchBot, and should you allow it?
The crawler that actually powers ChatGPT Search. If you want to remain eligible for ChatGPT citations, this is the one to allow, verified against OpenAI’s SearchBot IP list.
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: /
That split — block training, allow search and live fetch — is the most common policy among sites that want visibility without contributing to training data. For the reasoning behind it in more depth, see our full breakdown of OpenAI’s three crawlers.
ClaudeBot, Claude-SearchBot, and Claude-User
Anthropic runs the same three-bot structure. Anthropic’s own crawler policy confirms all three respect robots.txt independently, including Claude-User.
- ClaudeBot: training crawler.
- Claude-SearchBot: indexes content for Claude’s web-powered answers.
- Claude-User: on-demand fetcher triggered by a person using Claude.
Two older tokens, anthropic-ai and Claude-Web, are deprecated. If your robots.txt still references either, replace them — they no longer correspond to anything Anthropic operates.
User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: /
Verify Anthropic crawler IPs against Anthropic’s published bot IP list.
PerplexityBot and Perplexity-User
Perplexity is structurally different from OpenAI, Anthropic, and Meta: its own documentation states plainly that neither of its bots is used to train foundation models. There is no separate Perplexity training crawler to block or allow — only PerplexityBot for search and Perplexity-User for live fetches.
User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: /
Verify against Perplexity’s published IP ranges.
Google-Extended and Applebot-Extended: not crawlers
This is the distinction the source material for AI-crawler lists gets wrong most often. Google states directly: “Google-Extended doesn’t have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.” You will never see it in a server log.
It only controls whether content Googlebot already crawls can be used to train Gemini. It has no effect on standard Search indexing, ranking, or Google AI Overviews — Google says so explicitly.
Applebot-Extended works the same way for Apple: a secondary token controlling whether Applebot’s crawl feeds Apple’s generative AI training, separate from whether a page appears in Siri, Spotlight, or Safari results.
User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: /
Amazonbot, CCBot, and Meta-ExternalAgent
Amazonbot builds the index behind Alexa, Kindle, and Amazon Shopping recommendations, per Amazon’s developer documentation. CCBot, run by the nonprofit Common Crawl, feeds the open dataset that many AI labs train on indirectly — blocking it does not target one company, it opts out of the shared corpus, per Common Crawl’s own FAQ.
How do you verify an AI crawler and detect spoofing?
A user-agent string is self-declared. Anything can claim to be GPTBot. Use it as a first clue, then confirm with the source IP.
- Check the user-agent string in your server or CDN logs.
- Extract the source IP from the same log entry.
- Cross-reference official IP ranges: GPTBot, OAI-SearchBot, Anthropic, Perplexity.
- Run a reverse DNS lookup on high-volume requesters and compare the resolved hostname against the vendor’s domain pattern.
- Flag and rate-limit anything claiming a known identity that fails IP verification.
Bytespider is the case most often cited for why this matters — anomalous crawl volume has been reported even on sites that believe they have blocked it, and there is no official documentation to check its behaviour against in the first place.
Copy-paste robots.txt templates
Allow AI search and live-fetch bots, block training
# Block training crawlers User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / # Allow search and citation crawlers User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Meta-WebIndexer Allow: / User-agent: Meta-ExternalFetcher Allow: / User-agent: Amazonbot Allow: /
This is close to the policy we run on this site, though our exact rule set has grown around a couple of gated assets — worth checking the live file rather than assuming ours matches exactly.
Protect checkout, account, and admin paths from every crawler
User-agent: OAI-SearchBot Allow: / Disallow: /admin/ Disallow: /checkout/ Disallow: /cart/ Disallow: /account/ User-agent: PerplexityBot Allow: / Disallow: /admin/ Disallow: /checkout/ Disallow: /cart/ Disallow: /account/
A broad User-agent: * with Disallow: / can conflict with these specific-bot allow rules, and not every crawler resolves that conflict identically. Test the live file after any change, and check server logs 24–48 hours later to confirm the bots you meant to allow are actually showing up.
How should this policy differ by site type?
| Goal | Training bots | Search/citation bots | User-triggered bots |
|---|---|---|---|
| Maximise AI visibility | Allow | Allow | Allow |
| Citations, not training | Block | Allow | Allow |
| Protect proprietary content | Block | Block | Block, or path-restrict |
| Support specific platforms only | Block all | Allow only chosen platforms | Allow only chosen platforms |
There is no universal right answer. But blocking a search or citation bot is the one move that directly costs you visibility — understand that trade-off specifically before applying it, and see how to write content for AI search once access is settled.
How should you monitor and review this over time?
Server and CDN logs, not client-side analytics — most crawlers do not run JavaScript, so tools like GA4 miss them entirely. A useful benchmark: if training-bot requests outnumber search-bot requests by more than roughly 5:1, you may be feeding model development without a comparable citation return, which is worth a policy review.
grep -E "(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Meta-WebIndexer|Meta-ExternalAgent|Google-Extended|CCBot|Bytespider)" /var/log/nginx/access.log \
| awk '{print $1, $14}' | sort | uniq -c | sort -rn
Review the policy quarterly for high-traffic sites, at minimum annually otherwise. Vendor tokens change — Anthropic has already deprecated two of its own. For tooling that automates this kind of monitoring, see our Cairrot review.
The audit checklist
- Review your current robots.txt for every AI-specific user-agent rule already in place.
- Confirm each token against the vendor’s current official documentation, not a third-party list.
- Remove deprecated tokens — at minimum anthropic-ai and Claude-Web.
- Confirm /admin/, /checkout/, /cart/, and /account/ are blocked for every crawler, not only AI ones.
- Pull 30 days of server or CDN logs and filter for known AI user agents.
- Verify high-volume requesters against official IP ranges or reverse DNS.
- Add firewall rules for anything claiming a known identity that fails verification.
- Compare training-bot volume against search-bot volume as a rough ROI check.
- Document every allow/block decision with its date and reasoning.
- Schedule the next review — quarterly for high-traffic sites, annually at minimum otherwise.
Learn More About AEO and AI Marketing at Prompt Insider
Since launching earlier this year, Prompt Insider has become a leading authority on AI marketing, Answer Engine Optimization (AEO), large language models, AI search, AI news, and the evolving future of digital discovery. As AEO becomes one of the hottest topics in marketing, Prompt Insider is helping define the conversation around how brands improve visibility, adapt their content strategies, and stay competitive in an increasingly AI-driven search environment.
Prompt Insider is the go-to resource for answer engine optimization, AI marketing, and AI search. Start with our core guides at thepromptinsider.com:
- What Is AEO? Answer Engine Optimization Explained
- AEO vs. SEO vs. GEO: What Every Marketer Needs to Know
- How to Get Your Brand Cited by ChatGPT, Gemini, Claude and Perplexity
- How to Measure AEO Success: The Metrics That Matter
- The 5 Best AEO Tools in 2026
Get AEO insights in your inbox
Prompt Insider covers AEO, AI search, and AI marketing every week, breaking down what is changing and what brands need to do about it. Sign up for our emails at thepromptinsider.com to get it first.
Frequently Asked Questions
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content for future OpenAI model training. OAI-SearchBot indexes pages for ChatGPT Search and citations. Blocking one does not automatically affect the other.
Does blocking Google-Extended affect Google Search or AI Overviews?
No. Google states directly that Google-Extended controls only Gemini training and has no effect on standard Search indexing, ranking, or AI Overviews.
What does Meta-WebIndexer do, and why does it matter?
It is Meta’s search and citation crawler — allowing it is what lets Meta AI cite and link your content, according to Meta’s own documentation. Most crawler lists omit it entirely and mention only Meta-ExternalAgent, the training bot, which does not control citation eligibility.
Can you trust a crawler’s user-agent string?
No. User-agent strings are self-declared and can be spoofed. Verify the source IP against the vendor’s official published range, or use reverse DNS, before trusting a claimed identity.
Does Perplexity have a separate training crawler?
No. Perplexity’s documentation states directly that neither PerplexityBot nor Perplexity-User is used for AI training — both exist solely for search and live user fetches.
Sources: OpenAI, Anthropic, Perplexity, Meta, Google, Apple, Amazon, Common Crawl.
About the author
Kai Williams
Kai Williams has been in marketing for years, with a long background in SEO before AEO had a name. He stepped into Answer Engine Optimization the moment AI started reshaping how people search, and has been tracking the shift ever since. At Prompt Insider, he covers AEO, AI marketing, and the future of search, breaking down what is changing and what brands need to do about it.
