# The AI Crawler Reference

**Source:** https://simplyai.work/learn/ai-crawler-reference/
**Publisher:** SimplyAI (https://simplyai.work/) · **Language:** en · **Updated:** 2026-08-22

> **TL;DR** — AI crawlers do three different jobs (training, search indexing, on-demand user
> fetching) and conflating them is the most common and most expensive robots.txt mistake. Blocking a
> training crawler costs you nothing in live answers; blocking a search crawler makes you uncitable.
> Two facts most references miss: DeepSeek publishes no user-agent at all, so robots.txt cannot
> govern it, and the Chinese-model crawlers matter more than OpenAI's for any business serving Hong
> Kong, where ChatGPT and Claude are geofenced out.

## Three jobs, not one

An AI crawler is doing one of three jobs, and the distinction decides what blocking it costs.

- **Training crawlers** collect content that becomes part of a model's baseline knowledge. Effects
  are slow, diffuse, and arrive with the next model version. GPTBot, ClaudeBot, Google-Extended,
  Applebot-Extended, Meta-ExternalAgent, CCBot and TongyiBot.
- **Search crawlers** build the index an assistant queries at answer time — the group that decides
  whether you can appear in a live grounded answer today. OAI-SearchBot, Claude-SearchBot,
  PerplexityBot, Amazonbot, QwenBot.
- **User-fetch agents** retrieve a specific page because a conversation demanded it. ChatGPT-User,
  Claude-User, Perplexity-User, Qwen-User.

Consequence: blocking GPTBot while allowing OAI-SearchBot keeps you out of training data but fully
citable in ChatGPT Search. A blanket wildcard block makes you invisible in live answers, usually
unintentionally.

## The crawlers

Current as of 22 August 2026. "HK reach" marks whether the assistant behind the crawler is available
to a consumer in Hong Kong without a VPN.

| User-agent | Operator | Job | Honours robots.txt | HK reach |
|---|---|---|---|---|
| `GPTBot` | OpenAI | Training | Stated | No |
| `OAI-SearchBot` | OpenAI | Search index | Stated | No |
| `ChatGPT-User` | OpenAI | User fetch | Stated | No |
| `ClaudeBot` | Anthropic | Training | Stated | No |
| `Claude-SearchBot` | Anthropic | Search index | Stated | No |
| `Claude-User` | Anthropic | User fetch | Stated | No |
| `Google-Extended` | Google | Training (Gemini, Vertex) | Stated | Yes |
| `PerplexityBot` | Perplexity | Search index | Stated | Yes |
| `Perplexity-User` | Perplexity | User fetch | Stated | Yes |
| `QwenBot` | Alibaba | Search index | Stated | Yes |
| `TongyiBot` | Alibaba | Training | Stated | Yes |
| `Qwen-User` | Alibaba | User fetch | Stated | Yes |
| `AliyunBot` | Alibaba Cloud | Mixed | Stated | Yes |
| `Bytespider` | ByteDance | Training | Contested | Yes |
| `PetalBot` | Huawei | Search index | Stated | Yes |
| *none published* | DeepSeek | User fetch | N/A | Yes |
| `Applebot-Extended` | Apple | Training | Stated | Partial |
| `Amazonbot` | Amazon | Search index | Stated | Partial |
| `Meta-ExternalAgent` | Meta | Training | Stated | Partial |
| `CCBot` | Common Crawl | Training (feeds many) | Stated | Yes |
| `cohere-ai` | Cohere | Training | Stated | Partial |

## DeepSeek publishes no user-agent

DeepSeek does not publish a crawler user-agent, and its web fetches present as ordinary browser
traffic in server logs. Because robots.txt operates by matching user-agent tokens, there is no
directive that can target DeepSeek — neither to allow nor to block it. Tokens like `DeepSeekBot`
circulate in community blocklists but are not officially published, so a rule naming one matches
nothing.

For anyone selling into Hong Kong or the wider Chinese-speaking market, where DeepSeek carries
substantial consumer usage, this means DeepSeek visibility cannot be managed through crawler
permissions at all. It is governed by what the model retrieves and what third-party sources say
about you — putting the weight on directory listings, review platforms and consistent public facts
rather than on anything in your robots file.

## Why the Chinese crawlers outrank OpenAI's in Hong Kong

Most AI-crawler references are written for American businesses and treat GPTBot as the most
important token on the list. For a business serving Hong Kong that ordering is inverted. ChatGPT and
Claude are geofenced out, so GPTBot and ClaudeBot govern access to assistants local customers cannot
open, while QwenBot, Qwen-User, Bytespider and PetalBot govern access to the ones they can. A Hong
Kong clinic with a meticulous OpenAI robots policy and no Alibaba rules has optimised for the wrong
audience.

## A robots.txt that welcomes all of them

For most businesses the right answer is to allow everything: a practice that wants AI to recommend
it has little to gain from blocking the crawlers that make recommendation possible. Publishers
licensing their content are the real exception, and should make that call deliberately.

The mistake worth avoiding is subtler than blocking: a `Disallow` under a wildcard `User-agent: *`,
or a Cloudflare AI-bot toggle left on, quietly excludes every crawler above regardless of the rest
of the file. Check the platform setting as well as the file — the setting usually wins.

## Verifying a crawler is who it claims

User-agent strings are trivially forged, so a log hit claiming to be GPTBot may be a scraper wearing
its name. Vendors that care publish IP ranges, and reverse DNS on the requesting IP is the standard
verification. This matters if you are making decisions from crawler traffic; for simply being
allowed, forgery is not the concern.

## Frequently asked

**Does DeepSeek have a crawler user-agent?** No. Its fetches present as ordinary browser traffic, so
no robots.txt rule can target it. `DeepSeekBot` is community folklore, not an official token.

**What is the difference between GPTBot and OAI-SearchBot?** GPTBot collects training content, so it
affects what a model knows generally. OAI-SearchBot builds the index ChatGPT Search queries at answer
time, so it affects whether you appear in a live grounded answer. ChatGPT-User fetches a page on
demand during a conversation.

**Which AI crawlers should I allow?** For most businesses, all of them. The search and user-fetch
crawlers matter most because they decide live citability: OAI-SearchBot, Claude-SearchBot,
PerplexityBot, Google-Extended, QwenBot, Qwen-User.

**Do AI crawlers respect robots.txt?** OpenAI, Anthropic, Google, Apple and Perplexity all state
that theirs do. It is stated policy rather than a technical guarantee, and crawlers publishing no
user-agent cannot be governed by robots.txt at all.

## Sources

1. OpenAI. *Overview of OpenAI Crawlers.* https://platform.openai.com/docs/bots
2. Anthropic. *Does Anthropic crawl data from the web?* https://support.anthropic.com/en/articles/8896518
3. Google Search Central. *Google crawlers and user-agent strings.* https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
4. xSeek Docs. *DeepSeek user agents.* https://www.xseek.io/docs/deepseek-user-agents
5. xSeek Docs. *Qwen user agents.* https://www.xseek.io/docs/qwen-user-agents

---

**Cite this page:** SimplyAI (2026). *The AI Crawler Reference.* Updated 22 August 2026.
https://simplyai.work/learn/ai-crawler-reference/