The AI Crawler Reference

Three jobs, not one #

An AI crawler is doing one of three jobs, and the distinction decides what blocking it costs you.

• Training crawlers collect content that becomes part of a model's baseline knowledge. Effects are slow, diffuse, and arrive with the next model version. GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot and TongyiBot are in this group.
• Search crawlers build the index an assistant queries at answer time. This is the group that decides whether your business can appear in a live, grounded answer today. OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amazonbot and QwenBot sit here.
• User-fetch agents retrieve a specific page because a user's conversation demanded it — someone pasted your URL, or the assistant followed a link. ChatGPT-User, Claude-User, Perplexity-User and Qwen-User do this.

The practical consequence: a business that blocks GPTBot to keep its content out of training data, but leaves OAI-SearchBot allowed, stays fully citable in ChatGPT Search. A business that blanket-blocks everything with one wildcard rule has made itself invisible in live answers, usually without intending to.

The crawlers #

Current as of 22 August 2026. "HK reach" marks whether the assistant behind the crawler is available to a consumer in Hong Kong without a VPN — for a Hong Kong business, a crawler whose assistant cannot be opened locally is worth less than one whose can.

DeepSeek publishes no user-agent #

DeepSeek does not publish a crawler user-agent, and its web fetches present as ordinary browser traffic in server logs. Because robots.txt operates by matching user-agent tokens, there is no directive that can target DeepSeek — neither to allow it nor to block it. Tokens like DeepSeekBot circulate widely in community blocklists and vendor documentation, but they are not officially published, which means a rule naming one is matching nothing.

This matters more than it sounds for anyone selling into Hong Kong or the wider Chinese-speaking market, where DeepSeek carries substantial consumer usage. Your DeepSeek visibility cannot be managed through crawler permissions at all. It is governed entirely by what the model retrieves and what third-party sources say about you — which puts the weight on directory listings, review platforms and consistent public facts rather than on anything in your own robots file.

Why the Chinese crawlers outrank OpenAI's in Hong Kong #

Most AI-crawler references are written for American businesses and treat GPTBot as the most important token on the list. For a business serving Hong Kong that ordering is inverted. ChatGPT and Claude are geofenced out of Hong Kong, so GPTBot and ClaudeBot govern access to assistants your local customers cannot open. Meanwhile QwenBot, Qwen-User, Bytespider and PetalBot govern access to the assistants they can — and those are the ones carrying Chinese-language queries about your business. A Hong Kong clinic with a meticulous OpenAI robots policy and no Alibaba rules has optimised for the wrong audience.

A robots.txt that welcomes all of them #

For most businesses the right answer is to allow everything: a practice that wants AI to recommend it has little to gain from blocking the crawlers that make recommendation possible. Publishers who license their content are the real exception, and they should be making that decision deliberately rather than inheriting it from a default.

The mistake worth avoiding is subtler than blocking: a Disallow under a wildcard User-agent: *, or a Cloudflare AI-bot toggle left on, will quietly exclude every crawler in the table above regardless of what the rest of the file says. Check the platform setting as well as the file — the setting usually wins.

Verifying a crawler is who it claims #

User-agent strings are trivially forged, so a hit in your logs claiming to be GPTBot may be a scraper wearing its name. The vendors that care about this publish IP ranges for their crawlers, and a reverse DNS lookup on the requesting IP is the standard verification. This matters mainly if you are making decisions from crawler traffic — measuring which assistants actually visit you, or rate-limiting. For simply being allowed, forgery is not the concern.