Directory
AI Crawler Directory
DataSaaS recognizes 56 crawlers from 21 operators. For the ones that publish IP ranges, we confirm every hit against the operator's own list. The rest are recorded and flagged unverified.
AI answers
15 crawlersChatGPT-User
OpenAI
UA + IP rangeClaude-User
Anthropic
UA + IP rangeGoogle-Agent
Google NotebookLM
Google Read Aloud
Perplexity-User
Perplexity
UA + IP rangeMicrosoft Copilot
Microsoft
UA + IP rangeAmzn-User
Amazon
UA + IP rangeDuckAssistBot
DuckDuckGo
UA + IP rangeMeta External Fetcher
Meta
User agent onlyxAI SearchBot
xAI
User agent onlyGrok DeepSearch
xAI
User agent onlyMistralAI-User
Mistral AI
User agent onlyKimi-User
Moonshot AI
User agent onlyQwen-User
Alibaba
User agent onlySearch index
15 crawlersOAI-SearchBot
OpenAI
UA + IP rangeClaude-SearchBot
Anthropic
UA + IP rangeGooglebot
Google-InspectionTool
PerplexityBot
Perplexity
UA + IP rangeBingbot
Microsoft
UA + IP rangemsnbot
Microsoft
UA + reverse DNSApplebot
Apple
UA + IP rangeAmzn-SearchBot
Amazon
UA + IP rangeBaiduspider
Baidu
UA + reverse DNSMeta Web Indexer
Meta
User agent onlyMistralAI-Index
Mistral AI
User agent onlyTikTokSpider
ByteDance
User agent onlyKimi-SearchBot
Moonshot AI
User agent onlyYouBot
You.com
User agent onlyTraining
14 crawlersGPTBot
OpenAI
UA + IP rangeClaudeBot
Anthropic
UA + IP rangeAmazonbot
Amazon
UA + reverse DNSCCBot
Common Crawl
UA + IP rangeERNIE Bot
Baidu
User agent onlyMeta External Agent
Meta
User agent onlyBytespider
ByteDance
User agent onlycohere-ai
Cohere
User agent onlyCohere Training Data Crawler
Cohere
User agent onlyKimiBot
Moonshot AI
User agent onlyQwenBot
Alibaba
User agent onlyChatGLM-Spider
Zhipu AI
User agent onlyDeepSeekBot
DeepSeek
User agent onlyAI2Bot
Allen Institute for AI
User agent onlyOther
12 crawlersOAI-AdsBot
OpenAI
UA + IP rangeGoogleOther
YiyanBot
Baidu
User agent onlyMeta External Ads
Meta
User agent onlyFacebook External Hit
Meta
User agent onlyFacebookBot
Meta
User agent onlyGrokBot
xAI
User agent onlyxAI-Bot
xAI
User agent onlyxAI Web Crawler
xAI
User agent onlyDoubaobot
ByteDance
User agent onlyTongyiBot
Alibaba
User agent onlyAliyunBot
Alibaba
User agent onlyTrack AI crawlers on your own site
Most AI crawlers fetch raw HTML and never run JavaScript, so a browser tag cannot see them. The DataSaaS server package records every one and verifies the ones that publish IP ranges.
Read the setup guide