Common Crawl · Training

CCBot

CCBot is Common Crawl's crawler. This crawler collects page content used to train models.

Crawler facts

Operator	Common Crawl
User agent token	`ccbot`
Traffic type	Training
Verification	UA + IP range
IP source	https://index.commoncrawl.org/ccbot.json

What "verified" means for CCBot

For CCBot, "verified" means DataSaaS compares the request IP against Common Crawl's published ranges. A request carrying this user agent from any other IP is recorded as unverified.

Track it

CCBot fetches raw HTML and never runs JavaScript, so no browser tag can see it. Install the DataSaaS server-side package to record every request and verify the ones that publish IP ranges.

Read the bot-traffic setup guide

See which crawlers hit your site

DataSaaS records every AI crawler and search bot, then verifies the ones that publish IP ranges. Browse the full crawler directory.

Start free trial