Common Crawl: The Open Search Engine

[blockquote type="blockquote_line" align="left"]Our mission is to democratize access to web information by producing and maintaining an open repository of web crawl data that is universally accessible. We store the crawl data on Amazon’s S3 service, allowing it to be bulk downloaded as well as directly accessed for map-reduce processing in EC2.[/blockquote] Somewhere in the IP range between and lurks a fresh new bot, crawling and weaving an open web free from the shroud of mystery and secret algorithms of Google and Bing. Ladies and gentlemen meet Common Crawl’s ccBot identified as: User-Agent string:CCBot/1.0 (+

common-crawlCommon Crawl host their data with Amazon’s S3 service and their crawl data can be accessed directly for map-reduce processing in EC2 or downloaded as a bulk. They obey common search engine commands such as nofollow tags (for link metrics not accessing the content), meta data, robots.txt and there is support for gzip encoding format.

This could be a wonderful new start for development of search engines in the climate of Google’s dominance.


Dan Petrovic, the managing director of DEJAN, is Australia’s best-known name in the field of search engine optimisation. Dan is a web author, innovator and a highly regarded search industry event speaker.

More Posts - Website