ClueWeb09 and ClueWeb12 dataset consists of ~1 billion web pages crawled in January and February 2009 and 2012.
It is used by several tracks of the TREC conference.
This library is a fork of internetarchive/warc cdegroc/warc-clueweb that can handle both ClueWeb09 and ClueWeb12 dataset.
Why I change the code?
- there are certain caveats with ClueWeb09's files. it does not use the standard \r\n end-of-line markers. Moreover, some records are ill-formed.
- cdegroc/warc-clueweb This fork repo could not handle Clueweb12 dataset and another issue is that it could miss 1-2 document when dealiing every .warc.gz file
Only minor modifications to the original library were made.
The original documentation of the warc library still holds.
The documentation of the warc library is available at http://warc.readthedocs.org/.
For simply analyze warc format files such as clueweb09 and clueweb12, you can just use warc/warc.py
For other usage, just reference original document.
This software is licensed under GPL v2. See LICENSE file for details.