0

I currently parsing WARC files from CommonCrawl corpus and I would like to know upfront, without iterating through all WARC records, how many records are there.

Does WARC 1.1 standard defines such information?

dzieciou
  • 4,049
  • 8
  • 41
  • 85

1 Answers1

1

The WARC standard does not define a standard way to indicate the number of WARC records in the WARC file itself. The number of response records in Common Crawl WARC files is usually between 30,000 and 50,000 - note that there are also request and metadata records. The WARC standard recommends 1 GB as target size of WARC files which puts a natural limit to the number of records.

Sebastian Nagel
  • 2,049
  • 10
  • 10