0

I am attempting to create a database of Digital Object Identifier (DOI) found on the internet.

By manually searching the CommonCrawl Index Server manually I have obtained some promising results.

However I wish to develop a programmatic solution.

This may result in my process only requiring to read the index files and not the underlying WARC data files.

The manual steps I wish to automate are these:-

1). for each CommonCrawl Currently available index collection(s):

2). I search ... "Search a url in this collection: (Wildcards -- Prefix: http://example.com/* Domain: *.example.com) " e.g. link.springer.com/*

3). this returns almost 6MB of json data that contains approx 22K unique DOIs.

How can I browse all available CommonCrawl indexes instead of searching for specific URLs?

From reading the API documentation for CommonCrawl I cannot see how I can browse all the indexes to extract all DOIs for all domains.

UPDATE

I found this example java code https://github.com/Smerity/cc-warc-examples/blob/master/src/org/commoncrawl/examples/S3ReaderTest.java

that shows how to access a common crawl dataset.

However when I run it I receive this exception

"main" org.jets3t.service.S3ServiceException: Service Error Message. -- ResponseCode: 404, ResponseStatus: Not Found, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>common-crawl/crawl-data/CC-MAIN-2016-26/segments/1466783399106.96/warc/CC-MAIN-20160624154959-00160-ip-10-164-35-72.ec2.internal.warc.gz</Key><RequestId>1FEFC14E80D871DE</RequestId><HostId>yfmhUAwkdNeGpYPWZHakSyb5rdtrlSMjuT5tVW/Pfu440jvufLuuTBPC25vIPDr4Cd5x4ruSCHQ=</HostId></Error>

In fact every file I try to read results in the same error. Why is that?

what is the correct common crawl uri's for their datasets?

Has QUIT--Anony-Mousse
  • 76,138
  • 12
  • 138
  • 194
Hector
  • 4,016
  • 21
  • 112
  • 211

2 Answers2

2

The data set location has changed since more than one year, see announcement. However, many examples and libraries still contain the old pointers. You can access the index files for all crawls back to 2013 on s3://commoncrawl/cc-index/collections/CC-MAIN-YYYY-WW/indexes/cdx-00xxx.gz - replace YYYY-WW with year and week of the crawle and expand xxx to 000-299 to get all 300 index parts. New crawl data is announced on the Common Crawl group, or read more about how to access the data.

Sebastian Nagel
  • 2,049
  • 10
  • 10
  • I can now get https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-18/segments/1461860106452.21/warc/CC-MAIN-20160428161506-00000-ip-10-239-7-51.ec2.internal.warc.gz however when I try with s3://commoncrawl/ prefix my java code (show above) still gives the same error. Do I need to set up an Amazon account event to access public s3 datasets via this code? – Hector Jul 28 '17 at 08:51
  • 1
    It's possible to access the data without an AWS account. Easiest, using the [AWS CLI](https://aws.amazon.com/cli/) and `aws --no-sign-request s3 ...`. The [cc-warc-examples](https://github.com/commoncrawl/cc-warc-examples) are supposed to run from a Hadoop cluster on AWS in the us-east-1 region where the Common Crawl data is hosted. – Sebastian Nagel Jul 28 '17 at 11:18
1

To get the example code to work replace lines 24 and 25 with:

String fn = "crawl-data/CC-MAIN-2013-48/segments/1386163035819/warc/CC-MAIN-20131204131715-00000-ip-10-33-133-15.ec2.internal.warc.gz";
S3Object f = s3s.getObject("commoncrawl", fn, null, null, null, null, null, null);

Also note that the commoncrawl group have an updated example.

Mark Lapierre
  • 1,067
  • 9
  • 15
  • how to read to an offset within a WARC file using s3 java library? I get Mark not supported when calling "archiveReader.get(10L)" – Hector Feb 13 '18 at 12:37
  • I don't know. You may want to post a new question. But if you have a look at the docs for [S3Object.getObject](https://jets3t.s3.amazonaws.com/api/org/jets3t/service/S3Service.html#getObject-org.jets3t.service.model.S3Bucket-java.lang.String-java.util.Calendar-java.util.Calendar-java.lang.String:A-java.lang.String:A-java.lang.Long-java.lang.Long-) you might find an answer - it looks like the method accepts offset arguments. – Mark Lapierre Feb 14 '18 at 15:24