Scrapy Spider which reads from Warc file

Question

I am looking for a Scrapy Spider that instead of getting URL's and crawls them, it gets as input a WARC file (preferably from S3) and send to the parse method the content.

I actually need to skip all the download phase, that means that from start_requests method i would like to return a Response that will then send to the parse method.

This is what i have so far:

class WarcSpider(Spider):

    name = "warc_spider"

    def start_requests(self):
        f = warc.WARCFile(fileobj=gzip.open("file.war.gz"))
        for record in f:
            if record.type == "response":
                payload = record.payload.read()
                headers, body = payload.split('\r\n\r\n', 1)
                url=record['WARC-Target-URI']
                yield Response(url=url, status=200, body=body, headers=headers)


    def parse(self, response):
        #code that creates item
        pass

Any ideas of what is the Scarpy way of doing that ?

score 1 · Accepted Answer · answered Nov 27 '14 at 20:21

1

What you want to do is something like this:

class DummyMdw(object):

    def process_request(self, request, spider):
        record = request.meta['record']
        payload = record.payload.read()
        headers, body = payload.split('\r\n\r\n', 1)
        url=record['WARC-Target-URI']
        return Response(url=url, status=200, body=body, headers=headers)


class WarcSpider(Spider):

    name = "warc_spider"

    custom_settings = {
            'DOWNLOADER_MIDDLEWARES': {'x.DummyMdw': 1}
            }

    def start_requests(self):
        f = warc.WARCFile(fileobj=gzip.open("file.war.gz"))
        for record in f:
            if record.type == "response":
                yield Request(url, callback=self.parse, meta={'record': record})


    def parse(self, response):
        #code that creates item
        pass

answered Nov 27 '14 at 20:21

nramirezuy

171
5

Thanks @nramirezuy. I have 2 issues: 1. the custom_settings here doenst add the middlware. i have to override it in settings.py ? 2. the response is then processed in all midllwares and i get an exception. can i prevent from all other middlware to precess the response ? – Udy Nov 30 '14 at 16:14
@Udy I use master branch, and I don't keep track of when a change is released. 1. So `custom_settings` might not be present on your version, add it to the project settings and filter by spider name I guess. 2. The request and response are processed by SpiderMiddlewares; if you set this mdw as 1 it will be the first called for process the request, if a exception is raised documentation says that process_exception is called, but doesn't say in which order is done, I think you will have to experiment with it a bit to figure it out. – nramirezuy Dec 01 '14 at 17:57

Scrapy Spider which reads from Warc file

1 Answers1