How do you open a file stream for reading using Scrapy?

Question

Using Scrapy, I want to use my extracted url to read a binary file into memory and extract the contents.

Currently, I can find the URL on the page using a selector e.g.

myFile = response.xpath('//a[contains(@href,".interestingfileextension")]/@href').extract()

How do I then read that file into memory so that I can look for content in that file?

Many thanks

score 0 · Accepted Answer · answered Mar 25 '16 at 19:39

0

Make a request and explore the content in the callback:

def parse(self, response):
    url = response.xpath('//a[contains(@href,".interestingfileextension")]/@href').extract_first()
    return scrapy.Request(url, callback=self.parse_file)

def parse_file(self, response):
    # response here is the contents of the file
    print(response.body)

answered Mar 25 '16 at 19:39

alecxe

462,703
120
1,088
1,195

Perfect. Thank you! Scrapy makes it very easy. – John Smith Mar 25 '16 at 20:01

How do you open a file stream for reading using Scrapy?

1 Answers1

Linked