0

I'm trying to get a full picture of my crawls in a database (mySQL).

So I need any errback stuff to get logged to the database.

Is it possible to pass errback to the pipelines? I currently have it set up like so:

Reponse -> (Item) -> Pipeline

When the pipeline starts it creates an entry for that crawl (through open_spider). That entry's table ID is then stored as a variable in the pipeline. Which is why I can't log the errback through the spider itself.

Or is there a way to pass the crawl_id ID from the pipeline to the spider's errback?

Akustik
  • 43
  • 8
  • Ok, I've found out that you can pass variables from the pipeline to the spider. It's possible by using `spider.{yourvariable} = {yourvalue}` inside the `open_spider` function of the pipeline. Still, I'd prefer to handle the error logging in the pipelines if there's a way to do that. – Akustik Nov 02 '17 at 09:27
  • 1
    I imagine you already have an Item subclass for your items. You could create a second Item subclass for yours messages to be logged. Your errback function will than yield this another item with the desired message. Then you must have specific pipelines for each item and all pipelines must check the item type received. – Djunzu Nov 02 '17 at 22:57
  • Thanks, I wasn't aware that errback can also return items that would be passed to the pipelines. – Akustik Nov 06 '17 at 08:55

0 Answers0