Scrapy pipeline open_sider not being called

Question

I see this question, and I see this question, but the official Scrapy page on pipelines mention nothing about signals.

This is the code I have so far:

import sqlite3
import datetime
from scrapy import signals
from scrapy.xlib.pydispatch import dispatcher
from scrapy import log

class Sqlite3StorePipeline(object):
    def __init__(self):
        print 'wtf'
        dispatcher.connect(self.open_spider, signals.spider_opened)
        dispatcher.connect(self.close_spider, signals.spider_closed)

    def open_spider(self, spider):
        print 'wtf2'
        ...

When I run my spider using the command scrapy crawl <spider name>, I only see "wtf" but not "wtf2" being output.

How can I get my open_spider function to be called?

score 0 · Answer 1 · answered Oct 09 '12 at 00:23

0

I'm playing with scrapy right now, and I've had odd issues with stdout so far. Have you tried logging 'wtf'2? i.e.

    def open_spider(self,spider):
        spider.log('wtf2 from spider %s' % spider.name)

answered Oct 09 '12 at 00:23

Winawer

671
8
26

Huh, interesting. I tried it with spider.log as well. No luck there. I'm pretty sure that function is not called because if it was, my database would be initialized in a certain way that it isn't. – wrongusername Oct 09 '12 at 15:25

score 0 · Answer 2 · answered Oct 09 '12 at 10:21

0

Which version of scrapy are you using? the signals API was changed around 7 months ago, affecting 0.15 and later.

answered Oct 09 '12 at 10:21

Shane Evans

2,234
16
15

I'm using Scrapy 14.4. Should I upgrade to the development branch? – wrongusername Oct 09 '12 at 15:24

Scrapy pipeline open_sider not being called

2 Answers2