0

I write few spiders in the same project, which call the same pipeline class.

In pipeline code like this:

def process_item(self, item, spider):
    if spider.name == 'spider1':
        # do something
    if spider.name == 'spider2':
        # do something
    ...

And it works, before I let my new spiedrX call it.

And the code my spiderX call just like how other spider call:

# just like this kind of way
for book in books:
    item = BookItem();
    # assignment
    yield item

Then it doesn't call pipeline but others can, I can't figure out why.

When I run spiderX, there is a line scrapy output:

2016-07-17 22:30:15 [scrapy] INFO: Enabled item pipelines:
['dgut_spider.pipelines.CustomPipeline']

It means that pipeline can work I guess

Any suggestions will appreciate.

Edit: just my new spiderX can't call pipeline, others can.

Smith John
  • 1,035
  • 1
  • 10
  • 19
  • First, a spider does not call a pipeline in Scrapy. Second, by default Scrapy runs one single spider (you can't run two spiders together). Finally, are you sure you have two spiders with different `name` values? – Djunzu Jul 20 '16 at 22:58
  • @Djunzu I post a little bit more detail question in [here](http://stackoverflow.com/questions/38432786/scrapy-a-weird-bug-code-that-cant-call-pipeline/38459442#38459442), and my code is the reason. – Smith John Jul 21 '16 at 02:02
  • you edit your `settings.py` file?? – oldboy Jun 25 '18 at 01:59

0 Answers0