0

Does passing spider instance variables between pipelines work? Unfortunately I do not have the code but I'll try to explain as short and clear as possible.

Order is the following: Pipeline_1: high priority (@700) Pipeline_2: low priority (@900)

In Pipeline_1 I'm defining a spider instance variable with spider.variable=[] in init method and filling it in close_spider method of the same pipeline.

In Pipeline_2 I'm accessing it in spider_closed method (tried close_spider too) but it is empty.

In pipeline_1 I'm processing some items and I need to pass them all at once in pipeline_2 and this is the only solution I've been thinking about.

The Doctor
  • 17
  • 5
  • Could it work? Maybe. But it certainly isn't a good solution. Pipelines should be indipendent entities and should not be communicating with other pipelines. They also shouldn't set instance attributes for spiders they process items for. – Alexander Feb 15 '23 at 23:51
  • I also export the processed items in pipeline_1, would be a better practice to just open the file in pipeline_2 and access the items? – The Doctor Feb 16 '23 at 05:05

1 Answers1

0

So after reading a bit more I found that defining an instance variable in the pipelines is not that great of an idea (also what Alexander mentioned) - I've solved the issue by implementing the pipeline_2 methods into pipeline_1 and do everything in close_spider method.

The Doctor
  • 17
  • 5