0

I'm working with scrapy. I have a pipieline that starts with:

class DynamicSQLlitePipeline(object):

    @classmethod
    def from_crawler(cls, crawler):
        # Here, you get whatever value was passed through the "table" parameter
        table = getattr(crawler.spider, "table")
        return cls(table)

    def __init__(self,table):
        try:
            db_path = "sqlite:///"+settings.SETTINGS_PATH+"\\data.db"
            db = dataset.connect(db_path)
            table_name = table[0:3]  # FIRST 3 LETTERS
            self.my_table = db[table_name]

My spider starts with:

class For_Spider(Spider):

    name = "for"
    table = 'hello' # creating dummy attribute. will be overwritten

    def start_requests(self):

        self.table = self.dc # dc is passed in

When I start the spider with:

scrapy crawl for -a dc=mystring -a records=1

dc is passed in as a instance attribute (not a class attribute)

How do I pass this instance attribute to my pipeline so that I can dynamically set the table name in my pipeline? What I have now works for a class attribute.

user1592380
  • 34,265
  • 92
  • 284
  • 515
  • aren't you already doing that in the pipeline constructor? – eLRuLL Dec 24 '17 at 19:36
  • If I understand correctly : table = getattr(crawler.spider, "table") in the pipeline constructor is getting a class attribute ( which I have to set statically). I want to be able to pass in the table name in as a parameter so I can set in from the command line. – user1592380 Dec 24 '17 at 21:13
  • not correct, because `crawler.spider` already has the `Spider` instance, not the class. – eLRuLL Dec 24 '17 at 21:14
  • Sorry, I don't have a good understanding of this as yet. Would you mind expanding on that. My original question yesterday btw was https://stackoverflow.com/questions/47955005/attributeerror-spider-object-has-no-attribute-table – user1592380 Dec 25 '17 at 00:34
  • I'm assuming you are talking about https://doc.scrapy.org/en/latest/topics/api.html#scrapy.crawler.Crawler – user1592380 Dec 25 '17 at 00:59
  • see [Using arguments in scrapy pipeline on __init__](https://stackoverflow.com/questions/27513707/using-arguments-in-scrapy-pipeline-on-init) – furas Dec 25 '17 at 02:32

0 Answers0