-1

I am trying to pass paramters to my spider (ideally a Dataframe or csv) with:

self.client = ScrapinghubClient(apikey)
self.project = self.client.get_project()
job = spider.jobs.run()

I tried using the *args and **kwargs argument type but each time I only get the last result. For example:

data = ["1", "2", "3"]
job = spider.jobs.run(data=data)

When I try to print them from inside my spider I only get the element 3:

def __init__(self, **kwargs):
     for key in kwargs:
        print kwargs[key]



2018-05-17 08:39:28 INFO    [stdout] 3

I think that there is some easy explanation that i just can't seem to understand.

Thanks in advance!

parik
  • 2,313
  • 12
  • 39
  • 67
Emilz
  • 73
  • 1
  • 8

1 Answers1

1

For passing arguments and tags you can do like this

priority = randint(0, 4)
job = spider.jobs.run(
                        units=1,
                        job_settings=setting,  
                        add_tag=['auto','test', 'somethingelse'], 
                        job_args={'arg1': arg1,'arg2': arg2,'arg3': arg3},
                        priority=priority
                    )
parik
  • 2,313
  • 12
  • 39
  • 67
  • Thanks for the anwswer but i still have the same problem, i'm only able to acces the last element of the list. Maybe the spider can only accept string arguments? I used the job_args argument. Is there a way to send lists to the spider? or do i need to serliaze my csv into a string? – Emilz May 17 '18 at 09:46
  • I didn't undrestand your question, if you search how to pass arguments to scrapy spider on scrapinghub, I gave you an example that works for me, if it's not your question, it's not clear your question for me – parik May 17 '18 at 12:52
  • My question was: How to properly pass parameters so that i can print the whole list and not the last element and i had the same result with your method, but i found another solution to it. thx for your reply! – Emilz May 17 '18 at 13:20
  • My method is a good answer for your question :) If you found another solution please post your method if someone else has the same problem :) – parik May 17 '18 at 13:30