3

I got error while using scrapy

ValueError: not enough values to unpack (expected 2, got 1) while json.dumps(form_data)

My code is below is below:

    form_data = {"directory_search_id":"12093",
                 "elements":{ "0" : {"id":"38",
                                   "label":"Name  First",
                                   "attribute_id":"0",
                                   "attribute_type_label_id":"0",
                                   "attribute_part_id": "31",
                                   "value":""},
                              "1" : {"id": "39",
                                   "label":"Name  Last",
                                   "attribute_id":"0",
                                   "attribute_type_label_id":"0",
                                   "attribute_part_id":"34",
                                   "value":""},
                              "2" : {"id":"37",
                                   "label":"PFB License Number",
                                   "attribute_id":"10028676",
                                   "attribute_type_label_id":"0",
                                   "attribute_part_id":"0",
                                   "value":""},
                              "3" : {"id":"35",
                                   "label":"Counties of Practice",
                                   "attribute_id":"10028670",
                                   "attribute_type_label_id":"0",
                                   "attribute_part_id":"0",
                                   "value_ids":{"0" : "235934"}},
                              "4" : {"id":"33",
                                   "label":"Fiduciary Specialties",
                                   "attribute_id":"10028672",
                                   "attribute_type_label_id":"0",
                                   "attribute_part_id":"0",
                                   "value_ids":{"" : ""}},
                              "5" : {"id":"34",
                                   "label":"Languages",
                                   "attribute_id":"10028671",
                                   "attribute_type_label_id":"0",
                                   "attribute_part_id":"0",
                                   "value_ids":{"" : ""}}}
    }
    print(self.start_urls)
    response = FormRequest(url=self.start_urls[0],
                           formdata=json.dumps(form_data),
                           callback=self.parse_contents)

def parse_contents(self, response):
    open_in_browser(response)

I got these errors:

   2019-09-17 21:04:19 ERROR scraper 158: Spider error processing <GET 
   https://pfac.memberclicks.net/fiduciary-search#//> (referer: None)
   Traceback (most recent call last):
     File "/home/fairoos/projects/venvs/truelink/lib/python3.6/site- 
  packages/twisted/internet/defer.py", line 654, in _runCallbacks
       current.result = callback(current.result, *args, **kw)
     File "/home/fairoos/projects/truelinkscraping/TrueLinkScraping/spiders/pfac.py", line 73, in parse
    callback=self.parse_contents)
     File "/home/fairoos/projects/venvs/truelink/lib/python3.6/site-packages/scrapy/http/request/form.py", line 31, in __init__
    querystr = _urlencode(items, self.encoding)
     File "/home/fairoos/projects/venvs/truelink/lib/python3.6/site-packages/scrapy/http/request/form.py", line 66, in _urlencode
    for k, vs in seq
     File "/home/fairoos/projects/venvs/truelink/lib/python3.6/site-packages/scrapy/http/request/form.py", line 66, in <listcomp>
    for k, vs in seq
ValueError: not enough values to unpack (expected 2, got 1)

I converted all the int in the dict to str but still got these errors.

MyNameIsCaleb
  • 4,409
  • 1
  • 13
  • 31
muhammed fairoos nm
  • 652
  • 7
  • 12
  • at a guess `FormRequest` returns more than 1 value – DrBwts Sep 17 '19 at 19:02
  • you need to send a 1-1 dictionary, not nested dictionary – Umair Ayub Sep 18 '19 at 10:41
  • 3
    I think your `formdata` should be the dictionary itself, not a string – tomjn Sep 18 '19 at 13:01
  • 3
    Your issue comes from `json.dumps`. You cannot use it with [`FormRequest`](https://stackoverflow.com/questions/11236632/scrapy-formrequest-sending-json). You can use it with [`Request`](https://stackoverflow.com/a/41747616/10314460). I tried this (notice the url ): `fetch(scrapy.Request('https://pfac.memberclicks.net/ui-directory-search/v2/search-directory/', method="POST", body=json.dumps(form_data),headers={'Content-Type':'application/json'})) ` and it works. But it will return a json response, not an html. You will need to parse it differently. – AvyWam Sep 18 '19 at 13:30
  • 1
    Thanks for @AvyWam and all who supported. thanks for your time an – muhammed fairoos nm Sep 25 '19 at 07:12

0 Answers0