Where to place params for a request url in scrapy request

Question

I have a tuple of params which I have to pass along with my request in scrapy but I can't find any parameter that carries params in scrapy.Request or response.follow. Where should I place this?

params = (
            ('search_criteria[current_page]', '1'),
            ('search_criteria[excludeAggregations]', 'false'),
            ('search_criteria[filter_groups][0][filters][0][field]', 'status'),
            ('search_criteria[filter_groups][0][filters][0][value]', '1'),
            ('search_criteria[filter_groups][1][filters][0][field]', 'visibility'),
            ('search_criteria[filter_groups][1][filters][0][value]', '4'),
            ('search_criteria[filter_groups][2][filters][0][field]', 'category_ids'),
            ('search_criteria[filter_groups][2][filters][0][value]', '474'),
            ('search_criteria[page_size]', '20'),
            ('search_criteria[request_index]', 'product'),
            ('search_criteria[sort_orders][0][direction]', 'ASC'),
            ('search_criteria[sort_orders][0][field]', 'price'),
            ('tracker', 'a2zxlqwzj6-1567316733-1'),
        )
URL= "https://magento.ackermans.co.za/rest/default/V1/pepkor/searchV2"
scrapy.Request(URL,params=???)

Possible duplicate of [How to specify parameters on a Request using scrapy](https://stackoverflow.com/questions/33747174/how-to-specify-parameters-on-a-request-using-scrapy) — CodeIt, Sep 01 '19 at 06:14

score 1 · Answer 1 · answered Sep 01 '19 at 06:09

Possible duplicate of How to specify parameters on a Request using scrapy .

For your particular situation, the below would work, assuming you are using python 3.6+

params = (
            ('search_criteria[current_page]', '1'),
            ('search_criteria[excludeAggregations]', 'false'),
            ('search_criteria[filter_groups][0][filters][0][field]', 'status'),
            ('search_criteria[filter_groups][0][filters][0][value]', '1'),
            ('search_criteria[filter_groups][1][filters][0][field]', 'visibility'),
            ('search_criteria[filter_groups][1][filters][0][value]', '4'),
            ('search_criteria[filter_groups][2][filters][0][field]', 'category_ids'),
            ('search_criteria[filter_groups][2][filters][0][value]', '474'),
            ('search_criteria[page_size]', '20'),
            ('search_criteria[request_index]', 'product'),
            ('search_criteria[sort_orders][0][direction]', 'ASC'),
            ('search_criteria[sort_orders][0][field]', 'price'),
            ('tracker', 'a2zxlqwzj6-1567316733-1'),
        )

def to_url_params(params):
  param_string = ""
  for param in params:
    # iterate over params and append each into 
    param_string = f'{param_string}&{param[0]}={param[1]}'  
  # get rid of first & symbol
  return param_string[1:]

param_string = to_url_params(params)

URL= f"https://magento.ackermans.co.za/rest/default/V1/pepkor/searchV2?{param_string}"

I had an idea of this solution but I was wondering if there is any other way but Thanks man — Muhammad Ahmad, Sep 01 '19 at 06:17

score 1 · Answer 2 · answered Sep 01 '19 at 07:19

If you don't want to write a custom function for this, you can change the tuple into a dictionary and use urlencode from urllib.parse (python3):

from urllib.parse import urlencode

params = {
            'search_criteria[current_page]': '1',
            'search_criteria[excludeAggregations]': 'false',
            'search_criteria[filter_groups][0][filters][0][field]': 'status',
            'search_criteria[filter_groups][0][filters][0][value]': '1',
            'search_criteria[filter_groups][1][filters][0][field]': 'visibility',
            'search_criteria[filter_groups][1][filters][0][value]': '4',
            'search_criteria[filter_groups][2][filters][0][field]': 'category_ids',
            'search_criteria[filter_groups][2][filters][0][value]': '474',
            'search_criteria[page_size]': '20',
            'search_criteria[request_index]': 'product',
            'search_criteria[sort_orders][0][direction]': 'ASC',
            'search_criteria[sort_orders][0][field]': 'price',
            'tracker': 'a2zxlqwzj6-1567316733-1',
        }

encoded_params = urlencode(params)
url = f"https://magento.ackermans.co.za/rest/default/V1/pepkor/searchV2?{encoded_params}"

Where to place *params* for a request url in scrapy request

2 Answers2

Where to place params for a request url in scrapy request