0

I am trying to extract list of cities from https://www.lalpathlabs.com/find-a-lab.aspx which can be extracted one by one by selecting different states.

To achieve that through a shell program, I inspected the webpage to find the URL used to fetch the data. Now I am trying to simulate it via curl.

curl -X POST https://www.lalpathlabs.com/CascadingDropdown.asmx/BindCityDetails -d '{knownCategoryValues: "State:delhi-ncr;", category: "City"}'

However it returns me below error:

This website is secured against online attacks. Your request was blocked due to suspicious behavior

This data is a public data - how should I simulate this call in curl?

gyan
  • 111
  • 4

1 Answers1

0

I got it!

Actually I was missing to set the other header elements, which wee having security tokens specific to the session.

Check the 2nd line in below command, this is what I missed.

curl 'https://www.lalpathlabs.com/CascadingDropdown.asmx/BindCityDetails' 

-H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'X-Requested-With: XMLHttpRequest' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36' -H 'Content-Type: application/json; charset=UTF-8' -H 'Accept: */*' -H 'Origin: https://www.lalpathlabs.com' -H 'Sec-Fetch-Site: same-origin' -H 'Sec-Fetch-Mode: cors' -H 'Referer: https://www.lalpathlabs.com/find-a-lab.aspx' -H 'Accept-Language: en-US,en;q=0.9' -H 'Cookie: ASP.NET_SessionId=q5xjgrexqpj2ouq2tyvjbgxn; selectedCity=delhi; _gcl_au=1.1.468492529.1589600556; _ga=GA1.2.1259164458.1589600556; _gid=GA1.2.562777668.1589600556; _hjid=b605df81-acc8-43d6-bf57-558566a7c608; _nv_vi_vt=1; nv_push_error=201; WZRK_G=c7e8252ae2b64c02a5bd8f02a1aa8eb6; _fbp=fb.1.1589641923307.1530780058; _hjIncludedInSample=1; _hjAbsoluteSessionInProgress=1; _nv_ttsOnpage_5193={"d":0,"h":0,"m":11,"s":7,"ms":317}; _nv_pv=4; _nv_utm=209183885.1589641924.4.1.utmsrc=google|utmccn=(not set)|utmcmd=organic|utmctr=(not provided)|utmcct=(not set)|gclid=(not set); _nv_uid=209183885.1589641924.1bfc3e00-b238-44d8-bddc-2cf3698f3f74.1589642341.1589642349.4; WZRK_S_675-RZ7-6Z5Z=%7B%22p%22%3A4%2C%22s%22%3A1589641929%2C%22t%22%3A1589642356%7D; _gat=1; _nv_ts=14373'

--data-binary '{"knownCategoryValues":"State:andhra-pradesh;","category":"City"}' --compressed

Best way to do it, inspect the URL in browser developer tool,e.g. chrome developer tool and copy URL as "cURL" command.

gyan
  • 111
  • 4