I have to scrape a webpage in c# and i am using Httpclient,My problem is that when i scrape a url e.g https://somethng.com/search/?query=mobile,its give me result no products found ,but when i search manually in website and give a location in search text box in webpage e.g: some city ,it shows products on page but url is same as i wanna scrape in my application https://somethng.com/search/?query=mobile,there is no location querystring in url. My question is how to handle it in webscraping? because url shows no pruducts in my app but shows products when i search manually and both urls are same.
Asked
Active
Viewed 200 times
0
-
Sounds like it runs Javascript. You need a headless browser. – SLaks Oct 10 '17 at 19:57
-
Or send its AJAX requests directly and parse their responses. – SLaks Oct 10 '17 at 19:57
-
Or sends some special headers/cookies. – orhtej2 Oct 10 '17 at 19:57
-
how can it be done? when i clicked on search ,whole page is posted back. – csharplearner Oct 10 '17 at 20:01
-
The first step I would take is to figure out how the web browser is handling the page loads. I'd do this by reading the code in the page, and by watching what it loads and when. I like to use Fiddler for the monitoring. It will show you all of the pages being loaded, and should also allow you to look at the contents of each response. – theGleep Oct 10 '17 at 20:16
-
i have no idea how to use fiddler or any other tool for this purpose :( – csharplearner Oct 11 '17 at 13:58