Asp.net HttpWebRequest I can not source code

Question

I want html source code with asp.net HttpWebRequest but there is a problem.

I can get it. No problem that : https://www.nesine.com/iddaa/default.aspx

But I do not get it : http://www.iddaa.com/program/futbol.html

in my opinion problem url routing.

I use the code

public static string GetSourceCode(string url)
{

    HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);


    HttpWebResponse resp = (HttpWebResponse)req.GetResponse();


    StreamReader sRead = new StreamReader(resp.GetResponseStream(), Encoding.UTF8);


    return sRead.ReadToEnd();
}

help me please. Thanks.

Answer 1

The http://www.iddaa.com/program/futbol.html url is heavily using javascript and AJAX in order to load its dynamic content. It's a SPA application. The HttpWebRequest class in .NET doesn't execute any javascript. It is simply loading the contents returned by the server at the specified location.

By inspecting the network traffic you will notice that the information you are interested in is located on the following location: http://www.iddaa.com/program/data?sportId=1&date=&sortType=&marketType=1

So you can go ahead and scrape this url:

using System;
using System.IO;
using System.Net;

static class Program
{
    static void Main()
    {
        var request = WebRequest.Create("http://www.iddaa.com/program/data?sportId=1&date=&sortType=&marketType=1");
        using (var response = request.GetResponse())
        using (var stream = response.GetResponseStream())
        using (var reader = new StreamReader(stream))
        {
            string responseHtml = reader.ReadToEnd();
            Console.WriteLine(responseHtml);
        }
    }
}

Obviously, as always with scraping, if the website changes the location of this dynamic url your application will stop working. That's why it's better to use an API if the website provides one. Contact the authors of the website for more information.