WebClient returning 403 error only for this website?

Question

I am trying to download file from these links by using C# WebClient, but I am getting 403 error.

https://www.digikey.com/product-search/download.csv?FV=ffe00035&quantity=0&ColumnSort=0&page=5&pageSize=500

https://www.digikey.com/product-search/download.csv?FV=ffe00035&quantity=0&ColumnSort=0&page=4&pageSize=500

I tried to use different user agents, accept encoding etc. I replaced and tried https to http from url, but no success. When I paste these urls in Chrome or FireFox or IE, I am able to download file, sometimes it give 403 error, then I replace https to http from url, it downloads. But no success in webclient Tried Fiddler to inspect, no success Can someone try in your system, solve this problem.

Here is my code:

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
WebClient client= new WebClient();
Uri request_url = new Uri("https://www.digikey.com/product-search/download.csv?FV=ffe00035&quantity=0&ColumnSort=0&page=5&pageSize=500);
//tried http also http://www.digikey.com/product-search/download.csv?FV=ffe00035&quantity=0&ColumnSort=0&page=5&pageSize=500
client.Headers.Add("user-agent", " Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0");
client.DownloadFile(request_url, @"E:\123.csv");

I know there are many threads related to this topic, I tried all of them, no success, please don't mark duplicate. Try in your system, this <10 lines of code.

Note: the same code is working for other websites, only for this website it is giving error.

Answer 1

As I mentioned in my comment the issue here is that the server is expecting a cookie (specifically 'i10c.bdddb') to be present and is giving a 403 error when it's not. However, the cookie is sent with the 403 response. So you can make an initial junk request that will fail but give you the cookie. After this you can then proceed as normal.

Through some trial and error I was able to get the CSV using the code below:

System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;

CookieContainer cookieContainer = new CookieContainer();
Uri baseUri = new Uri("https://www.digikey.com");

using (HttpClientHandler handler = new HttpClientHandler() { CookieContainer = cookieContainer })
using (HttpClient client = new HttpClient(handler) { BaseAddress =  baseUri})
{
    //The User-Agent is required (what values work would need to be tested)
    client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0");

    //Make our initial junk request that will fail but get the cookie
    HttpResponseMessage getCookiesResponse = await client.GetAsync("/product-search/download.csv");

    //Check if we actually got cookies
    if (cookieContainer.GetCookies(baseUri).Count > 0)
    {
        //Try getting the data
        HttpResponseMessage dataResponse = await client.GetAsync("product-search/download.csv?FV=ffe00035&quantity=0&ColumnSort=0&page=4&pageSize=500");

        if(dataResponse.StatusCode == HttpStatusCode.OK)
        {
            Console.Write(await dataResponse.Content.ReadAsStringAsync());
        }
    }
    else
    {
        throw new Exception("Failed to get cookies!");
    }
}

Notes

Even with the right cookie if you don't send a User-Agent header the server will return a 403. I'm not sure what the server expects in terms of a user agent, I just copied the value my browser sends.

In the check to see if cookies have been set it would be a good idea to verify you actually have the 'i10c.bdddb' cookie instead of just checking if there are any cookies.

This is just a quick bit of sample code so it's not the cleanest. You may want to look into FormUrlEncodedContent to send the page number and other parameters.

Answer 2

I tested with your URL and I was able to reproduce your error. Any requests that I try with the querystring parameter quantity=0 seems to fail with a HTTP Error 403.

I would suggest requesting a quantity greater than zero.

Answer 3

A HTTP 403 status code mean forbidden, so there is a problem with your credentials. It doesn't seem to be like you're sending any. If you add them into your header this should work fine like this:

client.Headers.Add("Authorization", "token");

or sending them like this:

 client.UseDefaultCredentials = true;
 client.Credentials = new NetworkCredential("username", "password");

Most likely the links are working through web browsers is because you have already authenticated and the browser is sending the credentials/token.

Answer 4

I have this issue with Digi-key too.

The solution for me is to turn off my VPN service.