0

I'm scraping data from some sites and one of theses use headers for some information. I'm using Smart Proxy Manager from Zyte and having problems with a request with headers. When I use the zyte proxy, I receive a response, but the site that I'm scraping return an unsuccess message. If I not use the proxy, the request works like a charm. Can anyone help me?

Here is my code

string zyteKey = _configuration["ZyteKey"];
            var proxy = new WebProxy("myproxy");
            proxy.Credentials = new NetworkCredential(zyteKey, "");

            var clientHandler = new HttpClientHandler()
            {
                Proxy = proxy,
                UseProxy = true,
                PreAuthenticate = true,
                AllowAutoRedirect = false
            };

            using (var client = new HttpClient(clientHandler))
            {
                string json = JsonConvert.SerializeObject(jsonObject);
                var content = new StringContent(json, Encoding.UTF8, "application/json");

                client.Timeout = new TimeSpan(0, 1, 0);
                client.DefaultRequestHeaders.Add("cookie", BuildCookieRequest(latitude, longitude, restaurantName));
                client.DefaultRequestHeaders.Add("Accept", "application/json, text/plain, */*");
                client.DefaultRequestHeaders.Add("x-csrf-token", "x");

                return await client.PostAsync(url, content);
            }
Vinícius da Cruz Maia
  • 347
  • 6
  • 17
  • 1
    Use a sniffer like wireshark to get more info on failure. You should be getting a response with an error status. If the request is never sent than you have an TLS authentication issue. Not sure why a proxy should be using a setting : AllowAutoRedirect = false. I think it should be true. – jdweng Jun 12 '21 at 21:04
  • https://stackoverflow.com/a/30605900/12888024 `UseDefaultCredentials = false` – aepot Jun 12 '21 at 21:53
  • Also [_HttpClient is intended to be instantiated once per application, rather than per-use._](https://learn.microsoft.com/en-us/dotnet/api/system.net.http.httpclient?view=netcore-3.1#examples), in your case that would be "once per proxy". – aepot Jun 12 '21 at 22:06

0 Answers0