Retry HTTP request from .NET with different proxy server

Question

I can issue HTTP requests through a proxy in a .NET app. There are a number of proxy servers I can use and sometimes one or more will go down. How can I have my app retry the HTTP request using a different proxy? I am open to any suggestion and have heard good things about Polly for adding resiliency.

System.Net.WebClient with the DownloadString method. I’m open to any suggestions. — Brett Rowberry, Jul 09 '18 at 14:35

score 3 · Answer 1 · answered Jul 09 '18 at 15:56

3

If you were to use Polly, maybe something like this:

public void CallGoogle()
{
    var proxyIndex = 0;

    var proxies = new List<IWebProxy>
    {
        new WebProxy("proxy1.test.com"),
        new WebProxy("proxy2.test.com"),
        new WebProxy("proxy3.test.com")
    };

    var policy = Policy
                 .Handle<Exception>()
                 .WaitAndRetry(new[]
                     {
                         TimeSpan.FromSeconds(1),
                         TimeSpan.FromSeconds(2),
                         TimeSpan.FromSeconds(3)
                     }, (exception, timeSpan) => proxyIndex++);

    var client = new WebClient();

    policy.Execute(() =>
    {
        client.Proxy = proxies[proxyIndex];
        client.DownloadData(new Uri("https://www.google.com"));
    });
}

answered Jul 09 '18 at 15:56

Stevo

1,424
11
20

You really set me up for success here. Thank you very much. – Brett Rowberry Jul 11 '18 at 02:34
How would you be able to access the proxyIndex if you create your policy in another class than the one that you consume it? Foe example in apsnet core, you crate your policy in the Startup and you inject it through the construction on your implementation. – Vergil C. Apr 27 '20 at 21:02
Perhaps see the accepted answer, it's simply calling a function. – Brett Rowberry Jul 21 '21 at 14:17

score 0 · Accepted Answer · answered Jul 11 '18 at 02:34

For my use case, it turns out that I was better off without Polly.

public static string RequestWithProxies(string url, string[] proxies)
{
    var client = new WebClient
    {
        Credentials = new NetworkCredential(username, password)
    };
    var result = String.Empty;

    foreach (var proxy in proxies)
    {
        client.Proxy = new WebProxy(proxy);
        try
        {
            result = client.DownloadString(new Uri(url));
        }
        catch (Exception) { if (!String.IsNullOrEmpty(result)) break; }
    }

    if (String.IsNullOrEmpty(result)) throw new Exception($"Exhausted proxies: {String.Join(", ", proxies)}");
    return result;
}

Brett Rowberry · Answer 3 · 2018-07-11T02:53:28.200

Here is an answer with Polly, but I don't like it as much as the answer without Polly or Polly with wait to retry.

public static string RequestWithProxies(string url, string[] proxies)
{
    var client = new WebClient { Credentials = new NetworkCredential(username, password) };
    var result = String.Empty;
    var proxyIndex = 0;

    var policy = Policy.Handle<Exception>()
        .Retry(
            retryCount: proxies.Length,
            onRetry: (exception, _) => proxyIndex++);

    policy.Execute(() =>
    {
        if (proxyIndex >= proxies.Length) throw new Exception($"Exhausted proxies: {String.Join(", ", proxies)}");

        client.Proxy = new WebProxy(proxies[proxyIndex]);
        result = client.DownloadString(new Uri(url));
    });

    return result;
}

Retry HTTP request from .NET with different proxy server

3 Answers3

Linked