WebClient returns 500 (Internal Server Error) in DownloadFile while browsers do load the image

Question

I would like to download an image from a website using WebClient. I checked with a browser if the image exists, and it does.

I'm using this code to download the image:

Private Function pDownload(ByVal uFrom As String, ByVal uTo As String) As Boolean

    Try
        Using Client As New WebClient
            Client.DownloadFile(uFrom, uTo)
        End Using
        Return True
    Catch ex As Exception
        Debug.Print("Failed: " + uFrom + vbNewLine + ex.ToString)
        Return False
    End Try

End Function

Does anybody see why it might return a 500 - Internal Server Error?

@DogeAmazed Yes, that was the solution in my case. Can you make your comment the answer? — tmighty, Oct 15 '17 at 19:24

score 3 · Accepted Answer · answered Oct 16 '17 at 06:07

Sometimes servers deny requests because of a non-existent User-Agent.

You can add it with the following line of code.

Client.Headers.Add ("user-agent", "Your User-Agent")

Full example and documentation: https://learn.microsoft.com/en-us/dotnet/api/system.net.webclient?view=netframework-4.7.

score 0 · Answer 2 · answered Oct 15 '17 at 19:07

500 - Internal Server Error is a server response, not a local error in .net. Your downloading code is a very simple call for downloading the URL uFrom to a local file uTo-- and it sounds like it is working correctly.

The problem is the server is responding with a 500 INTERNAL SERVER ERROR and you expect it respond with a 200 OK. WebClient sees the 500 response from the server and knows that it is an non-successful response so it throws that exception. If the file was missing, the server should respond with a 404 NOT FOUND. If it were a matter of not having permission, we would expect a well-behaved server to respond with 401 NOT AUTHORIZED. The generic 500 response simply means, "something unexpected went wrong".

There are many reasons why a server would respond with a 500-- such as a file system error, database error, a bug in the server application, etc. I am assuming that since you are not posting any server-side code, that you did not write the server application and that this is an existing (and presumably well-behaved) server. Since you are not doing anything fancy-- just trying to download a file-- then the most likely reason I can think of might be your url in uFrom is not valid, or is not considered safe by the server.

An easy way to test your url it to open a new browser window in incognito/private browsing mode (to make sure you have no cookies causing the server behavior to be different) and pasting the value of uFrom directly into the address bar. If you get the same results in the browser (500) then you know there is something wrong with the server or your URL. If you get different results in your browser test than you do from running code, then there is something different in the request headers coming from your browser than what WebClient is sending by default. If that is the case, I would suggest installing fiddler or similar tool to examine the differences in what is happening when your browser requests the url versus when your code requests the same url.

WebClient returns 500 (Internal Server Error) in DownloadFile while browsers do load the image

2 Answers2