Encoding problems using StreamReader and WebResponse

Question

I'm trying to read an XML stream using the WorldBank API. The URL in question is http://api.worldbank.org/country?per_page=100 now in my web browser this runs fine and returns an XML document.

However, this will not play nicely in FSI. The stream will basically be a bunch of garbage, according to both the XML document and the response headers the encoding of the content is UTF-8 and I've set the StreamReader to use that encoding but it doesn't seem to help.

Here is the code:

let downloadUrl(url:string) = 
    async { let request = HttpWebRequest.Create(url)
            use! response = request.AsyncGetResponse()
            let stream = response.GetResponseStream()
            use reader = new StreamReader(stream, Text.Encoding.UTF8)
            return! Async.AwaitTask <| reader.ReadToEndAsync() }

Answer 1

I figured it out, apparently the stream returned was GZip compressed so I had to change the code so it would use automatic decompression.. I should mention.. This isn't very robust code as HttpWebRequest.Create could return something else besides a HttpWebRequest ideally I would use type checking in a pattern match to determine the type returned but this will work for my purposes because I know that I will only be using it for HttpWebRequests.

let downloadUrl(url:string) = 
    async { let request = HttpWebRequest.Create(url) :?> HttpWebRequest
            request.AutomaticDecompression <- DecompressionMethods.GZip
            use! response = request.AsyncGetResponse()
            let stream = response.GetResponseStream()
            use reader = new StreamReader(stream, Text.Encoding.UTF8)
            return! Async.AwaitTask <| reader.ReadToEndAsync() }