Why is Fiddler affecting the encoding of the response?

Question

Please see the code below:

private static async Task<string> GetResponse(string url)
        {

            using (HttpClient _HttpClient = new HttpClient())
            {

                using (var request = new HttpRequestMessage(HttpMethod.Get, new Uri(url)))
                {
                    request.Headers.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0");
                    request.Headers.TryAddWithoutValidation("Accept", "*/*");
                    request.Headers.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate, br");
                    request.Headers.TryAddWithoutValidation("Connection", "keep-alive");
                    request.Headers.TryAddWithoutValidation("Accept-Charset", "ISO-8859-1");

                    using (var response = await _HttpClient.SendAsync(request).ConfigureAwait(false))
                    {
                        response.EnsureSuccessStatusCode();
                        return await response.Content.ReadAsStringAsync();
                    }
                }
            }

        }

The response returned by the method is garbage i.e. invalid characters. I loaded Fiddler to analyse the response to more detail and then the response returned was valid i.e. the response contains valid characters when Fiddler is active and invalid characters when Fiddler is inactive i.e. closed.

Why is Fiddler affecting the encoding of the response? I have spent half an hour or so Googling this and I have looked here: https://forums.asp.net/t/1778112.aspx?Data+returned+from+an+api+call+has+strange+characters

Why is Fiddler affecting the encoding of the response? I get the correct response when I use Postman.

It sounds like (a) Fiddler decompresses the gzip or deflate encoded stream, and (b) you can achieve the same by configuring your HttpClient instance as in [this answer](https://stackoverflow.com/a/27327208/1220550). — Peter B, Aug 31 '21 at 10:42
It sounds like Fiddler *fixes* the response. To request compressed responses in .NET Framework you need to set the [AutomaticDecompression](https://learn.microsoft.com/en-us/dotnet/api/system.net.http.httpclienthandler.automaticdecompression?view=net-5.0) property. Even then, Brotli is only available in .NET Core 3.1 and later. This HttpClient code has problems - HttpClient should *not* be disposed, it's meant to be reused. All APIs and web sites use UTF8, not Latin1. Almost all of the code could be replaced with `HttpClient.GetStringAsync()` with a properly configured HttpClient — Panagiotis Kanavos, Aug 31 '21 at 10:49
Which runtime are you targeting? .NET Framework? .NET Core? Which versions? In all supported .NET Core versions (ie .NET Core 3.1 and later) you can use [the HttpClientFactory](https://learn.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/use-httpclientfactory-to-implement-resilient-http-requests) middleware to properly handle HttpClient's lifetime, pooling and configuration. Brotli is only available in .NET Core anyway. You can set `AutomaticDecompression` to `All` to accept all possible compression methods — Panagiotis Kanavos, Aug 31 '21 at 10:57
@Panagiotis Kanavos, thanks. That has sorted it. I am using .net core 3.1. If you would like to post an answer, then I will give some credit, else I will delete the question. — w0051977, Aug 31 '21 at 13:21

Why is Fiddler affecting the encoding of the response?

0 Answers0