20

I wrote a multithread program in c# that crawls a web site, but when I started Fiddler in the background request completed 12x faster, it is really strange for me and when I close Fiddler download rates slows downs. how it is possible please help, (there is no proxy setting for my connection to the ineternet and for Fiddler too) If i can inject the performance of fiddler in my application it would be wonderful, any solution? Is there any magic behind the scenes ?? :)

Thanks

Ehsan
  • 1,662
  • 6
  • 28
  • 49

2 Answers2

30

The reason is the limit on the number of simultaneous http connections which is ignored when using Fiddler.

I've run into the same behavior while using System.Net.Http.HttpClient to perform multiple (~80) concurrent requests. With Fiddler running, all the requests were completing much faster. Keep-alive was certainly enabled.

I used Wireshark to see what's happening and first thing I noticed that the manner of the http traffic was different. With Fiddler requests were thrown all at once and responses followed nicely grouped as well. Without Fiddler the requests were interleaving with responses.

Secondly, tcpview showed that my code without Fiddler opened only 2 tcp connections to the server. With Fiddler started, the number of connections significanly increased. There were tens of them from my app to Fiddler and then from Fiddler to the server.

It is well known that the http standard recommends the number of http connections should be no more than 2 and it seems the limit is implemented as a default setting in http clients.

In .NET applications we can control the limit with ServicePointManager.DefaultConnectionLimit static property. As an experiment, having it set to 100 made the requests completing with the same time with and without Fiddler.

The setting can also be controlled through app.config:

    <system.net>
    <connectionManagement>
        <add address="*" maxconnection="100" />
    </connectionManagement>
</system.net>

Now, why is the default connection limit not respected while using Fiddler? Looks like the limit is different when an http client uses a proxy and Fiddler does act as a proxy. I did not find much information about the proxy connection limit besides this old article.

v.shashenko
  • 716
  • 6
  • 11
12

Can you show some sample code so people can confirm this? Otherwise it'll become wild guessing.

My best guess: Fiddler uses keepalive which will save the trouble of opening the connection over and over again. You can confirm this by disabling both Reuse client connections and Reuse connections to servers: if it's then as slow as usual (or slower), the benefit is gained from keeping the connection alive.

CodeCaster
  • 147,647
  • 23
  • 218
  • 272
  • In my program when I create a connection and download the content then I'll close the response and abort the request in final, but I'm wondering that this is the same code that is running with fiddler in the background and its fast. do the Fiddler change my request and use the same connection for each request ? – Ehsan May 03 '12 at 09:33
  • @Eshan _"Does Fiddler change my request and use the same connection for each request?"_ Yes. – CodeCaster May 03 '12 at 09:46
  • 1
    so how can I use this feature in my program with HttpWebRequest ? – Ehsan May 03 '12 at 09:54
  • @Eshan by not recycling the client and enabling [KeepAlive](http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.keepalive.aspx). – CodeCaster May 03 '12 at 09:56
  • I should provide this header for my target server and I enabled it, the fiddler shows that it is enable too, but the problem remains (thanks for your quick response ;) – Ehsan May 03 '12 at 10:03
  • by the way Fiddler is not changing those request that I'm not enable KeepAlive for them and inject that to my request as you say. – Ehsan May 03 '12 at 10:06
  • 3
    This is the correct answer. Your client app is probably tearing down its connection on every request, but Fiddler's keeping its connections to the server alive. Your app makes a new connection to Fiddler for each request, but because it's a local request, it's basically instantaneous. You need to enable the KeepAlive property on the HTTPWebRequest in the client app. – EricLaw May 20 '12 at 22:44