I have a HTTPS based API which I need to call a lot of times. Using HttpWebRequest.Create(uri).GetResponse() takes from 50ms up to 500ms or more to execute. In order to check the response time I implemented it like this:
private void Action()
{
WebRequest request = HttpWebRequest.Create("https://.....");
using (WebResponse response = request.GetResponse()) { }
}
and then calling it:
private long getTime()
{
Stopwatch sw = new Stopwatch();
sw.Start();
Action();
return sw.ElapsedMilliseconds;
}
Output for several calls:
Time: 746 ms
Time: 51 ms
Time: 50 ms
Time: 50 ms
Time: 51 ms
Time: 49 ms
Time: 2417 ms ???
Time: 52 ms
Time: 52 ms
Time: 51 ms
Time: 50 ms
Time: 201 ms
Time: 234 ms
Time: 204 ms
Time: 51 ms
Time: 50 ms
Time: 50 ms
Time: 52 ms
Time: 280 ms
Time: 264 ms
First QUESTION: I was wondering if there was any way to speed up GetResponse in order for it to take as less time as possible?
Now.. because I need to make A LOT of different requests with different URLs, in order to speed up the process I decided to use TPL Dataflow Block (instead of Parallel.Foreach), since Parallel.Foreach is mainly used for CPU bound work, and what I am doing is I/O bound (there's also processing of the response, so also a bit of CPU work). And when I use TPL Dataflow Block the processing of 250 URLs takes up to 7 seconds to execute, whereas Parallel Foreach takes 15 seconds or more, so I'm positive TPL Dataflow Block usage is the right way to go. How I implemented it:
//CALL:
var block = new ActionBlock<string>(uri => Action(uri), new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 200 });
foreach (var URL in urlArray)
{
block.Post(URL);
}
block.Complete();
await block.Completion;
//Action(uri):
private void Action(string uri)
{
WebRequest request = HttpWebRequest.Create(uri);
using (WebResponse response = request.GetResponse()) { }
}
And since I am not happy with 7s execution I've tried tweaking ServicePointManagerin order to speed it up, things I've tried so far and NONE of it worked:
ServicePointManager.UseNagleAlgorithm = false;
ServicePointManager.Expect100Continue = false;
ServicePointManager.SetTcpKeepAlive(false, 0, 0);
ServicePointManager.DefaultConnectionLimit = 1000;
Second QUESTION: If it is not possible speeding up GetResponse() in order to achieve faster execution, is there any way to tweak TPL Dataflow Block into having better performance?
EDIT: My goal is to execute all calls AS FAST AS possible.
