0

I've been working on my first TPL project recently, and I'm trying to mix a couple of patterns that apparently aren't used together very much.

Given a huge volume of IO-bound http request tasks, and decent but not unlimited concurrent server capacity, I need a producer-consumer implementation that allows the blocking collection to precipitate tasks as they complete. The distinction here from the usual BlockingCollection<> scenario is, an arbitrary Take() method will not remove tasks in the order they complete. Essentially, I need some mix of BlockingCollection<>, .WhenAny(), and .Remove().

My options so far are:

  1. I can kitbash a wrapper for BlockingCollection<byte> with a paired ConcurrentDictionary<int, Task<T>> using the Task.Id for the key: .ToDictionary(t => t.Id). The blocking collection is just a token bag for traffic management, and the dictionary provides a when-any enumerable and a remove. This works, but its a bit of duct tape & bailing wire.

  2. I could try passing .GetConsumingEnumerable() to .WhenAny(), which looks great in the editor, but is bound to fail as the blocking collections gets arbitrarily consumed when the enumeration is scanned for completed tasks. I have not tried this yet, but it shouldn't work the way I intend.

  3. ?

This seems to be a somewhat unusual scenario; I'm bumping the .DefaultConnectionLimit up between 120 to 720 depending on the endpoint I'm targeting, and then spamming it with as much load as it can handle. I can find no obvious concurrent collection and/or blocking wrapper that provides .Add(), .Remove(), and a non-removing IEnumerable<> together. Am I missing something?

svick
  • 236,525
  • 50
  • 385
  • 514
KurtisR
  • 45
  • 5
  • Can you explain more of your situation, I am having trouble understanding the model of what you are doing. Perhaps a code example of how you you would do #2 if `WhenAny` worked the way you wished it worked. – Scott Chamberlain Nov 05 '13 at 00:13
  • Hard to pin down, seems you are just not committed to the notion of *consuming*. Possibly induced by the idea that you don't want to consume when the operation fails. That can't work, no collection class will take on the job of dealing with failure. That's the consumer's job. Yours. – Hans Passant Nov 05 '13 at 00:18
  • I'm also not sure what exactly are you asking for, but wouldn't something like [`OrderByCompletion()`](https://nitoasyncex.codeplex.com/wikipage?title=TaskExtensions) work for you? – svick Nov 05 '13 at 08:50
  • What real problem are you trying to solve? You are describing a complex solution but what is the problem? If you are trying to build a pipeline of tasks, perhaps you should look at [TPL DataFlow](http://msdn.microsoft.com/en-us/library/hh228603(v=vs.110).aspx) – Panagiotis Kanavos Nov 07 '13 at 11:05

0 Answers0