Stack Overflow
Stack Overflow

Questions tagged [tpl-dataflow]

TPL Dataflow (TDF) is a .NET library for building concurrent applications. It promotes actor/agent-oriented designs through primitives for in-process message passing, dataflow, and pipelining. TDF builds upon the TPL (Task Parallel Library) in .NET 4 and integrates with async language support in C#, Visual Basic, and F#. TDF lacks join/merge by key (like SSIS) and time-based windowing (available in Rx).

Platform: .NET Framework 4.0 / .NET Core / .NET 6

Promotes: actor/agent programming model

Strengths: limiting parallelism (for example reading from disk bound vs. CPU compute bound tasks would be limited differently), data flow via message passing, integration with other .NET libraries like Rx and TPL.

Weaknesses: no data flow merge by key (like SSIS merge operation), and only the most trivial of time windowing (compare BroadcastBlock with Rx BufferWithTime() or CEP/StreamInsight windowing operators)

629 questions
5
votes
2 answers

More effecient way of calling GetStringAsync multiple times?

I have (my url list is about 1000 urls), I was wondering if there is a more effecient call multiple urls from same site (already changing the ServicePointManager.DefaultConnectionLimit). Also is it better to reuse the same HttpClient or create new…
Zoinky
  • 4,083
  • 11
  • 40
  • 78
5
votes
3 answers

TPL Dataflow - Parallel&Async processing, while keeping order

I created a TPL Dataflow pipeline which consist of 3 TransformBlock's and an ActionBlock at the end. var loadXml = new TransformBlock(job => { ... }); // I/O var validateData = new TransformBlock(job => { ... }); //…
Peter
  • 135
  • 2
  • 13
5
votes
1 answer

.NET queued tasks (with async/await)

I have a large number of tasks (~1000) that need to be executed. I am running on a 4-core processor, so I'd like to process 4 tasks at a time, in parallel. To give you a starting point, here is some sample code. class Program { public class…
Paul Knopf
  • 9,568
  • 23
  • 77
  • 142
5
votes
2 answers

BroadcastCopyBlock for TPL Dataflow with guaranteed delivery

I would be glad for some input on the following implementation of a BroadcastCopyBlock in TPL Dataflow, which copies a received message to all consumers, that registered to the BroadcastCopyBlock and guarantees delivery to all consumers, which are…
crazy_crank
  • 659
  • 4
  • 17
5
votes
2 answers

TPL Dataflow - very fast producer, not so fast consumers OutOfMemory exception

I'm experimenting with TPL Dataflow before porting it into my production code. The production code is a classical producer/consumer system - producer(s) produce messages (related to financial domain), consumers process those messages. What I'm…
Michael
  • 2,961
  • 2
  • 28
  • 54
5
votes
2 answers

Understanding TPL Dataflow Degree of Parallelism ordering

I was reading Dataflow (Task Parallel Library), and there is a portion which says: When you specify a maximum degree of parallelism that is larger than 1, multiple messages are processed simultaneously, and therefore, messages might not be…
SimonSays
  • 477
  • 5
  • 13
5
votes
3 answers

Why do blocks run in this order?

This is short code sample to quickly introduce you what is my question about: using System; using System.Linq; using System.Threading.Tasks; using System.Threading.Tasks.Dataflow; namespace DataflowTest { class Program { static void…
kseen
  • 359
  • 8
  • 56
  • 104
5
votes
2 answers

Memory issue in TPL Dataflow implementation of IO read write operation

I have tried to implement Read write operation using File IO operations and encapsulated those operations into TransformBlock so as to make these operation thread safe instead of using locking mechanism. But the problem is that when I try to write…
Balraj Singh
  • 3,381
  • 6
  • 47
  • 82
5
votes
1 answer

"bounded" BatchBlock => ActionBlock. How to complete the proper way?

I'm trying to use a bounded batchblock linked to an action block. I know when the feeding of items in the batchblock end and I want to trigger a completion chain. The problem is: if my BatchBlock is of a given BoundedCapacity I won't get all my…
Jeremy D
  • 465
  • 5
  • 21
5
votes
2 answers

TPL Dataflow block never completes on PropagateCompletion

Since the last alteration to my propagated-completion pipeline, one of my buffer blocks never completes. Let me summarize what was working and what isn't anymore: Previously working: A.LinkTo(B, PropagateCompletion); B.LinkTo(C,…
Luis Ferrao
  • 1,463
  • 2
  • 15
  • 30
5
votes
1 answer

Nested Async/Await Doesn't Appear To Be Scaling

I have the following (simplified) code: public async Task GetData(DomainObject domainObject, int depth) { // This async operation is really quick, and there's usually like five. IEnumerable tierOnes = await…
Cameron
  • 2,574
  • 22
  • 37
5
votes
2 answers

How should I use DataflowBlockOptions.CancellationToken?

How could I use DataflowBlockOptions.CancellationToken? If I create instance of BufferBlock like this: var queue = new BufferBlock(new DataflowBlockOptions { BoundedCapacity = 5, CancellationToken = _cts.Token }); then having consumer/producer…
eXavier
  • 4,821
  • 4
  • 35
  • 57
5
votes
2 answers

TPL Dataflow async scheduling

The scheduling of async Tasks does not work as I expected in TPL Dataflow. In the example below, I expected the ActionBlock to process data from the TransformBlock as soon as it is available. But it is waiting on the second (delayed) result before…
Petter T
  • 3,387
  • 2
  • 19
  • 31
5
votes
5 answers

How to mark a TPL dataflow cycle to complete?

Given the following setup in TPL dataflow. var directory = new DirectoryInfo(@"C:\dev\kortforsyningen_dsm\tiles"); var dirBroadcast=new BroadcastBlock(dir=>dir); var dirfinder = new TransformManyBlock
Poul K. Sørensen
  • 16,950
  • 21
  • 126
  • 283
5
votes
1 answer

Multiple Short-lived TPL Dataflows versus Single Long-Running Flow

I'm using TPL dataflow to process items off a queue in an Azure worker role. Should I have a single long running dataflow, or spawn a new flow for every messages I receive? If an error is thrown in a block, that block will stop accepting new…
pnewhook
  • 4,048
  • 2
  • 31
  • 49
1 2 3
…
41 42