Stack Overflow
Stack Overflow

Questions tagged [tpl-dataflow]

TPL Dataflow (TDF) is a .NET library for building concurrent applications. It promotes actor/agent-oriented designs through primitives for in-process message passing, dataflow, and pipelining. TDF builds upon the TPL (Task Parallel Library) in .NET 4 and integrates with async language support in C#, Visual Basic, and F#. TDF lacks join/merge by key (like SSIS) and time-based windowing (available in Rx).

Platform: .NET Framework 4.0 / .NET Core / .NET 6

Promotes: actor/agent programming model

Strengths: limiting parallelism (for example reading from disk bound vs. CPU compute bound tasks would be limited differently), data flow via message passing, integration with other .NET libraries like Rx and TPL.

Weaknesses: no data flow merge by key (like SSIS merge operation), and only the most trivial of time windowing (compare BroadcastBlock with Rx BufferWithTime() or CEP/StreamInsight windowing operators)

629 questions
9
votes
1 answer

Difference between a greedy and a non-greedy dataflow block with boundedcapacity defined

I have a BatchBlock with BoundedCapacity defined on it var _batchBlock = new BatchBlock(2, new GroupingDataflowBlockOptions {BoundedCapacity = 100 }); So if the queue capacity reaches a 100 the block…
Varunkumar Manohar
  • 887
  • 3
  • 11
  • 29
9
votes
1 answer

TPL Dataflow how to remove the link between the blocks

I would like to know. How can I remove the link between the blocks? In other words. I want to get opposite of LinkTo. I want to write a logger based on tlp dataflow. I wrote this interface and want to delete a subscription for ILogListener when it…
user45245
  • 845
  • 1
  • 8
  • 18
9
votes
1 answer

How do you set/get/use the name of a block in TPL Dataflow?

MSDN documentation shows that there is a NameFormat attribute on the DataflowBlockOptions class, described as: Gets or sets the format string to use when a block is queried for its name. So ... how do you set the name? How is the name available? …
davidbak
  • 5,775
  • 3
  • 34
  • 50
9
votes
2 answers

Good approach for hundreds of comsumers and big files

I have several files (nearly 1GB each) with data. Data is a string line. I need to process each of these files with several hundreds of consumers. Each of these consumers does some processing that differs from others. Consumers do not write…
shda
  • 729
  • 7
  • 19
9
votes
1 answer

tpl dataflow: fixed buffer size without throwing items away

After playing around with dataflow I encountered a new problem. I would like to limit the inputqueue of all blocks. My producingblock (ActionBlock) is creating 5000 elements really fast and posts them to an broadcastblock. So if i set the…
Daffi
  • 506
  • 1
  • 7
  • 20
9
votes
1 answer

Customizing ActionBlock

I want to implement a prioritised ActionBlock. So that i can Conditionally give priority to some TInput items by using a Predicate. I read Parallel Extensions Extras Samples and Guide to Implementing Custom TPL Dataflow Blocks. But Still…
Rzassar
  • 2,117
  • 1
  • 33
  • 55
9
votes
2 answers

Dealing with a very large number of files

I am currently working on a research project which involves indexing a large number of files (240k); they are mostly html, xml, doc, xls, zip, rar, pdf, and text with filesizes ranging from a few KB to more than 100 MB. With all the zip and rar…
Martijn
  • 521
  • 1
  • 8
  • 17
8
votes
1 answer

How to run an async delegate on captured ExecutionContext

As Stephen Toub explained in this post, when you submit a message to an ActionBlock, you can ExecutionContext.Capture before calling ActionBlock.Post, pass a DTO holding both message and ExecutionContext into the block, then inside the message…
Andriy Volkov
  • 18,653
  • 9
  • 68
  • 83
8
votes
1 answer

How to properly manage Completion in TPL Dataflow

I've created something similar to a web crawler to create a report of the 1000+ Webservices I need to manage. I therefore created a TPL Dataflow Pipeline to manage getting and processing the data. The Pipeline I imagined looks a little bit like this…
Brezelmann
  • 359
  • 3
  • 16
8
votes
1 answer

Defining BoundedCapacity degrades performance

Is there any way to limit performance degradation with TPL Dataflow throttling? I have a complicated pipeline of components and trying to limit a memory requirements needed. I read in parallel from multiple files, components in a pipeline might do…
Jan Pluskal
  • 103
  • 1
  • 5
8
votes
2 answers

Use TPL Dataflow to encapsulate pipeline ending in an action block

TPL Dataflow provides a very useful function: public static IPropagatorBlock Encapsulate( ITargetBlock target, ISourceBlock source) to enable you to, well, encapsulate multiple blocks into…
bornfromanegg
  • 2,826
  • 5
  • 24
  • 40
8
votes
2 answers

How to limit the number of items that are passing concurrently through an entire Dataflow pipeline?

I want to limit the number of items posted in a Dataflow pipeline. The number of items depends of the production environment. These objects consume a large amount of memory (images) so I would like to post them when the last block of the pipeline…
n3bula
  • 83
  • 1
  • 4
8
votes
1 answer

Way to know if TPL Dataflow Block busy?

TPL Dataflow block has .InputCount and .OutputCount properties. But it can perform execution over item right now, and there is no property like .Busy [Boolean]. So is there a way to know if block is now operating and one of item still there?…
AsValeO
  • 2,859
  • 3
  • 27
  • 64
8
votes
2 answers

Network Command Processing with TPL Dataflow

I'm working on a system that involves accepting commands over a TCP network connection, then sending responses upon execution of those commands. Fairly basic stuff, but I'm looking to support a few requirements: Multiple clients can connect at the…
Dan Bryant
  • 27,329
  • 4
  • 56
  • 102
8
votes
2 answers

TPL DataFlow vs BlockingCollection

I understand that a BlockingCollection is best suited for a consumer/producer pattern. However, when do I use a ActionBlock from the TPL DataFlow library? My initial understanding is for IO operations, keep the BlockingCollection while CPU intensive…
poy
  • 10,063
  • 9
  • 49
  • 74
1 2 3
…
41 42