Stack Overflow
Stack Overflow

Questions tagged [tpl-dataflow]

TPL Dataflow (TDF) is a .NET library for building concurrent applications. It promotes actor/agent-oriented designs through primitives for in-process message passing, dataflow, and pipelining. TDF builds upon the TPL (Task Parallel Library) in .NET 4 and integrates with async language support in C#, Visual Basic, and F#. TDF lacks join/merge by key (like SSIS) and time-based windowing (available in Rx).

Platform: .NET Framework 4.0 / .NET Core / .NET 6

Promotes: actor/agent programming model

Strengths: limiting parallelism (for example reading from disk bound vs. CPU compute bound tasks would be limited differently), data flow via message passing, integration with other .NET libraries like Rx and TPL.

Weaknesses: no data flow merge by key (like SSIS merge operation), and only the most trivial of time windowing (compare BroadcastBlock with Rx BufferWithTime() or CEP/StreamInsight windowing operators)

629 questions
17
votes
3 answers

TransformBlock never completes

I'm trying to wrap my head around "completion" in TPL Dataflow blocks. In particular, the TransformBlock doesn't seem to ever complete. Why? Sample program My code calculates the square of all integers from 1 to 1000. I used a BufferBlock and a…
Steven Liekens
  • 13,266
  • 8
  • 59
  • 85
17
votes
1 answer

Apparent BufferBlock.Post/Receive/ReceiveAsync race/bug

cross-posted to http://social.msdn.microsoft.com/Forums/en-US/tpldataflow/thread/89b3f71d-3777-4fad-9c11-50d8dc81a4a9 I know... I'm not really using TplDataflow to its maximum potential. ATM I'm simply using BufferBlock as a safe queue for message…
spender
  • 117,338
  • 33
  • 229
  • 351
16
votes
1 answer

What are the use cases for TPL Dataflow over Reactive Extensions (Rx)

I'm specifically looking at writing some signal processing algorithms in one or other, or maybe some combination of both of these. Performance isn't a big concern, clarity of expressing intent is more important. I'd be looking to implement the…
stevenrcfox
  • 1,547
  • 1
  • 14
  • 37
16
votes
2 answers

Problems with references to TPL Dataflow and TPL in VS 2012 RC

I just upgraded Visual Studio 11 Beta to the new Visual Studio 2012 RC and have problems referencing TPL Dataflow. First, I tried to reference Dataflow as I did previously, by adding a reference from the framework. But when I try to do that, I get…
svick
  • 236,525
  • 50
  • 385
  • 514
15
votes
1 answer

Does BoundedCapacity include items currently being processed in TPL Dataflow?

Does the BoundedCapacity limit only includes items in the input queue waiting to be processed or does it also count items being processed at the moment? Lets take for example this ActionBlock: var block = new ActionBlock( i =>…
i3arnon
  • 113,022
  • 33
  • 324
  • 344
14
votes
3 answers

Dataflow with splitting work to small jobs and then group again

I need to do this kind of work: Get Page object from database For each page get all images and process them (IO bound, for example, upload to CDN) If all images proceeded successfully then mark Page as processed in database Since I need to control…
Michael Logutov
  • 2,551
  • 4
  • 28
  • 32
14
votes
2 answers

How do I link multiple target blocks with a source block in TPL Dataflow?

I expected the following to produce output from both publishers, but it only produces output from the first one: var broadcastBlock = new BroadcastBlock(null); var transformBlock = new TransformBlock(i => i*10); var publish1 = new…
Amit G
  • 5,165
  • 4
  • 28
  • 29
14
votes
1 answer

Entity Framework and Parallelism

Background I have an application that receives periodic data dumps (XML files) and imports them into an existing database using Entity Framework 5 (Code First). The import happens via EF5 rather than say BULK INSERT or BCP because business rules…
Eric J.
  • 147,927
  • 63
  • 340
  • 553
14
votes
1 answer

TPL Dataflow and Rx Combined example

I just want to learn both and how to use them together. I understand that they can complement each other I just could not find an example of someone actually doing it.
naeron84
  • 2,955
  • 3
  • 30
  • 37
13
votes
3 answers

Batching on duration or threshold using TPL Dataflow

I have implemented a producer..consumer pattern using TPL Dataflow. The use case is that code reads messages from the Kafka bus. For efficiency, we need to process messages in batches when going to the database. Is there a way in TPL data flow to…
Ashish Bhatia
  • 161
  • 1
  • 6
13
votes
1 answer

Difference between DataflowBlockOptions.BoundedCapacity and BufferBlock

Let's assume i have a simple ActionBlock var actionBlock = new ActionBlock(_ => Console.WriteLine(_)); I can specify a bounded capacity to enable buffering: var actionBlock = new ActionBlock( _ => Console.WriteLine(_), new…
i3arnon
  • 113,022
  • 33
  • 324
  • 344
12
votes
1 answer

Global per-block error handling in a Dataflow pipeline

I am designing a long-running Dataflow pipeline that consists of multiple blocks. Items are fed to the input block of the pipeline, eventually make their way through it, and are displayed in the UI at the end (as a courtesy to the user -- the…
Bugmaster
  • 1,048
  • 9
  • 18
12
votes
2 answers

TPL Dataflow exception in transform block with bounded capacity

I need to construct TPL dataflow pipeline which will process a lot of messages. Because there are many messages I can not simply Post them into infinite queue of the BufferBlock or I will face memory issues. So I want to use BoundedCapacity = 1…
Michael Logutov
  • 2,551
  • 4
  • 28
  • 32
12
votes
3 answers

Retry policy within ITargetBlock

I need to introduce a retry policy to the workflow. Let's say there are 3 blocks that are connected in such a way: var executionOptions = new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 3 }; var buffer = new BufferBlock(); var…
Oleks
  • 31,955
  • 11
  • 77
  • 132
12
votes
5 answers

Am I doing something wrong or is it not possible to extract a zip file in parallel?

I created this to test out a parallel extract: public static async Task ExtractToDirectoryAsync(this FileInfo file, DirectoryInfo folder) { ActionBlock block = new ActionBlock((entry) => { …
Poul K. Sørensen
  • 16,950
  • 21
  • 126
  • 283
1
2
3
…
41 42