Stack Overflow
Stack Overflow

Questions tagged [tpl-dataflow]

TPL Dataflow (TDF) is a .NET library for building concurrent applications. It promotes actor/agent-oriented designs through primitives for in-process message passing, dataflow, and pipelining. TDF builds upon the TPL (Task Parallel Library) in .NET 4 and integrates with async language support in C#, Visual Basic, and F#. TDF lacks join/merge by key (like SSIS) and time-based windowing (available in Rx).

Platform: .NET Framework 4.0 / .NET Core / .NET 6

Promotes: actor/agent programming model

Strengths: limiting parallelism (for example reading from disk bound vs. CPU compute bound tasks would be limited differently), data flow via message passing, integration with other .NET libraries like Rx and TPL.

Weaknesses: no data flow merge by key (like SSIS merge operation), and only the most trivial of time windowing (compare BroadcastBlock with Rx BufferWithTime() or CEP/StreamInsight windowing operators)

629 questions
6
votes
1 answer

TPL Dataflow: Persist previous data

I have been using TPL dataflow for an image processing pipeline using the producer/consumer pattern. I'm trying to work out the best approach to allow for algorithms that require either the previous frame or a persistent object. An example of one…
John Redmond
  • 63
  • 3
6
votes
2 answers

How can I specify an unordered Execution Block using the TPL Dataflow Library?

I want to set up a TransformBlock that processes its item in parallel. Thus, I'm setting ExecutionDataflowBlockOptions.MaxDegreeOfParallelism to > 1. I don't care about the order of the messages but the documentation says: When you specify a…
Dejan
  • 9,150
  • 8
  • 69
  • 117
6
votes
1 answer

Hashed/Sharded ActionBlocks

I have a constant flow of certain items that I need to process in parallel so I'm using TPL Dataflow. The catch is that the items that share the same key (similar to a Dictionary) should be processed in a FIFO order and not be parallel to each other…
i3arnon
  • 113,022
  • 33
  • 324
  • 344
6
votes
2 answers

How to implement continuously running dataflow blocks in TPL?

I have producer/consumer dataflow block set-up using BufferBlock and ActionBlock and it is working fine inside Console application; After adding all items into BurfferBlock and Linking BufferBlock with other Action Items; it is working good. now I…
user2757350
  • 311
  • 3
  • 12
6
votes
1 answer

Task Dataflow, can a data block be changed from completion state?

I would like to know whether it is possible to change the completion state of data blocks? For example, I marked a var block = new BufferBlock(); data block complete with block.Complete(). The block is linked to other data blocks. I would like…
Matt
  • 7,004
  • 11
  • 71
  • 117
6
votes
1 answer

await async lambda in ActionBlock

I have a class Receiver with an ActionBlock: public class Receiver : IReceiver { private ActionBlock _receiver; public Task Send(T item) { if(_receiver!=null) return _receiver.SendAsync(item); //Do some…
obi111
  • 61
  • 1
  • 2
6
votes
1 answer

TPL Dataflows LinkTo multiple consumers not working

I have a BufferBlock to which I post messages: public class DelimitedFileBlock : ISourceBlock { private ISourceBlock _source; _source = new BufferBlock(new DataflowBlockOptions() { BoundedCapacity = 10000 }); …
user1685304
  • 91
  • 1
  • 5
5
votes
1 answer

AsyncLocal values not correct with TPL Dataflow

Consider this example: class Program { private static readonly ITargetBlock Mesh = CreateMesh(); private static readonly AsyncLocal AsyncLocalContext = new AsyncLocal(); static async Task Main(string[]…
Azat
  • 2,275
  • 1
  • 28
  • 32
5
votes
0 answers

Using Polly with TPL Dataflow

Data processing pipelines and transient fault handling seem to go hand in hand, so I'm interested in seeing if I can get 2 of the best libraries for these - TPL Dataflow and Polly, respectively - to play nicely together. As a starting point, I'd…
Todd Menier
  • 37,557
  • 17
  • 150
  • 173
5
votes
2 answers

Is TPL Dataflow BufferBlock thread safe?

I have a fairly simple producer-consumer pattern where (simplified) I have two producers who produce output that is to be consumed by one consumer. For this I use System.Threading.Tasks.Dataflow.BufferBlock A BufferBlock object is created. One…
Harald Coppoolse
  • 28,834
  • 7
  • 67
  • 116
5
votes
1 answer

TPL Dataflow and exception handling in downstream blocks

I have the following pseudo code: var queue = new BufferBlock(new DataflowBlockOptions { BoundedCapacity = 5 }); var a = new ActionBlock(async item => { await Task.Delay(500); Trace.TraceInformation( …
Tomasz Jaskuλa
  • 15,723
  • 5
  • 46
  • 73
5
votes
1 answer

Write to open FileStream using reactive programming

I am writing a small logger and I want to open the log file once, keep writing reactively as log messages arrive, and dispose of everything on program termination. I am not sure how I can keep the FileStream open and reactively write the messages as…
heltonbiker
  • 26,657
  • 28
  • 137
  • 252
5
votes
2 answers

How to wait until item goes through pipeline?

So, I'm trying to wrap my head around Microsoft's Dataflow library. I've built a very simple pipeline consisting of just two blocks: var start = new TransformBlock(); var end = new ActionBlock(); start.LinkTo(end); Now I can…
Nikita B
  • 3,303
  • 1
  • 23
  • 41
5
votes
1 answer

A datablock to join a single result with multiple other results

In my application I want to join multiple strings with a dictionary of replacement values. The readTemplateBlock gets fed with FileInfos and returns their contents as string. The getReplacersBlock gets fed (once) with a single replacers…
Boris Callens
  • 90,659
  • 85
  • 207
  • 305
5
votes
1 answer

How to create never ending DataFlow Mesh with exception handling?

I am creating a Task processor which uses TPL DataFlow. I will follow a producer consumer model where in Producer produces some items to be processed once in a while and consumers keep waiting for new items to arrive. Here is my code: async Task…
Amit
  • 25,106
  • 25
  • 75
  • 116
1 2 3
…
41 42