Stack Overflow
Stack Overflow

Questions tagged [tpl-dataflow]

TPL Dataflow (TDF) is a .NET library for building concurrent applications. It promotes actor/agent-oriented designs through primitives for in-process message passing, dataflow, and pipelining. TDF builds upon the TPL (Task Parallel Library) in .NET 4 and integrates with async language support in C#, Visual Basic, and F#. TDF lacks join/merge by key (like SSIS) and time-based windowing (available in Rx).

Platform: .NET Framework 4.0 / .NET Core / .NET 6

Promotes: actor/agent programming model

Strengths: limiting parallelism (for example reading from disk bound vs. CPU compute bound tasks would be limited differently), data flow via message passing, integration with other .NET libraries like Rx and TPL.

Weaknesses: no data flow merge by key (like SSIS merge operation), and only the most trivial of time windowing (compare BroadcastBlock with Rx BufferWithTime() or CEP/StreamInsight windowing operators)

629 questions
8
votes
2 answers

How can a TPL Dataflow block downstream get data produced by a source?

I'm processing images using TPL Dataflow. I receive a processing request, read an image from a stream, apply several transformations, then write the resulting image to another stream: Request -> Stream -> Image -> Image ... -> Stream For that I use…
chase
  • 1,623
  • 1
  • 16
  • 21
7
votes
0 answers

Why does BufferBlock.ReceiveAsync() hang when data is available?

I am new to TPL Dataflow. I am trying to build a throttled async update to a fairly fast flowing input stream. BufferBlock seemed a nice match for this with the idea that I could call the ReceiveAll() to grab everything off the buffer and on the…
Wizbit
  • 421
  • 1
  • 4
  • 13
7
votes
2 answers

How to make fast producer paused when consumer is overwhelmed?

I have producer / consumer pattern in my app implemented used TPL Dataflow. I have the big dataflow mesh with about 40 blocks in it. There are two main functional parts in the mesh: producer part and consumer part. Producer supposed to continuosly…
kseen
  • 359
  • 8
  • 56
  • 104
7
votes
2 answers

ActionBlock vs Task.WhenAll

I would like to know what is the recommended way to execute multiple async methods in parallel? in System.Threading.Tasks.Dataflow we can specify the max degree of parallelism but unbounded is probably the default for Task.WhenAll too ? this : var…
fred_
  • 1,486
  • 1
  • 19
  • 31
7
votes
1 answer

BufferBlock and ActionBlock with BoundedCapacity does not use max DOP

I have this code: var data = new BufferBlock(new DataflowBlockOptions { BoundedCapacity = 1 }); var action = new ActionBlock(async id => { Console.WriteLine("[{0:T}] #{1}: Start", DateTime.Now, id); await Task.Delay(1000); …
Michael Logutov
  • 2,551
  • 4
  • 28
  • 32
7
votes
1 answer

Alternate to Dataflow BroadcastBlock with guaranteed delivery

I need to have some kind of object that acts like a BroadcastBlock, but with guaranteed delivery. So i used an answer from this question. But i don't really clearly understand the execution flow here. I have a console app. Here is my code: static…
shda
  • 729
  • 7
  • 19
7
votes
3 answers

using TPL Dataflow, can I cancel all posts and then add one?

With the TPL Dataflow library, I would like to do something like this: myActionBlock.Post(newValue, cancelAllPreviousPosts: true); It appears that the cancellation token on ActionBlock cancels the whole thing; I'd have to make a new ActionBlock if…
Brannon
  • 5,324
  • 4
  • 35
  • 83
7
votes
1 answer

Awaiting ActionBlock - TPL DataFlow

I am using TPL DataFlow and an ActionBlock to create parallelism. The reason for using TPL DataFlow is because it supports asynchronicity, except I can't get it to work. var ab = new ActionBlock(async group => { try { …
James Jeffery
  • 12,093
  • 19
  • 74
  • 108
6
votes
2 answers

Send parallel requests but only one per host with HttpClient and Polly to gracefully handle 429 responses

Intro: I am building a single-node web crawler to simply validate URLs are 200 OK in a .NET Core console application. I have a collection of URLs at different hosts to which I am sending requests with HttpClient. I am fairly new to using Polly and…
Collin Barrett
  • 2,441
  • 5
  • 32
  • 53
6
votes
2 answers

TPL Dataflow vs plain Semaphore

I have a requirement to make a scalable process. The process has mainly I/O operations with some minor CPU operations (mainly deserializing strings). The process query the database for a list of urls, then fetches data from these urls, deserialize…
BornToCode
  • 9,495
  • 9
  • 66
  • 83
6
votes
1 answer

TPL Dataflow: Flatten incoming collection to sequential items

I'm building an application using TPL dataflow. Actually I've the following problem. I've one transformblock var tfb1 = new TranformBlock>. So tfb1 receives on in message and creates a list of out-messages. This…
Moerwald
  • 10,448
  • 9
  • 43
  • 83
6
votes
2 answers

TransformBlock posting to output

My scenario is that I have a BufferBlock receiving Stream's from an external source, let's say the file system or some FTP server. These file Streams will pass into another block and undergo processing. The only catch is that some of these…
Gideon
  • 18,251
  • 5
  • 45
  • 64
6
votes
1 answer

How to construct a TransformManyBlock with a delegate

I'm new to C# TPL and DataFlow and I'm struggling to work out how to implement the TPL DataFlow TransformManyBlock. I'm translating some other code into DataFlow. My (simplified) original code was something like this: private IEnumerable…
Matt L
  • 83
  • 5
6
votes
1 answer

Best practice for ITargetBlock.Completion.ContinueWith()

This question is about best practices when using ContinueWith() to handle a TPL datablock's completion. The ITargetBlock.Completion() method allows you to asynchronously handle a datablock's completion using ContinueWith(). Consider the…
Matthew Watson
  • 104,400
  • 10
  • 158
  • 276
6
votes
2 answers

How can I branch logic in TPL Dataflow?

Im brand new to TPL dataflow so forgive me if this is a simple question. I have an input buffer block that takes a base class. How can I branch from there to a block based on the derived type? So for example: var inputBlock = new…
jmichas
  • 1,139
  • 1
  • 10
  • 21
1 2 3
…
41 42