Stack Overflow
Stack Overflow

Questions tagged [tpl-dataflow]

TPL Dataflow (TDF) is a .NET library for building concurrent applications. It promotes actor/agent-oriented designs through primitives for in-process message passing, dataflow, and pipelining. TDF builds upon the TPL (Task Parallel Library) in .NET 4 and integrates with async language support in C#, Visual Basic, and F#. TDF lacks join/merge by key (like SSIS) and time-based windowing (available in Rx).

Platform: .NET Framework 4.0 / .NET Core / .NET 6

Promotes: actor/agent programming model

Strengths: limiting parallelism (for example reading from disk bound vs. CPU compute bound tasks would be limited differently), data flow via message passing, integration with other .NET libraries like Rx and TPL.

Weaknesses: no data flow merge by key (like SSIS merge operation), and only the most trivial of time windowing (compare BroadcastBlock with Rx BufferWithTime() or CEP/StreamInsight windowing operators)

629 questions
0
votes
1 answer

Start a background task from a Web Api request

I have an ASP.WEB Web Api controller that needs to fire and forget some slow code. What would be a good way to do that? That is I want the controller to return an HTML response to the browser, while the slow code keeps running somewhere. Is it a…
TEst16
  • 407
  • 1
  • 4
  • 6
0
votes
2 answers

c# Dataflow or Tasks, consuming messages to process in parallel

If I want to take messages from an external queue, say in Redis or similar. Is it it better to have one thread constantly checking the queue and sending messages to the relevant BroadcastBlock for processing (e.g.) if (message.type == "person") …
user2788307
  • 9
  • 1
0
votes
1 answer

Using Orderby on BatchedJoinBlock(Of T1, T2) - Dataflow (Task Parallel Library)

I'm just looking to be able to sort the results of a BatchedJoinBlock (http://msdn.microsoft.com/en-us/library/hh194683.aspx) so that the different results of the different targets stay together. I will explain! Example in some pseudo-code: Dim…
Finch042
  • 307
  • 3
  • 9
  • 18
0
votes
2 answers

Simpler solution than TPL Dataflow for parallel async blob deletion

I'm implementing a worker role on Azure which needs to delete blobs from Azure storage. Let's assume my list of blobs has about 10K items. The simplest synchronous approach would probably be: Parallel.ForEach(list, x => ((CloudBlob)…
talkol
  • 12,564
  • 11
  • 54
  • 64
0
votes
2 answers

Proper way to filter BlockBuffer.RecieveAsync

Good day. I have a TPL Dataflow mesh for rpc calls It has two unkinked flows which in simplified way looks like this: Output flow: BlockBuffer to store output ActionBLock to send output to server and produce sent id And input flow: while loop to…
xakpc
  • 1,709
  • 13
  • 27
0
votes
1 answer

Task Fails to Run

I wrote a small utility to read large text files and search for lines that contain a search term. I'm using this as an opportunity to learn TPL Dataflow. The code works fine, unless the search term is near the very end of the file. In that case,…
Eric J.
  • 147,927
  • 63
  • 340
  • 553
0
votes
1 answer

Can I "hot swap" one Func in TransfromBlock for another during runtime?

I have the following setup of TransformBlock: private void SetupTestModule() { Func func1 = new Func(input => { return (input + 1); }); Func func2 = new Func(input => …
Matt
  • 7,004
  • 11
  • 71
  • 117
0
votes
1 answer

TPL Data Flow Thread Local Data

Is there a good way to pass thread local data into an ActionBlock, such that if you specify MaxDegreeOfParallelization in its DataFlowExecutionOptions to be > 1, then each task that executes the action will have its own thread local data? Here is…
dmg
  • 608
  • 8
  • 15
0
votes
2 answers

Why isn't 'post' a method of Dataflow.TransformBlock? Doesn't compile

In examples on the web, I see that there is a method Dataflow.TransformBlock.Post(), yet I can't get it to compile: Dim q As New Dataflow.TransformBlock(Of Integer, Integer)(Function(x As Integer) As Integer …
Eyal
  • 5,728
  • 7
  • 43
  • 70
0
votes
1 answer

Cancel Operations / UI Notification and UI Information?

I'm currently working on a small project that use Tasks.Dataflow and I'm a little bit confused about UI notifications. I want to separate my "Pipeline" from the UI in another class called PipelineService, but I'm unable to notify the UI on cancelled…
varg
  • 3,446
  • 1
  • 15
  • 17
0
votes
1 answer

This block must only be used with the source from which it was created

I'm using a TPL Dataflow with this initial network: (a)CustomSource => (b)TransformBlock When a message arrives in b, b creates a new transform block with a filter and append it to itself (it does not that for each message). The network becomes…
Softlion
  • 12,281
  • 11
  • 58
  • 88
-1
votes
0 answers

Filtering in TPL dataflow to discard earlier messages with the same id

I've been using ConcurrentQueue and BroadcastBlock to discard all but the most recent messages when my queue gets backed up. I have a new situation where messages are received with group ids. Now instead of discarding all messages when a new…
compound eye
  • 1,898
  • 18
  • 23
-1
votes
2 answers

Will TransformBlock.TryReceiveAll(IList) return true when if and only if all the element(s) received?

Previously pasted picture is the official definition. But I am still confused. Does it return true if and only if all the element(s) received?
Potato
  • 491
  • 1
  • 4
  • 13
-1
votes
2 answers

How to restore the order of a shuffled Dataflow pipeline?

I have a Dataflow pipeline that consists of multiple blocks that are processing heterogeneous documents (XLS, PDF etc). Each type of document is processed by a dedicated TransformBlock. At the end of the pipeline I have an ActionBlock that receives…
Theodor Zoulias
  • 34,835
  • 7
  • 69
  • 104
-1
votes
1 answer

Each element of InputQueue in TransformBlock is overwritten whenever a record is read

Purpose I'm trying to create a pipeline where I read from a file one bytes record at a time and send it to a `BufferBlock', which append items in the buffer block. This is linked through the trivial LinkTo () method to TransformBlock
NickMan
  • 23
  • 4
1 2 3
…
41
42