Stack Overflow
Stack Overflow

Questions tagged [tpl-dataflow]

TPL Dataflow (TDF) is a .NET library for building concurrent applications. It promotes actor/agent-oriented designs through primitives for in-process message passing, dataflow, and pipelining. TDF builds upon the TPL (Task Parallel Library) in .NET 4 and integrates with async language support in C#, Visual Basic, and F#. TDF lacks join/merge by key (like SSIS) and time-based windowing (available in Rx).

Platform: .NET Framework 4.0 / .NET Core / .NET 6

Promotes: actor/agent programming model

Strengths: limiting parallelism (for example reading from disk bound vs. CPU compute bound tasks would be limited differently), data flow via message passing, integration with other .NET libraries like Rx and TPL.

Weaknesses: no data flow merge by key (like SSIS merge operation), and only the most trivial of time windowing (compare BroadcastBlock with Rx BufferWithTime() or CEP/StreamInsight windowing operators)

629 questions
12
votes
2 answers

Tasks vs. TPL Dataflow vs. Async/Await, which to use when?

I have read through quite a number technical documents either by some of the Microsoft team, or other authors detailing functionality of the new TPL Dataflow library, async/await concurrency frameworks and TPL. However, I have not really come across…
Matt
  • 7,004
  • 11
  • 71
  • 117
12
votes
2 answers

Is this a job for TPL Dataflow?

I run a pretty typical producer/consumer model on different tasks. Task1: Reads batches of byte[] from binary files and kicks off a new task for each collection of byte arrays. (the operation is batched for memory management purposes). Task 2-n:…
Matt
  • 7,004
  • 11
  • 71
  • 117
11
votes
1 answer

SingleProducerConstrained and MaxDegreeOfParallelism

In the C# TPL Dataflow library, SingleProducerConstrained is an optimisation option for ActionBlocks you can use when only a single thread is feeding the action block: If a block is only ever going to be used by a single producer at a time,…
Stig Schmidt Nielsson
  • 1,667
  • 20
  • 37
11
votes
2 answers

BroadcastBlock with guaranteed delivery in TPL Dataflow

I have a stream of data that I process in several different ways... so I would like to send a copy of each message I get to multiple targets so that these targets may execute in parallel... however, I need to set BoundedCapacity on my blocks because…
Brian Rice
  • 3,107
  • 1
  • 35
  • 53
11
votes
2 answers

Where can I find a TPL dataflow version for 4.0?

I am looking for the .NET 4.0 version of the TPL dataflow library. The Nuget package has a 4.0 version of the library, but it seems to target .NET 4.5. I found various references to a 4.0 version, like in this…
aKzenT
  • 7,775
  • 2
  • 36
  • 65
11
votes
3 answers

Async logging throwing a NullReferenceException

I am trying to asynchronously log some information to SQL Server inside of an MVC 4 controller action targeting .NET 4.0 using the AsyncTargetingPack. I would jump straight to .NET 4.5 but my app lives in Azure and we're still waiting for the…
David Peden
  • 17,596
  • 6
  • 52
  • 72
10
votes
2 answers

What is the difference between TPL dataflow and Akka.net?

I have worked with TPL dataflow. Really liked it. I had heard the term Akka many times from my java/scala friends so I tried to read about it and found out that akka has a .net port too. Great. When I continued reading about what akka is, I was…
Bilal Fazlani
  • 6,727
  • 9
  • 44
  • 90
10
votes
1 answer

Unexpected Behaviour - TPL DataFlow BatchBlock Rejects items while TriggerBatch is executing

When you create a batchblock with bounded capacity and call triggerBatch while (In parallel to) posting a new item - posting new item will fail during the trigger batch execution time. Calling Trigger batch (every X time) is made in order to ensure…
Al Yaros
  • 151
  • 3
10
votes
2 answers

TPL Dataflow pipeline design basics

I try to create well-designed TPL dataflow pipeline with optimal using of system resources. My project is a HTML parser that adds parsed values into SQL Server DB. I already have all methods of my future pipeline, and now my question is what is the…
AsValeO
  • 2,859
  • 3
  • 27
  • 64
10
votes
2 answers

Force a Task to continue on the current thread?

I'm making a port of the AKKA framework for .NET (don't take this too serious now, it is a weekend hack of the Actor part of it right now) I'm having some problems with the "Future" support in it. In Java/Scala Akka, Futures are to be awaited…
Roger Johansson
  • 22,764
  • 18
  • 97
  • 193
9
votes
1 answer

Prioritized TPL DataFlow BufferBlock

It should be something very natural to have, I was wondering if there is a ready implementation of Prioritized BufferBlock from TPL DataFlow library?
husayt
  • 14,553
  • 8
  • 53
  • 81
9
votes
4 answers

Using AsObservable to observe TPL Dataflow blocks without consuming messages

I have a chain of TPL Dataflow blocks and would like to observe progress somewhere inside the system. I am aware that I could just jam a TransformBlock into the mesh where I want to observe, get it to post to a progress updater of some variety and…
theStrawMan
  • 235
  • 2
  • 9
9
votes
1 answer

How to do async operations in a TPL Dataflow for best performance?

I wrote the following method to batch process a huge CSV file. The idea is to read a chunk of lines from the file into memory, then partition these chunk of lines into batches of fixed size. Once we get the partitions, send these partitions to a…
user330612
  • 2,189
  • 7
  • 33
  • 64
9
votes
1 answer

Cancelling specific items in a dataflow pipeline

I am building a Dataflows pipeline whose job it is to process large files. Each file is parsed, analyzed, and rendered; but every file may take a different path through the pipeline, depending on what type of file it is. The user interface for this…
Bugmaster
  • 1,048
  • 9
  • 18
9
votes
3 answers

TPL Dataflow block consumes all available memory

I have a TransformManyBlock with the following design: Input: Path to a file Output: IEnumerable of the file's contents, one line at a time I am running this block on a huge file (61GB), which is too large to fit into RAM. In order to avoid…
Brian Berns
  • 15,499
  • 2
  • 30
  • 40
1 2
3
…
41 42