Messages types with DataFlow Blocks

Question

I am self-training on the TPL-Dataflow, and I have read that using immutable objects for messages is the way to go.

To comply with this, I have designed specific classes for every block inputs and outputs.

Unfortunately, when I link my block each other, because the blocks input and output types are very different, it leads to a proliferation of TransformBlock:

var proc1 = new TransformBlock<proc1In,proc1Out>(...
var convertOut1toIn2 = new TransformBlock<proc1Out,proc2In>(p1 => new proc2In { ... 
var proc2 = TransformBlock<proc2In,proc2Out>(...

proc1.LinkTo(convertOut1ToIn2);
convertOut1ToIn2.LinkTo(proc2);

Using Batch and Join blocks later to merge results together makes me struggle with a very messy code.

Every samples I read on the internet uses simple types as int, string... I have not found anything that deals with a bit more complex types.

I feel the urge to use single big object and pass its reference through all the blocks. Before doing this mistake, I would like to know if there is some better way to do.

Answer 1

After some time musing with TPL-Dataflow, it turns out that:

• Envisioning Dataflow as a conveyor belt carrying manufacturing items towards different working stations where items are enriched and built is completely wrong: doing this way leads to excruciating hard concurrency issues. Dataflow is a messaging system.

• Instead, I feel it better picturing it as a mesh of people who deals with external facilities to make things (IO, Database persistence, CalculationEngines...)

The problem of messages types I dealt with is easly circumvented using Tuples. In general I dislike Tuples uglyness, but in this very situation, I feel like they really fits this place.

My problem is multiple picture analysis. Instead of having Blocks passing a "Workitem" object each other and mess with it, I rather use an separate "WorkItemSupplier" class instead. This class uses a ConcurrentDictionary of WorkItems and exposes methods to deals with workitems.

This way, my blocks in Dataflow only passes the ID of a workitem each other, so they can use the WorkItemSupplier as an external facility to store/retrieves, or change the state of any workitem.

By this way, code is running way smoothier, well separated and easier to read.