TPL Dataflow block never completes on PropagateCompletion

Question

Since the last alteration to my propagated-completion pipeline, one of my buffer blocks never completes. Let me summarize what was working and what isn't anymore:

Previously working:

A.LinkTo(B, PropagateCompletion);
B.LinkTo(C, PropagateCompletion);
C.LinkTo(D, PropagateCompletion);
D.Receive();

// everything completes

No longer working:

A.LinkTo(B, PropagateCompletion);
C.LinkTo(D, PropagateCompletion);

await A.Completion;
someWriteOnceBlock.Post(B.Count);
// B.Complete(); commented on purpose
B.LinkTo(C, PropagateCompletion);

D.Receive();

// Only A reaches completion
// B remains in 'waiting for activation'
// C executes but obviously never completes since B doesn't either

If I uncomment the commented line, everything works, but obviously that line should not be necessary.

Somehow my BufferBlock B never reaches completion, even though the block linked to it is completed and propagates its completion, and the block linked from it receives all the buffered items.

@i3arnon I can't put the whole code as the pipeline is pretty complex, and cleaning it up just to keep the relevant parts would take a long time, but more importantly, everything in my pipeline is pretty standard, all my blocks are linked and one thing goes in one thing comes out. Those lines around await are the only unusual thing, and they are exactly the way I have them, and I believe there's nothing in the rest of my perfectly working pipeline that explains that uncommenting the commented line makes it work, and commenting it prevents B from ever completing. — Luis Ferrao, Jun 22 '15 at 07:29

score 2 · Answer 1 · answered Jun 19 '15 at 11:08

2

By awaiting the completion of A none of the remaining code is executed until A completes. That's how await works - the code after it is wrapped in a continuation ready for the completion of the awaited code. So in this scenario B is linked to A after A completes so completion is not propagated I think.

answered Jun 19 '15 at 11:08

James Lucas

2,452
10
15

First, you probably mean A is linked to B (not B to A), second, your explanation only confirms that awaiting A.Completion is the problem, which obviously I already knew, but it doesn't say anything about whatever happened to the propagated completion, was it simply ignored? – Luis Ferrao Jun 19 '15 at 11:40

Liam · Answer 2 · 2021-02-10T09:48:43.400

I had the same problem - my final block was a TransformBlock that was just sitting there awaiting completion.

A block will only complete if it has has no inputs remaining to process, all it's outputs have been processed and it has Completed, either by calling block.Complete or the completion being propagated down the chain.

I fixed this by making the final block an ActionBlock, as this doesn't produce any outputs. Here's my code, that takes a series of text files and deserialises their contents:

Dim maxDop = Environment.ProcessorCount
Dim executionOptions = New ExecutionDataflowBlockOptions With {.MaxDegreeOfParallelism = maxDop}
Dim linkOptions = New DataflowLinkOptions With {.PropagateCompletion = True}

Dim inputBlock = New BufferBlock(Of String)
Dim transformBlock = New TransformBlock(Of String, IEnumerable(of JsonObject))(Function(fileName) DeserialiseFromFileAsync(fileName)
Dim outputBlock As New BufferBlock(Of IEnumerable(of JsonObject))
Dim combineBlock = New ActionBlock(Of IEnumerable(of JsonObject))(Sub(col) ' Do something to add all the collections together)

inputBlock.LinkTo(transformBlock, linkOptions)
transformBlock.LinkTo(outputBlock, linkOptions)
outputBlock.LinkTo(combineBlock, linkOptions)

For fileNo = 1 To 10
    inputBlock.Post(String.Concat("JsonFile", fileNo, ".txt"))
Next

inputBlock.Complete() 'Complete the first block, propogation will handle the rest
Await combineBlock.Completion 'Await the last block completing

TPL Dataflow block never completes on PropagateCompletion

2 Answers2