How to prevent OTL Pipeline from making its input sealed?

Question

Many years ago in the old forum I asked Primozh if Pipeline pattern can be kind of Uroboros, feeding half-complete results back to itself.

Back then Primozh said it would be straight-forward and PipeLine stage can feed OmniValues not only to the OUTPUT but to INPUT too.

The problem is that initial feeding stages are run way too fast, they expire and seal the INPUT collection and there is no way to UN-SEAL it, and thus as soon as they try to feed half-baked packets back to themselves - voila! - OTL throws "cannot add to completed collection" exception.

So how can this self-exploding task by the link above be implemented via self-feeding Pipeline pattern?

UPD: Changed the example from "self-exploding" - generating huge amounts of intermediate half-calculated results - permutations generation, to a simple (i hope) calculating of factorial. This however has disadvantage of determinism: it always generate ONE intermediate job item, so the ability of pipeline to deal with growing collections was not tasted.

{$A+} // not $A8
type FactTask = record
  Goal, Curr: Cardinal;
  Value : Int64;
end;

procedure TForm6.Button1Click(Sender: TObject);
var Msg: string;
    f: FactTask;
    Results: TArray<Int64>;
    pipeOut: IOmniBlockingCollection;
    pipe:    IOmniPipeline;
begin
  lblResults.Caption := ' WAIT, we are producing...';
  Repaint;

  pipe := Parallel.Pipeline;
  f.Goal := edLen.Value; // 10
  f.Curr := 0;
  f.Value := 1;

  pipe.Stage(
     procedure ( const input, output: IOmniBlockingCollection )
     begin
       output.Add( TOmniValue.FromRecord( f ) );
     end
  );

  pipe.Stage(
     procedure ( const input, output: IOmniBlockingCollection )
     var f_in, f_out: FactTask; v: TOmniValue;
     begin
       for v in input do begin
         f_in := v.ToRecord<FactTask>;
         if f_in.Curr < f_in.Goal then begin
            f_out.Goal := f_in.Goal;
            f_out.Curr := Succ(f_in.Curr);
            f_out.Value := f_in.Value * f_out.Curr;
            input.Add( TOmniValue.FromRecord( f_out ) ); //  <<< Exception!
         end;
       end;
     end
  );

  pipe.Stage(
     procedure ( const input, output: IOmniBlockingCollection )
     var f_in: FactTask;  v: TOmniValue;
     begin
       for v in input do begin
         f_in := v.ToRecord<FactTask>;
         if f_in.Curr = f_in.Goal then begin
            Output.Add( f_in.Value );
         end;
       end;
     end
  );

  pipe.Run;
  pipeOut := pipe.Output;
//    pipe.WaitFor(INFINITE);  ToArray would efficiently do that
//    pipeOut.CompleteAdding;    ...without frozing on Pipeline/Collections SetThrottle
  Results := TOmniBlockingCollection.ToArray<Int64>(pipeOut);

  Msg := IntToStr(f.Goal) + '! = ' + IntToStr(Results[0]);
  lblResults.Caption := Msg;
  ShowMessage(Msg);
end;

It crashes with the pipeline stage trying to re-fill the input that unexpectedly got sealed by TOmniPipeline.Run. At the marked line the "Cannot add to competed collection" exception is unexpectedly thrown.

How to keep the Pipeline running when collection is balancing between empty and few ( it is not only the starting condition, it would be repeated near the calculations end )?

A bit of dreaming: https://plus.google.com/+AriochThe/posts/LCHnSCmZYtx
Bit more: https://github.com/gabr42/OmniThreadLibrary/issues/61

Answer 1

Your demo program works as it should.

Your first stage just outputs one record to its output (by the way, you could do that in the main thread by writing to pipe.Input) and then it closes its output pipe.

This in turn causes second stage to shut down. Before for v in input exits, second stage typically tries to write to input (unless you are really lucky with timing). Its input is, however, already closed and Add raises an exception. Calling TryAdd instead of Add will fix that.

I would guess that Pipeline is not really the abstraction you are looking for and that you would be better off by using something else. I would go with normal low-level task wrapping a TOmniBlockingCollection (for stage 2). You would have to create this blocking collection with Create(1) so that it will be aware that it is feeding itself and will unblock automatically. (See Parallel Search in a Tree example from the book for more information.)