TPL DataFlow One by one processing

Question

I am having system that continuously processing messages. I want to make sure that I request messages from an external queue only when previous message was processed. Lets imagine that GetMessages method requests messages from external queue.

• Got event 1. Will push it
• Pushed 1
• Got event 2. Will push it - my concert is here. As we get item before processing previous
• Processing 1
• Processed 1
• Deleted 1

Code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using System.Threading.Tasks.Dataflow;

namespace ConsoleApp1
{
    class Program
    {
        static void Main(string[] args)
        {
            EventProcessor a = new EventProcessor();
            Task task = Task.Run(async ()=> await a.Process());

            task.Wait();
        }
    }

    public class EventProcessor
    {
        private readonly TransformBlock<int, string> _startBlock;
        private readonly ActionBlock<string> _deleteBlock;
        private readonly ActionBlock<int> _recieveBlock;

        public EventProcessor()
        {
            var executionDataflowBlockOptions = new ExecutionDataflowBlockOptions {
                MaxDegreeOfParallelism = 1,
                BoundedCapacity = 1,
            };

            this._startBlock = new TransformBlock<int, string>(
                async @event => await this.ProcessNotificationEvent(@event),
                executionDataflowBlockOptions
            );

            this._deleteBlock = new ActionBlock<string>(async @event => {
                await this.DeleteMessage(@event);
            }, executionDataflowBlockOptions);
            var trashBin = DataflowBlock.NullTarget<string>();


            var dataflowLinkOptions = new DataflowLinkOptions {
                PropagateCompletion = true,
            };

            this._startBlock.LinkTo(
                this._deleteBlock,
                dataflowLinkOptions,
                (result => result != "o")
            );

            this._startBlock.LinkTo(
                trashBin,
                dataflowLinkOptions,
                (result => result == "o")
            );
        }

        private async Task<string> ProcessNotificationEvent(int @event)
        {
            Console.WriteLine($"Processing {@event}");
            await Task.Delay(5000);
            Console.WriteLine($"Processed {@event}");
            return @event.ToString();
        }

        public async Task Process()
        {

            //while (this._cancellationTokenSource.IsCancellationRequested == false) {
            foreach (var notificationEvent in GetMessages()) {
                Console.WriteLine($"Got event {notificationEvent}. Will push it");
                if (await this._startBlock.SendAsync(notificationEvent) == false) {
                    Console.WriteLine($"Failed to push {notificationEvent}");
                    return;
                }
                Console.WriteLine($"Pushed {notificationEvent}");
            }
            //}
            this._startBlock.Complete();
            this._deleteBlock.Completion.Wait();
        }

        private static IEnumerable<int> GetMessages() {
            return Enumerable.Range(1, 5);
        }

        private async Task DeleteMessage(string @event)
        {
            Console.WriteLine($"Deleted {@event}");
        }
    }
}

Output will be

Got event 1. Will push it
Pushed 1
Got event 2. Will push it
Processing 1
Processed 1
Deleted 1
Processing 2
Pushed 2
Got event 3. Will push it
Processed 2
Processing 3
Deleted 2
Pushed 3
Got event 4. Will push it
Processed 3
Deleted 3
Processing 4
Pushed 4
Processed 4
Deleted 4
Press any key to continue . . .

I thought that i can create TDL DataFlow for the each message, but i think it will be an overkill.

Answer 1

The problem is that you have a buffer of one so your producer loop will always be working on the next item while the first is processing. This is a natural consequence of using TPL Dataflow.

If you want to literally process one at a time, the easiest approach is probably to remove TPL Dataflow:

public class EventProcessor
{
  private async Task<string> ProcessNotificationEvent(int @event)
  {
    Console.WriteLine($"Processing {@event}");
    await Task.Delay(5000);
    Console.WriteLine($"Processed {@event}");
    return @event.ToString();
  }

  public async Task Process()
  {
    foreach (var notificationEvent in GetMessages()) {
      Console.WriteLine($"Got event {notificationEvent}. Will push it");
      var result = await this.ProcessNotificationEvent(notificationEvent);
      if (result != "o")
        await DeleteMessage(result);
    }
  }

  private static IEnumerable<int> GetMessages() => Enumerable.Range(1, 5);

  private async Task DeleteMessage(string @event) => Console.WriteLine($"Deleted {@event}");
}