0

I am consuming one data streaming source which pushes data in interval >= 5 seconds. I have one observer like below to receive push notifications:

public class ConsoleDataObserver : IObserver<DataPipeEvent>
    {
        private DataPipeType _dataPipeType;
        private LimitedConcurrencyLevelTaskScheduler _scheduler;//This is based on msdn parallel extensions examples
        private TaskFactory _factory;


        public ConsoleDataObserver(DataPipeType afDataPipeType)
        {
            _dataPipeType = afDataPipeType;
            _scheduler = new LimitedConcurrencyLevelTaskScheduler(5);
            _factory = new TaskFactory(_scheduler);
        }


        public void OnNext(DataPipeEvent dataPipeEvent)
        {
            _factory.StartNew(() => ProcessDataNow(dataPipeEvent));
        }

        private void ProcessDataNow(DataPipeEvent dataPipeEvent)
        {
            Thread.Sleep(8000); //just want to simulate long running tasks

        }

        public void OnError(Exception error)
        {
            Console.WriteLine("Provider has sent an error");
        }

        public void OnCompleted()
        {
            Console.WriteLine("Provider has terminated sending data");
        }
    }

I have following requirements:

In my OnNext, I don't want to block main thread and want to do long running processing tasks in other thread. I am using TaskScheduler to used ThreadPool. Is it good implementation? OnNext can get 500-1000 events per second. ProcessDataNow will log in case of any exception.

Abhay
  • 928
  • 1
  • 10
  • 28
  • 1
    You should be using [ActionBlock](https://msdn.microsoft.com/en-us/library/hh194684(v=vs.110).aspx) instead of implementing your own pipeline processing. That's not what Observer is about anyway – Panagiotis Kanavos Dec 21 '16 at 15:05
  • What would be the advantage? – Abhay Dec 21 '16 at 15:06
  • That Observer offers no functionality for pipeline processing, while ActionBlock and the TPL Dataflow library is purpose-built for that? With ActionBlock you specify a single lambda, your desired DOP and input limits, and just post messages to it for processing. It *already* implements what you'd have to implement by hand using Tasks – Panagiotis Kanavos Dec 21 '16 at 15:08
  • What is your *actual* problem? TPL Dataflow has many blocks that can be linked to create a processing pipeline. Eg the ActionBlock receives messages but doesn't produce output. TransformBlock generates output that is sent to the next block in the pipeline – Panagiotis Kanavos Dec 21 '16 at 15:10
  • You can specify conditions when linking the block so that eg. error messages are sent to a different block. This way the problem changes to how you can break your code to independent, communicating steps. Very similar to a Linux shell or Powershell pipeline. You don't have to solve how to glue the steps together, how to synchronize them or how to schedule them – Panagiotis Kanavos Dec 21 '16 at 15:11
  • I am not doing pipeline processing using observer. Observer is only giving me events which I want to process in parallel. Can you please provide some pointer or example? – Abhay Dec 21 '16 at 15:15
  • Did you check the link? It shows how to post events to the block and even emulates processing with a ... Sleep. – Panagiotis Kanavos Dec 21 '16 at 15:15

0 Answers0