Reading output from console application and WPF plotting async

Question

I have a console application which outputs about 160 lines of info every 1 second.

The data output is points that can be used to plot on a graph.

In my WPF application, I've successfully have this hooked up and the data output by the console application is being plotted, however, after about 500 or so data points, I see significant slow down in the application and UI thread lockups.

I assume this is due to the async operations I'm using:

BackgroundWorker worker = new BackgroundWorker();

worker.DoWork += delegate(object s, DoWorkEventArgs args)
{
    _process = new Process();
    _process.StartInfo.FileName = "consoleApp.exe";
    _process.StartInfo.UseShellExecute = false;
    _process.StartInfo.RedirectStandardOutput = true;
    _process.StartInfo.CreateNoWindow = true;
    _process.EnableRaisingEvents = true;
    _process.OutputDataReceived += new DataReceivedEventHandler(SortOutputHandler);
    _process.Start();
    _process.BeginOutputReadLine();
    _watch.Start();
};
worker.RunWorkerAsync();

And the handler that is taking care of parsing and plotting the data:

private void SortOutputHandler(object sendingProcess, DataReceivedEventArgs outLine)
{
    if (!String.IsNullOrEmpty(outLine.Data))
    {
            var xGroup = Regex.Match(outLine.Data, "x: ?([-0-9]*)").Groups[1];
            int x = int.Parse(xGroup.Value);

            var yGroup = Regex.Match(outLine.Data, "y: ?([-0-9]*)").Groups[1];
            int y = int.Parse(yGroup.Value);

            var zGroup = Regex.Match(outLine.Data, "z: ?([-0-9]*)").Groups[1];
            int z = int.Parse(zGroup.Value);


            Reading reading = new Reading()
            {
                Time = _watch.Elapsed.TotalMilliseconds,
                X = x,
                Y = y,
                Z = z
            };


            Dispatcher.Invoke(new Action(() =>
            {
                _readings.Enqueue(reading);
                _dataPointsCount++;

            }), System.Windows.Threading.DispatcherPriority.Normal);                    
    }
}

_readings is a custom ObservableQueue<Queue> as defined in this answer. I've modified it so that only 50 items can be in the queue at a time. So if a new item is being added and the queue count >= 50, a Dequeue() is called before an Enqueue().

Is there any way I can improve the performance or am I doomed because of how much the console app outputs?

Have you started the process from the main application thread? If so that could explain the UI lock up. — ChrisF, Feb 19 '11 at 22:54
@Chris - No, I'm using a background worker for the process. I've updated my question with more code. — Omar, Feb 19 '11 at 23:01
Thanks for clearing that up. At least we know to look elsewhere ;) — ChrisF, Feb 19 '11 at 23:03
So is it correct to assume you're only plotting the graph for 50 points at a time? — bug-a-lot, Feb 22 '11 at 13:15
Yes, the queue is an observable collection which will dequeue prior to enqueue if and only if the current number of points in it is >= 50 — Omar, Feb 22 '11 at 16:29
Is it an option to redirect the streams and read from them? I suppose that the event is coming faster than you can handle it and it somehow "piles up"... — Daniel Hilgarth, Feb 23 '11 at 13:38
I believe `_process.StartInfo.RedirectStandardOutput = true;` is redirecting the stream. I'm not sure if I can get an actual instance of the stream. — Omar, Feb 23 '11 at 16:30
Did you solve your perf problems by now? My solution (Alois Kraus) should give you a major perf boost with minor code changes. — Alois Kraus, Mar 04 '11 at 14:08

CCondron · Answer 1 · 2011-02-25T05:35:57.540

From what I can tell here is what it looks like is going on:

IU thread spins up a background worker to launch the console App.
It redirects the output of the Console and handles it with a handler on the UI thread
The handler on the UI thread then calls Dispatcher.Invoke 160 times a second to update a queue object on the same thread.
After 50 calls the queue starts blocking while items are dequeued by the UI

The trouble would seem to be:

Having the UI thread handle the raw output from the console and the queue and the update to the Graph.

There is also a potential problem with blocking between enqueue and dequeue once the UI is over 50 data items behind that might be leading to a cascading failure. (I can't see enough of the code to be sure of that)

Resolution:

Start another background thread to manage the data from the console app
The new thread should: Create the Queue; handle the OutputDataReceived event; and launch the console app process.
The Event Handler should not use Dispatcher.Invoke to update the Queue. A direct threadsafe call should be used.
The Queue really needs to be non blocking when updating the UI, but I don't really have enough information about how that's being implemented to comment.

Hope this helps -Chris

If I create the queue in the background thread, will the UI still have access to it? Because the queue is a `ObservableCollection`, I need the graph to notice when it is updated so that it can automatically update itself. — Omar, Feb 25 '11 at 20:17
Pass a delegate from the main thread in the arguments to the background worker. Invoke that when the queue hase enough data. You'll need to make sure the updates are threadsafe. — CCondron, Mar 02 '11 at 20:24

score 0 · Answer 2 · answered Mar 01 '11 at 00:52

I suspect that there's a thread starvation issue happening on the UI thread as your background thread is marshaling calls to an observable collection that is possibly forcing the underlying CollectionView to be recreated each time. This can be a pretty expensive operation.

Depending how you've got your XAML configured is also a concern. The measure / layout changes alone could be killing you. I would imagine that at the rate the data is coming in, the UI hasn't got a chance to properly evaluate what's happening to the underlying data.

I would suggest not binding the View to the Queue directly. Instead of using an Observable Queue as you've suggested, consider:

Use a regular queue that caps content at 50 items. Don't worry about the NotifyCollectionChanged event happening on the UI thread. You also won't have to marshal each item to the UI thread either.
Expose a CollectionViewSource object in your ViewModel that takes the Queue as its collection.
Use a timer thread on the UI to manually force a refresh of the CollectionViewSource. Start with once a second and decrease the interval to see what your XAML and machine can handle. In this fashion, you control when the CollectionView is created and destroyed.

score 0 · Answer 3 · answered Mar 01 '11 at 03:39

You could try passing the processed data onto the UI Thread from the BackgroundWorker ProgressChanged event.

Something like....

// Standard warnings apply: not tested, no exception handling, etc.

     var locker = new object();
     var que = new ConcurrentQueue<string>();
     var worker = new BackgroundWorker();
     var proc = new Process();

     proc.StartInfo.FileName = "consoleApp.exe";
     proc.StartInfo.UseShellExecute = false;
     proc.StartInfo.RedirectStandardOutput = true;
     proc.StartInfo.CreateNoWindow = true;
     proc.EnableRaisingEvents = true;

     proc.OutputDataReceived +=
        (p, a) =>
        {
           que.Enqueue(a.Data);
           Monitor.Pulse(locker);
        };

     worker.DoWork +=
        (s, e) =>
        {
           var watch = Stopwatch.StartNew();
           while (!e.Cancel)
           {
              while (que.Count > 0)
              {
                 string data;
                 if (que.TryDequeue(out data))
                 {
                    if (!String.IsNullOrEmpty(data))
                    {
                       var xGroup = Regex.Match(data, "x: ?([-0-9]*)").Groups[1];
                       int x = int.Parse(xGroup.Value);

                       var yGroup = Regex.Match(data, "y: ?([-0-9]*)").Groups[1];
                       int y = int.Parse(yGroup.Value);

                       var zGroup = Regex.Match(data, "z: ?([-0-9]*)").Groups[1];
                       int z = int.Parse(zGroup.Value);

                       var reading = new Reading()
                       {
                          Time = watch.Elapsed.TotalMilliseconds,
                          X = x,
                          Y = y,
                          Z = z
                       };

                       worker.ReportProgress(0, reading);
                    }
                 }
                 else break;
              }
              // wait for data or timeout and check if the worker is cancelled.
              Monitor.Wait(locker, 50);
           }
        };

     worker.ProgressChanged +=
        (s, e) =>
        {
           var reading = (Reading)e.UserState;
           // We are on the UI Thread....do something with the new reading...
        };

     // start everybody.....
     worker.RunWorkerAsync();
     proc.Start();
     proc.BeginOutputReadLine();

Alois Kraus · Answer 4 · 2011-03-02T19:31:44.550

You can simply store the points in a list and call the dispatcher only when you have e.g. reached 160 points so you do not create to many update messages. Currently you are causing a window message every 6ms which is way too much. When you update the UI e.g. every second or every 160 points things will be much smoother. If the notifications are still too much you need to have a look how you can suspend redrawing your control while you update the UI with 160 data points and resume drawing afterwards so you do not get heavy flickering.

List<Reading> _Readings = new List<Reading>();
DateTime _LastUpdateTime = DateTime.Now;
TimeSpan _UpdateInterval = new TimeSpan(0,0,0,0,1*1000); // Update every 1 second

private void SortOutputHandler(object sendingProcess, DataReceivedEventArgs outLine)
{
    if (!String.IsNullOrEmpty(outLine.Data))
    {
            var xGroup = Regex.Match(outLine.Data, "x: ?([-0-9]*)").Groups[1];
            int x = int.Parse(xGroup.Value);

            var yGroup = Regex.Match(outLine.Data, "y: ?([-0-9]*)").Groups[1];
            int y = int.Parse(yGroup.Value);

            var zGroup = Regex.Match(outLine.Data, "z: ?([-0-9]*)").Groups[1];
            int z = int.Parse(zGroup.Value);


            Reading reading = new Reading()
            {
                Time = _watch.Elapsed.TotalMilliseconds,
                X = x,
                Y = y,
                Z = z
            };

            // create a batch of readings until it is time to send it to the UI
            // via ONE window message and not hundreds per second. 
            _Readings.Add(reading);

            DateTime current = DateTime.Now;
            if( current -_LastUpdateTime > _UpdateInterval )  // update ui every second 
            {
                 _LastUpdateTime  = current;
                 List<Reading> copy = _Readings;  // Get current buffer and make it invisible to other threads by creating a new list. 
                // Since this is the only thread that does write to it this is a safe operation.


                 _Readings = new List<Reading>(); // publish a new empty list 

                 Dispatcher.Invoke(new Action(() =>
                 {
                    // This is called as part of a Window message in the main UI thread
                    // once per second now and not every 6 ms. Now we can upate the ui
                    // with a batch of 160 points at once. 
                    // A further optimization would be to disable drawing events 
                    // while we add the points to the control and enable it after
                    // the loop
                    foreach(Reading reading in copy)
                    {
                        _readings.Enqueue(reading);
                       _dataPointsCount++;
                    }

                 }),
                 System.Windows.Threading.DispatcherPriority.Normal);                    
            }
    }
}

Reading output from console application and WPF plotting async

4 Answers4