D3 plotter and file watcher, interference?

Question

I am using file watcher monitoring the file output, and then dynamically display the results using d3 (dynamic data display). Here is more details:

Data are series of tiff file generated one by one quickly (20-25 miliseconds per file generation);
Once there is a file coming in, event is fired, and the file is processed (some stats calculation);
The results will be sent to a d3 plotter, which will dynamically display the results on a charter.

When I close the d3 charter window, the file watcher catcher pretty much all the events, but here is my problem:

However, when I leave the d3 charter window open, even when there is not any stats calculation, the file watcher drop many events (some big gap occurs).

Here is what we have tried:

Increase the watcher buffer, but, it still drop events as long as the d3 charter window is open.
Using C++ dll to calculate stats, but it looks even more slow.

So I am wondering:

Do d3 plotter and file watcher conflict/interfere with each other?
Can I possibly use them together to to real time plot?
Is there anyway to get around my problem?

Any pointers are appreciated.

Here is the code for d3 plotter, including constructor, plotter initiator, and plotter update:

#region Constructor
public SignalStatsDisplay()
{
    InitializeComponent();

    //   timeDomainPlotter.Legend.Remove();

    _initialChildrenCount = timeDomainPlotter.Children.Count;

    int count = timeDomainPlotter.Children.Count;

    //do not remove the initial children
    if (count > _initialChildrenCount)
    {
        for (int i = count - 1; i >= _initialChildrenCount; i--)
        {
            timeDomainPlotter.Children.RemoveAt(i);
        }
    }

    _nMaxStatsOneChannel = Enum.GetNames(typeof(Window1.ROISignalList)).Length;
    _curveBrush = new Brush[_nMaxStatsOneChannel];
    _statsEnable = new int[_nMaxStatsOneChannel];

    for (int i = 0; i < _nMaxStatsOneChannel; i++)
    {
        _curveBrush[i] = new SolidColorBrush((Color)ColorConverter.ConvertFromString(_colorList[i]));
        _statsEnable[i] = 0;
    }

    _nActiveStatsOneChannel = 0;
}
#endregion Constructor

public void InitiateSignalAnalysisPlot()
{
    _statsName = Enum.GetNames(typeof(Window1.ROISignalList));

    int count = 0;
    _statsEnableIndex = new int[_nActiveStatsOneChannel];
    for (int i = 0; i < _nMaxStatsOneChannel; i++)  // assign color
    {
        if (_statsEnable[i] == 1)
        {
            _statsEnableIndex[count] = i;
            count++;
        }
    }



    if (_nActiveChannel > 0)    // timeDomainPlotter init
    {
        _dataX = new List<double[]>();
        _dataY = new List<double[]>();

        double[] dataXOneCh = new double[_signalLength];
        double[] dataYOneCh = new double[_signalLength];

        dataXOneCh[0] = 0;
        dataYOneCh[0] = 0;

        for (int i = 0; i < _nActiveChannel; i++)
        {
            for (int j = 0; j < _nActiveStatsOneChannel; j++)
            {
                _dataX.Add(dataXOneCh);    // data x-y mapping init
                _dataY.Add(dataYOneCh);

                EnumerableDataSource<double> xOneCh = new EnumerableDataSource<double>(dataXOneCh);
                EnumerableDataSource<double> yOneCh = new EnumerableDataSource<double>(dataYOneCh);

                xOneCh.SetXMapping(xVal => xVal);
                yOneCh.SetXMapping(yVal => yVal);

                CompositeDataSource dsOneCh = new CompositeDataSource(xOneCh, yOneCh);

                LineAndMarker<MarkerPointsGraph> lam = timeDomainPlotter.AddLineGraph(dsOneCh,
                    new Pen(_curveBrush[_statsEnableIndex[j]], 2),
                    new CirclePointMarker { Size = 5, Fill = _curveBrush[_statsEnableIndex[j]] },
                    new PenDescription(_statsName[_statsEnableIndex[j]]));

            }
        }

        Action FitToView = delegate()
        {
            timeDomainPlotter.FitToView();
        };
        this.Dispatcher.Invoke(System.Windows.Threading.DispatcherPriority.Normal, FitToView);
    }
    else
    {
        return;
    }
}

public void RedrawSignalAnalysisPlot()
{
    int startIndex = _initialChildrenCount;

    if ((_nActiveStatsOneChannel > 0) && (_dataX != null) && (_dataY != null))
    {
        CompositeDataSource[] dsCh = new CompositeDataSource[_nActiveStatsOneChannel];
        int m, n;
        int index;

        for (int i = 0; i < _nActiveChannel; i++)
        {
            for (int j = 0; j < _nActiveStatsOneChannel; j++)
            {
                index = i * _nActiveStatsOneChannel + j;
                if (_dataX[index].Length == _dataY[index].Length)
                {
                    EnumerableDataSource<double> xOneCh = new EnumerableDataSource<double>(_dataX[index]);
                    xOneCh.SetXMapping(xVal => xVal);
                    EnumerableDataSource<double> yOneCh = new EnumerableDataSource<double>(_dataY[index]);
                    yOneCh.SetYMapping(yVal => yVal);
                    CompositeDataSource ds = new CompositeDataSource(xOneCh, yOneCh);

                    Action UpdateData = delegate()
                    {
                        m = i * 2;
                        n = j * 2;

                        // ((LineGraph)timeDomainPlotter.Children.ElementAt(startIndex + n + m * _nActiveStatsOneChannel)).DataSource = ds;
                        // ((LineGraph)timeDomainPlotter.Children.ElementAt(startIndex + n + m * _nActiveStatsOneChannel)).LinePen
                        //    = new Pen(new SolidColorBrush(_curveBrush[j]), 1);

                        ((MarkerPointsGraph)timeDomainPlotter.Children.ElementAt(startIndex + n + 1 + m * _nActiveStatsOneChannel)).DataSource = ds;
                        //  ((MarkerPointsGraph)timeDomainPlotter.Children.ElementAt(startIndex + n + 1 + m * _nActiveStatsOneChannel)).Marker
                        //      = new CirclePointMarker { Size = 5, Fill = Brushes.Green };
                    };

                    this.Dispatcher.Invoke(System.Windows.Threading.DispatcherPriority.Normal, UpdateData);
                }

            }


        }

        /* Action PlotFitToView = delegate()
          {
              timeDomainPlotter.FitToView();
          };

         this.Dispatcher.Invoke(System.Windows.Threading.DispatcherPriority.Normal, PlotFitToView);*/
    }
}

Here is how files are monitored: Where events (file written) are filed

void tiffWatcher_EventChanged(object sender, WatcherExEventArgs e)
{
    string fileName = ((FileSystemEventArgs)(e.Arguments)).FullPath;
    string fileExt = StringExtension.GetLast(fileName, 4);

    if (!IsFileLocked(fileName))
    {

        Action EventFinished = delegate()
        {
            CreateListViewItem(fileName, "Finished", DateTime.Now.ToString("HH:mm:ss.fff"));
        };

        listView1.Dispatcher.Invoke(System.Windows.Threading.DispatcherPriority.Normal, EventFinished);

        _tiffName.Add( fileName);
        _tiffFilledCount++;    
    }
}

tiff data processing, and data communication to d3 plotter:

void tiffTimer_Tick(object sender, EventArgs e)
{
    //throw new NotImplementedException();
    byte[] image = new byte[_signalManager.ImgWidth * _signalManager.ImgHeight];

    if (_tiffFilledCount - _tiffProcessedCount >= 1)
    {


        string fileName = _tiffName[_tiffProcessedCount++];
        char filePre = fileName[49];
        int indexBeigin = fileName.LastIndexOf("_");
        int indexEnd = fileName.LastIndexOf(".");

        _signalIndex = Convert.ToInt32(fileName.Substring(indexBeigin + 1, indexEnd - indexBeigin - 1)) - 1; // 0 based//
        _deltaT = ExtractTiffDeltaT(fileName, "DeltaT=", 1);
        _channelIndex = (int)Enum.Parse(typeof(ChannelList), Convert.ToString(filePre));
        TIFFIImageIO.LoadTIFF(fileName, ref image);
        _signalManager.Image = image;
        for (int i = 0; i < _nActiveStatsOneChannel; i++)
        {
            _signalManager.GetSignal( _signalDisplay.StatsEnableIndex[i], ref _signal);
            UpdateSignal(_channelIndex, i, _tiffProcessedCount-1, _deltaT, _signal);
        }

      // if( _tiffProcessedCount % 5 == 0)
        _signalDisplay.SetData(_XList, _YList, true);
    }
}

I use a Timer.Tick to process the file every 50-100 miliseconds, still testing.

wow that's a lot of text, and Im too lazy to read. Either post some code or you'll get downvotes quickly. Also, why is this tagged WPF? — Federico Berasategui, Mar 05 '13 at 21:24
I am just asking a general concept. I know that there text, but I really have no idea how to use code to describe the problem. — Nick Tsui, Mar 05 '13 at 21:26
then at least try to use some punctuation (bullets or something) — Federico Berasategui, Mar 05 '13 at 21:27
Does it look better? I hope people can read this with their least effort, so that they can easily share some opinions if they have similar experience. — Nick Tsui, Mar 05 '13 at 21:36
Ok I still insist. You will have to post some code. Post the code used to generate the plot and the code you're using to monitor the filesystem. Otherwise I don't think anyone can help you. — Federico Berasategui, Mar 05 '13 at 21:43

score 0 · Accepted Answer · answered Mar 07 '13 at 15:43

The way I handled it was to setup a Timer, cyclically handling calculation, display. So far, it works fine. Of course there will be many events are missing, but we handle large amount of points (thousands of them), so it is OK to hundreds to do the plot. Here is the code:

private Timer _tiffTimer;
void Window1_Loaded(object sender, RoutedEventArgs e)
        {
            //throw new NotImplementedException();
            _tiffTimer = new Timer();
            _tiffTimer.Interval = 50;  // change interval to change performance
            _tiffTimer.Tick += new EventHandler(tiffTimer_Tick);
            _tiffTimer.Start();

        }

void tiffTimer_Tick(object sender, EventArgs e)
        {
            //do your stuff here

        }

D3 plotter and file watcher, interference?

1 Answers1