0

I have a logic analyser project that records several hundred million 16bit values (~100-500 million) and I need to display anything from a few hundred samples to the entire capture as the user zooms.

When you zoom out the whole system gets a huge performance hit as it's loading a massive chunk from the file.

I just though this morning that it would be more efficient to "stride" through the file at the users screen resolution. You can't physically display anything between pixels anyways. This doesn't solve the massive file size hit in memory though.

Is there away I can take a huge data set and stream chunk it down efficiently?

I was thinking streaming from start to start + view size by horiz resolution. This makes a very choppy zoom though.

Program uses python but I am open to calling something in c if it already exists.

uMinded
  • 595
  • 1
  • 9
  • 21

2 Answers2

0

Well, I don't know if this is actually question on programming or design overall.

For "zooming" problem with vizualizations I suggest:

  1. Have pre-computed/cached version for some zoom levels. Ideally, gradation should be calculated based on user behaviour.
  2. When user zooms-in, you simultaneously
    • calculate "proper" data or load pre-computed aggregated data of deeper zoom layer and crop it by your view frame
    • cheat by rendering low-res data from previous layer or smooth it by some approximation (but make sure to somehow tell user that data is not finalized)

Aside of it, think if you can optimize the way you store data. Trees may make your life way easier, both for partial disk read/search and for storing aggregated data.

Slam
  • 8,112
  • 1
  • 36
  • 44
0

In my opinion, there is no point to display even a few hundred samples unless they form some kind of image/shape. I guess one can look at hundred numbers if they are properly structured (colored). Several hundred - doubt it - here you replace actual data with some visualization (plots, charts, maps, ...).

To approach the problem you may define some rule to stop displaying actual data at all. For instance, if digit height becomes less than, say, 10 pixels you display some kind of message selected numbers are from rows 200...300, columns 400..500 or some graphical alterantive with corner coordinates and amount of numbers.