1

I am working on a C# SW for Windows which control an acquisition device with 256 channels, 10kHz, 16 bit (5MB/s). The acquisition generally lasts between 1minute and 1hour; it is saved as raw data in a binary file whose resulting size may reach many GB.

After the acquisition, using the same SW, the user should be able to elaborate the data; e.g., he should be able to select two channels and add them together generating a new trace (saved to disk); to discard (remove) the final part of the acquisition in all the channels; to discard a whole bunch of channels...

In the old version of the SW, when the HW sampling frequency was slower and the number of channels limited to 16, the acquired data was saved in a file. A new file was created for each combination of existing traces to store the elaboration result. All the files were finally zipped together in a single archive, with additional xml files containing the acquisition parameters.

Now that the data size increased this approach is very slow and seems to me not efficient at all: after the acquisition a lot of time is spent compressing the files, and, every time an old acquisition if opened, it takes ages to unzip all the files in a temporary folder, modify them and re-zip everything.

My first idea was to switch to hdf5, but the fact that the file is not shrank when a dataset is deleted is a big limitation/complication in this case. (Delete or update a dataset in HDF5?)

Another (maybe simpler) idea I had is to store all files in a folder (using the old format and simplifying a lot the development) and to declare a "custom verb" so that double clicking the folder opens the SW. http://msdn.microsoft.com/en-us/library/cc144171%28v=VS.85%29.aspx#custom_verbs_desktop I have a feeling that this solution is fragile (also because I never saw it used in other applications...) but I couldn't find real problems... What are your suggestions? is the "custom verb" a good solution for use in production systems? do you see big drawbacks?

Community
  • 1
  • 1
Paolo
  • 76
  • 1
  • 4
  • Why not register a new file-type, a kind of "project file" that describes where the acquisition files (and other files) are situated? That's how visual studio does it. The project file could live in the same folder as the other files. It's always worth remembering that additional storage is considerably cheaper than developer time! – spender Jan 13 '15 at 18:37
  • Thank you @spender for your answer. The main reason it to avoid the "basic" user to mess up the acquired files; in particular we don't want the use to delete by accident some acquired files (while keeping others, thus having incomplete sets) and, more important, we would like to protect the system from situations where the user places in the same folder two acquisitions: that would be like placing in the same folder two Visual Studio projects with the same names. Anyway, using guids for the raw file names that could become an option :) Anyone tryed to use custom verbs in the way I described? – Paolo Jan 13 '15 at 23:30

0 Answers0