1

I am writing a serial data logger in Python and am wondering which data type would be best suited for this. Every few milliseconds a new value is read from the serial interface and is saved into my variable along with the current time. I don't know how long the logger is going to run, so I can't preallocate for a known size.

Intuitively I would use an numpy array for this, but appending / concatenating elements creates a new array each time from what I've read.

So what would be the appropriate data type to use for this?

Also, what would be the proper vocabulary to describe this problem?

LimaKilo
  • 196
  • 9
  • This is both a software and a hardware question. If you're using mainstream data acquisition hardware, many of those vendors publish libraries specific to their equipment. The best solution may depend on the capabilities of such a library. Are you looking for a more generic answer? – mayosten Oct 04 '19 at 13:45
  • Yes, I'm looking for a generic answer, the data is acquired from an Arduino that is just reading ADC values and sending them via serial. – LimaKilo Oct 04 '19 at 13:47
  • 1
    It sounds like you want to keep everything in memory and not write to a file, but you might encounter a limit: _"Every few milliseconds a new value is read"_ mixed with _"I don't know how long the logger is going to run"_ sounds like an efficient path to frustrating crashes. – Joe Friedrichsen Oct 04 '19 at 18:57
  • @JoeFriedrichsen good point, but in my case it's only running for an hour or so, where memory is not an issue at roughly 20 kB/s (and I'm saving to disk every few seconds). But I suppose then I could also preallocate for something like 500 MB, which I know I won't fill completely. – LimaKilo Oct 04 '19 at 22:42

1 Answers1

0

Python doesn't have arrays as you think of them in most languages. It has "lists", which use the standard array syntax myList[0] but unlike arrays, lists can change size as needed. using myList.append(newItem) you can add more data to the list without any trouble on your part.

Since you asked for proper vocabulary in a useful concept to you would be "linked lists" which is a way of implementing array like things with varying lengths in other languages.

conner wallace
  • 196
  • 4