1

I have a question about file output in Python. I was designing a software that reads values from 3 sensors. Each sensors read 100 values for 1 second, and between each process, I have to print them in file. 

time_memory = [k + i/100 for i in range(100)] # dividing 1 second into 100 intervals
x = [100 elements]
y = [100 elements]
z = [100 elements]

Below is the code of writing into the file. 

for i in range(self.samples):
    self.time_memory[i] = file_time + self.time_index[i] 
    f.write("{0} {1} {2} {3}\n".format(self.time_memory[i], x[i], y[i], z[i]))

So the result in the file will look like

time_value, x, y, z
time_value, x, y, z
...

However, when the measurement time is over 8000 seconds, the software stops. I think it is due to so many data the device must proceed since the device I am using is kind of an old one. (I cannot change the device because the computer is connected to NI DAQ device.)

I tried to find many alternative ways to change the code above, but I couldn't find it. Is there anyone who can help me with this problem??

Ilja Everilä
  • 50,538
  • 7
  • 126
  • 127
jdub Kim
  • 23
  • 5
  • 1
    I'd wager it is not your file handling that causes the possible out of memory situation (please include the full traceback when asking for debugging help), but how you're storing the measurements before writing. – Ilja Everilä Sep 14 '16 at 08:18
  • Not sure If I understand it correctly but, instead of writing every second, can't you write a batch of several seconds (10-30s) ? – Floran Gmehlin Sep 14 '16 at 08:27
  • _when the measurement time is over 8000 seconds, the software stops_ 2 questions: (a) How does your python script fill `self.samples`? (b) What happens when the software _stops_? Do you see an exception, a silent hang, something else? Please update your question with this information. – Joe Friedrichsen Sep 14 '16 at 19:02
  • Actually, I am using the DAQ Device from National Instruments. Is model is NI-6281, and I will upload the error message on the question. Thank you for lots of helps – jdub Kim Sep 16 '16 at 01:28
  • To Floran Gmehlin, I think it will not make any difference since I have to use for loops to make that format (time_value, x, y, z \n for several times). Is there any other way to write in that format(or similar one) without for loops? Again, thank you for your helps guys – jdub Kim Sep 16 '16 at 01:29

1 Answers1

-1

One suggestion would be to write the data in binary mode. This should be faster than the text mode (it also needs less space). Therefore you have to open a file in binary mode like this:

f = open('filename.data', 'wb')
Maxim
  • 21
  • 6
  • Depending on python version, the formatted string might already be binary. – Ilja Everilä Sep 14 '16 at 08:15
  • Since we don't know whether the bottleneck is the CPU or the memory, it might be faster to use binary mode explicitly. Also one could try to avoid the "".format() method of the string and write the four values one after each other with four successive f.write() calls. – Maxim Sep 14 '16 at 08:26