I'm regularly writing a 5 GB NumPy ndarray to disc using H5py. If I close the file during this write, it gets corrupted. Is there any way for me to tell the program something along the lines of, "if somebody ends your program, wait until you finish the file first"? Alternatively, can I revert to the previous file in this case?
Asked
Active
Viewed 44 times
0
-
3Store in temporary file, rename it after success? – Stefan Pochmann Feb 07 '16 at 01:30
-
I guess I could do that if I wanted to use 10 GB instead of 5 GB... at 16 GB of RAM would that cause any problems? – Eric Gorlin Feb 07 '16 at 02:22
-
I don't see why RAM matters, or why it would be a problem with the temp file if it isn't one with the real file. (Then again, I know nothing about H5py.) – Stefan Pochmann Feb 07 '16 at 03:31
-
@StefanPochmann, took your suggestion! Thanks. – Eric Gorlin Feb 07 '16 at 04:42
1 Answers
0
try:
#do stuff
finally:
#save stuff
Might allow you to capture what has already been processed before app exits?
-
I think I'm already accounting for what you're talking about- I have a progress file, and only update the progress file once I've saved. The problem is that the save process of the main file takes 50 seconds or so, and if I exit during the save process itself the file can't be loaded again. (and will be smaller than 5 GB size) – Eric Gorlin Feb 07 '16 at 02:20
-
when you say: "the file can't be loaded again", are you referring to the progress file? And when you update the progress file, is it an accumulation of processed data or does it just contain one "data dump" from a single application run? – Logic1 Feb 07 '16 at 02:50