Locking of HDF files using h5py

Question

I have a whole bunch of code interacting with hdf files through h5py. The code has been working for years. Recently, with a change in python environments, I am receiving this new error message.

IOError: Unable to open file (unable to lock file, errno = 11, error message = 'Resource temporarily unavailable')

What is interesting is the error occurs intermittently in some places and persistently in others. In places where it is occuring routinely, I have looked at my code and confirm that there is no other h5py instance connected to the file and that the last connection was properly flushed and closed. Again this was all working fine prior to the environment change.

Heres snippets from my conda environment:

h5py 2.8.0 py27h470a237_0 conda-forge hdf4 4.2.13 0 conda-forge hdf5 1.10.1 2 conda-forge

Can you construct a minimal example that also shows this error? — Tom de Geus, Jun 25 '18 at 11:12
Possible sources of the problem: disk full, shared filesystem, permissions problems. As @TomdeGeus mentions, a minimal example would help! — Pierre de Buyl, Jun 26 '18 at 13:42

score 2 · Answer 1 · edited Jan 28 '23 at 17:34

2

In terms of my version of this issue, it failed to close the file in an obscure method. Interesting thing is that unlocking the file in some cases just took a restart of ipython, other times took a full reboot.

edited Jan 28 '23 at 17:34

Nike

1,223
2
19
42

answered Jun 27 '18 at 20:49

user2611761

169
1
1
11

score 1 · Answer 2 · answered Aug 08 '19 at 12:33

1

with h5py.File(), the same .h5 file can be open for read ("r") multiple times. But h5py doesn't support more than a single thread. You can experience bad data with multiple concurrent readers.

answered Aug 08 '19 at 12:33

Machuck

11
1

score 1 · Answer 3 · answered Jan 12 '21 at 16:22

1

I had other process running that I did not realize. How did I solved my problem:

Used ps aux | grep myapp.py to find the process number that was running myapp.py.
Kill the process usill the kill command
Run again

answered Jan 12 '21 at 16:22

Matheus Araujo

5,551
2
22
23

score 0 · Answer 4 · answered Mar 29 '22 at 11:20

0

Similar as with the other answers I had already opened the file, but for me it was in a separate HDF5 viewer.

answered Mar 29 '22 at 11:20

Florian Brucker

9,621
3
48
81

score 0 · Answer 5 · answered Apr 08 '22 at 02:37

For me, I used multiprocessing to parallelise my data processing, and the file handle is passed to the multiprocessing pool. As a result, even if I called close(), the file would not be closed until all the subprocesses spawned by the multiprocessing pool are terminated.

Remember to call join and close if you are using multiprocessing.

        pool = multiprocessing.Pool(processes=multiprocessing.cpu_count())
        task_iter = pool.imap(...) # <- file is used in pool!
        ...
        pool.close()
        pool.join()

Locking of HDF files using h5py

5 Answers5