Cannot open matlab files using the latest HDF5

Question

I recently upgrade tables on my python installation and some strange things seem to be happening with the HDF5 libraries.

I've got a bunch of data that was originally saved as a .mat file, which uses the HDF5 format. I've been reading this into python using pyTables, and this has worked fine up until now.

I upgrade pyTables to version 3.0, and it seems like this must have upgraded my HDF5 installation as well (or something like this). Now, any time I try to read the .mat file into memory, I get the following error:

Warning! ***HDF5 library version mismatched error***
The HDF5 header files used to compile this application do not match
the version used by the HDF5 library to which this application is linked.
Data corruption or segmentation faults may occur if the application continues.
This can happen when an application was compiled by one version of HDF5 but
linked with a different version of static or shared HDF5 library.
You should recompile the application or check your shared library related
settings such as 'LD_LIBRARY_PATH'.
You can, at your own risk, disable this warning by setting the environment
variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'.
Setting it to 2 or higher will suppress the warning messages totally.
Headers are 1.8.5, library is 1.8.9
        SUMMARY OF THE HDF5 CONFIGURATION
        =================================

General Information:
-------------------
           HDF5 Version: 1.8.9
          Configured on: Tue Sep 11 15:02:44 CDT 2012
          Configured by: ilan@centos5x86
         Configure mode: production
            Host system: x86_64-unknown-linux-gnu
          Uname information: Linux centos5x86 2.6.18-308.el5 #1 SMP Tue Feb 21 20:06:06 EST 2012 x86_64 x86_64 x86_64 GNU/Linux
               Byte sex: little-endian
              Libraries:
         Installation point: /opt/anaconda1anaconda2anaconda3

Compiling Options:
------------------
               Compilation Mode: production
                     C Compiler: /usr/bin/gcc ( gcc (GCC) 4.1.2 20080704 )
                         CFLAGS:
                      H5_CFLAGS: -std=c99 -pedantic -Wall -Wextra -Wundef -Wshadow -Wpointer-arith -Wbad-function-cast -Wcast-qual -Wcast-align -Wwrite-strings -Wconversion -Waggregate-return -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Wredundant-decls -Wnested-externs -Winline -Wno-long-long -Wfloat-equal -Wmissing-format-attribute -Wmissing-noreturn -Wpacked -Wdisabled-optimization -Wformat=2 -Wunreachable-code -Wendif-labels -Wdeclaration-after-statement -Wold-style-definition -Winvalid-pch -Wvariadic-macros -Wnonnull -Winit-self -Wmissing-include-dirs -Wswitch-default -Wswitch-enum -Wunused-macros -Wunsafe-loop-optimizations -Wc++-compat -Wvolatile-register-var -O3 -fomit-frame-pointer -finline-functions
                      AM_CFLAGS:
                       CPPFLAGS:
                    H5_CPPFLAGS: -D_POSIX_C_SOURCE=199506L   -DNDEBUG -UH5_DEBUG_API
                    AM_CPPFLAGS: -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_BSD_SOURCE
               Shared C Library: yes
               Static C Library: no
  Statically Linked Executables: no
                        LDFLAGS:
                     H5_LDFLAGS:
                     AM_LDFLAGS:
        Extra libraries:  -lz -lrt -lm
               Archiver: ar
             Ranlib: ranlib
          Debugged Packages:
            API Tracing: no

Languages:
----------
                        Fortran: no

                            C++: no

Features:
---------
                  Parallel HDF5: no
             High Level library: yes
                   Threadsafety: no
            Default API Mapping: v18
 With Deprecated Public Symbols: yes
         I/O filters (external): deflate(zlib)
         I/O filters (internal): shuffle,fletcher32,nbit,scaleoffset
                            MPE: no
                     Direct VFD: no
                        dmalloc: no
Clear file buffers before write: yes
           Using memory checker: no
         Function Stack Tracing: no
                           GPFS: no
      Strict File Format Checks: no
   Optimization Instrumentation: no
       Large File Support (LFS): yes
Bye...
Aborted

It then breaks python and dumps me back to the shell.

I've tried downgrading my pyTables to the pre 3.0 version, but this doesn't help. I've also tried re-opening in matlab and saving again, but this still doesn't work.

Any ideas what is going wrong? It sounds like Matlab is saving to a different HDF5 version than the one pyTables is assuming. It seems like I should either update Matlab's HDF5 library, or downgrade python's, but I can't figure out how to do this...

Answer 1

OK, I figured out what the problem was. Basically, it was a version conflict. (why is it always a version conflict?)

Apparently pip and anaconda have two different naming conventions for PyTables. Pip calls it "tables", and Anaconda calls it "pytables".

So, when I ran conda install tables, anaconda looked for tables, didn't find it, and defaulted to using pip. Pip then found tables and installed it.

The problem is that PyTables was already installed in the anaconda distribution, it's just that it was called "pytables".

So now I had two different versions of tables on the same distribution, one installed via pip and one installed via anaconda. This was doing all kinds of wonky things and resulted in the above error.

So, lesson learned: never assume that giant package repositories make sure that they call things the same thing. :P