Stack Overflow
Stack Overflow

Questions tagged [hdf5]

The Hierarchical Data Format (HDF5) is a binary file format designed to store large amount of numerical data.

HDF5 refers to:

  • A binary file format designed to store efficiently large amount of numerical data
  • Libraries of function to create and manipulate these files

Main features

  • Free
  • Completely portable
  • Very mature
  • No limit on the number and size of the datasets
  • Flexible in the kind and structure of the data and meta-data
  • Complete library in C and Fortran well documented
  • A lot of wrappers and tools are available (Python, Matlab, Java, …)

Some links to get started

2598 questions
1
vote
1 answer

Recursively read an HDF5 file into R

I have a fairly deeply nested HDF5 file which I would like to read into R recursively into a list data structure. I can already read single nodes using the rhdf5 package. Is there a library that can do this or must I parse the tree myself?
Thomas Browne
  • 23,824
  • 32
  • 78
  • 121
1
vote
1 answer

How to use H5Sselect_elements to read data

I have a HDF file which contains a simple array of compound types. To read all elements in the array i do hid_t hDataSet = H5Dopen(hSpecies,AGENT_DATASET_NAME, H5P_DEFAULT); herr_t status = H5Dread(hDataSet, agent_type, H5S_ALL, H5S_ALL,…
user1479670
  • 1,145
  • 3
  • 10
  • 22
1
vote
2 answers

HDF5 .Net wrapper

I'm getting ( http://www.hdfgroup.org/projects/hdf.net/) The specified module could not be found. (Exception from HRESULT: 0x8007007E) from the dependency walker i'm seeing that SZLIBDLL.DLL is missing i tried to download it from random place…
UshaP
  • 1,271
  • 2
  • 18
  • 32
1
vote
0 answers

Error importing H5PY "undefined symbol: H5FD_mpio_init"

I've installed H5PY, but when trying to import the h5py module, I get this error: >>> import h5py Traceback (most recent call last): File "", line 1, in File…
Turms
  • 155
  • 1
  • 6
1
vote
1 answer

How can I turn on HDF5 Dense Attribute Storage in PyTables/Python?

Since 1.8.8 HDF5 offers dense storage of attributes for faster handling of HDF5 nodes with many or large attributes. By default the limit (H5Pset_attr_phase_change) to start dense storage is set to 8 attributes. Is there any way how can change this…
SmCaterpillar
  • 6,683
  • 7
  • 42
  • 70
1
vote
1 answer

Pandas HDF5 store unicode error on select query

I have unicode data as read from this file: Mdt,Doccompra,OrgC,Cen,NumP,Criadopor,Dtcriacao,Fornecedor,P,Fun 400,8751215432,2581,,1,MIGRAÇÃO,01.10.2004,75852214,,TD 400,5464282154,9874,,1,MIGRAÇÃO,01.10.2004,78995411,,FO I have two problems: When…
mvbentes
  • 1,022
  • 12
  • 24
1
vote
0 answers

HDF5 vs Panel or Panel4D in Pandas

In pandas, one can keep data in a Panel of multiple dataframes. That Panel can even be 4D or of higher dimensions. When saving data, one can save to HDF5 format, which allows an internal directory structure, with nested directories. Thus, it is…
jeffalstott
  • 2,643
  • 4
  • 28
  • 34
1
vote
1 answer

appending rows with pandas' to_hdf multiples H5 file size?

I have an HDF5 with about 13,000 rows × 5 columns, these rows were appended over time to the same file with DF.to_hdf(Filename, 'df', append=True, format='table') and here's the size: -rw-r--r-- 1 omnom omnom 807M Mar 10 15:55…
Om Nom
  • 169
  • 1
  • 2
  • 11
1
vote
1 answer

Read HDF5 based file as a numpy array in Python

How can I load in a .hws file as a numpy array? Based on the description in http://kingler.net/2007/05/22/90 which says it is a HDF5 based format, so I found https://confluence.slac.stanford.edu/display/PSDM/How+to+access+HDF5+data+from+Python might…
Francis
  • 6,416
  • 5
  • 24
  • 32
1
vote
0 answers

How to tell pytables the amount of RAM to be used (using it for paging)?

I am storing several numpy arrays in a pytables file. Each individual array (size ~1MB - 100MB) fits into RAM but not all (N ~10 - 1000) arrays together fit. In the application I operate repeatedly on these arrays, also changing their shapes etc. So…
Marc Morkinson
  • 11
  • 3
1
vote
1 answer

Error using HDF5 data for training models in caffe

I am working with caffe and I have been trying to train the caffenet model using HDF5 data I used the prototxt files from ~/../caffe/examples/hdf5_classification. But I get the following error I0219 19:06:43.038563 5773 caffe.cpp:103] Use CPU. I0219…
sohail siddique
  • 23
  • 5
1
vote
2 answers

find lat/long point in a hdf5

I have an HDF5 files, global coverage of temperature. The file was converted from netcdf. The conversion process set longitude from 0 to 360 and additionally flipped the map upside down, so north is now south. I have used HDFView and I can display…
Herman Toothrot
  • 1,463
  • 3
  • 23
  • 53
1
vote
0 answers

accessing dataset distributed over many hdf5 files

Is there a simple way to transparently access a data set distributed over several hdf5 files in python? Assume, I have two hdf files, h1 and h2. Both contain 1-dimensional datasets dd and cc, say the date in dd and temperature of this date in cc. I…
monos
  • 75
  • 1
  • 6
1
vote
1 answer

Python Pandas storing error: (HDF5ExtError: Problems creating the Array)

I'm having problems storing a large amount of data using HDF5 functionality in Pandas. What I'm trying to do is download a huge amount of data (Millions of rows) from a SQL sever and have it stored locally for easy and quick access. I defined some…
Jemme
  • 303
  • 2
  • 12
1
vote
2 answers

JHDF5 - How to avoid dataset being overwritten

I am using JHDF5 to log a collection of values to a hdf5 file. I am currently using two ArrayLists to do this, one with the values and one with the names of the values. ArrayList valueList = new ArrayList(); ArrayList…
HigiPha
  • 742
  • 9
  • 19
1 2 3
…
99 100