Stack Overflow
Stack Overflow

Questions tagged [hdf]

Hierarchical Data Format (HDF, HDF4, or HDF5) is a set of file formats and libraries designed to store and organize large amounts of numerical data.

Hierarchical Data Format (HDF, HDF4, or HDF5) is a set of file formats and libraries designed to store and organize large amounts of numerical data.

Originally developed at the National Center for Supercomputing Applications, it is supported by the non-profit HDF Group, whose mission is to ensure continued development of HDF5 technologies, and the continued accessibility of data stored in HDF.

In keeping with this goal, the HDF format, libraries and associated tools are available under a liberal, BSD-like license for general use. HDF is supported by many commercial and non-commercial software platforms, including Java, MATLAB/Scilab, Octave, IDL, Python, and R. The freely available HDF distribution consists of the library, command-line utilities, test suite source, Java interface, and the Java-based HDF Viewer (HDFView).

There are two major versions of HDF; HDF4 and HDF5, which differ significantly in design and API.

Wikipedia: http://en.wikipedia.org/wiki/Hierarchical_Data_Format

344 questions
2
votes
2 answers

How to merge two or three 3D arrays in python?

I have time series data in hdf format. I use the code below to read the data from the hdf files. Now I tried to join data on the basis of latitude and longitude for those data having same jdn (julian day number). Data with same julian day number…
bikuser
  • 2,013
  • 4
  • 33
  • 57
2
votes
1 answer

Is auto-chunking enabled by default when GZIP compression is used in the HDF5 C++ api?

I am writing an HDF5 file using the C++ HDF api and performing a few comparisons against the H5py Python library. In the H5py Python library autochunking is applied by default when a compression algorithm such as GZIP or LZF is used. Does the same…
zd5151
  • 395
  • 1
  • 3
  • 15
2
votes
1 answer

Empty pandas DataFrames saved with HDFStore make very large HDF files

Storing empty DataFrames using pandas HDFStore consumes a lot of disk space. Here's an example: import pandas as pd for i in range(1000): with pd.HDFStore('/tmp/test_empty.hdf') as s: key = 'E{:03d}'.format(i) s[key] =…
user1281657
  • 31
  • 3
2
votes
2 answers

How do I write a list/array of custom types into HDF5 file?

I need to write some data into an HDF5 file using C#. I am not tied to any specific library but the only accessible library I could find is HDF5DotNet (http://hdf5.net/). I have a simple class containing some data (just using default value…
SCBuergel
  • 1,613
  • 16
  • 26
2
votes
1 answer

Opening a corrupted PyTables HDF5 file

I am hoping for some help in opening a corrupted HDF5 file. I am accessing PyTables via Pandas, but a pd.read_hdf() call produces the following error. I don't know anything about the inner workings of PyTables. I believe the error was created…
Wapiti
  • 1,851
  • 2
  • 20
  • 40
2
votes
3 answers

Fields not found when using pyhdf

I am currently working with HDF files (version 4), and I use the pyhdf module (http://hdfeos.org/software/pyhdf.php). When I open one of my HDF files in MATLAB using the nctoolbox, I get the following variables: >> a =…
Holt
  • 36,600
  • 7
  • 92
  • 139
2
votes
0 answers

Using HDF5DotNet in C#, how do I retrieve and access cell arrays from a .mat file

Using HDF5DotNet in C#, I'd like to read in a 900x1 Cell array, then access 2D double arrays inside of each cell. I found how to map an H5Array to an array in C# (See below), but I can't figure out how to map an H5Array if the array is made of…
CodyF
  • 4,977
  • 3
  • 26
  • 38
2
votes
0 answers

read HDF file with pandas containing special characters in the index

How can I store data with spacial characters in the index (e.g: äüö) into a HDF file. Create a DataFrame: import pandas as pd import numpy as np df = pd.DataFrame(np.random.randn(8, 4), columns=['A','B','C','D'], …
omun
  • 113
  • 1
  • 7
2
votes
2 answers

Delete dataset attribute from HDF5 file

I have a hdf5 file I want to modify by deleting an attribute of one of the datasets and save the file without further changes. I can do this in hdfview, but I need something that's scriptable because it needs to be applied to a large number of…
ventsyv
  • 3,316
  • 3
  • 27
  • 49
2
votes
0 answers

Pandas - Optimal persistence strategy for highest compression ratio?

Question Given a large series of DataFrames with a small variety of dtypes, what is the optimal design for Pandas DataFrame persistence/serialization if I care about compression ratio first, decompression speed second, and initial compression speed…
bazel
  • 299
  • 7
  • 20
2
votes
2 answers

How to read .hdf file in R?

I have a large number of files in .hdf format. These are sadly not the hdf5 files that I know are readable in R. Is there a way to load and read hdf files in R? Or is there a way to convert .hdf to hdf5? I downloaded the C-based h4toh5 tool, but it…
Animesh
  • 43
  • 1
  • 6
2
votes
1 answer

Read from dataset in HDF5 file with datatype detection

I currently try to read some data from a hdf5 dataset in C which looks like this. dataset = H5Dopen(ic_group, 'vx', H5P_DEFAULT); status = H5Dread(dataset, H5T_NATIVE_FLOAT, memspace,H5S_ALL, H5P_DEFAULT, vx_ptr); status =…
jrsm
  • 1,595
  • 2
  • 18
  • 39
2
votes
1 answer

How do you concatenate all the HDF5 files in a given directory?

I have many HDF5 files in a directory and I want to concatenate all of them. I tried the following: from glob import iglob import shutil import os PATH = r'C:\Dropbox\data_files' destination = open('data.h5','wb') for filename in…
okarin
  • 239
  • 2
  • 11
1
vote
2 answers

HDF5 files and plotting using chunks

I'm new to HDF5 files and I don't understand how to access chunks in a dataset. I have quite a big dataset (1536, 2048, 11, 18, 2) which is chunked into (768, 1024, 1,1,1), each chunk represents half of an image. I want to plot the dataset, giving…
Amhm
  • 19
  • 5
1
vote
0 answers

How can I install HDF4 on Mac and use it for GDAL?

I would like to convert a series of HDF4-files via GDAL, but it needs the possibility to read this format (HDF4). This can probably be done by installing the package hdf4. The easiest way possible would be: brew install hdf4 But that results in No…
LiWa
  • 51
  • 2
1 2 3
…
22 23