Stack Overflow
Stack Overflow

Questions tagged [hdf]

Hierarchical Data Format (HDF, HDF4, or HDF5) is a set of file formats and libraries designed to store and organize large amounts of numerical data.

Hierarchical Data Format (HDF, HDF4, or HDF5) is a set of file formats and libraries designed to store and organize large amounts of numerical data.

Originally developed at the National Center for Supercomputing Applications, it is supported by the non-profit HDF Group, whose mission is to ensure continued development of HDF5 technologies, and the continued accessibility of data stored in HDF.

In keeping with this goal, the HDF format, libraries and associated tools are available under a liberal, BSD-like license for general use. HDF is supported by many commercial and non-commercial software platforms, including Java, MATLAB/Scilab, Octave, IDL, Python, and R. The freely available HDF distribution consists of the library, command-line utilities, test suite source, Java interface, and the Java-based HDF Viewer (HDFView).

There are two major versions of HDF; HDF4 and HDF5, which differ significantly in design and API.

Wikipedia: http://en.wikipedia.org/wiki/Hierarchical_Data_Format

344 questions
1
vote
1 answer

Problem reading HDF file with Pandas Python

I have a link to a NASA dataset, formatted as .hdf (download below) https://reason.gesdisc.eosdis.nasa.gov/data/Vegetation_Indices/MODVI.005/2001/MODVI.200101.005.hdf After downloading this locally, I tried the following code import pandas as…
Stephen
  • 89
  • 2
  • 6
1
vote
1 answer

How to filter orbit paths by the ones that intersect a specific country?

I have a ton of files of orbit paths which each contain a list of 63,000 latitude and longitude coordinates which make up an orbit path like this. The files are HDF files and contain two lists, one of 63k latitude points and one of 63k longitude…
iraz
  • 11
  • 2
1
vote
2 answers

Read Variable Length String From HDF5 Dataset Using C API

I'm attempting to read a variable length string from an HDF5 dataset using the C API. The original C++ code that worked uses v1.8.15.1 of the HDF5 library. I decided to drop to C for debugging, as I have more control. Unfortunately, I'm in a very…
ajr
  • 11
  • 4
1
vote
1 answer

Nifi Content Vs Attribute Modification Techniques

In Nifi we can design a flow in two ways : Content Based Modification (UpdateContent) - In this approach we are directly modifying the content of flowfiles . With this at each stage , the flowfile content will get persisted in flow file…
Aviral Kumar
  • 814
  • 1
  • 15
  • 40
1
vote
0 answers

UnicodeDecodeError when trying to read an hdf file made with python 2.7

I have a bunch of hdf files that I need to read in with pandas pd.read_hdf() but they have been saved in a python 2.7 environment. Nowadays, I'm on python 3.7, and when trying to read them with data = pd.read_hdf('data.h5', 'data'), I'm…
JC_CL
  • 2,346
  • 6
  • 23
  • 36
1
vote
1 answer

Space efficient data store for list of list of lists. Elements are integers, and size of all lists varies in length

Say my data looks like this thisList = [ [[13, 43, 21, 4], [33, 2, 111, 33332, 23, 43, 2, 2], [232, 2], [23, 11]] , [[21, 2233, 2], [2, 3, 2,1, 32, 22], [3]], [[3]], [[23, 12], [55, 3]], .... ] What is the most…
SantoshGupta7
  • 5,607
  • 14
  • 58
  • 116
1
vote
2 answers

How to convert gridded satellite swath data in hdf to netcdf4 which is available from the NASA GPM satellite?

I am trying to access the gridded satellite swath data which is available in hdf from https://disc.gsfc.nasa.gov/datasets?keywords=LATENT%20HEATING&page=1 You can see any level 3 data which is what I am trying to use. I am familiar with netcdf…
tech_climate
  • 153
  • 6
1
vote
1 answer

Fast iterative writing to hdf5 file

Some time ago I wrote a parallel FORTRAN code that diagonalises very large dense matrices on a supercomputer. These matrices are read from dense chunked hdf5 datasets. Now I want to use this code for very sparse matrices that are constructed using…
Baranas
  • 71
  • 1
  • 6
1
vote
1 answer

HDFql Adding Data to Existing Dataset

I want to create a HDF5 file in C++ and dynamically add data to a dataset contained in that file. I have been having some trouble getting this work however, with this minimal example as my current best attempt (based mainly on an answer by…
Mr Squid
  • 1,196
  • 16
  • 34
1
vote
1 answer

Using HDF data in r

I am trying to download and plot the modis data from here http://orca.science.oregonstate.edu/1080.by.2160.monthly.hdf.vgpm.s.chl.a.sst.php I regularly use .nc files from the ocean colour website but I can't seem to work with the hdf files the same…
liv
  • 93
  • 10
1
vote
1 answer

How to store data from multiple files to a single variable

I have 22 CALIPSO data files in my folder. I have accessed the parameters required for 1 files and stacked it. Now I want to do it for all other 22 files and store it in a single variable called outData. I created and empty array outData[]. I want…
Sreerag Ek
  • 13
  • 3
1
vote
1 answer

Can you store and query compound data of Matlab arrays and structures into a database?

How do I store Matlab arrays located in a 'struct within struct within struct' into a database so that I can then retrieve the fields and arrays? More detail on why do I need this below: I have tons of data saved as .mat files....the hassle is that…
Ahmad Khaled
  • 387
  • 3
  • 17
1
vote
1 answer

pandas: write empty DataFrame to HDF file

Is there a way to force pandas to write an empty DataFrame to an HDF file? import pandas as pd df = pd.DataFrame(columns=['x','y']) df.to_hdf('temp.h5', 'xxx') df2 = pd.read_hdf('temp.h5', 'xxx') Output: Traceback (most recent call last): File…
S.V
  • 2,149
  • 2
  • 18
  • 41
1
vote
1 answer

Why is Pandas df.to_hdf("a_file", "a_key") output increases in size when executed multiple times

Pandas has a method .to_hdf() to save a dataframe as a HDF table. However each time the command .to_hdf(path, key) is run, the size of the file increases. import os import string import pandas as pd import numpy as np size = 10**4 df =…
Adrien Pacifico
  • 1,649
  • 1
  • 15
  • 33
1
vote
0 answers

How to store resultset into hdf5 file?

i had created a java project which is getting a resultset from mysql and i have to store my resultset into a hdf5 file, so how to do it properly? Right now i am getting the data from sql in resultset and than storing it into List of Map and than…
user3458271
  • 638
  • 12
  • 31
1 2 3
…
22 23