Stack Overflow
Stack Overflow

Questions tagged [hdf]

Hierarchical Data Format (HDF, HDF4, or HDF5) is a set of file formats and libraries designed to store and organize large amounts of numerical data.

Hierarchical Data Format (HDF, HDF4, or HDF5) is a set of file formats and libraries designed to store and organize large amounts of numerical data.

Originally developed at the National Center for Supercomputing Applications, it is supported by the non-profit HDF Group, whose mission is to ensure continued development of HDF5 technologies, and the continued accessibility of data stored in HDF.

In keeping with this goal, the HDF format, libraries and associated tools are available under a liberal, BSD-like license for general use. HDF is supported by many commercial and non-commercial software platforms, including Java, MATLAB/Scilab, Octave, IDL, Python, and R. The freely available HDF distribution consists of the library, command-line utilities, test suite source, Java interface, and the Java-based HDF Viewer (HDFView).

There are two major versions of HDF; HDF4 and HDF5, which differ significantly in design and API.

Wikipedia: http://en.wikipedia.org/wiki/Hierarchical_Data_Format

344 questions
1
vote
1 answer

How to save each file in a for loop using matlab

I have three files named as modis1.hdf, modis2.hdf and modis3.hdf in one folder. I am able to read files individually using my command. for i=1:3 or for i=1 lst_try=['D:\lst2016\lst_try\modis',num2str(i),'.hdf']; lst_3(:,:,:,:,i)=hdfread(lst_try,…
Vaishali Jain
  • 17
  • 5
1
vote
0 answers

R package gdalUtils does not work for hdf4 files

when I load rgdal and GdalUtils packages to read hdf4 file, with the following library(raster) library(raster) library(sp) library(rgdal) library(proj4) library(gdalUtils)` it gives the following warning message rgdal: version: 1.2-12, (SVN…
Getachew
  • 29
  • 3
1
vote
1 answer

Reading hdf5 file using h5py slows down over time

I have an hdf5 file of shape 80000 * 401408. I need to read data from it in batches of size 64 but the indices can be random say (5, 0, 121, .., 2). The problem is that while initially the reads are quite consistent and a batch takes say 0.5 seconds…
user3682478
  • 21
  • 4
1
vote
0 answers

HDF Object Package method unavailable

Working on creating an .h5 parser for a project. Attempting to implement this tutorial code here. I'm running into an issue Group root = (Group) ((javax.swing.tree.DefaultMutableTreeNode) testFile.getRootNode()).getUserObject(); where the…
Ben Turner
  • 11
  • 1
1
vote
2 answers

Python: Read compressed (.gz) HDF file without writing and saving uncompressed file

I have a large number of compressed HDF files, which I need to read. file1.HDF.gz file2.HDF.gz file3.HDF.gz ... I can read in uncompressed HDF files with the following method from pyhdf.SD import SD, SDC import os os.system('gunzip < file1.HDF.gz…
hm8
  • 1,381
  • 3
  • 21
  • 41
1
vote
1 answer

Why the memory usage of Series is about 1.5x of DataFrame with the same data?

Here is the code: In [1]: import pandas as pd In [2]: import numpy as np In [3]: from itertools import product In [4]: index = list(map(''.join, product(*['ABCDEFGH']*4))) In [5]: columns = list(map(''.join, product(*['xyzuvw']*3))) In [6]: df =…
Eastsun
  • 18,526
  • 6
  • 57
  • 81
1
vote
0 answers

HDF5 dataspace interface - What does it do and what is it's real world application

I am currently building some applications which make use of HDF5 files. My language of choice is R. I have already taken a look at the hdfgroup website with regards to dataspace and I think I understand the concept. But I am very much unable to…
FoldedChromatin
  • 217
  • 1
  • 4
  • 12
1
vote
0 answers

Issues when restarting NiFi cluster

I have Ambari 2.4.2 with the latest HDF installed. I mainly installed it to moderate my NiFi cluster. I have configured the NiFi to work with LDAP although I don't think there's any connection to it. Now each time I try to restart the NiFi service…
BigdataWow
  • 11
  • 2
1
vote
2 answers

Reading and writing numpy arrays to and from HDF5 files

I am building simulation software, and I need to write (thousands of) 2D numpy arrays into tables in an HDF5 file, where one dimension of the array is variable. The incoming array is of float32 type; to save disk space every array is stored as a…
Patrickens
  • 321
  • 3
  • 14
1
vote
1 answer

HDF parsing using Apache Tika

How to parse HDF files(.h5) using Apache Tika. Apache Tika provides parser for .h5 files, but Using that I'm not able to parse the data. Parser parser=new HDFParser(); Metadata metadata=new Metadata(); ContentHandler handler=new…
ketankk
  • 2,578
  • 1
  • 29
  • 27
1
vote
1 answer

Apache nifi Rest Api Issue

when I try running this below url into the browser, I am not getting a 404 response. can any one please guide me on this, as how to use it Thanks. http://localhost:8080/nifi/provenance/search-options apache nifi rest api issue screenshot
ahmed
  • 273
  • 2
  • 8
  • 16
1
vote
0 answers

Data Loss Issue Replace Text and Put sql Processor

We have created a flow in nifi, -- From hive(selecthiveql) we are getting 4257 records -- based on output of selecthiveql we are passing the data based on attribute to replaceText Processor with the insert query into it and passing it to putsql…
priyanka
  • 287
  • 2
  • 5
  • 11
1
vote
2 answers

Comparing h5 files

I often have to compare hdf files. How I do it is either with a binary diff (which tells me files are different even though the actual numbers inside are the same) or by dumping the content into a txt file with h5dump and the comparing the content…
Manfredo
  • 1,760
  • 4
  • 25
  • 53
1
vote
0 answers

Deleting HDF file if writing does not complete (Pandas)?

I am saving a Dataframe as an HDF file in Pandas, using: store = pandas.HDFStore('myfile.h5', table=True) store.append('df', df, chunksize=100000) store.close() I have this code in a Luigi pipeline, and it works fine - except when there is some…
user1566200
  • 1,826
  • 4
  • 27
  • 47
1
vote
0 answers

libjpeg.8.dylib can't be loaded when using Python tool

I have faild to run a tool using for satellite data process CCPLOT After successfully installed, I tried the first case. Here is the error information. >Traceback (most recent call last): File "/Users/HYF/anaconda/bin/ccplot", line 56,…
Han Zhengzu
  • 3,694
  • 7
  • 44
  • 94
1 2 3
…
22 23