0

For the following THREDDS OPeNDAP server:

http://data.nodc.noaa.gov/thredds/catalog/ghrsst/L2P/MODIS_T/JPL/2015/294/catalog.html

I would like to note four Attributes of every file in there. The attributes are: northernmost lattitude; easternmost lattitude; westernmost lattitude; southernmost lattitude. These can be found under the Global attributes under:

http://data.nodc.noaa.gov/thredds/dodsC/ghrsst/L2P/MODIS_T/JPL/2015/294/20151021-MODIS_T-JPL-L2P-T2015294235500.L2_LAC_GHRSST_N-v01.nc.bz2.html

At first I tried this with MATLAB. Problem is: all the netcdf files on the server are compiled to .bz2 files. This makes calling for the Global attributes take around 15 seconds (the server is extracting the file). I would like javascript to run these server requests parallel to save me time. In total I need 90,000 files.

Is there a way to code this using javascript?

user5612440
  • 1
  • do you have to download each *.bz2 and then process them to get the lat, long boundaries ? – trk Nov 27 '15 at 13:51
  • No, because it is an OPeNDAP server you don't have to download the entire file to see the boundaries. They are in the Global attributes section. You can do this simple operation with Matlab. The problem however is: when you call for a global attribute the OPeNDAP server first needs to extract the .bz2 file to a .nc file. This takes 15 seconds. So if you use MATLAB to process the 90,000 files this will take at least 15 days. I was told javascript can make parallel computations, so make alot of server requests parallel, speeding up the processing. – user5612440 Nov 27 '15 at 14:03
  • how do you request ? are you familiar with node.js ? you can use its request module. If you don't know how to request in node I can write that section/help. – trk Nov 27 '15 at 14:10
  • I do not know how to request in node.js. – user5612440 Nov 27 '15 at 14:19
  • Is there a matlab snippet I can use for reference ? – trk Nov 27 '15 at 14:23
  • nc_file_list = {http://data.nodc.noaa.gov/thredds/dodsC/ghrsst/L2P/MODIS_T/JPL/2015/294/20151021-MODIS_T-JPL-L2P-T2015294235500.L2_LAC_GHRSST_N-v01.nc.bz2.html}; temp.sl = ncreadatt(nc_file_list,'/','southernmost_latitude'); – user5612440 Nov 27 '15 at 14:34

1 Answers1

0

You can use the THREDDS DAS service. DAS

Change the OPenDAP link you have above replacing the .html extension with .das This is a small text file with metadata about the file which could be easily parsed with javascript and includes a section with the global attributes:

 NC_GLOBAL {
    . . .
    Float32 northernmost_latitude 89.9942;
    Float32 southernmost_latitude 66.9853;
    Float32 easternmost_longitude -121.445;
    Float32 westernmost_longitude 76.7485;
    . . .
  }

This metadata is cached by THREDDS and the above DAS link responds instantly.

Edit:

Re: the correct comments below, (cache exists only after the first request) one alternative might be to use the source data at the NASA JPL OPeNDAP Server (Hyrax): http://podaac-opendap.jpl.nasa.gov/opendap/allData/ghrsst/data/L2P/MODIS_T/JPL/

My browser only tests (i.e. subjective) seem to show that a random DAS responses are quicker, than 15 seconds.

http://podaac-opendap.jpl.nasa.gov/opendap/allData/ghrsst/data/L2P/MODIS_T/JPL/2015/294/20151021-MODIS_T-JPL-L2P-T2015294084500.L2_LAC_GHRSST_N-v01.nc.bz2.das

Eric Bridger
  • 3,751
  • 1
  • 19
  • 34
  • 1
    Can someone confirm? If I try, it is equally slow for new requests and responds fast only after being cached. – kakk11 Nov 30 '15 at 08:51
  • 1
    Yeah, if you're the first one to hit it, the file still has to be decompressed to get the metadata. I don't think there's a magic bullet here. (No idea about the js part) – DopplerShift Nov 30 '15 at 18:02
  • My browser tests confirm that this is indeed true. – Eric Bridger Nov 30 '15 at 21:58