0

I have a code as below. So basically i have predefined a grid, and i make a loop every single small grid to this larger predefined grid. But the code stuck at the reindex part.it just using too much memory and it crushed.Is there other way i can snap the small grid to a larger grid?

latitudes=np.linspace(20.0000,50.0000,100001)

longitudes=np.linspace(-130.0000,-100.0000,100001)

AREASCORE=np.full((100001,100001),255,dtype=np.uint8)

AREAFUEL=np.full((100001,100001),255,dtype=np.uint8)

datasets_WFHS= xr.DataArray(AREASCORE,name='AREASCORE',dims='latitude','longitude'],coords={'latitude':latitudes,'longitude':longitudes})

print ("created empty array")





for d in glob.glob(r'Z:\travelers\shp\test\*WFHS.nc'):

    d=xr.open_dataset(d)

    d=d.reindex({'latitude': latitudes, 'longitude': longitudes}, method='nearest', tolerance=0.0001)

    print ('done reindex')

    d=d.fillna(255).astype(np.uint8)

    print ("done fillna")

    datasets_WFHS = xr.where(d==255, datasets_WFHS['AREASCORE'], d['AREASCORE'])

    print ("done np where")
alice
  • 225
  • 5
  • 14

1 Answers1

0

You are making some gigantic arrays here -- an array with shape (100001, 100001) and uint8 dtype requires 10 GB to represent in memory. Typically you need at least 3-4x the size of your largest arrays to comfortably manipulate arrays with NumPy, so I would not be surprised if this fails on any machine with less than about 64 GB of total RAM.

You should try opening your datasets with dask, which works with xarray to support working with arrays larger than fit into memory. This could be as simple as adding a chunks argument to your call to xr.open_dataset().

shoyer
  • 9,165
  • 1
  • 37
  • 55
  • but i only tried with 2 small grid. The code not even pass one loop. the create empty array part is works fast. but not reindex part – alice Aug 29 '18 at 16:40
  • It's also quite likely that `reindex` uses more memory than simply allocating an array with `np.full()`. It's doing a lot more work to reorder all of the data. – shoyer Aug 29 '18 at 17:13
  • so xarray no other packages can do this. – alice Aug 29 '18 at 17:47