Querying with Large FeatureCollection in Google Earth Engine using python

Question

I am trying process Tanzania shape file downloaded from here.

    # im -> {Image} ee.Image({...})
    # self.geom_coll -> {FeatureCollection} ee.FeatureCollection({...}). containing 
    # 3000 features.
    # spacereducer() -> ee.Reducer.mean
    # self.scale -> 10 #Changing this value to small number gives error

    feats = im.reduceRegions(self.geom_coll, spacereducer(), self.scale)
    flist = getInfo_werrorcontrol(feats,
                          self.errorcheck)['features']

.

def getInfo_werrorcontrol(featureCollection, errorcontrolon=True):
    """
    Wrapper to add error control to GEE evaluations.

    For large computations GEE sometimes times out and needs to be
    restarted. This does so in a controlled manner with out 
    interrrupting the program flow.
    """
    if errorcontrolon:
        i=0
        while True:
            try:
                with timeout.timeout(10*60):
                    return featureCollection.getInfo() # In this line I am getting exception.
            except NameError:
                exc_type, exc_value, exc_traceback = sys.exc_info()
                lines = traceback.format_exception(exc_type, exc_value, exc_traceback)
                print ''.join('!! ' + line for line in lines)
                i+=1
                print 'attempts: '+str(i)
                if i > 20:
                    raise ValueError('to many attempts') 
                elif i > 10:
                    print 'waiting 2 minutes'
                    time.sleep(60*2)
    else:
        return featureCollection.getInfo()

Changing self.scale to 10 throws the following error for line:

featureCollection.getInfo()

ee.ee_exception.EEException: Server returned HTTP code: 413

Changing self.scale to 1000 throws this error:

ee.ee_exception.EEException: Computation timed out

What is the correct way to process shape file with the larger region?

Answer 1

Note that requests to the Earth Engine API are already wrapped with exponential backoff. So your code is not going to do much to solve the problem. It's not obvious how those errors arise from the code you posted, but in either case the answer is probably the same: export the result. Example:

import ee
ee.Initialize()
image = ee.Image('srtm90_v4')
geometry = ee.Geometry.Polygon([[[-113.64, 39.97], [-113.64, 38.13],[-109.42, 38.13],[-109.42, 39.97]]], None, False)
dict = image.reduceRegion(reducer=ee.Reducer.mean(), geometry=geometry, scale=1000)
featureCollection = ee.FeatureCollection([ee.Feature(None, dict)])
task = ee.batch.Export.table.toDrive(collection=featureCollection, description='foo', fileNamePrefix='foo', fileFormat='CSV')
task.start()
print task.status()

The output of the export will be materialized in your Google Drive folder. To learn more about scale, please see this doc.