Google Earth Engine download problems, is this caused by immutable server side objects?

Question

I have a function that will download an image collection as a TFrecord or a geotiff.

Heres the function -

def download_image_collection_to_drive(collection, aois, bands, limit, export_format):
    if collection.size().lt(ee.Number(limit)):
        bands = [band for band in bands if band not in ['SCL', 'QA60']]
        for aoi in aois:
            cluster = aoi.get('cluster').getInfo()
            geom = aoi.bounds().getInfo()['geometry']['coordinates']
            aoi_collection = collection.filterMetadata('cluster', 'equals', cluster)

            for ts in range(1, 11):
                print(ts)
                ts_collection = aoi_collection.filterMetadata('interval', 'equals', ts)
                if ts_collection.size().eq(ee.Number(1)):
                    image = ts_collection.first()
                    p_id = image.get("PRODUCT_ID").getInfo()
                    description = f'{cluster}_{ts}_{p_id}'
                    task_config = {
                        'fileFormat': export_format,
                        'image': image.select(bands),
                        'region': geom,
                        'description': description,
                        'scale': 10,
                        'folder': 'output'
                    }
                    if export_format == 'TFRecord':
                        task_config['formatOptions'] = {'patchDimensions': [256, 256], 'kernelSize': [3, 3]}
                    task = ee.batch.Export.image.toDrive(**task_config)
                    task.start()
                else:
                    logger.warning(f'no image for interval {ts}')
    else:
        logger.warning(f'collection over {limit} aborting drive download')

It seems whenever it gets to the second aoi it fails, Im confused by this as if ts_collection.size().eq(ee.Number(1)) confirms there is an image there so it should manage to get product id from it.

line 24, in download_image_collection_to_drive
    p_id = image.get("PRODUCT_ID").getInfo()
  File "/lib/python3.7/site-packages/ee/computedobject.py", line 95, in getInfo
    return data.computeValue(self)
  File "/lib/python3.7/site-packages/ee/data.py", line 717, in computeValue
    prettyPrint=False))['result']
  File "/lib/python3.7/site-packages/ee/data.py", line 340, in _execute_cloud_call
    raise _translate_cloud_exception(e)
ee.ee_exception.EEException: Element.get: Parameter 'object' is required.

am I falling foul of immutable server side objects somewhere?

score 1 · Accepted Answer · answered Mar 24 '20 at 15:04

This is a server-side value, problem, yes, but immutability doesn't have to do with it — your if statement isn't working as you intend.

ts_collection.size().eq(ee.Number(1)) is a server-side value — you've described a comparison that hasn't happened yet. That means that doing any local operation like a Python if statement cannot take the comparison outcome into account, and will just treat it as a true value.

Using getInfo would be a quick fix:

if ts_collection.size().eq(ee.Number(1)).getInfo():

but it would be more efficient to avoid using getInfo more than needed by fetching the entire collection's info just once, which includes the image info.

...
ts_collection_info = ts_collection.getInfo()
if ts_collection['features']:  # Are there any images in the collection?
    image = ts_collection.first()
    image_info = ts_collection['features'][0]  # client-side image info already downloaded
    p_id = image_info['properties']['PRODUCT_ID']  # get ID from client-side info
    ...

This way, you only make two requests per ts: one to check for the match, and one to start the export.

Note that I haven't actually run this Python code, and there might be some small mistakes; if it gives you any trouble, print(ts_collection_info) and examine the structure you actually received to figure out how to interpret it.

Im running it now with `if ts_collection.size().eq(ee.Number(1)).getInfo()` but Im confused as to why it happily goes thorough 10 ts loops using this if statement fine, then only fails when it gets to the second aoi, and the other if statement `if collection.size().lt(ee.Number(limit)):` seems to work as intended with no getInfo — Javrel, Mar 24 '20 at 16:46
@Javrel It must be working only by accident. As I said, these conditions will always be treated as **true** — so your code will proceed that way. An exception is raised only if something that is _separately_ a problem happens, like `ts_collection` being empty and therefore not having any `first` element. — Kevin Reid, Mar 24 '20 at 17:17
yeah it worked with the getInfo() on the if statement, I have seen the other if be true or false and work though so Im a bit confused, maybe I changed the code after testing that method, thanks for the help — Javrel, Mar 24 '20 at 18:10

Google Earth Engine download problems, is this caused by immutable server side objects?

1 Answers1