1

I am trying to upload some large geometries to google earth engine.

This is the example code given in the ?sf_as_ee description:

# # 3. Upload large geometries to EE asset
# ee_Initialize(gcs = TRUE)
# assetId <- sprintf("%s/%s", ee_get_assethome(), 'toy_poly_gcs')
#  eex <- sf_as_ee(
#  x = toy_poly,
#  overwrite = TRUE,
#  assetId = assetId,
#  bucket = 'rgee_dev',
#  monitoring = FALSE,
#  via = 'gcs_to_asset'
#  )

I just need to know how to set bucket?

From description:

bucket - Character. Name of the bucket (GCS) to save intermediate files (ignore if via is not defined as "gcs_to_asset").

I think it is a local space for saving intermediate files, but I cannot get it to work with the folder I usually use for temporary files (bucket = 'tempfiles').

It is within my working directory, but is throwing this error:

Error in value[[3L]](cond) : The data bucket was not found.

File structure for reference: (on MacOS)

-R Project (working directory)

-- scripts

--- script

--tempfiles

I tried setting it to a google drive folder name as well thinking it might be asking for that, but it didn't work either. Any help will be much appreciated!

Thanks!

Luke McDonald
  • 21
  • 3

1 Answers1

1

sf_as_ee(..., via = "gcs_to_asset") is a two-step function. Firstly, the geometry is uploaded to Google Cloud Storage (GCS) and after that using manifests upload is moved from GCS to Earth Engine.

To use sf_as_ee(..., via = "gcs_to_asset") is necessary to save a service account key (SAK) with privileges to write/read your GCS resources. Upload files via Google Drive is not possible.

library(rgee)
library(googleCloudStorageR)
ee_Initialize("csaybar", gcs = TRUE)
#> ── rgee 1.0.7 ─────────────────────────────────────── earthengine-api 0.1.245 ── 
#>  ✓ email: csaybar 
#>  ✓ GCS credentials: NOT FOUND
#>  ✓ Initializing Google Earth Engine:  DONE!
#>  ✓ Earth Engine user: users/csaybar 
#> ────────────────────────────────────────────────────────────────────────────────
#> Unable to find a service account key (SAK) file in: /home/csaybar/.config/earthengine//csaybar
#> Please, download and save the key manually on the path mentioned
#> before. A compressible tutorial to obtain their SAK file is available in:
#> > https://github.com/r-spatial/rgee/tree/help/gcs
#> > https://cloud.google.com/iam/docs/creating-managing-service-account-keys
#> > https://console.cloud.google.com/apis/credentials/serviceaccountkey
#> Until you do not save a SKA file, the following functions will not work:
#> - rgee::ee_gcs_to_local()
#> - ee_as_raster(..., via = "gcs")
#> - ee_as_stars(..., via = "gcs")
#> - ee_as_sf(..., via = "gcs")
#> - sf_as_ee(..., via = "gcs_to_asset")
#> - gcs_to_ee_image
#> - raster_as_ee
#> - local_to_gcs
#> - stars_as_ee

This guide explains how to generate the SAK, then, you need to save it in:

rgee::ee_get_earthengine_path()
#> [1] "/home/csaybar/.config/earthengine//csaybar/"

This specific error (Error in value[[3L]](cond): The data bucket was not found.) appears because googleCloudStorageR is not able to recognize your SAK (and therefore your GCS bucket names).

library(googleCloudStorageR)
gcs_get_bucket('rgee_dev') # I have privileges to modify the bucket (rgee will work!)
#> ==Google Cloud Storage Bucket==
#> Bucket:          rgee_dev 
#> Project Number:  XXXXXXXXXXXXXX 
#> Location:        US 
#> Class:           STANDARD 
#> Created:         2020-02-08 01:00:23 
#> Updated:         2020-12-10 15:00:47 
#> Meta-generation: 8 
#> eTag:            XXX

gcs_get_bucket('tempfiles') # I do not have privileges privileges to modify the bucket (rgee will not work!)
#> ℹ 2020-12-23 16:16:22 > Request Status Code:  403
#> Error: API returned: gcs-auth-file@rgee-267600.iam.gserviceaccount.com does not
#> have storage.buckets.get access to the Google Cloud Storage bucket.
csaybar
  • 179
  • 1
  • 5
  • Thanks, that's exactly what I was confused about! This worked for my smaller files, but one of my sf objects has ~12000 rows. Is that just too big or should I search for an error in my object? The error I am getting is: `State: FAILED ERROR in Earth Engine servers: File access denied: 'file3a6d2f74078.zip'. Error in ee_monitoring() : ee_monitoring was forced to stop before getting results` It happens after the third task each time I retry. Is the task ordering based on the order of features in the sf object? – Luke McDonald Dec 24 '20 at 17:10
  • 1
    If you are using GCS, there are no limits on the size of the files. It looks like a problem with your GCS credentials (when the [CLI command is invoked](https://developers.google.com/earth-engine/guides/command_line)). Unfortunately without a reproducible example is not possible to help you. – csaybar Dec 25 '20 at 11:31