1
gsutil -m rm gs://{our_bucket}/{dir}/{subdir}/*

...
Removing gs://our_bucket/dir/subdir/staging-000000000102.json...
Removing gs://our_bucket/dir/subdir/staging-000000000101.json...
CommandException: 103 files/objects could not be removed.

The command is able to find the directory with the 103 .JSON files, and "tries" removing them per the Removing gs://... being output. For what reason might we be receiving CommandException: 103 files/objects could not be removed.?

  • This works on my local machine
  • This works in our docker container run locally
  • This does not work in our docker container on the GCP compute engine where we need it to be working.

Perhaps this is a permissions issue with the compute engine not having permission to remove files in our GCS?

Edit: We have a service account JSON in the /config folder of our Airflow project, and that service account is shared to an IAM user with Storage Admin permission. Perhaps having the JSON in the /config folder is not sufficient for assigning permissions to the entire GCP compute engine? I am particularly confused because this server is able to query from our BQ database, and WRITE to GCS, but cannot delete from GCS...

Gourav B
  • 864
  • 5
  • 17
Canovice
  • 9,012
  • 22
  • 93
  • 211
  • The first step is to determine the permissions (roles) that the instance has. Edit your question with those details. – John Hanley Aug 03 '21 at 18:52
  • @JohnHanley see edit. I am beginning to think it is not enough for the service account JSON to be *somewhere* on the server... do we need to do something else to assign permissions to the server? – Canovice Aug 03 '21 at 18:58
  • when i try to use `gsutil cp`, as opposed to `rm`, i do get a `AccessDeniedException: 403 Insufficient Permission` error – Canovice Aug 03 '21 at 19:02

1 Answers1

2

The solution in this link - https://gist.github.com/ryderdamen/926518ddddd46dd4c8c2e4ef5167243d was exactly what we needed:

  • Stop the instance
  • Edit the settings
  • Remove gsutil cache
Canovice
  • 9,012
  • 22
  • 93
  • 211