8

I am trying to transfer data from S3 to GCS by using a Java client but I got this error.

Failed to obtain the location of the Google Cloud Storage (GCS) bucket ___ due to insufficient permissions. Please verify that the necessary permissions have been granted.

I am using a service account with the Project Owner role, which should grant unlimited access to all project resources.

Son Truong
  • 13,661
  • 5
  • 32
  • 58
Dmytro Kulaiev
  • 271
  • 2
  • 6

3 Answers3

19

Google Transfer Service is using an internal service account to move the data back and forth. This account is created automatically and should not be confused with the service accounts you create yourself.

You need to give this user a permission called "Legacy bucket writer".

This is written in the documentation, but it's VERY easy to miss:

https://cloud.google.com/storage-transfer/docs/configure-access

Dmytro Kulaiev
  • 271
  • 2
  • 6
  • Great to see you found the solution! You can accept your own answers after 48 hours of being posted – Federico Panunzio Oct 30 '18 at 10:48
  • 1
    Also, these roles should be added on the specific source and target buckets only, not on the project level. This is what the Data Transfer service itself does, when configured manually via the GUI. – thnee May 11 '20 at 18:53
  • FYI the user seems to be `project-${var.project_number}@storage-transfer-service.iam.gserviceaccount.com` – Maximilian Oct 09 '20 at 05:14
  • 1
    can use this API https://cloud.google.com/storage-transfer/docs/reference/rest/v1/googleServiceAccounts/get to get the service account email being used by Data transfer service. – JD-V Oct 23 '20 at 11:59
1

Thanks to @thnee's comment I was able to piece together a terraform script that adds the permissions to the hidden Storage Transfer service account:

data "google_project" "project" {}

locals {
  // the project number is also available from the Project Info section on the Dashboard
  transfer_service_id = "project-${data.google_project.project.number}@storage-transfer-service.iam.gserviceaccount.com"
}

resource "google_storage_bucket" "backups" {
  location      = "us-west1"
  name          = "backups"
  storage_class = "REGIONAL"
}

data "google_iam_policy" "transfer_job" {
  binding {
    role = "roles/storage.legacyBucketReader"

    members = [
      "serviceAccount:${local.transfer_service_id}",
    ]
  }

  binding {
    role = "roles/storage.objectAdmin"

    members = [
      "serviceAccount:${local.transfer_service_id}",
    ]
  }

  binding {
    role = "roles/storage.admin"

    members = [
      "user:<GCP console user>",
      "serviceAccount:<terraform user doing updates>",
    ]
  }
}

resource "google_storage_bucket_iam_policy" "policy" {
  bucket      = "${google_storage_bucket.backups.name}"
  policy_data = "${data.google_iam_policy.transfer_job.policy_data}"
}

Note that this removes the default acls of OWNER and READER present on the bucket. This would prevent you from being able to access the bucket in the console. We therefore add the roles/storage.admin back to owner users and the terraform service account that's doing the change.

Richard Nienaber
  • 10,324
  • 6
  • 55
  • 66
1

I was logged into my work account on the gcloud CLI. Changing the auth to gcloud auth login helped solve my random issues.

Kyle Pennell
  • 5,747
  • 4
  • 52
  • 75