0

Im trying to export logs in to bigquery using sink from the cloud shell. I did the following steps: bq mk dataset

gcloud beta logging sinks create my-bq-sink \ bigquery.googleapis.com/projects/my-project/datasets/\ my_dataset --log-filter='resource.type="gce_instance"'

I created a service account for the sink and bind him to bigQuery.dataEditor and logging.logWriter

The problem is that if im not going to the console-> edit sink -> update sink im getting that my access to the dataset was denied. how can i solve that from cloud shell?

Sapir Pe'er
  • 1
  • 3

1 Answers1

2

Like in many products, creating a service is separate from the IAM authorization. For the logging sink, the "strange" decision from Google has been to generate a service account by the logging service and to send you the name of this service account in the command result

Created [https://logging.googleapis.com/v2/projects/My_PROJECT/sinks/test].
Please remember to grant `serviceAccount:p78401601954-957849@gcp-sa-logging.iam.gserviceaccount.com` the BigQuery Data Editor role on the dataset.
More information about sinks can be found at https://cloud.google.com/logging/docs/export/configure_export

Not very usable if you want to script something. So, add the parameter --format=json into the sink creation command and the result is the following

{
  "createTime": "2020-05-21T19:27:36.599050569Z",
  "destination": "bigquery.googleapis.com/projects/My_PROJECT/datasets/asset_eu",
  "filter": "resource.type=cloud_function",
  "name": "test",
  "updateTime": "2020-05-21T19:27:36.599050569Z",
  "writerIdentity": "serviceAccount:p78401601954-465055@gcp-sa-logging.iam.gserviceaccount.com"
}

Now you can get the writerIdentity and grant the role that you need on it. However, I repeat, this choice is strange for Google (and not consistant with other products) and I won't be surprised that this behavior change in the future.

guillaume blaquiere
  • 66,369
  • 2
  • 47
  • 76
  • I used sink description to get the writerIdentity and i created a SA with the biqeury permission ans logs writes permission. the problem is that its working only after im going to the GCP console and click on UPDATE SINK – Sapir Pe'er May 23 '20 at 17:08
  • What do you mean by "I created the SA"? You don't have to create it, only grant the role that you want. – guillaume blaquiere May 23 '20 at 18:52
  • i created a new service account (Under the IAM) using the unique writer identity. should i just give permissions to: p78401601954-465055@gcp-sa-logging.iam.gserviceaccount.com? – Sapir Pe'er May 24 '20 at 06:17
  • Yes!! You simply have to grant permissions to the provided service account after the sink creation! No service account to create – guillaume blaquiere May 24 '20 at 08:14