2

I want to export relevant BigQuery (audit) logging to a BQ dataset.

So I have to create a log sink for that.

I found a blog that sets up the BQ logging only using bigquery_project or bigquery_dataset:
https://dataform.co/blog/exporting-bigquery-usage-logs

I noticed that there are several more resource.types to query on:

  • bigquery_project
  • bigquery_resource
  • bigquery_dataset
  • bigquery_table
  • bigquery_dts_config
  • bigquery_dts_run
  • bigquery_biengine_model

I found that list here:
https://cloud.google.com/logging/docs/api/v2/resource-list#resource-types

So the last three have to do with data transfer and biengine.

But I wonder what is the difference between project, resource and dataset?

What info is in those 3 different kinds of logs? And: which of the three (or which of the seven resource types) should I use?

log sink for exporting BQ logs

Sander van den Oord
  • 10,986
  • 5
  • 51
  • 96
  • AuditData payload will return resource.type set to bigquery_resource, not bigquery_dataset. In BigQueryAuditMetadata messages, resource.type is set to one of these values: `bigquery_dataset` for operations to datasets such as `google.cloud.bigquery.v2.DatasetService.*`, resource.labels.dataset_id contains the encapsulating dataset and `bigquery_project` for all other called methods, such as jobs resource.labels.location contains the location of the job. For more information you can refer this [document](https://cloud.google.com/bigquery/docs/reference/auditlogs#bigqueryauditmetadata_format). – Prajna Rai T May 19 '22 at 14:03
  • Hi @SandervandenOord, Is the above comment helpful? – Prajna Rai T May 23 '22 at 08:55
  • @PrajnaRaiT it was somewhat helpful, but am also hoping for examples of useful data to extract from all these different resource.types – Sander van den Oord May 23 '22 at 13:49
  • Hi @SandervandenOord, If you find my answer helpful, please consider to accept it as per [Stack Overflow guidelines](https://stackoverflow.com/help/someone-answers), helping more Stack contributors with their researches. – Prajna Rai T Jun 28 '22 at 09:46

1 Answers1

2

AuditData payload will return resource.type set to bigquery_resource, not bigquery_dataset. In BigQueryAuditMetadata messages, resource.type is set to one of the following values: bigquery_dataset for operations to datasets such as google.cloud.bigquery.v2.DatasetService.* such as resource.labels.dataset_id contains the encapsulating dataset and bigquery_project for all other called methods, such as jobs resource.labels.location contains the location of the job.

resource.type="bigquery_project" AND logName:"cloudaudit.googleapis.com" is used to get BigQuery audit logs for a project.

resource.type="bigquery_dataset" AND logName:"cloudaudit.googleapis.com" is used to get BigQuery audit logs for a dataset.

resource.type="bigquery_dts_config" AND logName:"cloudaudit.googleapis.com" is used to get BigQuery audit logs for a Data Transfer Service configuration.

resource.type="bigquery_dts_run" AND logName:"cloudaudit.googleapis.com" is used to get BigQuery audit logs for a Data Transfer Service.

resource.type="bigquery_biengine_model" AND logName:"cloudaudit.googleapis.com" is used to get BigQuery audit logs for BI Engine Model.

Example:

To show log entries from a given transfer config_id, in the Query builder, add the following filter:

resource.type="bigquery_dts_config"
labels.run_id="transfer_config_id"

For more information you can refer to this document.

Prajna Rai T
  • 1,666
  • 3
  • 15