0

I am building a similar approach to bring all the logs into BigQuery dataset. As per this https://dev.to/alvardev/gcp-cloud-logging-the-basis-45eh I am able to bring the project log details into the corresponding BigQuery. But how do we bring all the projects logs into a single Bigtable i.e for example lets say An organisation has 50 projects and with minimal configuration I want all the logs from 50 projects to come under a single Bigtable as consolidated project logs.

Is this possible in GCP, can someone please help me on this

enter image description here

Mabel A.
  • 1,775
  • 4
  • 14
Alex Man
  • 4,746
  • 17
  • 93
  • 178

1 Answers1

1

Currently, the only supported destinations based on the documentation are Cloud Storage, Pub/Sub, BigQuery and another Cloud Logging Bucket . However I found this another approach where the architecture is: Cloud Logging -> Pub/Sub -> DataFlow -> BigTable.

Basically, you will create a sink and choose Pub/Sub topic as the destination then from there, you can stream your data from Pub/Sub to Bigtable using Dataflow. You can check this SO answer that you can use as reference on how to stream data from Pub/Sub to BT.

Mabel A.
  • 1,775
  • 4
  • 14
  • Thanks for the answer, also I would like to know if it possible to transfer Bigtable to BigQuery. i.e lets say I want all data greater than 1 year should be moved from Datatable to BigQuery. Is this doable? – Alex Man Dec 24 '21 at 08:21
  • @AlexMan here's a [documentation](https://cloud.google.com/bigquery/external-data-bigtable) on how to query Bigtable data on BigQuery. If you need a detailed answer, please post a new question so other people can contribute. – Mabel A. Dec 27 '21 at 07:48
  • I have created an [SO](https://stackoverflow.com/questions/70507851/gcp-moving-bigtable-data-to-bigquery) for this – Alex Man Dec 28 '21 at 13:37