Stack Overflow
Stack Overflow

Questions tagged [google-datastream]

Use this tag when facing Google Datastream issues or questions like: set up, running, viewing logs, handle error, recovery, working with datastream public API

50 questions
0
votes
0 answers

Cloud Google Datastream file_rotation_interval it does not change

i change the rotation interval in cloud storage, I take the change, however it doesn't create the object with the interval set, but every 60s this is my code: { "file_rotation_mb":20, "file_rotation_interval":"480s", "avro_file_format":…
Mery
  • 1
  • 1
0
votes
2 answers

Google DataStream character varying not being replicated

I have setup Google DataStream to replicate data from PostgresSQL using CDC. It works fine, but I have noticed that all character varying columns are not being replicated. I can see them in the source schema, but the destination table that been…
Akram
  • 453
  • 4
  • 21
0
votes
0 answers

Datastream in Project-A, CloudSQL in Project-B

I'm trying to configure Datastream to enable streaming data from CloudSQL to the BigQuery. Our CloudSQL instance in the project-A, Datastream in the project -B. Project-A has vpc-A which is peered with vpc-B in the Project-B. vpc-A has cloudsql_auth…
Sharip Alikhanov
  • 21
  • 3
0
votes
0 answers

How to set-up the same stream for cross domain tracking (GA4) in two different GTM container

I'd like to set-up cross domain tracking between two domains (let's say website A and website B). Each website involved has its' own GTM installed and configured a bunch of custom events. Each website has also own GA4 property (different once) with…
ACM
  • 1
  • 1
0
votes
0 answers

Google Datastream doesn't show before-image with the RDS mysql source

https://cloud.google.com/datastream/docs/configure-your-source-mysql-database#rdsformysql I followed the doc above building a datastram with RDS mysql. The stream was built successfully, but there is a problem, source_metadata.change_type only has…
peter huang
  • 21
  • 2
0
votes
0 answers

Datastream cannot read UPDATE binary log in Google cloud Datastream

I use Datastream to bigquery. I sync data from oracle mysql to bigquery use datastream. when use oracle mysql version 8.0.28 -> working. when use oracle mysql version 8.0.30 -> working but I only saw data with change_type is INSERT. There is no…
Anh Tú Cấn
  • 1
  • 1
0
votes
1 answer

I am using GCP datastream for MySQL to Bigquery for historical data load and CDC/incremental load. I am getting historical load but not incremental?

GCP Datastream is used for historical and CDC data. I am getting historical load but not incremental/CDC data into Bigqery. All the configuration for datastream and Mysql is configured properly as I have tested the same in other gcp cloud project…
tanmay sakpal
  • 1
0
votes
0 answers

Why has Datastream throughput dropped to near zero after the backfill completed?

I have a stream running from MySQL to BigQuery. The backfill completely perfectly but when it was finished the ongoing stream is just not picking up changes to almost all of the tables. Just a few here and there, almost nothing is going though. The…
CClarke
  • 503
  • 7
  • 18
0
votes
0 answers

Configure Google Datastream - Shared VPC

According Google Datastream documentation https://cloud.google.com/datastream/docs/create-a-private-connectivity-configuration#before-you-begin, to configure private connectivity within a shared VPC is required to grant compute.networkadmin role on…
Ivano Indrio
  • 1
0
votes
0 answers

Google Cloud Datastream Stopped Replicating

We've been using Google's Datastream service for a little over 1 years with 2 Streams with about 150 tables in total. Yesterday 2022/10/21 around 18:30 both Streams stopped replicating. No error messages appear in the logs. No operations were…
Artur Alves
  • 1
0
votes
0 answers

GCP - CDC fetch failed: We couldn't connect to the MySQL server

I configured a DataStream in GCP to replicate data from MySQL to BigQuery. It was working successfully for some time and initial data sump is successful to BQ, but after that, I keep on getting the error, { "insertId":…
Stranger
  • 10,332
  • 18
  • 78
  • 115
0
votes
0 answers

How should we configure Datastream in Google Cloud

We have c. 60 tables whose changes we would like to capture (CDC) using GCP's datastream and ingest into our data lake. Are there any drawbacks to using Datastream? And should I set up one stream that ingests all tables? Or should I create a stream…
Negm
  • 57
  • 6
0
votes
1 answer

Shared VPC and Datastream SA Network Admin Role

According to this document, if using a shared VPC, the Datastream service account requires the COMPUTE.NETWORKADMIN role on the host project. However, I was wondering if there were more granular permissions/roles that can be assigned instead since…
Stephen Hwang
  • 1
0
votes
1 answer

GA4 - filtering retail and wholesale traffic

So, I would like to filter retail and wholesale metrics on a site, but I am having issues finding the best way of doing this in GA4. I tried looking into data streams but I am not sure if that will work with my current set-up. The way my site…
Nathan
  • 1
  • 1
0
votes
1 answer

Not able to connect the oracle database to GCP data-stream

I am trying to create a connection profile in GCP data-stream for oracle database. When I try to connect to oracle database, it is showing hostname or port configuration are not correct, even though everything is correct. Can anyone assist me how to…
Subhash bhat
  • 21
  • 2
1 2
3
4