Stack Overflow
Stack Overflow

Questions tagged [google-datastream]

Use this tag when facing Google Datastream issues or questions like: set up, running, viewing logs, handle error, recovery, working with datastream public API

50 questions
1
vote
1 answer

Datastream can not connect to Cloud SQL with private IP using private connectivity

My Datastream is not able to connect to Cloud SQL (with private IP) using a private connectivity. I have setup my cloud SQL (Postgres) for replication as it is already streaming events for debezium. I have created a private connectivity in my VPC…
Jonathan Chevalier
  • 993
  • 1
  • 9
  • 18
1
vote
0 answers

Datastream streamed (backfilled) table to bigquery missing rows

So I'm using Datastream's new feature to stream data from Cloud SQL MySQL database to BigQuery, I've set the maximum data staleness of the stream to 8 hours. THE PROBLEM: To initialise the stream I performed a backfill on my tables. Datastream…
Firas Dhaha
  • 19
  • 2
1
vote
3 answers

Datastream Troubleshoot: "An unknown error occurred. Please try again. If the error persists, contact Google support"

We are trying to replicate data from AlloyDB to Bigquery using Datastream. We Get "An unknown error occurred. Please try again. If the error persists, contact Google support." In the Datastream console --> objects list, we see all source tables with…
N.N.
  • 3,094
  • 21
  • 41
1
vote
1 answer

Google Cloud DataStream to Bigquery template not able to sync data to big query

I am trying to design CDC pipeline to stream data from cloud SQL to BigQuery using DataStreams and Dataflow on GCP, the datastream part is working fine and I can see data being transferred to CloudStorage successfully in avro format. When it comes…
Karim Tawfik
  • 1,286
  • 1
  • 11
  • 21
1
vote
1 answer

Two New Fields Added on the Dataflow Job from a Template

I created a Dataflow job from a Template (Cloud Datastream to BigQuery) several weeks ago. I stopped the job and then tried to create a new job with the same Template (Cloud Datastream to BigQuery). Now, I see two new fields that are required (The…
Brian Pembroke
  • 11
  • 3
0
votes
0 answers

What is the source code of gcr.io/dms-images/tcp-proxy image?

I am trying to establish a connectivity between Datastream and AlloyDB in Google Cloud. In the documentation article it is recommended to set up a TCP proxy due to the inability to have transitive peering between Datastream AlloyDB private…
vilozio
  • 111
  • 1
  • 8
0
votes
1 answer

Is Datastream able to write on an already existing table?

I created a stream job having mysql and bigquery as my source and destination connection profiles respectively, and the job was able to successfully replicate the source tables and data into the bigquery tables(which were created by datastream…
theDataEngineerGuy
  • 11
  • 3
0
votes
0 answers

Datastream `POSTGRES_PUBLICATION_DOES_NOT_EXIST`

When I tried to create Data stream with Terraform according to the Google Cloud official blog[1], then I got the following error. Enter a value: yes module.test-db.google_datastream_stream.stream:…
Ryo Matsuzaka
  • 135
  • 1
  • 1
  • 9
0
votes
0 answers

Datastream bigquery_profile is empty

I am trying to create Datastream with terraform according to the Google Cloud official blog [1]. There, bigquery_profile is empty. Is it okay? When I check the official documentation[2], the bigqueryprofile section is also empty. resource…
Ryo Matsuzaka
  • 135
  • 1
  • 1
  • 9
0
votes
1 answer

How can I pass a list of values to a block within a resource

I am trying to set up a Datastream connection from MySQL to BigQuery using Terraform (docs here). The docs show this structure for specifying which tables to include: resource "google_datastream_stream" "default" { depends_on = [ …
CClarke
  • 503
  • 7
  • 18
0
votes
0 answers

GCP PostgreSQL Instance: How to stop WAL continuously increasing in size since implementing datastream

PostgreSQL version: PostgreSQL 12.12 Since implementing datastream to replicate 2 rather large tables to GCP BigQuery, the SQL instances WAL size has been steadily increasing each day. This has been going on for just shy of a month now and it is…
Logan
  • 1
  • 1
0
votes
1 answer

Datastream created table and overwrites historic data

I have a datastream between a PostrgreSQL and Google BigQuery which was running smoothly over the last couple weeks. Suddenly all historic data was gone and only incremental data was added again. I've check the activity and found that the Google key…
Michieldo
  • 169
  • 1
  • 2
  • 15
0
votes
1 answer

How can I remove duplicate events when using Datastream to transfer data from PostgreSQL to Cloud Storage?

I use Datastream to transfer data from PostgreSQL to Cloud Storage. In the documentation it is written that there can be overlap between backfill and CDC, resulting in duplicate events. Event metadata should be used to remove duplicates. In article…
vilozio
  • 111
  • 1
  • 8
0
votes
0 answers

Datastream MYSQL to BigQuery Is Not Replicating OnDeleteCascade

We have a MySQL/BigQuery stream that replicates data in 0 seconds. Data backfilled with no issues and stream went to Running state as expected. RDS MySQL source was configured as stated in Google Docs with following configurations binlog_format:ROW,…
Yunus Einsteinium
  • 1,102
  • 4
  • 21
  • 55
0
votes
1 answer

Google Datastream (mysql to bigquery), no is_deleted field with tables with primary key

I am using datastream in GCP with MySQL source and BQ destination. I have noticed that if the source table has a primary key defined, then the datastream metadata fields do not include the "is_deleted" column. Also, the BQ table (when queried)…
ssm
  • 620
  • 6
  • 24
1
2
3 4