1

I am configuring the MongoDB to BigQuery CDC Template. The job is able to connect to MongoDB and starts up. But it does not process any Change Streams automatically. When I manually publish a message to the Pub/Sub topic, only then it processes and writes to BigQuery.

https://cloud.google.com/dataflow/docs/guides/templates/provided-streaming#mongodb-to-bigquery-cdc

My understanding is that, if the configuration is asking for MongoDB connection URI, database and collection names, shouldn't it directly connect to the Change Streams and populate the data to BigQuery? It kind of doesn't make sense that I would need to have a separate process that reads the change streams from mongodb, extract the record and then send it to Pub/Sub.

If this is the case, why is the configuration asking for MongoDB parameters?

Khusro Siddiqui
  • 11
  • 1

1 Answers1

0

Not too familiar with that template, it was introduced by MongoDB and they are the ones who usually support it.

However, looking at https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v2/mongodb-to-googlecloud/docs/MongoDbToBigQueryCDC/README.md, it appears to be the case that it requires Changestream running that pushes the changes from MongoDb to Pub/Sub topic., which appears to be the missing piece here.

Maybe this is helpful: https://www.mongodb.com/developer/products/mongodb/stream-data-mongodb-bigquery-subscription/

Bruno Volpato
  • 1,382
  • 10
  • 18