0

I am trying to use Debezium server (Standalone without Kafka) to load data (incremental) from GCP Cloud SQL Server to PubSub and then BigQuery.
I am running a test on my laptop to see if that's doable. I set up debezium server on docker this is my docker-compose.yaml

If I remove SQL Server it works (debezium server starts on docker), am I even on the right track? Should I define the source (GCP SQL Server) on here?

sqlserver:
image: mcr.microsoft.com/mssql/server:2019-latest
container_name:sqlserver
ports:
 - 1433:1433
environment:
 - ACCEPT_EULA=Y
 - MSSQL_PID=Standard
 - SA_PASSWORD=Password!
 - MSSQL_AGENT_ENABLED=true

services:

debezium-server:
image: quay.io/debezium/server
container_name: debezium-server
environment:
  GOOGLE_APPLICATION_CREDENTIALS: /tmp/credentials/service-account.json
ports:
  - "8080:8080"
volumes:
  - ./demo-sa.json:/tmp/credentials/service-account.json
  - ./conf:/debezium/conf
depends_on:
  - sqlserver
restart: "always"
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Ariox66
  • 620
  • 2
  • 9
  • 29
  • Not sure I understand the question. No, you can't configure GCP database within the compose file... What's in your conf folder there? That's where you'd configure Debezium to read from any database – OneCricketeer Aug 31 '22 at 13:37
  • I got around that issue and set up the Debezium server, on GCP we have instance name and no hostname. it doesn't seem to recognise the instance name as hosname for property debezium.source.database.server.name – Ariox66 Sep 01 '22 at 04:32
  • If both Debezium and your database are on the same VPC, then the DNS hostname should work fine. – OneCricketeer Sep 01 '22 at 15:47
  • I got around that issue too by using the same name as database for hostname. now it throws but the table is not whitelisted by the connector error! – Ariox66 Sep 01 '22 at 22:34
  • this is the exact message I am getting: After applying the include/exclude list filters, no changes will be captured. Please check your configuration! I filtered on only one table by debezium.source.schema.include.list=dbo.Table1 – Ariox66 Sep 01 '22 at 22:57
  • Please [edit] your question to add more information rather than use comments. Specifically, sounds like you're no longer using SQL Server Docker container – OneCricketeer Sep 02 '22 at 14:46

0 Answers0