Stack Overflow
Stack Overflow

Questions tagged [debezium]

Debezium is an open source distributed platform to stream database changes. Use this tag for questions related to using, configuring or troubleshooting Debezium.

Overview

From [https://debezium.io/]:

Debezium is an open source distributed platform for change data capture.

Debezium uses Kafka to record all change events. It supports connectors for:

  • MySQL
  • Oracle
  • MongoDB
  • PostgreSQL
  • SQL Server (colloquially "MSSQL" or "MS SQL")

References

Related Tags

1029 questions
5
votes
1 answer

Kafka connect Debezium Postgres Cloud SQL

I am trying to connect to Cloud SQL postgres using Debezium Postgres connector. And I am getting the following error in logs. connect_1 | org.apache.kafka.connect.errors.ConnectException: org.postgresql.util.PSQLException: FATAL: must be…
Amit Basuri
  • 563
  • 5
  • 12
5
votes
1 answer

Cannot read the binlog filename and position via 'SHOW MASTER STATUS'

I am making some POC using debezium version 0.9 and I am running a standalon docker-ized mysql database using the embedded Debezium configuration. I am running into this issue: Caused by: java.lang.IllegalStateException: Cannot read the binlog…
Squake
  • 370
  • 2
  • 10
5
votes
4 answers

Kafka producer can't create topics and throwing continuous error after creating Debezium MySQL connector

I am using Debezium as a CDC tool to stream data from MySql. After installing Debezium MySQL connector to Confluent OSS cluster, I am trying to capture MySQL bin_log changes into a Kafka topic. When I create a connector, after taking the snapshot of…
Rahul Gupta
  • 51
  • 1
  • 3
5
votes
2 answers

Authentication failed when connected to mongodb by debezium mongodb connector

Deployed the Strimzi Kafka, Strimzi Zookeeper and Debezium mongodb connector, and configured the Debezium mongodb, curl 'http://my-connect-cluster-connect-api:8083/connectors' -X POST -i -H "Content-Type:application/json" -d '{ "name":…
user3661933
  • 91
  • 3
  • 7
5
votes
1 answer

Apache kafka production cluster setup problems

We have been trying to set up a production level Kafka cluster in AWS Linux machines and till now we have been unsuccessful. Kafka version: 2.1.0 Machines: 5 r5.xlarge machines for 5 Kafka brokers. 3 t2.medium zookeeper nodes 1 t2.medium node for…
Ankur rana
  • 580
  • 10
  • 27
5
votes
3 answers

Debezium-contains no connector type

I am trying to use Debezium to connect to a mysql database on my local machine. Trying with the following command to call kafka: sudo kafka/bin/connect-standalone.shsh kafka/config/connect-standalone.properties kafka/config/connector.properties Here…
Pravudutta Patra
  • 51
  • 3
5
votes
1 answer

debezium table whitelist with regular expression

I have some database table with bg_ and cp_ prefix like "bg_table1", "bg_table2" and "cp_table1".There are also some other tables without any prefix such as my_table1, user_action and so forth. I have two debezium postgreSQL connectors and trying to…
Khorshed Alam
  • 314
  • 2
  • 11
5
votes
3 answers

How can a org.apache.kafka.connect.data.Decimal stored in an avro file be converted to a python type?

I am trying to interpret a Avro record stored by Debezium in Kafka, using Python { "name": "id", "type": { "type": "bytes", "scale": 0, "precision": 64, …
Ravindranath Akila
  • 209
  • 3
  • 35
  • 45
4
votes
1 answer

Debezium Oracle Connector - jmx metrics attribute type string not populating to Prometheus

We configured successfully the Oracle Debezium connector and are receiving messages for any related database changes. We also activated the jmx exporter to get the metrics into prometheus using kafka-connect to build Grafana Dashboard. The issue we…
Sandy
  • 53
  • 3
4
votes
1 answer

Multiple postgres connections with debezium connector

When I create a kafka connect connector with the debezium connector, it results in four database connections. Three of them remain idle, while one works as the actual replication slot. I can't find any information about the expected number of…
Avocado
  • 871
  • 6
  • 23
4
votes
1 answer

Debezium Kafka Connect - Error Loading Plugin From Confluent Hub via Docker

I've recently tried installing the Debezium PostgreSQL Connector v1.6.0 via confluent-hub. When the plugin is loaded, the connector fails to load, raised by loadProperties in IoUtil, per this error: [2021-07-13 00:45:06,715] INFO Loading plugin…
alexcu
  • 361
  • 5
  • 14
4
votes
1 answer

(KAFKA Oracle DEBEZIUM) Unable to connect: Failed to resolve Oracle database version

i have setup kafka debezium on UBUNTU server and created sql connector and that works perfectly fine. then i created oracle connector and i m getting this error "Unable to connect: Failed to resolve Oracle database version" i have followed strictly…
AliM
  • 41
  • 3
4
votes
2 answers

Cannot start debezium MySQL connector due to Error code 1236

When I check the status of my debezium connector via the kakfa-connect's REST API, I see this error message for the connector: org.apache.kafka.connect.errors.ConnectException: The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION =…
lollerskates
  • 964
  • 1
  • 11
  • 28
4
votes
1 answer

JOINS in Debezium : MySQL to Elasticsearch

I've been trying to set up a MySQL to Elasticsearch data pipeline for real-time data replication. The MySQL database has around 10 different tables that are highly normalized. But in Elasticsearch, I'm in need to have all of the data from these…
dev_x
  • 41
  • 5
4
votes
1 answer

How to disable JSON schema in Kafka Source Connector (e.g. Debezium)

I followed Debezium tutorial (https://github.com/debezium/debezium-examples/tree/master/tutorial#using-postgres) and all received CDC data from Postgres are sent to Kafka topic in JSON format with schema - how to get rid of schema? Here is config of…
ALZ
  • 1,997
  • 2
  • 27
  • 44
1 2
3
…
68 69