StreamSets JDBC Producer CDC - Change Log Format Error edit

Question

The idea behind my pipeline is to reflect changes from a MySQL to a PostgreSQL DB. In the future I'll also have a Oracle to PostgreSQL replication.

So, from this forum and SDC documentation, I saw that the right way to do it is to use a CDC origin. So I'm using a MySQL Binary Log. I was able to build a pipeline that process the 3 CRUD operations (INSERT, DELETE, UPDATE), but it uses several processors (Field remover, flattener, stream selector, field renamer and so on):

SDC Pipeline - CRUD Operations

From what I saw in the config of the JDBC Producer, this destination should be able to process MySQL Binary Log directly from a Stream that reads from a MySQL Binary log Origin, right? Just setting the Change Log Format in the JDBC Producer to MySQL Binary Log:

SDC Pipeline - MySQL Binary Log Option

But even though I do this, the pipeline runs with no error, but the data is NOT changed in the PostgreSQL destination.

Am I missing something? Is it necessary to process the stream from the MySQL Binary Log origin before sending it to a JDBC Producer? If so, what must be done?

Answer 1

This was the answer given at Ask StreamSets:

You are right that JDBC Producer can process CDC records directly from MySQL Binlog Origin. What kind of records do you see when you run preview or take snapshot? Also do you see INSERT, DELETE, UPDATE in sdc.log??