5

How can I make Kafka Connect JDBC connector to predefined Avro schema ? It creates a new version when the connecter is created. I am reading from DB2 and putting into Kafka topic. I am setting schema name and version during creation but it does not work!!! Here is my connector settings :


    {
      "name": "kafka-connect-jdbc-db2-tst-2",
      "config": {
        "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
        "tasks.max": "1",
        "connection.url": "jdbc:db2://mydb2:50000/testdb",
        "connection.user": "DB2INST1",
        "connection.password": "12345678",
        "query":"SELECT CORRELATION_ID FROM TEST.MYVIEW4 ",
        "mode": "incrementing",
        "incrementing.column.name": "CORRELATION_ID",
        "validate.non.null": "false",
        "topic.prefix": "tst-4" ,
        "auto.register.schemas": "false",
        "use.latest.version": "true",
        "transforms": "RenameField,SetSchemaMetadata",
        "transforms.RenameField.type": "org.apache.kafka.connect.transforms.ReplaceField$Value",
        "transforms.RenameField.renames": "CORRELATION_ID:id",
        "transforms.SetSchemaMetadata.type": "org.apache.kafka.connect.transforms.SetSchemaMetadata$Value",
        "transforms.SetSchemaMetadata.schema.name": "foo.bar.MyMessage",
        "transforms.SetSchemaMetadata.schema.version": "1"
    
      }
    
    }

And here are the schemas : V.1 is mine, and V.2 is created by JDBC source connector:


    $ curl localhost:8081/subjects/tst-4-value/versions/1 | jq .
    
    {
      "subject": "tst-4-value",
      "version": 1,
      "id": 387,
      "schema": "{"type":"record","name":"MyMessage",
    "namespace":"foo.bar","fields":[{"name":"id","type":"int"}]}"
    }
    
    $ curl localhost:8081/subjects/tst-4-value/versions/2 | jq .
    {
      "subject": "tst-4-value",
      "version": 2,
      "id": 386,
      "schema": "{"type":"record","name":"MyMessage","namespace":"foo.bar",
       "fields":[{"name":"id","type":"int"}],
       "connect.version":1,
       "connect.name":"foo.bar.MyMessage"
    }"
    }

Any idea how can I force Kafka connector to use my schema ? Thanks in advance,

usernotexist
  • 51
  • 1
  • 5
  • AFAIK, Transforms happen after the Message is serialized as Avro and registered in the registry – OneCricketeer Aug 03 '20 at 17:06
  • I cannot imagine how that could work for transforms that drop certain fields for example, that needs to happen before serialization. – peedee Mar 08 '23 at 13:54

0 Answers0