How can I send data without schema to kafka - confluent jdbc - sink usage?

Question

I load my data from kafka to oracle with conluent jdbc-sink.

But I write my schema on value with data.

I do not want to write schema with data , how can I write schema on kafka topic and then I want to send just data from my client?

thanks in advance

json data

{
    "schema": {
        "type": "struct",
        "fields": [
            {
                "field": 'ID',
                "type": "int32",
                "optional": False
            },
            {
                "field": 'PRODUCT',
                "type": "string",
                "optional": True
            },
            {
                "field": 'QUANTITY',
                "type": "int32",
                "optional": True
            },
            {
                "field": 'PRICE',
                "type": "int32",
                "optional": True
            }
        ],
        "optional": True,
        "name": "myrecord"
    },
    "payload": {
        "ID": 1071,
        "PRODUCT": 'ersin',
        "QUANTITIY": 1071,
        "PRICE": 1453
   }

python code:

producer.send(topic, key=b'1071'
              , value=json.dumps(v, default=json_util.default).encode('utf-8'))

how can I solve this?

thanks in advance

You need to publish message with schema from producer and also need to enable schema converter for sink message using kafka-connect. — Atif, Apr 15 '20 at 13:14
I do not want to send schema with data everytime , it is ok when I send schema and payload together — CompEng, Apr 15 '20 at 13:21
It is not possible to sink data using kafka-connect without a schema. Without a schema, kafka-connect will fail to map with database table columns. And you do not need to send schema every time because schema registry will register or store schemas in _schemas topic in broker and bind with that schema with a specific topic. — Atif, Apr 15 '20 at 13:27
thanks for answer so how can I use schema register for specific one topic , my schema is json — CompEng, Apr 15 '20 at 13:30
When you publish your object, you need to set ''io.confluent.kafka.serializers.KafkaJsonSchemaSerializer.class'' for serializer in producer and need to enable json schema converter in kafka sink connector property file. — Atif, Apr 15 '20 at 13:37
my producer is python, so I set schema registry for my json for one time — CompEng, Apr 15 '20 at 13:42
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/211720/discussion-between-ersin-gulbahar-and-atif). — CompEng, Apr 15 '20 at 13:42

Giorgos Myrianthous · Accepted Answer · 2020-04-15T08:52:08.363

5

If you want to use the JDBC sink connector you must provide a schema. This can be achieved in three ways:

Use JSON with schemas enabled
Use Avro and Schema Registry
Use JSON schemas with Schema Registry

You are currently using JSON with schemas enabled that requires to send the schema along with the actual payload. The only way you can achieve your requirement is by using Avro and Confluent Schema Registry so that your schemas are registered in the schema registry. In this way, you won't be required to send the payload schema every time.

Another option would be to use JSON with schema registry (#1289). For Kafka Connect you can use JsonSchemaConverter and for Java Consumers and Producers you can use KafkaJsonSchemaSerializer and KafkaJsonSchemaDeserializer.

edited Apr 15 '20 at 08:52

answered Apr 15 '20 at 08:43

Giorgos Myrianthous

36,235
20
134
156

@ErsinGülbahar I updated my answer but I am not sure if JSON with schema registry is supported by the python clients yet as this is a relatively new feature. – Giorgos Myrianthous Apr 15 '20 at 08:52
thanks for answer, my question is how can I make json schema registry ? – CompEng Apr 15 '20 at 08:55
@ErsinGülbahar In your sink connector use `JsonSchemaConverter` for your values. – Giorgos Myrianthous Apr 15 '20 at 08:57
@ErsinGülbahar `value.converter=` – Giorgos Myrianthous Apr 15 '20 at 09:11
@ErsinGülbahar Yes but as I said you need to use `JsonSchemaConverter ` – Giorgos Myrianthous Apr 15 '20 at 09:13
@ErsinGülbahar All you need is here: https://docs.confluent.io/1.0/schema-registry/docs/intro.html :) – Giorgos Myrianthous Apr 15 '20 at 09:14
how can I define schema on topic as json ? – CompEng Apr 15 '20 at 12:11
org.apache.kafka.connect.json.JsonSchemaConverter is not valid it gives error – CompEng Apr 15 '20 at 13:56
@ErsinGülbahar As far as I know the PR was merged a couple of months back. You need to make sure you have the version that supports this. I've attached the link to the PR in my answer for your reference. – Giorgos Myrianthous Apr 15 '20 at 13:58
what do you mean by saying 'PR' – CompEng Apr 15 '20 at 14:17
@ErsinGülbahar Pull Request. – Giorgos Myrianthous Apr 15 '20 at 14:18
What about protobuf? – Mark Mar 17 '23 at 12:01

How can I send data without schema to kafka - confluent jdbc - sink usage?

1 Answers1