2

I am using Confluent JDBC Kafka connector to publish messages into topic. The source connector will send data to topic along with schema on each poll. I want to retrieve this schema.

Is it possible? How? Can anyone suggest me

My intention is to create a KSQL stream or table based on schema build by Kafka connector on poll.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Achaius
  • 5,904
  • 21
  • 65
  • 122

1 Answers1

2

The best way to do this is to use Avro, in which the schema is stored separately and automatically used by Kafka Connect and KSQL.

You can use Avro by configuring Kafka Connect to use the AvroConverter. In your Kafka Connect worker configuration set: 

key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://schema-registry:8081

(Update schema-registry to the hostname of where your Schema Registry is running)

From there, in KSQL you just use 

CREATE STREAM my_stream WITH (KAFKA_TOPIC='source_topic', VALUE_FORMAT='AVRO');

You don't need to specify the schema itself here, because KSQL fetches it from the Schema Registry.

You can read more about Converters and serialisers here.

Disclaimer: I work for Confluent, and wrote the referenced blog post.

Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92
  • I saw your post. But I have 2 questions here. 1. I am using confluent JDBC connector which sends records in delimited format. Is it still work? 2. What happened if schema was updated. Is KSQL able to refer the updated schema? – Achaius Nov 23 '18 at 05:59
  • The answer for first questtion is yes, but confirm it. Kindly respond for the 2nd question – Achaius Nov 23 '18 at 06:22
  • 1
    1. The JDBC Connector uses the Kafka Connect API, which supports multiple converters, including JSON, Avro, etc. So it's not the case that "JDBC connector sends records in delimited format". It is configurable the format in which it sends the data. – Robin Moffatt Nov 23 '18 at 09:01
  • 2
    2. KSQL should pull the latest version of the schema when you create a new stream against the topic, yes. – Robin Moffatt Nov 23 '18 at 09:01