1

If we have written into a Kafka topic data using a SourceTask, in which we have defined a Schema for SourceRecords different than Schema.STRING_SCHEMA, that is a custom schema for example:

private final Schema valueSchema = SchemaBuilder.struct()
.field("dim0", Schema.FLOAT64_SCHEMA)
.field("dim1", Schema.FLOAT64_SCHEMA)
.field("dim2", Schema.FLOAT64_SCHEMA)
.field("dim3", Schema.FLOAT64_SCHEMA)
.build();

then is it possible for a KafkaConsumer to be able to read data from that topic?

Or only SinkTask can read data from that topic since you can define Schema for SinkRecords as you can for SourceRecords?

Thanks in advance!

Novemberland
  • 530
  • 3
  • 8
  • 25

1 Answers1

1

You would rely on the Converter interface to handle the serialization for you.

The method in the source code for this is fromConnectData.

enter image description here

Ones I know of for Struct objects.

  • JsonConverter is included with Apache Kafka

  • AvroConverter is from Confluent

  • ProtobufConverter is from BlueApron.

If you used StringConveter, you'd end up with records that look like Struct{dim0=1,dim1=2,dim2=3,dim3=4}, which would have to be manually parsed as compared to getting proper structured objects using the other options.

The Converters are often wrappers around Serializer and Deserializer interfaces, so you would include the corresponding Deserializer class in your consumer code.

img src: Deep Dive blog from Confluent.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • According to the documentation provided in the book: *Kafka: The Definitive Guide*: "When the connector returns a Data API record to the worker, the worker then uses the configured converter to convert the record to either an Avro object, JSON object, or a string, and the result is then stored into Kafka." I would like to ask if there is support for custom binary objects or only for those three kinds of object? – Novemberland May 04 '19 at 17:34
  • ByteArrayConverter exists. And, as I mentioned, there's one for Protobuf as well – OneCricketeer May 04 '19 at 17:58