0

i am new of the KAFKA protocol's world and i would like to ask you some inportant information related to my project.

I am using AVRO file for producing and consuming messages, i want to know if i can use the same avro file for multiple Topics maybe for example by using a different "name" attribute into the producer and by using a specific "name" attribute in the consumer.

Thanks a lot. Stefano

kkflf
  • 2,435
  • 3
  • 26
  • 42
SteVizzo
  • 29
  • 5
  • Avro uses the topic name as schema name(subject). The Avro subject could essentially be anything. You can by default use the same Avro schema in multiple topics, but you cannot use multiple Avro schemas for one topic. You can however change the Avro schema subject to something unique, such as the canonical name of the class you wish to serialize. Which schema registry do you use? – kkflf Jul 20 '18 at 11:01

1 Answers1

1

You can use one file to send data to multiple topics, yes, although I'm not sure why one would do that

I would be cautious about merging multiple topics into one Avro file because the schema must match in every topic for that file

It would be suggested that you use the Confluent Schema Registry, for example, rather than sending individual Avro events because if you are not using some registry, then you're likely sending the Avro schema as part of every message, which will slow down the possible throughput of your topic. And then, the name of the Avro schema record in the register will correspond to the topic name

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245