1

Can someone give advices on how to convert a String into a GenericRecord? I would like to convert my records to avro and put them into a Kafka topic.

I used to have it as a String (I replaced it with GenericRecord).

def producerMethod(socket: Seq[Socket], prot: String): KafkaProducer[String, GenericRecord] =
producer(params(socket, prot))

def producerMethod(params: Map[String, String]): KafkaProducer[String, GenericRecord] =
new KafkaProducer[String, GenericRecord](params.asInstanceOf[Map[String, Object]].asJava)

I also have an apply method, which has an iterator in it. It iterates over strings and adds the values in a tuple.

In the end I send the data to a topic:

The results variable is a String but I added "asInstanceOf"

result.send(new ProducerRecord[String,GenericRecord](topic,key,results.asInstanceOf[GenericRecord]))

Here is my iterator:

iterator.foreach { tuple =>
      val (keyinfo, info, Worked(outinfo, _)) = tuple
      val (key, results) = keyResultToString(keyinfo, info, outinfo)
      result.send(new ProducerRecord[String,GenericRecord](topic,key,results.asInstanceOf[GenericRecord]))
    }

Everything works, when I replace the GenericRecord back to String. But not if I would like to convert it to a GenericRecord

Nika
  • 145
  • 1
  • 13
  • few things are not clear here: you used to put Strings into your topic and you want to put in "things" serialized with Avro? What's you're Kafka producer configuration? – mfirry Jul 23 '19 at 08:17
  • do you mean this Information? "bootstrap.servers" -> servers, "security.protocol" -> prot, "max.request.size" -> maxMessageSize, "key.serializer" -> stringSerializer, "value.serializer" -> stringKafkaAvroSerializer, "schema.registry.base.url " -> schemaURL – Nika Jul 23 '19 at 08:22
  • I would like to convert the data into binary avro datatype. I receive the data as String. I use following serializer: "org.apache.kafka.common.serialization.StringSerializer" and "io.confluent.kafka.serializers.KafkaAvroSerializer" – Nika Jul 23 '19 at 08:24
  • Did you take a look at this example https://docs.confluent.io/current/schema-registry/serializer-formatter.html ? (though it's in Java, it should be easy to port it to Scala) – mfirry Jul 23 '19 at 08:46
  • Yes I saw this example. But I don't know how to correctly convert my tupel that stores the values in String (to GenericRecords) – Nika Jul 23 '19 at 08:51
  • I don't see any tuple in your example... but assuming you have a tuple representing key and value, `val x = ("k", "v")` and assuming you've got your Avro schema with you, you can just do `let avroRecord = new GenericData.Record(schema)` and then `avroRecord.put(x._1, x._2)` – mfirry Jul 23 '19 at 08:54
  • I added the tupel – Nika Jul 23 '19 at 09:08
  • What does `keyResultToString` do? – mfirry Jul 23 '19 at 09:55
  • keyResultToString: ((Key, Info, Result) => (String, String)) with Serializable) – Nika Jul 23 '19 at 10:30
  • Try ``` val (key, results) = keyResultToString(keyinfo, info, outinfo) val avroRecord = new GenericData.Record(schema) avroRecord.put(key, results) result.send(new ProducerRecord[String, GenericRecord](topic, key, avroRecord)) ``` – mfirry Jul 23 '19 at 11:39

0 Answers0