How to get a Map [String, String] returned by a Kafka Consumer (Alpakka)?

Question

I should get a Map [String, String] back from a Kafka Consumer, but I don't really know how. I managed to configure the consumer, it works fine, but I don't understand how I could get the Map.

implicit val system: ActorSystem = ActorSystem()

    val consumerConfig = system.settings.config.getConfig("akka.kafka.consumer")

    val = kafkaConsumerSettings =
      ConsumerSettings(consumerConfig, new StringDeserializer, new StringDeserializer)
        .withBootstrapServers(localhost:9094)
        .withGroupId(group1)
    Consumer
      .plainSource(kafkaConsumerSettings, Subscriptions.topics(entity.entity_name))
      .toMat(Sink.foreach(println))(DrainingControl.apply)
      .run()

Start by writing what you've tried and what you are getting returned right now. Some code would be very helpful — GamingFelix, Aug 17 '20 at 14:30
a `Map` of which keys and which values? A map of every key and associated value in the topic? — Levi Ramsey, Aug 17 '20 at 14:52
If that's what you want (i.e. you're reading finitely many values from the topic), I'd really question whether Alpakka Kafka is a great fit: it's much more oriented towards a stream (i.e. an unbounded number of values from the topic) approach. — Levi Ramsey, Aug 17 '20 at 14:54

score 0 · Answer 1 · answered Aug 17 '20 at 15:08

Lightbend's recommendation is to deal with byte arrays while deserializing incoming data from Kafka

The general recommendation for de-/serialization of messages is to use byte arrays (or Strings) as value and do the de-/serialization in a map operation in the Akka Stream instead of implementing it directly in Kafka de-/serializers. When deserialization is handled explicitly within the Akka Stream, it is easier to implement the desired error handling strategy as the examples below show.

To do so, you may setup a consumer using this setting:

val consumerSettings = ConsumerSettings(consumerConfig, new StringDeserializer, new ByteArrayDeserializer)

And get the results by calling the .value() method from your Record class. To deserialize it, i would recommend using circe + jawn. This code should do the trick.

import io.circe.jawn
import io.circe.generic.auto._

val bytes = record.value()

val data = jawn.parseByteBuffer(ByteBuffer.wrap(bytes)).flatMap(_.as[Map[String, String]])

How to get a Map [String, String] returned by a Kafka Consumer (Alpakka)?

1 Answers1