Connecting list of case classes to kafka producer?

Question

I have the below case class:

case class Alpakka(id:Int,name:String,animal_type:String)

I am trying to connect a list of these case classes to a producer in kafka by using the following code:

  def connectEntriesToProducer(seq: Seq[Alpakka]) = {


    val producerSettings = ProducerSettings(system, new StringSerializer, new StringSerializer)
      .withBootstrapServers("localhost:9092")

seq.map(alpakka => new ProducerRecord[String, String]("alpakkas", alpakka.asJson.noSpaces))   
      .runWith(Producer.plainSink(producerSettings))
  }

I am using circe to convert the case class to json. However I keep getting a compiler error saying this:

Error:(87, 34) type mismatch;
 found   : akka.stream.scaladsl.Sink[org.apache.kafka.clients.producer.ProducerRecord[String,String],scala.concurrent.Future[akka.Done]]
 required: org.apache.kafka.clients.producer.ProducerRecord[String,String] => ?
      .runWith(Producer.plainSink(producerSettings))

I'm not sure whats going on!

Answer 1

You are trying to build a Graph from a Seq instead of a Source.

Your method connectEntriesToProducer should look like

def connectEntriesToProducer(seq: Source[Alpakka]) = {

Note, Source instead of Seq.

In alternative, you can build a source from a Seq, but you'll have to use immutable.Seq since Source.apply would only take an immutable iterable.

def connectEntriesToProducer(seq: scala.collection.immutable.Seq[Alpakka]) = {
val producerSettings = ProducerSettings(system, new StringSerializer, new StringSerializer)
  .withBootstrapServers("localhost:9092")

Source(seq).
  map(alpakka => new ProducerRecord[String, String]("alpakkas", alpakka.asJson.noSpaces))
  .runWith(Producer.plainSink(producerSettings))
}