0

I have an application that receives an api request and relays its to a Kafka API Producer. Each request calls the producer to send a message to Kafka. The producer exists throughout the application lifetime and is shared for all requests.

producer.send(new ProducerRecord[String, String](topic, requestBody))

This works OK. Now I want to use instead, an alpakka Producer for the job. The code looks like this:

val kafkaProducer = producerSettings.createKafkaProducer()
val settingsWithProducer = producerSettings.withProducer(kafkaProducer)

val done = Source.single(requestBody)
  .map(value => new ProducerRecord[String, String](topic, value))
  .runWith(Producer.plainSink(settingsWithProducer))

What are the advantages of the alpakka Producer over the plain, vanilla Producer? I don't know whether the new approach can help me handle a large number of API requests in order at the same time.

Levi Ramsey
  • 18,884
  • 1
  • 16
  • 30
landau
  • 41
  • 1
  • 6

1 Answers1

1

For that case of producing a single message to a Kafka topic, the Alpakka Producer sink that you're using doesn't really offer a benefit (the only tangential one might be if you're interested in using Akka Discovery to discover your Kafka brokers). Alpakka's SendProducer might be useful in your Scala code for that case: it exposes a Scala Future instead of a Java Future.

Where the Alpakka Producer sinks and flows shine is in a stream context where there's a sequence of elements that you want produced in order with backpressure, especially if the messages to be produced are the output of a complex stream topology.

I'm taking "large number of API requests" to mean HTTP/gRPC requests coming into your service and each request resulting in producing at most one message to Kafka. You can contort such a thing into a stream (e.g. feeding a stream via a Source.actorRef), but that's probably getting over-elaborate.

As for "in order at the same time": that's kind of a contradiction, as "in order" somewhat rules out simultaneity. Are you thinking of a situation where you can partition the requests and then you want ordering within that partition of requests, but are OK with any ordering across partitions (note that I'm not necessarily implying anything about partitioning of the Kafka topic you're producing to)? In that case, Akka Streams (and likely actors) will come in handy and the Producer sinks/flows will likely come in handy.

Levi Ramsey
  • 18,884
  • 1
  • 16
  • 30
  • Thanks a lot @LeviRamsey. By "in order simultaneously", I meant messages of the same topic get to Kafka in order across partitions (by the received timestamp of Kafka) and messages of the different topics may get to my producer at the same time. – landau Nov 21 '22 at 03:28
  • Do you mean in order based on when the messages get to Kafka by "by the received timestamp of Kafka"? – Levi Ramsey Nov 21 '22 at 11:31
  • (or do you mean that a consumer will see the messages in the same order as they were produced, across partitions?) – Levi Ramsey Nov 21 '22 at 11:32
  • I meant the time that Kafka noted the message. I configured the producer to make messages arrive to Kafka in order. – landau Nov 21 '22 at 12:52
  • Kafka basically gives that to you for free (with the caveat that there's no guaranteed consumption order across partitions, so in some sense the order of production across partitions does not matter). – Levi Ramsey Nov 21 '22 at 15:49