-2

I have to publish messages from a kafka topic to lambda to process them and store in a database using a springboot application, i did some research and found something to consume messages from kafka public Function<KStream<String, String>, KStream<String, String>> process(){} however, im not sure if this is only used to publish the consumed messages to another kafka topic or can be used as an event source to lambda, I need some guidance on consuming and converting the consumed kafka message to event source.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Are you using auto managed Kafka cluster or you're using Amazon MSK? – Felipe Bonfante Aug 11 '22 at 12:49
  • MSK, I'm using something like this ` public class LambdaRequestHandler implements RequestHandler, Person> { @Override public Person handleRequest(? event, Context context) { // process event return processes event; } }` Here the '?' is the place holder for kafka message, how i get the kafka messsage as input – –  Aug 11 '22 at 15:38

1 Answers1

1

Brokers do not push. Consumers always poll.

Code shown is for Kafka Streams API, which primarily writes to new Kafka topics. While you could fire HTTP events to start a lambda, that's not recommended.

Alternatively, Kafka is already supported as an event source. You don't need to write any consumer code.

https://aws.amazon.com/about-aws/whats-new/2020/12/aws-lambda-now-supports-self-managed-apache-kafka-as-an-event-source/

This is possible from MSK or a self managed Kafka

process them and store in a database

Your lambda could process the data and send to a new Kafka topic using a producer. You can then use MSK Connect or run your own Kafka Connect cluster elsewhere to dump records into a database. No Spring/Java code would be necessary.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • ah that makes sense, I'm using something like this ` public class LambdaRequestHandler implements RequestHandler, Person> { @Override public Person handleRequest(? event, Context context) { // process event return processes event; } }` Here the '?' is the place holder for kafka message, how i get the kafka messsage as input –  Aug 11 '22 at 14:47
  • Seems fine if you only want to print json to Cloudwatch logs – OneCricketeer Aug 11 '22 at 14:49