Not able to publish messages to Kafka from Lambda function

Question

I am trying to publish some messages to kafka from a aws lambda function. When I'm trying to test this feature on my local, the messages are not getting published and the function is getting timed out. I was able to connect to the local kafka instance from the same lambda function using a consumer and list the topics. Is there anything I'm missing here?

String bootstrapServer = "host.docker.internal:9092";
String topic = "test-topic";
Properties properties = new Properties();
properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServer);
properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
properties.setProperty(ProducerConfig.ACKS_CONFIG, "all");
properties.put(ProducerConfig.BATCH_SIZE_CONFIG, 16384);

Properties props = new Properties();
props.put("bootstrap.servers", bootstrapServer);
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> simpleConsumer = new KafkaConsumer<>(props);
// THIS IS WORKING
simpleConsumer.listTopics().forEach((t, v) -> logger.log(t + "\n"));

try (KafkaProducer<String, String> kafkaProducer = new KafkaProducer<>(properties)) {
ProducerRecord<String, String> record = new ProducerRecord<>(topic, value);
logger.log("Publishing '" + value + "' to kafka topic: " + topic + "\n");
Future<RecordMetadata> future = kafkaProducer.send(record);
logger.log("Waiting for flushing data\n");
kafkaProducer.flush();
logger.log("Waiting for the response\n");
RecordMetadata recordMetadata = future.get();

logger.log("Published record to kafka topic: " + recordMetadata.topic() + " partition: " + recordMetadata.partition() + " offset: " + recordMetadata.offset() + "\n");

I'm triggering the function in my local using sam cli as follows:

sam local invoke "MyLambdaFunction" -e events/event.json

This is the timeout message I get.

Invoking helloworld.App::handleRequest (java11)
Skip pulling image and use local one: public.ecr.aws/sam/emulation-java11:rapid-1.59.0-x86_64.

Mounting .aws-sam/build/MyLambdaFunction as /var/task:ro,delegated inside runtime container
START RequestId: c07609a4-7cab-4b58-97fc-cdddf20afd5c Version: $LATEST
Picked up JAVA_TOOL_OPTIONS: -XX:+TieredCompilation -XX:TieredStopAtLevel=1
test-topic
__consumer_offsets
Publishing 'abc' to kafka topic: test-topic
Function 'MyLambdaFunction' timed out after 20 seconds
No response from invoke container for MyLambdaFunction

Not able to publish messages to Kafka from Lambda function

0 Answers0