I am new at R2DBC , sorry for a basic question.
My business scenario is that , I have one kafka consumer and after reading the data from a topic, I have to insert data in to a postgress table using R2DBC DatabaseClient.
I am using a simple insert into SQL for insertion with param binding.
But my problem is that huge back presser.
My kafka consumer reading huge amount of data and forward it my service class , written in reactive way , for data insertion into database.
But my App server and DB server not able to handle that huge amount of client request and after running sometime it is giving to many connection error.
I tried to slow down the kafka consumer by sleeping the consumer thread by 5 sec , and by that solution no error in server and each record consumed by the consumer written properly into DB.
But I believe sleeping the consumer thread is not acceptable way.
Is there any why through which I can keep waiting for my insert into query to complete the insertion in R2DBC and then my kafka consumer will go for the next record.
Please advice....
My current code in kafka consumer is...
@KafkaListener(topics = {"${spring.kafka.topic}"}, containerFactory = "kafkaListenerContainerFactory", concurrency = "1")
public void geoMessage(@Payload ConsumerRecord<String, geographyMessage> consumerRecord,
@Header(KafkaHeaders.RECEIVED_TIMESTAMP) String receivedTimeStamp,
Acknowledgment acknowledgment) throws IOException {
// log.warn("Received customer payload at offset : "+consumerRecord.offset() + " at partition: " + consumerRecord.partition());
// System.out.println("GEO ID ::: "+consumerRecord.value().getGeography().getGeoId());
KafkaConsumerGeographyDTO kafkaDTO = locationMasterMapper.kafkaConsumerToKafkaConsumerDTO(consumerRecord.value().getGeography());
locationMasterService.saveLocation(kafkaDTO);
//System.out.println("kafkaDTO JSON Body : "+kafkaDTO.getProjectionMsg());
System.out.println("Counter : "+counter++);
try {
Thread.sleep(5000);
}
catch (Exception exx)
{
exx.printStackTrace();
}
//System.out.println("================== All Data Saved ==================");
}