- About the application: I have a spring application deployed to AWS Fargate Service. This application consumes data from SQS queue. Then, apply some logic and send those data to Kinesis Stream. The application gets extreme amount of request at some point of time in a day. Here are the how my application looks like.
@Component
public class MessageConsumer {
private Logger log = LoggerFactory.getLogger(this.getClass());
// some class fields and autowired beans
@Autowired
public KPLService kplService;
@SqsListener(value="sqs-queue")
public void consumingSqs(SqsMessage message) {
// applying some logic
// also doing some mapping here
// kinesis part
kplService.addRecord(mappedSqsRecord);
}
}
@Service
public class KPLService {
private Logger log = LoggerFactory.getLogger(this.getClass());
@Autowired
public KinesisProducer producer;
public void addRecord(MappedSqsRecord record) {
producer.addUserRecord(
"kinesis-stream-name",
record.uuid,
ByteBuffer.wrap(new ObjectMapper().writeValueAsBytes(record))
);
}
public void flush() {
producer.flushSync();
}
@PreDestroy
public void onShutdown() {
flush();
}
}
What I am trying to achieve: Before sending to Kinesis stream, I would like to add a set number of records by calling
producer.addUserRecord
and send them as a batch by callingflushSync
, instead of sending the record one by one. I created a function calledonShutdown
. So for example after adding 500 records by callingaddUserRecord
then I would like to callonShutdown
once to send them as a batch of 500.What I am having trouble with: What I am trying to achieve is have a function(like onShutdown in the example) that gets called when the application gets no request from sqs in a given amount of time and call the function to
flush
to send the records to kinesis.
I tried it by using @PreDestroy
annotation. But I realized that my application deployed to AWS Fargate does not go to sleep or showdown ever. It keeps running all the time and seems like @PreDestroy
never gets called. How can I resolve this situation. Any suggestion or advice is much appreciated. Thank you in advance.
I have seen the example of counting the number of records. And if it is more than 500(as an example) call flush
, otherwise keep adding by calling addUserRecord
. However, I would like to know the other approach for now if that is possible.