0
  • About the application: I have a spring application deployed to AWS Fargate Service. This application consumes data from SQS queue. Then, apply some logic and send those data to Kinesis Stream. The application gets extreme amount of request at some point of time in a day. Here are the how my application looks like.
@Component
public class MessageConsumer {
  private Logger log = LoggerFactory.getLogger(this.getClass());
  // some class fields and autowired beans
  @Autowired
  public KPLService kplService;
  
  @SqsListener(value="sqs-queue")
  public void consumingSqs(SqsMessage message) {
    // applying some logic
    // also doing some mapping here

    // kinesis part
    kplService.addRecord(mappedSqsRecord);
  }
}
@Service
public class KPLService {
  private Logger log = LoggerFactory.getLogger(this.getClass());

  @Autowired
  public KinesisProducer producer;
  
  public void addRecord(MappedSqsRecord record) {
    producer.addUserRecord(
      "kinesis-stream-name",
      record.uuid,
      ByteBuffer.wrap(new ObjectMapper().writeValueAsBytes(record))
    );
  }

  public void flush() {
    producer.flushSync();
  }

  @PreDestroy
  public void onShutdown() {
    flush();
  }
}
  • What I am trying to achieve: Before sending to Kinesis stream, I would like to add a set number of records by calling producer.addUserRecord and send them as a batch by calling flushSync, instead of sending the record one by one. I created a function called onShutdown. So for example after adding 500 records by calling addUserRecord then I would like to call onShutdown once to send them as a batch of 500.

  • What I am having trouble with: What I am trying to achieve is have a function(like onShutdown in the example) that gets called when the application gets no request from sqs in a given amount of time and call the function to flush to send the records to kinesis.

I tried it by using @PreDestroy annotation. But I realized that my application deployed to AWS Fargate does not go to sleep or showdown ever. It keeps running all the time and seems like @PreDestroy never gets called. How can I resolve this situation. Any suggestion or advice is much appreciated. Thank you in advance.

I have seen the example of counting the number of records. And if it is more than 500(as an example) call flush, otherwise keep adding by calling addUserRecord. However, I would like to know the other approach for now if that is possible.

Rue Honda
  • 13
  • 3

0 Answers0