I am trying to create a SQS queue processor that processes messages from a SQS queue and streams them to a client using RSocket.
The problem is that I don't know how to do this because the list of messages isn't known beforehand.
From what I have read online is that I probably have to use a Sink in order to store the received messages from the queue, but I don't know how to do it cleanly (based on this post)
If someone could help me out, or point me in the right direction, that would be great.
The code below works. When a message is sent to the queue, it gets read by the receiveMessage method, which emits the received message to the sink. If a client wants to receive messages, the stream method gets called, which subscribes on the sink. But what I don't understand is that the code within the subscribe part log.info("receive {} in stream, message) doesn't get executed. But if I remove the subscribe part it doesn't work at all.
private Many<Message> sink = Sinks.many().multicast().directBestEffort();
private AmazonSQSAsync amazonSQSAsync;
@SqsListener(value = "${custom.sqs}", deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS)
public void receiveMessage(Message message) {
log.info("Received message from sqs: " + message);
sink.emitNext(message, Sinks.EmitFailureHandler.FAIL_FAST);
}
@MessageMapping("messages")
Flux<Message> stream(final int delay) {
log.info("Received stream request. With delay of {} seconds between messages", delay);
this.sink.asFlux().subscribe(message -> {
log.info("receive {} in stream", message);
}).dispose()
return this.sink.asFlux();
}
Based on Yuri's answer, the code below has been made!
private Many<Message> publisher = Sinks.many().multicast().directBestEffort();
@SqsListener(value = "${custom.sqs}", deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS)
public void receiveMessage(Message message) {
log.info("Received message from sqs: " + message);
this.publisher.emitNext(message, Sinks.EmitFailureHandler.FAIL_FAST);
}
@MessageMapping("messages")
Flux<Message> stream(final int delay) {
log.info("Received stream request. With delay of {} seconds between messages", delay);
return Flux.<Message>create(sink -> {
this.publisher.asFlux().subscribe(message -> {
sink.next(message);
});
});
}
To fix the sink emission exception I used the following code below which I found from this link
// From: https://stackoverflow.com/questions/65029619/how-to-call-sinks-manyt-tryemitnext-from-multiple-threads/65185325#65185325
public static EmitFailureHandler retryOnNonSerializedElse(EmitFailureHandler fallback) {
return (signalType, emitResult) -> {
if (emitResult == EmitResult.FAIL_NON_SERIALIZED) {
LockSupport.parkNanos(10);
return true;
} else
return fallback.onEmitFailure(signalType, emitResult);
};
}
And I used this failureHandler in the emitNext() method which is
called in the receiveMessages() method.
publisher.emitNext(message, retryOnNonSerializedElse(Sinks.EmitFailureHandler.FAIL_FAST));
This works, although I don't completely understand why
