I am trying to implement a kinesis consumer reactive. My current understanding is that a kinesis shard is always consumed by a thread so the thread consuming it will subscribe.
Here is my code:
private static CompletableFuture<Void> responseHandlerBuilderReactor(KinesisAsyncClient client, SubscribeToShardRequest request) {
SubscribeToShardResponseHandler responseHandler = SubscribeToShardResponseHandler
.builder()
.onError(t -> System.err.println("Error during stream - " + t.getMessage()))
.onEventStream(p -> Flux.from(p)
.ofType(SubscribeToShardEvent.class)
.flatMapIterable(SubscribeToShardEvent::records)
.limitRate(2)
.buffer(2)
.subscribe(e -> {
// decoder.decode(record.data()).toString();
e.forEach(el-> {
try {
System.out.println(decoder.decode(el.data().asByteBuffer()));
} catch (CharacterCodingException e1) {
e1.printStackTrace();
}
});
}))
.build();
return client.subscribeToShard(request, responseHandler);
}
public static void main(String[] args) {
KinesisAsyncClient client = KinesisAsyncClient.create();
SubscribeToShardRequest request = SubscribeToShardRequest.builder()
.consumerARN(CONSUMER_ARN)
.shardId("shardId-000000000000")
.startingPosition(StartingPosition.builder().type(ShardIteratorType.TRIM_HORIZON).build())
.build();
responseHandlerBuilderReactor(client, request).join();
client.close();
}
I think there are many details I don't understand about kinesis and even if I keep reading it is not becoming more clear.
The documentation says that this connection will have a low-latency HTTP/2 connection but I have no idea how to handle this between runs, what I mean if I restart the app I don't want to process already consumed messages. How can I implement a RecordProcessorCheckpointer for the reactive version?
Also after a few minutes, the process exits without any error. I remember reading in the documentation that I need to resubscribe. Is it enought if I run in infinite loop: responseHandlerBuilderReactor(client, request).join();?
