Spring Boot Data Redis Stream consume messages in Batch

Question

I am trying to consume messages from Redis Stream in Batch using Imperative StreamMessageListenerContainer of Spring Boot Data Redis.

But this have method signature of 'MapRecord<String, String, String> message'

How to use List or Set for this. I added the Batch size 3 in Subscription .

.batchSize(3)

Subscription Code

public Subscription subscribeToEvents(RedisConnectionFactory redisConnectionFactory) throws InterruptedException {


    StreamMessageListenerContainer<String, MapRecord<String, Object, Object>> listenerContainer =
            StreamMessageListenerContainer.create(Objects.requireNonNull(redisConnectionFactory),
                    StreamMessageListenerContainer.StreamMessageListenerContainerOptions.builder()
                            .hashKeySerializer(new Jackson2JsonRedisSerializer<>(String.class))
                            .hashValueSerializer(new Jackson2JsonRedisSerializer<>(Object.class))
                            .pollTimeout(Duration.ofSeconds(1))
                            .batchSize(3)
                            .build());

    StreamMessageListenerContainer.StreamReadRequest<String> streamReadRequest =
            StreamMessageListenerContainer.StreamReadRequest
                    .builder(StreamOffset.create(redisStreamName, ReadOffset.lastConsumed()))
                    .consumer(Consumer.from(consumerGroup, redisStreamName))
                    .cancelOnError(ex -> false)
                  
                    .autoAcknowledge(false)
                    .build();

    Subscription subscription = listenerContainer.register(streamReadRequest, streamListener2);

    listenerContainer.start();
    LOGGER.info("Registered subscriber for stream key: {} with consumer group: {} & consumer name: {}", redisStreamName, consumerGroup);
    return subscription;
}

StreamListener Code

public class RedisStreamListener2 implements StreamListener<String, MapRecord<String, Object, Object>> {

    @Value("${redis-stream-name2}")
    private String redisStreamName;

    @Value("${redis-stream-consumer2}")
    private String consumerGroup;

    private static final Logger LOG = LoggerFactory.getLogger(RedisStreamListener2.class);

    @Autowired
    private RedisTemplate redisTemplate;

    @Override
    public void onMessage(MapRecord<String, Object, Object> message) {
        LOG.debug(" - message Received from Redis :");
        String stream = message.getStream();
        RecordId id = message.getId();
        String data=  (String)message.getValue().get("testdata");
        try {
            EventEFRModel eventEFRModel = CommonUtils.readStringToObject(data,EventEFRModel.class);
            LOG.debug(" - consumed data:" +eventModel);
          

           acknowledgeAndDeleteRecord(stream,stream,id);
        }
        catch (JsonProcessingException e) {
            e.printStackTrace();
        }

    }

How ever with Redis template archive this But need to add Scheduler to listen the messages for infinity times.

   List<MapRecord<String, Object, Object>> messages = redisTemplate.opsForStream().read(Consumer.from(consumerGroup, redisStreamName)
            , StreamReadOptions.empty().autoAcknowledge().count(2),
            StreamOffset.create(redisStreamName, ReadOffset.lastConsumed()));

Is It even Possible in Spring Boot or I Miss understood the batch size value in Subscription .

score 0 · Answer 1 · answered Jul 24 '23 at 19:05

From what I understand the batchSize that you can set on the listening container is the number of messages specified as the COUNT in the underlying XREADGROUP command. Increasing this batch size reduces the number of Redis calls your app needs to make if your stream is large or has a high throughput.

It doesn't really make sense why we can't then consume this as a list in the consumer onMessage() but maybe they'll add that functionality someday. What you can do if you're wanting to process messages in a batch (if you're single-threaded) is to simply add the messages to an instance variable list then process the list when it reaches the batchSize.

...
private List<MapRecord<String, Object, Object>> messageBatch = new ArrayList<>();

@Override
public void onMessage(MapRecord<String, Object, Object> message) {
  messageBatch.add(message);
  if(messageBatch.size() >= batchSize){
    processBatch();
    messageBatch.clear();
  }
}

private void processBatch(){
  // do stuff over entire batch like pipelined Redis calls, etc
}
...

If you have multiple subscriptions (threads) registered for the consumer which are all using the same StreamListener class then this approach won't work since it's not thread safe. In this case you could try using something like a ThreadLocal list so each consumer thread has their own batch.

Spring Boot Data Redis Stream consume messages in Batch

1 Answers1