Redis consumer unable to consume a particular event

Question

I have a producer in a spring boot project that is producing two events to the same redis consumer group and a consumer (another spring boot project) consuming the same. The redis consumer is able to consume only one type of event. I see the below exception while consuming the 2nd type of event :-

Exception seen in consumer spring application :-

2021-01-06 19:14:22,580 ERROR [SimpleAsyncTaskExecutor-1] DefaultStreamMessageListenerContainer$LoggingErrorHandler: Unexpected error occurred in scheduled task.
java.lang.IllegalArgumentException: Value must not be null!
    at org.springframework.util.Assert.notNull(Assert.java:198)
    at org.springframework.data.redis.connection.stream.Record.of(Record.java:81)
    at org.springframework.data.redis.connection.stream.MapRecord.toObjectRecord(MapRecord.java:147)
    at org.springframework.data.redis.core.StreamObjectMapper.toObjectRecord(StreamObjectMapper.java:132)
    at org.springframework.data.redis.core.StreamObjectMapper.map(StreamObjectMapper.java:158)
    at org.springframework.data.redis.core.StreamOperations.read(StreamOperations.java:377)
    at org.springframework.data.redis.stream.DefaultStreamMessageListenerContainer.lambda$getReadFunction$2(DefaultStreamMessageListenerContainer.java:232)
    at org.springframework.data.redis.stream.StreamPollTask.doLoop(StreamPollTask.java:138)
    at org.springframework.data.redis.stream.StreamPollTask.run(StreamPollTask.java:123)
    at java.lang.Thread.run(Thread.java:748)

Redis Stream Task :-

@Component
public class RedisStreamTask<T> {

  @Autowired
  private RedisTemplate<String, Object> redisTemplate;

  @Async
  public void publish(final StreamingEvent<T> event) {

    final ObjectRecord<String, T> record = StreamRecords.newRecord()
        .ofObject(event.getMessage())
        .withStreamKey(event.getKey());

    final RecordId recordId = redisTemplate
        .opsForStream()
        .add(record);
  }
}

Producer in 1st spring application :-

final StreamingEvent<MyBean> streamingEvent = new StreamingEvent<>();
streamingEvent.setKey("CONSUMER_GROUP_KEY");
streamingEvent.setMessage(myBean);
redisStreamTask.publish(streamingEvent);

Consumer in 2nd spring application :-

imports...

@Component
public class Consumer implements StreamListener<String, ObjectRecord<String, MyBean>>  {

    private AtomicInteger atomicInteger = new AtomicInteger(0);

    private final Logger log = LoggerFactory.getLogger(Consumer.class);

    @Autowired
    private ReactiveRedisTemplate<String, String> redisTemplate;

    @Autowired
    public RedisConfig config;

    @Autowired
    private MyService myService;


    @Override
    public void onMessage(ObjectRecord<String, MyBean> record) {
        record.getValue();
        tempCreateFile.log(record.getValue());

        try {
            myService.saveEvent(record.getValue());
        } catch (Exception e) {
            log.error("Error while storing redis events", e);
        }
        atomicInteger.incrementAndGet();
    }

    @Scheduled(fixedRate = 10000)
    public void showPublishedEventsSoFar() {
        log.info("Total Consumed :: " + atomicInteger.get());
    }

    @PostConstruct
    public void afterPropertiesSet() throws Exception {
        try {
            log.info("Create Consumer group: {}", config.getConsumerGroupName());
            redisTemplate.opsForStream().createGroup(config.getStreamName(), config.getConsumerGroupName()).block();
        } catch (RedisSystemException e) {
            if (e.getRootCause().getClass().equals(RedisBusyException.class)) {
            } else if (e.getRootCause().getClass().equals(RedisCommandExecutionException.class)) {
                log.info("STREAM - Stream does not yet exist, creating empty stream: " + config.getStreamName());
                redisTemplate.opsForStream().add( config.getStreamName(), Collections.singletonMap("", "")).block();
                redisTemplate.opsForStream().createGroup(config.getStreamName(), config.getConsumerGroupName()).block();
            } else {
                e.printStackTrace();
                throw e;
            }
        }
    }
}

Please let me know if any additional info is required. Any help on how to fix this issue would be grateful!

score 0 · Answer 1 · answered Aug 20 '21 at 14:56

I had this error and it turned out to be a deserialization issue relating to the stream record object's _class not matching exactly between producer and consumer (since it uses the fully qualified class name). I was able to fix it by adding a @TypeAlias on the producer-side object so it matched the path of the same object on consumer-side:

@TypeAlias("com.example.consumer.model.MyStreamRecord")
public class MyStreamRecord {
...

I also had to use a RedisTemplate<String,String> instead of <String,MyStreamRecord> on both producer and consumer side because of the way Redis handles default object-to-hash mappings (more info here and here).

Redis consumer unable to consume a particular event

1 Answers1