1

I have a requirement where I have to read a file which contains around 1 million records each record in a separate line. Each record will be validated and then will be saved in Redis cache. I implemented this in a normal traditional way by reading each line and saving it in the Redis cache but it is hampering performance very badly. Then I came to know about Redis pipeline feature in which I will process records in a batch say 10k at a time in order to improve performance.

How can I use this feature in my current scenario? Any small simple example appreciated. I am using Redis-2.1.1.RELEASE, spring boot-2.0.8.RELEASE

pan1490
  • 939
  • 1
  • 7
  • 25
  • Can you give the sample code of your existing solution? I believe redis pipelines by default, so it should just be a matter of passing your lines in in batches. Just make sure your batches are small enough that you don't explode your RAM. – DylanYoung Aug 23 '19 at 17:37
  • @DylanYoung Yes you are right but what we want is put many ObjectRedisRepository.save(object) in single pipeline so we need to stripe off the wrapping of this save() and get the core Redis commands If I pass 10k objects in a batch like below redisTemplate.executePipelined()( new RedisCallback() { for(10k times){ ObjectRedisRepository.save(object)}} Then all 10k objects will get executed in a single pipeline or separate pipeline for each object gets created? Also 1 doubt.Unless and until I call connection.closePipeline(); the commands in pipeline will not execute is it true? – pan1490 Aug 26 '19 at 13:28
  • I need to know what redis library you're using if you want me to take a look. Otherwise, I'd just be guessing ;) – DylanYoung Sep 03 '19 at 17:04
  • We are using lettuce – pan1490 Sep 04 '19 at 07:58
  • 1
    A quick look here (https://lettuce.io/core/release/reference/#_pipelining_and_command_flushing) says that you need to (1) use the async api, (2) disable automatic flushing, and (3) manually flush after each batch. – DylanYoung Sep 05 '19 at 16:17
  • I also ave similar requirements. Were you able to acheive this using spring data redis.? – Priya Tanwar Mar 06 '20 at 03:29
  • @PriyaTanwar No. I didn't get proper information anywhere so I used the executor framework to achieve this – pan1490 Apr 02 '20 at 13:01

0 Answers0